Is the EPG getting too big for dialup users?

[DX30]

My sister uses dialup to download the Sky lineup listings. Her TiVo was unplugged for a couple of months due to building work and so now has to download the lot, but it is failing.

Looking at the log files it seems the problem is the download is taking longer than an hour, so the phone line is disconnected at the other end before it completes. The next call starts to download it all again and again fails after an hour.

Over time we have added extra channels and extra days to the EPG downloads so I'm wondering if we have pushed it to far for dialup users. It will probably be OK if the TiVo downloads every day but if for any reason the TiVo runs out of guide data, or needs to rerun guided setup, it won't be able to.

Going back to less days in the EPG would penalise everyone so that doesn't seem a good thing. I'm not sure of the work involved but off the top of my head a couple of other possible solutions.

1. Allow the maximum number of slices to download in a single daily call to be set in Account Management.

2. Stop downloading slices at say 45 minutes.

The theory being all the slices will eventually download over a few daily calls.

I can probably sort my sisters TiVo without this but I am worried about dialup users with unmodified TiVo's.

[mrtickle]

Hmm. I wonder why it's taking that long. In the Tribune days IIRC the full "data call" during Guided Setup was 45 mins I think? And that was with 3 weeks' data for more channels. I wonder why it's taking over an hour. Is it dropping back to 28.8k, or even lower; poor quality phoneline? ADSL in the house without filters on every socket, wrecking modem performance? Can't remember if the connection speed gets logged, but worth checking.

A key thing you should do if you haven't already: switch over to TmsId slices. Although they don't truncate the synopses which may lead you to think there'd be more data, they are packaged in a more efficient way and they are a smaller download.

Is it SD or the HD lineup? I got the SD lineup purged of all pay-HD channels in the non-swapped positions, which reduced the size of the lineup considerably, along with all the PPV movie channels and most channels we don't have data for along with the "crappy shopping channels" as I believe they were called.

[DX30]

I wonder why it's taking over an hour. Is it dropping back to 28.8k, or even lower; poor quality phoneline? ADSL in the house without filters on every socket, wrecking modem performance?

It could well be a problem with the line causing a slow connection.

There were missing ADSL filters from when broadband was first installed but I got that sorted out. There could be a dodgy filter I suppose, or dodgy wiring for the extension sockets.

Can't remember if the connection speed gets logged, but worth checking.

I'll check the TiVo logs when I next visit, although at some 500 miles distance that will probably be Christmas.

A key thing you should do if you haven't already: switch over to TmsId slices. Although they don't truncate the synopses which may lead you to think there'd be more data, they are packaged in a more efficient way and they are a smaller download.

It was on Tmsid slices but yesterday I switched back to standard slices yesterday to see if it made a difference. I'll switch back again.

Is it SD or the HD lineup? I got the SD lineup purged of all pay-HD channels in the non-swapped positions, which reduced the size of the lineup considerably, along with all the PPV movie channels and most channels we don't have data for along with the "crappy shopping channels" as I believe they were called.

It's using the "Sky / Freesat from Sky" SD lineup.

My current plan is to temporarily switch the TiVo over to the 08451112111 number as that doesn't have the 1 hour limit. Luckily I installed the altepg image on my sisters TiVo a while ago so I think I should be able to talk her through changing the dial-in configuration code over the phone. Anything more complicated will probably need to wait until I next visit.

I may also try a guided setup on my TiVo here using dialup rather than network and see how long it takes. If it is a one-off problem on my sisters TiVo that is OK but I am just worried it may be more widespread and from what I remember an unmodified TiVo has to use the 1 hour limit numbers.

[mrtickle]

Hmm yeah
I may be remembering wrong, but maybe the server side also logs the connection speed. If not it would certainly show what times that tivo requested which files and how long they are taking, which would indicated the speed.

[DX30]

If not it would certainly show what times that tivo requested which files and how long they are taking, which would indicated the speed.

I used this method and it was taking slightly under 9 minutes to download a slice of 1284617 bytes, or around 2400 bytes per second. There will be some overheads on the http_get and ppp layer, and a lot depends on how compressible the data is (presumably not much as it is a gz file), but on a rough calculation it looks like it isn't connecting at 33.6 Kbps, more like 24 Kbps.

[nick1austin]

As I understand edits (corrections) to the guide data only affect the latest set of slices so limiting the number of slice downloads means not getting the latest and greatest guide data. Maybe that's acceptable under the circumstances.

Can I drawn your attention to my previous suggestion (since you didn't comment on that thread so I'm not sure if you saw it):
http://www.tivoland.com/forum/viewtopic.php?f=17&t=1290&p=12742&hilit=madcap#p12742

[DX30]

Can I drawn your attention to my previous suggestion (since you didn't comment on that thread so I'm not sure if you saw it):

Thanks for the link. It would address this 1 hour limit problem as well as the original purpose of reducing call charges, but as pointed out an epg-lite does reduce the usefulness of wishlists etc.

What I was hoping to address with a limit on the number of slices wouldn't change the epg in normal operation. An up-to-date TiVo making a daily call normally only downloads the new slices, not slices it already has, and as there are only a few new slices wouldn't hit the limit.

It would only be where the TiVo has to download all the slices (e.g. after Guided Setup or a Service Interruption) that the limit would have an effect. So after the first call the user might see a e.g. 4 day EPG. After a second daily call another 4 days worth of slices get downloaded so they then see an 8 day EPG etc. After a few calls they would be caught up and normal service would be resumed.

Obviously this doesn't reduce the call charges, it is intended purely a workaround for the 1 hour download limit.

[mrtickle]

The major issue with that is, the system doesn't have "day's worth of slices", as if each "day" has 1 small slice with all the stations for that day and a daily call downloads lots of small files. The system uses 1 slice per whole lineup.

Eg My TiVo is set up with the Sky SD lineup + the Sky Radio lineup. Each daily call, it downloads just two files. Today's, from the Account Management / View Logfiles section of altepg.com were:

Code: Select all

Fri Nov 01 01:47:19 2013   FILE=/tivo-static/tmsid-listings/UK01003_16005-16024.slice.gz STATUS=New
Fri Nov 01 01:47:20 2013   FILE=/tivo-static/tmsid-listings/UK01023_16005-16024.slice.gz STATUS=New


If you are on Sky SD, you'll download UK01003_16005-16024.slice.gz. A tivo will either manage to get the file in an hour, or it won't

[DX30]

I think I am misunderstanding how slices are working.

Here is the log from a failed call where there is no epg data

Code: Select all

tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:26:27 2013   FILE=/tivo-static/ukheadends/message-20000038.slice STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:26:33 2013   FILE=/tivo-static/ukheadends/DBS~01-60.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:30:44 2013   FILE=/tivo-static/tmsid-listings/UK01003_15995-16014.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:37:59 2013   FILE=/tivo-static/tmsid-listings/UK01003_15994-16013.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:37:59 2013   FILE=/tivo-static/tmsid-listings/UK01006_15992-16011.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:38:00 2013   FILE=/tivo-static/tmsid-listings/UK01006_15993-16012.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:38:01 2013   FILE=/tivo-static/tmsid-listings/UK01006_15999-16018.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:42:02 2013   FILE=/tivo-static/tmsid-listings/UK01003_15993-16012.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:48:01 2013   FILE=/tivo-static/tmsid-listings/UK01003_15992-16011.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:48:01 2013   FILE=/tivo-static/tmsid-listings/UK01006_15996-16015.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:52:42 2013   FILE=/tivo-static/tmsid-listings/UK01003_15997-16016.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:52:42 2013   FILE=/tivo-static/tmsid-listings/UK01006_15994-16013.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Mon Oct 28 23:59:57 2013   FILE=/tivo-static/tmsid-listings/UK01003_15998-16017.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Tue Oct 29 00:04:06 2013   FILE=/tivo-static/tmsid-listings/UK01003_15996-16015.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Tue Oct 29 00:09:45 2013   FILE=/tivo-static/tmsid-listings/UK01003_15999-16018.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Tue Oct 29 00:09:46 2013   FILE=/tivo-static/tmsid-listings/UK01006_16000-16019.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Tue Oct 29 00:09:47 2013   FILE=/tivo-static/tmsid-listings/UK01006_15995-16014.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Tue Oct 29 00:18:40 2013   FILE=/tivo-static/tmsid-listings/UK01003_16001-16020.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Tue Oct 29 00:18:41 2013   FILE=/tivo-static/tmsid-listings/UK01006_16001-16020.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Tue Oct 29 00:18:42 2013   FILE=/tivo-static/tmsid-listings/UK01006_15997-16016.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Tue Oct 29 00:18:42 2013   FILE=/tivo-static/tmsid-listings/UK01006_15998-16017.slice.gz STATUS=New
tclient_download   Mon Oct 28 23:21:34 2013   Tue Oct 29 00:24:15 2013   FILE=/tivo-static/tmsid-listings/UK01003_16000-16019.slice.gz STATUS=Failed


As you can see it downloads multiple slices for an hour until disconnected and the last slice fails. A second daily call will just attempt to download all the same slices again, even those that previously succeeded.

So what would happen if the server only downloaded say the first 4 slices?

My assumption was those would be processed and you would get a partial epg. The next daily call would start downloading at the 5th slice in the list rather than starting back again at the 1st slice.

[excoriated]

As you can see it downloads multiple slices for an hour until disconnected and the last slice fails.

But they are not separate days. Look at the filenames for the slices you downloaded.
E.g. UK01003_15995-16014.slice
This says slice for lineup UK01003 (Sky SD) for days 15995-16014 (17/10/13 - 5/11/13).

Your next file download is for the same lineup but one day earlier (16/10/13 - 4/11/13).
Then it downloads another day earlier (15/10/13 - 3/11/13) (skip past the ones for UK01006 (Aerial))

Put them all in date order and you have tried:
UK01003_16001-16020
UK01003_16000-16019
UK01003_15999-16018
UK01003_15998-16017
UK01003_15997-16016
UK01003_15996-16015
UK01003_15995-16014
UK01003_15994-16013
UK01003_15993-16012
UK01003_15992-16011

You can see all the dates overlap. Basically the AltEPG server has sent you pretty much every slice it has for Sky SD!

Why it is doing this I don't know. I've seen mine occasionally download the current slice plus the one for the day before, but I've never seen it download 10 slices like yours has!

A second daily call will just attempt to download all the same slices again, even those that previously succeeded.

So there definitely seems something wrong with your Tivo. It's either requesting too many slices or not remembering which ones it's already downloaded. I don't know enough about how the slice "request" process works to be able to suggest which.

My assumption was those would be processed and you would get a partial epg. The next daily call would start downloading at the 5th slice in the list rather than starting back again at the 1st slice.

No because they are not "day" slices but each slice covers a 20 day period.

[DX30]

You can see all the dates overlap. Basically the AltEPG server has sent you pretty much every slice it has for Sky SD!

As I said in the original post the TiVo was unplugged for a couple of months so has no EPG data left. It is therefore trying to get all the available data.

Why it is doing this I don't know. I've seen mine occasionally download the current slice plus the one for the day before, but I've never seen it download 10 slices like yours has!

It used to do that, prior to being unplugged for a couple of months.

If you use "Clear Download Logs" in "Account Management" you can force a TiVo to download all available slices like this.

So there definitely seems something wrong with your Tivo. It's either requesting too many slices or not remembering which ones it's already downloaded. I don't know enough about how the slice "request" process works to be able to suggest which.

My assumption is the TiVo needs to successfully complete a daily call before the server knows it has processed the slices. Since the calls keep getting cut-off at 1 hour the daily calls never succeed.

No because they are not "day" slices but each slice covers a 20 day period.

I wasn't expecting a rigid day slice.

My assumption is every day a new slice is generated containing the changes to the guide data. Those changes will be from the fetch of a new day of guide data a couple of weeks ahead, plus any edits to the current guide data. So for any given slice you will get mostly the newest available day for each channel, plus a smattering from earlier days.

Or are you saying each slice actually contains the entire 20 day EPG? If so why would we ever download more than 1 slice?


The basic problem seems to be if a dialup TiVo needs to download all available slices it may not be able to because of the 1 hour phone cutoff. One way round that is to reduce the amount of data it needs to download in a daily call.

The server knows which slices have been successfully downloaded from the server logs. Clearing Download Logs forces all available slices to be downloaded again. From this I conclude the list of slices to download is generated by the server and because the server never sees the daily call succeeding it keeps generating the same list.

If so it should be possible for the server to trim the list of slices in some way. If each daily call successfully downloads a small number of slices after a few days the TiVo will catch up.

[millsb]

As you can see it downloads multiple slices for an hour until disconnected and the last slice fails.

But they are not separate days. Look at the filenames for the slices you downloaded.
E.g. UK01003_15995-16014.slice
This says slice for lineup UK01003 (Sky SD) for days 15995-16014 (17/10/13 - 5/11/13).

Your next file download is for the same lineup but one day earlier (16/10/13 - 4/11/13).
Then it downloads another day earlier (15/10/13 - 3/11/13) (skip past the ones for UK01006 (Aerial))

Put them all in date order and you have tried:
UK01003_16001-16020
UK01003_16000-16019
UK01003_15999-16018
UK01003_15998-16017
UK01003_15997-16016
UK01003_15996-16015
UK01003_15995-16014
UK01003_15994-16013
UK01003_15993-16012
UK01003_15992-16011

You can see all the dates overlap. Basically the AltEPG server has sent you pretty much every slice it has for Sky SD!

Why it is doing this I don't know. I've seen mine occasionally download the current slice plus the one for the day before, but I've never seen it download 10 slices like yours has!

A second daily call will just attempt to download all the same slices again, even those that previously succeeded.

So there definitely seems something wrong with your Tivo. It's either requesting too many slices or not remembering which ones it's already downloaded. I don't know enough about how the slice "request" process works to be able to suggest which.

This is quite normal - as you say you don't know enough about the process, so it's not correct to conclude that there is anything wrong with the TiVo (or the server for that matter). I'll try to explain.

When TiVo makes a daily call, the server responds with the names of all the available slices (for the lineups the TiVo wants). The TiVo then decides which slices to download. It will download a slice unless it has previously downloaded and processed that slice. Only once it has downloaded all the available slices will it start processing them. In the case cited above one of the downloads fails, so TiVo abandons the whole lot and does not do any processing.

Although the available slices cover overlapping days, each slice does not contain all the data for every day in the range it covers - it only includes data for a given station/day combination if the programme data for that station/day has changed since the last time a slice was built for that station/day. So, strictly speaking, to build full programme data from scratch, the TiVo needs all the slices.

To reduce the impact for dial-up users, what could be done would be for the server to be selective about how many days worth of slices it advertises. Currently it advertises 14 days worth (slices older than 14 days get deleted from the server so it can never advertise more than that). It could advertise (say) just the last 7 days if the TiVo is dialling in rather than on network connection. Just a thought at this stage.

[DX30]

Thanks for the explanation but I'm still puzzled a bit by the following

When TiVo makes a daily call, the server responds with the names of all the available slices (for the lineups the TiVo wants). The TiVo then decides which slices to download. It will download a slice unless it has previously downloaded and processed that slice.

What is puzzling me is I don't see how the "Account Management" "Clear Download Logs" fits in here. If the TiVo has already downloaded and processed all the slices how does this work? I had assumed it was the server that was telling the TiVo what download and the TiVo just did it.

[millsb]

Thanks for the explanation but I'm still puzzled a bit by the following

When TiVo makes a daily call, the server responds with the names of all the available slices (for the lineups the TiVo wants). The TiVo then decides which slices to download. It will download a slice unless it has previously downloaded and processed that slice.

What is puzzling me is I don't see how the "Account Management" "Clear Download Logs" fits in here. If the TiVo has already downloaded and processed all the slices how does this work? I had assumed it was the server that was telling the TiVo what download and the TiVo just did it.

Sorry, I missed a little detail there. The server only advertises slices that have been created since the TiVo last did a successful download.

[excoriated]

You can see all the dates overlap. Basically the AltEPG server has sent you pretty much every slice it has for Sky SD!

As I said in the original post the TiVo was unplugged for a couple of months so has no EPG data left. It is therefore trying to get all the available data.

Sorry I forgot that bit So it will download all available slices and until you Tivo gets all of them it won't process any of them, so next time you dial in it will start again.