UPC’s flaky network down again today, in Cork

Impetus

All services were affected by an outage on UPS’s network serving Cork city today from 13.30 to 15.30. Does this company have any backup for its core infrastructure, one might ask?

UPC’s contention ratio is appallingly high, leading to slow screen fills.

This company is taking advantage of the user, thanks to its monopoly position in "fat pipe infrastructure".

It is time to unbundle broadband services to allow multiple service providers operate over monopoly pipelines (eg cable and fibre optic).

moonboy52

The network was down for 2 hours and you call it flaky?

I find UPC's network to have at least a 99% uptime. It is superb on my end for the last 5 years.

Outkast_IRE

Impetus wrote: »

All services were affected by an outage on UPS’s network serving Cork city today from 13.30 to 15.30. Does this company have any backup for its core infrastructure, one might ask?

UPC’s contention ratio is appallingly high, leading to slow screen fills.

This company is taking advantage of the user, thanks to its monopoly position in "fat pipe infrastructure".

It is time to unbundle broadband services to allow multiple service providers operate over monopoly pipelines (eg cable and fibre optic).

Did you really sign up to just moan about upc.

I am with them about 7 years now and in terms of uptime its about 99.999% .
In 7 years I only recollect it being down 4 or 5 times for a few hours at most.

UPC made the investment in the infrastructure, they can do with it as they like, to force them to share the infrastructure would be madness.

Impetus

“Loyal customers” / employees even... Talking.


The telecommunications industry worldwide works to a “five nines” 99.999% uptime.


Which equates to 5 minutes of failure per annum.


Not 2 hours.


And not an entire city blacked out for two hours.


I have never heard of a two hour city wide failure affecting eircom, Deutsche Telekom, Swisscom, Telefonica, etc.


Hopefully the newly proposed EU telco passport system will be realised soon to bring serious competition to the dozy Irish broadband market. A market where the publicans dilute the whiskey with 20 to 40 x water (the broadband contention ratios).

Outkast_IRE

Impetus wrote: »

“Loyal customers” / employees even... Talking.


The telecommunications industry worldwide works to a “five nines” 99.999% uptime.


Which equates to 5 minutes of failure per annum.


Not 2 hours.


And not an entire city blacked out for two hours.


I have never heard of a two hour city wide failure affecting eircom, Deutsche Telekom, Swisscom, Telefonica, etc.


Hopefully the newly proposed EU telco passport system will be realised soon to bring serious competition to the dozy Irish broadband market. A market where the publicans dilute the whiskey with 20 to 40 x water (the broadband contention ratios).

I am in cork , I wasn't affected on the day your talking about.

And how would I be an employee, anyone with any bit of sense can see from my post history what I am.

Impetus

Were you using the UPC network between 13.30 and 15.30 on 11.09.2013?

When I called UPC to report the issue, there was an RVA stating that ALL services were down in "the Cork area" during that period. The outage is by UPC's own admission.

(Of course the "Cork area" is as vague as mud -- there is no postcode system where they could specify a range of affected postcodes - eg 2000 to 2050 range. Yet another infrastructural deficit in Ireland's communications platform).

Maybe you were lucky, but judging by the postings to this threat it seems to me that everybody seems happy to "roll in the mud" of a second rate service economy. Which is probably why the quality of services of all types from buying a cup of coffee, to the service one gets from a financial institution is generally very poor, with a few notable exceptions (in the coffee space).

keithclancy

Impetus wrote: »

“Loyal customers” / employees even... Talking.


The telecommunications industry worldwide works to a “five nines” 99.999% uptime.


Which equates to 5 minutes of failure per annum.


Not 2 hours.


And not an entire city blacked out for two hours.


I have never heard of a two hour city wide failure affecting eircom, Deutsche Telekom, Swisscom, Telefonica, etc.


Hopefully the newly proposed EU telco passport system will be realised soon to bring serious competition to the dozy Irish broadband market. A market where the publicans dilute the whiskey with 20 to 40 x water (the broadband contention ratios).

UPC are excellent in regards to uptime, have them here in the Netherlands and same in Ireland, simply because they have their own cabling.

DSL in comparison is terribly flaky.

If anything I would say their only fault is billing and customer service, technically they are superb.

I think your expecting business service from a consumer product.

Then again we have had outages on business lines from Deutsche Telecom (Dusseldorf)

2 hours outage on a consumer line .. lol .. if you cannot live with that then get an Eircom Business line in as you have no SLA with the product you are currently using.

Impetus

I have used DSL in Ireland, and I have never had a breakdown that I can recall. I have cable and fibre in other locations too, so I think I am in a position to compare QoS issues.

Have you as an entity or has the entire ort/city of Düsseldorf (PLZ 40210 > 40629) had a citywide telecom failure from DT? If so, when please?

I have my own backup. The key point I was trying to make is that the UPC network covering Cork city was down (for all services). A properly managed phone company would have backup infrastructure to ensure that wide-spread outages do not happen.

Also UPC has not told us what element of the network fabric failed leading to the outage in question.

roast

Impetus wrote: »

A properly managed phone company would have backup infrastructure to ensure that wide-spread outages do not happen.

Out of interest, can you name a service provider in Ireland that has a redundant network in place as a backup?

Paulw

Eircom have had major outages in the past. Full telecoms outage in Dublin (phone/DSL). They have also had international outages (no in/out traffic for the whole country).

And since almost every other DSL provider uses the Eircom network, it effects them too.

UPC have an excellent record. I have used them for the last 7 years and have only had one short outage.

Impetus

Paulw wrote: »

Eircom have had major outages in the past. Full telecoms outage in Dublin (phone/DSL). They have also had international outages (no in/out traffic for the whole country).

And since almost every other DSL provider uses the Eircom network, it effects them too.

UPC have an excellent record. I have used them for the last 7 years and have only had one short outage.

When was this outage on eircom please? Did it affect the entire city or a single switching entity? I did a quick google and can't find anything, except for a DNS server problem (which one could fix by using an alternative such as opendns), and a fault affecting approx. 22,000 PSTN channels in the SE.

dazdrog

Paulw wrote: »

Eircom have had major outages in the past. Full telecoms outage in Dublin (phone/DSL). They have also had international outages (no in/out traffic for the whole country).

And since almost every other DSL provider uses the Eircom network, it effects them too.

UPC have an excellent record. I have used them for the last 7 years and have only had one short outage.

yeah and im sure everyone remembers eircoms near constant DNS issues at one stage

allen175

Been with UPC/NTL for years for broadband and like everyone else never had any serious downtime or intermittent issues, only the odd outage now and again which is fine, its to be expected, no ISP is perfect.

No ISP has a backup infrastructure, there was a huge outage for the south-east about 2 weeks ago that was caused by a eircom fibre cable being dug up, nothing could be done about that other than to repair the cable, think it was out for about 12 hours.

roast

Impetus wrote: »

When was this outage on eircom please? Did it affect the entire city or a single switching entity? I did a quick google and can't find anything, except for a DNS server problem (which one could fix by using an alternative such as opendns), and a fault affecting approx. 22,000 PSTN channels in the SE.

Oh buddy....

This kind of info isn't publicly available, but the amount of outages and the affected footprint in the last year alone would make your toes curl.

Paulw

Impetus wrote: »

When was this outage on eircom please? Did it affect the entire city or a single switching entity?

You won't find it on google.

It took out the majority of Dublin (that's Dublin 1-24 areas).

I am not at liberty to give specific dates, but I am sure it's in the archive of boards.ie if you can find it. There were a few large threads at the time.

Impetus

roast wrote: »

Oh buddy....

This kind of info isn't publicly available, but the amount of outages and the affected footprint in the last year alone would make your toes curl.

Perhaps this information should be available publicly, along with the contention ratios. In the interest of creating an informed marketplace. A job for ComReg. It would lead to an improvement in the QoS for customers, and enable customers to go with the provider that has the lowest contention and breakdown ratios.

When one buys food, the label has to show one nutritional information - saturated fats, carbohydrates, sodium etc in g/100.

A city with say 50,000 customers offline for 2h = 100,000 outage units. 300 customers offline for 7h = 2100 units, etc.

Improvement starts with measurement.

roast

Impetus wrote: »

Perhaps this information should be available publicly, along with the contention ratios. In the interest of creating an informed marketplace. A job for ComReg. It would lead to an improvement in the QoS for customers, and enable customers to go with the provider that has the lowest contention and breakdown ratios.

When one buys food, the label has to show one nutritional information - saturated fats, carbohydrates, sodium etc in g/100.

A city with say 50,000 customers offline for 2h = 100,000 outage units. 300 customers offline for 7h = 2100 units, etc.

Improvement starts with measurement.

The majority of outages are due to faults, issues that are not anticipated.
If it is anticipated that routine maintenance or upgrades may affect a customers connection, they are notified in advance. If an error develops over the course of scheduled maintenance or an upgrade, it is declared a fault/outage and thus considered as such.

If an outage were declared publicly, it would be after the fact.
Yes, it would be a measurement, but I fail to see how it would be an improvement, if the issue has already occurred?

Contention ratios, while not always advertised, are advised if questioned.

Impetus

roast wrote: »

The majority of outages are due to faults, issues that are not anticipated.
If it is anticipated that routine maintenance or upgrades may affect a customers connection, they are notified in advance. If an error develops over the course of scheduled maintenance or an upgrade, it is declared a fault/outage and thus considered as such.

If an outage were declared publicly, it would be after the fact.
Yes, it would be a measurement, but I fail to see how it would be an improvement, if the issue has already occurred?

Contention ratios, while not always advertised, are advised if questioned.

I suspect that most if not all outages are unanticipated. They can be minimised or even eliminated by redundancy in the core network. I am not expecting a telco to provide 100% uptime to an individual subscriber, unless they themselves are paying for that last km of redundancy.

However if a company strings up a "network" and makes no provision for outage in the form of redundant paths and switches, it would be asking for trouble. And imposing trouble on the customer.

Which is why I say that outage performance should be published in a standardised basis routinely. Similarly for contention. It is not true to suggest that contention ratios will be advised to a potential customer on request, in my experience. Eircom for one, does not appear to have provided this information to its sales or support staff. Another favourite lie seems to be "we have no contention".

SpaceTime

Bear in mind that the cable network uses shared trunks that run along buildings / underground from a fibre-supplied local street cabinet. Every house on a given route connects to the same cable. They are not individual cables connected back to a central point like a telephone network.

A problem with one of those trunks can knock out a street or a problem with a cabinet can knock out several streets.

The problem seems to be that UPC gave a "in the Cork area" service message because of that usual Irish mentality that there's "Dublin" which has lots of areas and then other cities/major towns are just one big area to be lumped together when speaking about them.

The UPC service in Cork was not down, the UPC service in a small area of Cork may well have been.

Also, UPC's network's being progressively rebuilt, so some areas of Cork may have much better service levels than others depending on whether the local infrastructure has been replaced or not.

In most of these networks the core is pretty secure and redundant, the problem will usually be some piece of local infrastructure - typically a cable.

roast

Impetus wrote: »

I suspect that most if not all outages are unanticipated. They can be minimised or even eliminated by redundancy in the core network. I am not expecting a telco to provide 100% uptime to an individual subscriber, unless they themselves are paying for that last km of redundancy.

However if a company strings up a "network" and makes no provision for outage in the form of redundant paths and switches, it would be asking for trouble. And imposing trouble on the customer.

Which is why I say that outage performance should be published in a standardised basis routinely. Similarly for contention. It is not true to suggest that contention ratios will be advised to a potential customer on request, in my experience. Eircom for one, does not appear to have provided this information to its sales or support staff. Another favourite lie seems to be "we have no contention".

There is a certain amount of redundancy in place for the majority of networks.

However, you have to take into account the feasibility of switching to a workaround or (for example) a redundant fibre line. Will it take longer to switch routing through a different node than it will to fix the initial problem?

For example - consumer lines. Lets say, a PSTN line. Eircom have an SLA (by standard) of 5 working days to repair a "standard fault". In the event of an outage that's affecting 5,000 PSTN lines, is it quicker to focus on resolving the initial issue, than it is to have half the manpower working on the resolution and half switching everyone to a workaround? Yes, absolutely, if it's just a case that the exchange has been isolated from the core network.

In the case of an outage due to backhaul (be it a fibre feed to an exchange, or a wireless P2P link) in the majority of cases, regardless of redundancy, the issue is unavoidable.

Furthermore, on business/corporate lines, (while it's always advisable for a backup to be in place anyway) in the event of a fault, there may be a shorter SLA in place, depending on the tariff. Regardless, the same problem applies. An advanced SLA is what, two hours? The outage would be resolved within SLA anyway.

Impetus

Surely a modern network is built with re-routing already planned in the system fabric. If one path fails, traffic is automatically re-routed over a working path, until the fault is fixed. Re-routing should involve little or no time or human effort. Terms like "self-healing" and "fault tolerant" come to mind.

roast

Impetus wrote: »

Surely a modern network is built with re-routing already planned in the system fabric. If one path fails, traffic is automatically re-routed over a working path, until the fault is fixed. Re-routing should involve little or no time or human effort. Terms like "self-healing" and "fault tolerant" come to mind.

Easy to do on a small, local network. When you're talking WAN infrastructure, the game changes.

SpaceTime

d

Impetus wrote: »

Surely a modern network is built with re-routing already planned in the system fabric. If one path fails, traffic is automatically re-routed over a working path, until the fault is fixed. Re-routing should involve little or no time or human effort. Terms like "self-healing" and "fault tolerant" come to mind.

Not if the network element that has broken down is a cable running across the eves of 10 houses somewhere in Cork City.

They're hardly going to install two working lines to everyone's house just in case one fails.

The outage you're describing sounds like a very localised cable fault. There were no core network issues on UPC's network in recent months.

The most likely problem there is a broken coaxial cable in the local distribution network, not the core UPC fibre network which would generally speaking be very secure and have a lot of built-in redundancy.

In most networks in Ireland there would be multiple routes from one node to another, and if there was a fault on one of those links then traffic would flow the other way (albeit with possibly reduced capacity) but, on local lines to houses / groups of houses there's very little you can do other than repair the fault within a reasonable time.

....

You should report the fault to UPC and keep reporting it if it reoccurs or if you're getting speed issues.
They may need to replace a faulty cable or a tap or amp somewhere and sometimes these faults can only be identified by complaints.

You can check for explanations to faults here : http://www.upc.ie/customer_support/service_status/

roast

On a side note...
The below is a report from a server I use on a residential UPC connection over the period of a year. 2hrs, 55mins of downtime in total. The majority of this is due to server restarts and maintenance.

Seems good enough to me...

keithclancy

Impetus wrote: »

I suspect that most if not all outages are unanticipated. They can be minimised or even eliminated by redundancy in the core network. I am not expecting a telco to provide 100% uptime to an individual subscriber, unless they themselves are paying for that last km of redundancy.

However if a company strings up a "network" and makes no provision for outage in the form of redundant paths and switches, it would be asking for trouble. And imposing trouble on the customer.

Which is why I say that outage performance should be published in a standardised basis routinely. Similarly for contention. It is not true to suggest that contention ratios will be advised to a potential customer on request, in my experience. Eircom for one, does not appear to have provided this information to its sales or support staff. Another favourite lie seems to be "we have no contention".

If you want to be protected from something like that then get a business line.

You are using a consumer product.

Impetus

roast wrote: »

Easy to do on a small, local network. When you're talking WAN infrastructure, the game changes.

Rubbish! When one is running a big network, there is plenty of money to pay for alternative routing strategies. If UPC or eircom or anybody else was a tiny village network, your "excuse" might be valid.

roast

Impetus wrote: »

Rubbish! When one is running a big network, there is plenty of money to pay for alternative routing strategies. If UPC or eircom or anybody else was a tiny village network, your "excuse" might be valid.

Again, read my previous responses. As rubbish as it may seem to you, these are facts.

Impetus

SpaceTime wrote: »

d

Not if the network element that has broken down is a cable running across the eves of 10 houses somewhere in Cork City.

They're hardly going to install two working lines to everyone's house just in case one fails.

The outage you're describing sounds like a very localised cable fault. There were no core network issues on UPC's network in recent months.

The most likely problem there is a broken coaxial cable in the local distribution network, not the core UPC fibre network which would generally speaking be very secure and have a lot of built-in redundancy.

In most networks in Ireland there would be multiple routes from one node to another, and if there was a fault on one of those links then traffic would flow the other way (albeit with possibly reduced capacity) but, on local lines to houses / groups of houses there's very little you can do other than repair the fault within a reasonable time.

....

You should report the fault to UPC and keep reporting it if it reoccurs or if you're getting speed issues.
They may need to replace a faulty cable or a tap or amp somewhere and sometimes these faults can only be identified by complaints.

You can check for explanations to faults here : http://www.upc.ie/customer_support/service_status/

Read the thread.... UPC admitted that the fault was service-wide and city-wide. Not local.

Impetus

roast wrote: »

Again, read my previous responses. As rubbish as it may seem to you, these are facts.

http://www.open-mpi.org/papers/dapsys-2006-self-healing-network/dapsys-2006-self-healing-network.pdf

Impetus

keithclancy wrote: »

If you want to be protected from something like that then get a business line.

You are using a consumer product.

Consumer or business user, one is entitled to a quality of service. Hence the need for publicly available service reliability and contention ratio statistics for all, from ComReg. Give people the information they should have to make an informed choice.

cali_eire

Impetus wrote: »

“Loyal customers” / employees even... Talking.


The telecommunications industry worldwide works to a “five nines” 99.999% uptime.


Which equates to 5 minutes of failure per annum.


Not 2 hours.


And not an entire city blacked out for two hours.


I have never heard of a two hour city wide failure affecting eircom, Deutsche Telekom, Swisscom, Telefonica, etc.


Hopefully the newly proposed EU telco passport system will be realised soon to bring serious competition to the dozy Irish broadband market. A market where the publicans dilute the whiskey with 20 to 40 x water (the broadband contention ratios).

To be fair I just moved back from Silicon Valley in California and our UPC equivalent there which is called Comcast regularly had outages so I wouldn't be so quick to think far fields are greener.