SIRO and Vodafone: infuriating connectivity problems

RoundCube

A month ago I got SIRO installed and everything was great.

Since last Thursday I've noticed that my local VMs provisioning started failing massively due to connectivity issues. So, I made some investigations:

1) There are some problems with establishing TCP connections with North American IPs. E.g. curl google.com may hang forever and that happens often. This makes browsing inconvenient and it breaks my VM provisioning (and this is infuriating). Example hosts I've tried: twitter.com / reddit.com / some hosts us-west/us-east AWS zones.
1.1) The connections either not receive SYN-ACK (stuck in SYN-SENT state) or establish but do not receive any further responses. In case a connections establishes and receives some data there would be no any further problem with it.
2) ICMP works fine, there are no losses or anomalies when I ping the same addresses, even when I flood ping (with 0.01s pause)
3) Everything is fine with local Irish addresses (e.g. vodafone.ie, eu-west-1 AWS zone, etc)
4) This problem is specific to Vodafone. In case I replace my WAN connection with Three's LTE everything works just fine
5) In traceroutes first host after the gateway (89.19.72.21) sometimes shows stars (may be irrelevant but surprising)
6) I didn't observe such issues before last thursday, everything was great
7) Sometimes the issue may not manifest for prolonged periods of time (1-2 hours) but it always returns back

I use this simple bash line to test:

THOST=google.com ;  while true; do (curl -4 -q --max-time 3 $THOST > /dev/null 2>&1 && echo "$(date): $THOST replied") || (echo "$(date): $THOST timed out") && sleep 0.5 ; done

It just queries google.com with curl (plain old http, no TLS) with 3s timeout (more than enough for a roundtrip to USA) and prints if the server responded successfully or the request timed out.

Typical results on LTE (Three network) look like:

Sun  6 Jun 2021 21:56:54 IST: google.com replied
Sun  6 Jun 2021 21:56:54 IST: google.com replied
Sun  6 Jun 2021 21:56:55 IST: google.com replied
Sun  6 Jun 2021 21:56:55 IST: google.com replied
Sun  6 Jun 2021 21:56:56 IST: google.com replied
Sun  6 Jun 2021 21:56:57 IST: google.com replied
Sun  6 Jun 2021 21:56:57 IST: google.com replied
Sun  6 Jun 2021 21:56:58 IST: google.com replied
Sun  6 Jun 2021 21:56:58 IST: google.com replied
Sun  6 Jun 2021 21:56:59 IST: google.com replied
Sun  6 Jun 2021 21:56:59 IST: google.com replied
Sun  6 Jun 2021 21:57:00 IST: google.com replied
Sun  6 Jun 2021 21:57:01 IST: google.com replied
Sun  6 Jun 2021 21:57:01 IST: google.com replied
Sun  6 Jun 2021 21:57:02 IST: google.com replied
Sun  6 Jun 2021 21:57:02 IST: google.com replied
Sun  6 Jun 2021 21:57:03 IST: google.com replied
Sun  6 Jun 2021 21:57:03 IST: google.com replied
Sun  6 Jun 2021 21:57:04 IST: google.com replied
Sun  6 Jun 2021 21:57:05 IST: google.com replied
Sun  6 Jun 2021 21:57:06 IST: google.com replied
Sun  6 Jun 2021 21:57:06 IST: google.com replied
Sun  6 Jun 2021 21:57:07 IST: google.com replied
Sun  6 Jun 2021 21:57:08 IST: google.com replied
Sun  6 Jun 2021 21:57:09 IST: google.com replied
Sun  6 Jun 2021 21:57:09 IST: google.com replied
Sun  6 Jun 2021 21:57:10 IST: google.com replied
Sun  6 Jun 2021 21:57:11 IST: google.com replied
Sun  6 Jun 2021 21:57:11 IST: google.com replied
Sun  6 Jun 2021 21:57:12 IST: google.com replied
Sun  6 Jun 2021 21:57:13 IST: google.com replied
Sun  6 Jun 2021 21:57:13 IST: google.com replied
Sun  6 Jun 2021 21:57:14 IST: google.com replied
Sun  6 Jun 2021 21:57:15 IST: google.com replied
Sun  6 Jun 2021 21:57:15 IST: google.com replied
Sun  6 Jun 2021 21:57:16 IST: google.com replied
Sun  6 Jun 2021 21:57:17 IST: google.com replied
Sun  6 Jun 2021 21:57:17 IST: google.com replied
Sun  6 Jun 2021 21:57:18 IST: google.com replied
Sun  6 Jun 2021 21:57:18 IST: google.com replied
Sun  6 Jun 2021 21:57:19 IST: google.com replied
Sun  6 Jun 2021 21:57:20 IST: google.com replied
Sun  6 Jun 2021 21:57:21 IST: google.com replied
Sun  6 Jun 2021 21:57:21 IST: google.com replied
Sun  6 Jun 2021 21:57:22 IST: google.com replied
Sun  6 Jun 2021 21:57:23 IST: google.com replied
Sun  6 Jun 2021 21:57:23 IST: google.com replied
Sun  6 Jun 2021 21:57:24 IST: google.com replied
Sun  6 Jun 2021 21:57:24 IST: google.com replied
Sun  6 Jun 2021 21:57:25 IST: google.com replied
Sun  6 Jun 2021 21:57:26 IST: google.com replied
Sun  6 Jun 2021 21:57:26 IST: google.com replied

And on SIRO it's...

Sun  6 Jun 2021 22:33:13 IST: google.com replied
Sun  6 Jun 2021 22:33:13 IST: google.com replied
Sun  6 Jun 2021 22:33:17 IST: google.com timed out
Sun  6 Jun 2021 22:33:17 IST: google.com replied
Sun  6 Jun 2021 22:33:18 IST: google.com replied
Sun  6 Jun 2021 22:33:20 IST: google.com replied
Sun  6 Jun 2021 22:33:20 IST: google.com replied
Sun  6 Jun 2021 22:33:22 IST: google.com replied
Sun  6 Jun 2021 22:33:23 IST: google.com replied
Sun  6 Jun 2021 22:33:26 IST: google.com timed out
Sun  6 Jun 2021 22:33:27 IST: google.com replied
Sun  6 Jun 2021 22:33:28 IST: google.com replied
Sun  6 Jun 2021 22:33:31 IST: google.com timed out
Sun  6 Jun 2021 22:33:32 IST: google.com replied
Sun  6 Jun 2021 22:33:32 IST: google.com replied
Sun  6 Jun 2021 22:33:33 IST: google.com replied
Sun  6 Jun 2021 22:33:33 IST: google.com replied
Sun  6 Jun 2021 22:33:34 IST: google.com replied
Sun  6 Jun 2021 22:33:34 IST: google.com replied
Sun  6 Jun 2021 22:33:38 IST: google.com timed out
Sun  6 Jun 2021 22:33:38 IST: google.com replied

I've tried to contact VF support and got the following responses:

First attempt: you can't use anything apart of Vodafone Gigabox. Ah, you switched to it? Fine, we've reset your line, your numbers are great, everything will be fine.

Second attempt: we passed your issue/screenshots/diagnostics to our NOC team and they said the issues you experience are normal.

Third attempt: we passed your issue/screenshots/diagnostics to our team and eventually we'll be back.

Could you give me any advice how to explain them that such frequent timeouts can't be a "norm" (even damn LTE works without any issues!!!)?

Do you experience any similar problems? (Essentially, when you browse pages may stuck and never load)

Could you run this test for google.com/twitter.com/reddit.com and report here if there were any timeouts and what ISP/connection/plan you use please?

For me it looks like a faulty/incorrectly configured router (border one I guess) somewhere in Vodafone network, most likely improperly configured QoS/shaping rules. ICMP works well, it's definitely something TCP-specific.

RoundCube

I can't post screenshots here, but this is how my pings look like on my monitoring: imgur.com/5YLjgVn

Damn stable! The red box denotes the time when I've been switching between LTE and SIRO.

ED E

RoundCube wrote: »

I can't post screenshots here, but this is how my pings look like on my monitoring: imgur.com/5YLjgVn

Damn stable! The red box denotes the time when I've been switching between LTE and SIRO.

RoundCube

Fun fact: in case I connect to a VPN server here in Ireland (Node in eu-west-1, OpenVPN, UDP), I do not experience this problem.

VF support is just "passing my feedback to a relevant team". No help at all.

Even funnier: VPN to the US removes the problem too. So, both ICMP and UDP work well. TCP is broken.

ED E

Practically speaking you wont be able to speak to the people you need to without an enterprise account.
1) Wait and hope
2) VPN all the time
3) Change SP

RoundCube

ED E wrote: »

1) Wait and hope

It's continuing for 5 days in a row.

ED E wrote: »

2) VPN all the time

That's kind of an option, but it limits throughput. And, eghm, I'm kinda paying for a service which is supposed to be good without any tricks.

ED E wrote: »

3) Change SP

I'm on an 12-month contract. Though I won't mind terminating it because these issues hamper my work. And that's why I asked about other's experiences. It may be equally a Vodafone problem or SIRO problem, or some kind of an upstream operator issue and other ISPs may have similar issues too.

Praetorian

I've had awful connectivity the last few days with Vodafone as well. Changed to Google DNS which seemed to fix it for a while but the issues have come back. Sometimes multiple attempts needed to refresh a website.

Depressing as I really don't want to change provider as I'll lose my glorious 200 m/bit upload speed

Vodafone tech support is next to useless. Just have to hope it is eventually fixed. Really surprised how many periods of downtime or bad connectivity we've had with Siro / Vodafone. Always seem to be during weekends as well, when nobody is around to fix anything.

Famil is all on Eir ftth and receiving absolutely rock-solid reliability.

RoundCube

> Changed to Google DNS which seemed to fix it for a while but the issues have come back.
This is clearly not a DNS issue. Likely you just got into a "good timeframe".

>Sometimes multiple attempts needed to refresh a website.
Yes, that's it. Could you run my test please?

> Famil is all on Eir ftth
Eir is not available in my area and I would prefer to avoid them at all costs anyway. Some time ago I had to prove them my point with phone recordings, they were dealing very unfair.

RoundCube

Well, I had a long conversation with their techsupport and it's totally useless. I'll never subscribe for their service again nor recommend it.

Though I made an educated guess and removed my Static IP addon. Aaaand... the issue is gone. Completely. I have blazing fast network again.

So, the problem is defintely somewhere in their network. The only problem is that I actually need a static IP. And the real problem is that their techsupport is useless. Either incompetent or unwilling to help.

Praetorian

As per below same issues for us.

My IP is dynamic though. I might hit my head off the vodafone tech support wall again tomorrow.

Tue 8 Jun 2021 22:06:05 IST: google.com replied
Tue 8 Jun 2021 22:06:08 IST: google.com timed out
Tue 8 Jun 2021 22:06:09 IST: google.com replied
Tue 8 Jun 2021 22:06:09 IST: google.com replied
Tue 8 Jun 2021 22:06:10 IST: google.com replied
Tue 8 Jun 2021 22:06:12 IST: google.com replied
Tue 8 Jun 2021 22:06:13 IST: google.com replied
Tue 8 Jun 2021 22:06:13 IST: google.com replied
Tue 8 Jun 2021 22:06:14 IST: google.com replied
Tue 8 Jun 2021 22:06:17 IST: google.com timed out
Tue 8 Jun 2021 22:06:19 IST: google.com replied
Tue 8 Jun 2021 22:06:20 IST: google.com replied
Tue 8 Jun 2021 22:06:20 IST: google.com replied
Tue 8 Jun 2021 22:06:21 IST: google.com replied
Tue 8 Jun 2021 22:06:21 IST: google.com replied
Tue 8 Jun 2021 22:06:22 IST: google.com replied
Tue 8 Jun 2021 22:06:25 IST: google.com timed out
Tue 8 Jun 2021 22:06:26 IST: google.com replied

alan4cult

What time of day are you observing it at. Is it only evening?
There was earlier posts here about Vodafone seeing traffic peering issues in the evening time.

RoundCube

> What time of day are you observing it at. Is it only evening?

Any time of the day including 3am.

> in the evening time.
Nope, it doesn't seem to depend on the time.

Also once I turned static IP off the issue got "fixed". No timeouts, everything is perfect. I need a static IP though

RoundCube

Praetorian wrote: »

As per below same issues for us.

My IP is dynamic though. I might hit my head off the vodafone tech support wall again tomorrow.

Tue 8 Jun 2021 22:06:05 IST: google.com replied

Are you on FTTH or FTTC?

RoundCube

Here is the same test rewritten in Python.
Provides bit more info.

import socket

from contextlib import contextmanager
from timeit import default_timer
import http.client
import time

@contextmanager
def elapsed_timer():
    start = default_timer()
    elapser = lambda: default_timer() - start
    yield lambda: elapser()
    end = default_timer()
    elapser = lambda: end-start

host = "google.com"
timeout = 1.0
pause = 0.05
count = 100

c = 0
goodc = 0
badc = 0

while c < count:
    c = c+1
    with elapsed_timer() as elapsed:
        ip = "?"
        restime = "?"
        r2 = "?"
        reqtime = "?"
        sz = "?"
        code = "?"
        datalen = "?"
        status = "PASS"
        try:
            ip = socket.gethostbyname(host)
            restime = "{:.4f}".format(elapsed())
            conn = http.client.HTTPConnection(ip, timeout=timeout)
            conn.request("GET", "/")
            r2 = conn.getresponse()
            code = str(r2.status)
            data2 = r2.read()
            datalen = len(data2)
            conn.close()
            reqtime = "{:.4f}".format(elapsed())
            time.sleep(pause)
            goodc = goodc + 1
        except:
            status = "FAIL"
            badc = badc + 1
            pass
        print("status: &#37;s, host: %s, ip: %s, dns time: %s, code: %s, size: %s, request time: %s, timeout: %s" % (status, host, ip, restime, code, datalen, reqtime, timeout))

print("PASSED: %d, FAILED: %d" % (goodc, badc))

Example output:

status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0005, code: 301, size: 219, request time: 0.0164, timeout: 1.0
status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0007, code: 301, size: 219, request time: 0.0166, timeout: 1.0
status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0005, code: 301, size: 219, request time: 0.0165, timeout: 1.0
status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0006, code: 301, size: 219, request time: 0.0165, timeout: 1.0
status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0007, code: 301, size: 219, request time: 0.0177, timeout: 1.0
status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0005, code: 301, size: 219, request time: 0.0168, timeout: 1.0
status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0010, code: 301, size: 219, request time: 0.0177, timeout: 1.0
status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0005, code: 301, size: 219, request time: 0.0180, timeout: 1.0
status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0009, code: 301, size: 219, request time: 0.0194, timeout: 1.0
status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0005, code: 301, size: 219, request time: 0.0208, timeout: 1.0
status: PASS, host: google.com, ip: 74.125.193.113, dns time: 0.0008, code: 301, size: 219, request time: 0.0184, timeout: 1.0
PASSED: 100, FAILED: 0

RoundCube

So far that's what I got from them:

RoundCube

Actually I've just discovered a tool named httping which does the same job as my scripts

alan4cult

Why do you need a static IP, also I didn't realise there was a way to get an external Static IP without Vodafone switching you manually, how are you able to get one yourself?

RoundCube

alan4cult wrote: »

Why do you need a static IP, also I didn't realise there was a way to get an external Static IP without Vodafone switching you manually, how are you able to get one yourself?

You can turn static ip on/off in My Vodafone.

I need it because I work with a company which requires those who have access to their aws infrastructure to have a static ip to whitelist it.

VPN is an option, but it's lot slower and, for example, yesterday I had to transfer 200gb of statistical data for my local experiments. Usualy they collect about 20-100gb of stats per day and currently I'm working on their in-house analytics engine.

In order to make deployments and debugging faster and easier I use my local hardware (actually I have a threadripper 3970x specifically for these experiments) instead of aws during development phase. But I need to access huge chunks of real data to validate my outputs.

alan4cult

Ha this might be a bit of an XY problem but it would nearly make more sense to send your local hardware into work and remote onto it from home or wherever avoiding pushing 200G down your local link. Also I'm assuming that 200G is compressed?

RoundCube

alan4cult wrote: »

Ha this might be a bit of an XY problem but it would nearly make more sense to send your local hardware into work and remote onto it from home or wherever avoiding pushing 200G down your local link. Also I'm assuming that 200G is compressed?

Remote debugging is not that convenient.


Yeah, compressed.

RoundCube

So, how it ended: Vodafone acknowledged that they can't fix the issue and kindly terminated my contract w/o exit fees.

I still can't imagine how they corporate logic looks like - it makes no sense to lose clients because of an obvious misconfiguration at their side. Everything worked and it must be possible to track down and fix the problem instead of saying "that's how it works for now, in case you have a static ip your connection is broken".

But it is how it is.

KildareP

RoundCube wrote: »

So, how it ended: Vodafone acknowledged that they can't fix the issue and kindly terminated my contract w/o exit fees.

I still can't imagine how they corporate logic looks like - it makes no sense to lose clients because of an obvious misconfiguration at their side. Everything worked and it must be possible to track down and fix the problem instead of saying "that's how it works for now, in case you have a static ip your connection is broken".

But it is how it is.

Often with large corporates like Vodafone it's deemed cheaper to cut relatively isolated issues like yours loose than spend time and money escalating it up and beyond Tier 2. They want the mass market people who just pay the bill every month and maybe complain to friends and family (but not to vodafone).

I wouldn't touch them personally for any service, fixed or mobile, these days based on numerous past negative dealings with them. I find the independently owned ISP's far better, easier to contact and more willing to work with you to root out any issues that do arise than the likes of Vodafone, Eir or Virgin Media.

RoundCube

Almost the end of the story: Digiweb took my connection over this morning, everything looks great and there are no connectivity issues.

Now I just need to cancel Vodafone service and the story will be over.

Conclusion: use Digiweb.

httping -i 0.05 http://google.com

on Digiweb gives me:

--- http://google.com/ ping statistics ---
116 connects, 116 ok, 0.00% failed, time 8419ms
round-trip min/avg/max = 16.1/19.1/109.8 ms

Looks perfect.

TallGlass2

BTW if you needed a static address but are assigned Dynamic you could use something like DNS Duck to track your IP changes and link your DNS Duck address to the whitelist.

RoundCube

TallGlass2 wrote: »

BTW if you needed a static address but are assigned Dynamic you could use something like DNS Duck to track your IP changes and link your DNS Duck address to the whitelist.

Yes, I can. Though it may be hard to agree with our devops team to poll my DNS zone for my IP updates. Apart of that - yup, of course I would prefer dynamic IP, bit less tracking online, etc.

RoundCube

Also, regarding the dynamic/static problem. I didn't report it in this thread, but in fact I made bit more diagnostics before I switched and found that yes, when static IP is on the connectivity with North America is indeed broken. And indeed switching to dynamic IP fixes that.

But with dynamic IP connectivity with Easter Europe (Poland, Ukraine, Russia) and China is broken. Usually I don't need to visit their websites/etc but I did it as a part of investigation.

Surprisingly there were at least a couple IP ranges with good connectivity with all the world but in most of the cases Easter Europe connectivity was broken.

So, seems like on Vodafone you may either talk with the US hosts or eastern hosts. You may choose just a half of the (interconnected) world

UDP/ICMP worked great with all the regions, including Australia. And everything seems to be great on Digiweb.

McSim

Experience connectivity problems with SIRO and Vodafone myself.

My test case is:

nslookup -type=TXT intel.com. 8.8.8.8

In 98% of cases it just sits there, not receiving DNS reply.

The reason is that DNS reply is too large to fit inside UDP packet, so DNS query is retried over TCP.

And pretty much DNS does not work over TCP via Vodafone Gigabox. I tried to disable Firewall and mess with ALG settings without success.

More generic test case will be

nslookup -vc <name to lookup> <dns server to use>

-vc tells nslookup to use TCP queries for everything

Further to add, as noted in other threads on boards, Gigabox firewall logs the following for this particular DNS failure

SRC=192.168.X.X DST=8.8.8.8 PROTO=TCP SPT=59770 DPT=53 MARK=0x2 DROP

Also quite a few http outbound connections dropped with

SRC=192.168.X.X DST=X.X.X.X PROTO=TCP SPT=42222 DPT=443 MARK=0x3 DROP

McSim

Setting Secure DNS to OFF solves DNS over TCP issue with my Gigabox.

Finally, after 3 months!

Thanks to https://www.boards.ie/discussion/2058223043/vodafone-ftth-dns-hijacking-interception

With setting above Gigabox continue to serve own IP address as address of DNS server to DHCP clients.

Specifying additional DNS servers like below adds them to the list of DNS servers served by Gigabox to DHCP clients, after Gigaboxe's own IP address.

RoundCube

I dind't use their router at all, so my issue wasn't related to yours.

edenlt

i have a Synology router, having the same issues right now, tried to talk with support :D "your problem was escalated upstream and call u back tomorrow". useless.

at first, tried switching routers back to VF generic one, but no luck there.

restarted Siro to get another IP. no luck either.

DNS changing not helps

if I tracert to any IP and try to pingback all IP from the nearest. witch is SIRO it immediately starts dropping packets. like PC[ok]->synology[ok]->VF generic router[ok]->SIRO[FAILL]

my contract ends by the end of this month if this week the problem won't be solved going somewhere else.

edenlt

even pinging DNS fails