Wednesday, November 28, 2012, 15:41

Internet Backbone Issues Identified

Update: This issue has been resolved.

We are currently seeing connectivity problems in North America with the various service providers including: Level3.net, comcast.net, aliant.net, etc. If you are experience e-mail or other connectivity problems it may be related to this problem. We're seeing this problem in network hubs in New York, Chicago, Dallas, Los Angeles. If your network traffic goes through these hubs you may experience problems.

There is partial connectivity. In tests we are able to "connect" to servers, but the connectivity does not appear to be consistent and often fails.

Here's a sample of the packet loss we're seeing between Buffalo and Toronto, which gets routed through New York:

                                  Snt: 300   Loss%  Last   Avg  Best  Wrst StDev
...
BLINK-COMMU.car2.Toronto2.Level3.net          0.0%   1.0   3.3   0.8  90.2   7.9
ge-6-24.car2.Toronto2.Level3.net              0.0%   2.0  20.2   0.9 301.3  43.4
ae-3-3.bar1.Toronto1.Level3.net              53.7%   6.0   8.1   4.9  67.4  10.8
ae-7-1.ebr2.NewYork1.Level3.net              56.3%  19.0  18.7  17.9  21.2   0.6
ae-82-82.csw3.NewYork1.Level3.net            48.7%  18.0  21.8  17.9  31.1   4.0
ae-3-80.edge3.NewYork1.Level3.net            49.3%  51.0  22.0  17.9  81.0  12.0
VASPIAN.car1.Buffalo1.Level3.net              0.0%  26.0  27.3  25.6  46.0   1.5
...

                                  Snt: 120   Loss%  Last   Avg  Best  Wrst StDev
...
pos-3-15-0-0-cr01.ashburn.va.ibone.comcast.n 50.8%  24.0  25.5  22.0  49.0   4.4
pos-0-14-0-0-ar01.taylor.mi.michigan.comcast 53.3%  32.2  32.7  28.9  62.9   4.6
te-9-1-ur02.ypeast.mi.michigan.comcast.net   49.2%  29.8  30.8  29.7  49.4   3.1
te-18-10-cdn03.ypeast.mi.michigan.comcast.ne 48.3%  47.3  48.2  34.0 232.2  24.1
lns-mailgw.lantology.com                     59.2%  40.1  41.6  37.0  78.9   8.7
...

                                  Snt: 120   Loss%  Last   Avg  Best  Wrst StDev
...
ae6.bx01.mtrl.pq.aliant.net                  56.3%  24.4  25.9  23.2 109.1   8.9
ae5.cr02.stjh.nb.aliant.net                  53.7%  29.2  31.0  28.3  88.0   8.5
ae6.cr02.hlfx.ns.aliant.net                  52.7%  29.5  31.5  28.5  79.1   7.9
xe-0-0-0-new.gw01.hlfx.ns.aliant.net         51.3%  38.5  35.7  28.7 104.9  12.5
...

-- David Rutherford