Web Host Chat - The UK's host forum since 2001!
RECOMMENDED HOSTS
Colocation Rack Services
UK Web & Reseller Hosting
UK Leased Lines
VMWare Cloud Servers
Advertise here!
Results 1 to 7 of 7
Like Tree2Likes
  • 1 Post By connected
  • 1 Post By Ed-Freethought

Thread: iomart outage

  1. #1
    Array

    iomart outage

    Seems to be an issue here

    iomart hosting Service Status | Major Incident

    Have clients who use a system hosted here and are very annoyed.

    Nothing to do with me but wondering if any one here as more access to what issue might be and what steps could have been taken to prevent it.

    Mike
    Mike Clift
    Mandarin Business Solutions Limited | Hosting for the UK Travel and Tourism Industry

  2. #2
    132 rating
    23 reviews
    Posts 13826
    Post rank 1 (0)
    Array
    Quote Originally Posted by mandarin View Post
    Seems to be an issue here

    iomart hosting Service Status | Major Incident

    Have clients who use a system hosted here and are very annoyed.

    Nothing to do with me but wondering if any one here as more access to what issue might be and what steps could have been taken to prevent it.

    Mike
    They had two simultaneous fibre cuts on their network - one between Glasgow and Nottingham on the east side and one between Manchester and Maidenhead on the west side which effectively partitioned the northern part of their network.

    The only fix would be more (diverse) fibre paths, for example a direct Manchester to Nottingham link.
    Freethought Internet
    Freethought Internet Limited registered in London No. 5862996. Registered office: The Old Church Hall, 2A Cromwell Street, Lincoln, LN2 5LP. VAT number GB 987 0952 66.

  3. #3
    132 rating
    23 reviews
    Posts 13826
    Post rank 1 (0)
    Array
    Quote Originally Posted by Ed-Freethought View Post
    The only fix would be more (diverse) fibre paths, for example a direct Manchester to Nottingham link.
    I should have said, in these kind of situations it's very hard to strike an appropriate balance between redundancy and cost. For example, even with a Manchester to Nottingham link, if the fibre cut had been between Nottingham and Leicester instead of Glasgow and Nottingham then the network would still have been partitioned.

    Short of having 3 (or more) separate paths to each data centre, there's not much that you can do to protect against this kind of event.
    Freethought Internet
    Freethought Internet Limited registered in London No. 5862996. Registered office: The Old Church Hall, 2A Cromwell Street, Lincoln, LN2 5LP. VAT number GB 987 0952 66.

  4. #4
    Registered User Array
    I disagree, this is indicative of a somewhat poorly architected network. If the main role of your network is to get shuffle packets between your datacentres and the internet (as aposed to say a internal network without external connectivity) then you should be able to survive a spilt between which isolates Manchester and Glasgow from the southern (London fed) network.

    Transit picked up just in Manchester or Glasgow (both better) along with a sensible dynamically routed network design which doesn't leak /24s between datacentres would have allowed all systems to be reachable, even if some internal cross country system replication traffic was lost. I suspect Iomart probably do pickup some limited amount of transit in either Manchester or Glasgow but either not an adequate amount or for some reason they had poorly designed route redistribution or internal routing policy.
    mandarin likes this.

  5. #5
    132 rating
    23 reviews
    Posts 13826
    Post rank 1 (0)
    Array
    Quote Originally Posted by connected View Post
    I disagree, this is indicative of a somewhat poorly architected network. If the main role of your network is to get shuffle packets between your datacentres and the internet (as aposed to say a internal network without external connectivity) then you should be able to survive a spilt between which isolates Manchester and Glasgow from the southern (London fed) network.

    Transit picked up just in Manchester or Glasgow (both better) along with a sensible dynamically routed network design which doesn't leak /24s between datacentres would have allowed all systems to be reachable, even if some internal cross country system replication traffic was lost. I suspect Iomart probably do pickup some limited amount of transit in either Manchester or Glasgow but either not an adequate amount or for some reason they had poorly designed route redistribution or internal routing policy.
    That would be a very wasteful, inflexible addressing policy in the real world. It's unlikely to last for long in your average business, particularly a PLC.

    Having sufficient local transit available in each data centre in case of a partition through multiple fibre breaks would also be very expensive, particularly given that some of their data centres are not in areas where there a lot of carriers (and that's assuming that they have full table routers in each DC).
    You'd be much better off spending the extra money on adding extra diverse inter-DC paths to the core network.

    We peer with iomart at IXManchester, so they definitely have some external connectivity in Manchester. We saw a lot more BGP updates from them than normal during their network issues, so I suspect they were making some changes to do the best they could with what they had.
    Unfortunately we don't really exchange much traffic with them in Manchester normally, so it's hard to tell from our graphs whether there was any significant impact to that during the fibre breaks due to congestion etc.
    mandarin likes this.
    Freethought Internet
    Freethought Internet Limited registered in London No. 5862996. Registered office: The Old Church Hall, 2A Cromwell Street, Lincoln, LN2 5LP. VAT number GB 987 0952 66.

  6. #6
    91 rating
    4 reviews
    Posts 7974
    Post rank 12 (-10)
    Array
    Quote Originally Posted by Ed-Freethought View Post
    That would be a very wasteful, inflexible addressing policy in the real world. It's unlikely to last for long in your average business, particularly a PLC.

    Having sufficient local transit available in each data centre in case of a partition through multiple fibre breaks would also be very expensive, particularly given that some of their data centres are not in areas where there a lot of carriers (and that's assuming that they have full table routers in each DC).
    You'd be much better off spending the extra money on adding extra diverse inter-DC paths to the core network.

    We peer with iomart at IXManchester, so they definitely have some external connectivity in Manchester. We saw a lot more BGP updates from them than normal during their network issues, so I suspect they were making some changes to do the best they could with what they had.
    Unfortunately we don't really exchange much traffic with them in Manchester normally, so it's hard to tell from our graphs whether there was any significant impact to that during the fibre breaks due to congestion etc.
    Perhaps if they weren't both with the same fibre provider, they might have had more of a chance?
    We all know "diverse" routes north and south can head to the same POP before the physical diversity kicks in...

  7. #7
    132 rating
    23 reviews
    Posts 13826
    Post rank 1 (0)
    Array
    Quote Originally Posted by Pete-CCS View Post
    Perhaps if they weren't both with the same fibre provider, they might have had more of a chance?
    We all know "diverse" routes north and south can head to the same POP before the physical diversity kicks in...
    Depends how well you do your due diligence on what you're buying I guess (and whether your provider is being suitably forthcoming with you).

    My experience is that once you start involving multiple providers, it gets a whole lot harder to validate that everything is truly diverse, particularly given the industry's somewhat incestuous nature! That's not to say that it's impossible however.

    In this case, the fibre cuts were on two different legs on separate sides of the country, so it's not a case of one site being physically dependant upon a single PoP.
    Freethought Internet
    Freethought Internet Limited registered in London No. 5862996. Registered office: The Old Church Hall, 2A Cromwell Street, Lincoln, LN2 5LP. VAT number GB 987 0952 66.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Content Relevant URLs by vBSEO 3.6.1