Severe Flooding In The Northwest Knocks Out T-Mobile's Customer Service

Mary writes:

Hey ya’ll. I just wanted to alert your readers to the fact the T-Mobile USA’s customer care (1-800-937-8997) is offline right now. It has been since sometime yesterday. Apparently they house all their system stuff in Seattle, WA or Portland, OR or one of those currently waterlogged states. Well, it has caused all their stuff to crash. I called last night and then again this morning and waited past their automated system telling me that they could not view my account info thru the IVR and that the CSR’s were unable to view my account info and to call back later. I spoke with a polite rep Cassie who said that they did not have an ETR on when their systems would be back up and running and that I should try calling back later on today.

We called T-Mobile and sure enough they’re currently unable to pull up anyone’s account info due to the storms. No word on when it will be back up.

Get well soon, T-mobile.

(Photo:Rob Finch/The Oregonian)

Comments

Edit Your Comment

  1. iEddie says:

    When it works they’ve got good CS. (At least they still let you talk to a CSR – many companies would have just said “we can’t access our customer DB, bye.”)

  2. Sam2k says:

    Can they accept bill payments?

  3. Buran says:

    And with that, T-Mobile’s reputation for having decent customer service goes out the window.

    This is why you have BACKUPS.

  4. parad0x360 says:

    leave it to a huge corporation to not have a backup place where this work can be done from…

  5. MeOhMy says:

    So is the customer service actually any worse? I kid, I kid.

    @Buran: I’m sure they have backups. Depending on the nature of the problem, backups could be utterly useless. Depending on the nature of their redundancy, they may be hedging that riding out the storm will not take as long as bringing a DR site online.

  6. ottergal says:

    @Buran: Depending on the type of crash and the system/systems that failed, do you have any idea what it takes to restore the (potential) amount of data they are dealing with? I work in telecom billing (not for TMO) and can tell you that they’re probably not dealing with a simple pull from a tape.

    This is where you hope you have a good disaster recovery plan.

  7. coan_net says:

    Hum… when I worked for a “bigger” company, they had 4 offices – 1 in hurricane area, 1 in flooding area, 1 in tornado area, and 1 in Mexico. Any of the 4 could do the work of the other 3 if something were to happen – sounds strange that the only place that t-mobile has is in 1 area and has nothing as a backup plan.

  8. Buran says:

    @Troy F.: I mean redundant copies of all your data in widely-geographically-separated datacenters. Like what Coan is describing. The only thing that is like to knock all of those down at once is an asteroid impacting Earth.

  9. oneswellfoop says:

    Portland’s not waterlogged. I’m siting in Portland right now high and dry. Not high like that, but definitely dry like you’d think.

  10. MeOhMy says:

    @Buran: Like I said, depending on their redundancy, riding out the storm might be faster and cheaper. Having a second replicated data center is mind-bogglingly expensive. With a hot site contract if your systems are intact but unavailable, it may still be slower/more expensive than activating the hot site.

  11. MeOhMy says:

    @Troy F.: Er…it may still be slower more/expensive TO activate the hot site.

  12. yagisencho says:

    Western Washington, on the other hand, is soaked to the freaking bone. The image of dark, wet, and depressing has been greatly enhanced by the recent flooding, freeway closures, school closures, and state of emergency declaration.

    I’m a lifetime resident, and I’ve never seen a week quite like this. It normally requires a couple weeks of constant rain to get where we’re at now after four days. Bremerton (Navy town on a penninsula west of Seattle) saw eleven (11) inches of rain yesterday. That’s in addition to 6-8 inches of snow on Saturday.

    The real flooding is likely to hit by the weekend, when the ultra-saturated flood plains refuse to soak up any more water, and several major rivers spill over their banks, destroying hundreds more homes and businesses.

    Sorry about your customer service hotline being down.

  13. Hanke says:

    I am having a problem with them now myself; I am waiting for my new corporate leash, I mean Blackberry, and it didn’t arrive. THey can’t even tell me if it shipped yesterday.

  14. QuantumRiff says:

    A section of I5 in washington is closed because the highway is under 3 feet of water. Guess where all the fiber runs between PDX and Seattle? and honestly, Replicated datacenters can be more problems then their worth. Its not that easy to create the same copy in 4 places at the same time. And what happens if you have it all working, and a tech types the wrong command, and deletes the database? Then you get that delete replicated to 4 sites.

  15. DrGirlfriend says:

    Portland is dry now, but it’s been soaking wet since Saturday. It would not stop raining! And the coast got quite a battering.

  16. algormortis says:

    No, we *don’t* have a backup system and that’s half the bloody problem.

    If you get hit for a fee agitate like there’s no tomorrow. Get names and Rep IDs of anyone you speak to, and most of all make sure that they’ve entered it CS Remedy, which I assure you is up but doesn’t pull up customer info. Just FYI. Your calls can still be logged on our end.

  17. mondomondoman says:

    I used to work for T-mobile and the servers are in Bothell Washington. I was sent a picture of the server room and it has water waist deep in it. CCR have been told that they will be back up by tomorrow but somehow I doubt this will happen.

  18. DeliBoy says:

    @mondomondoman: Guess that’s why I can’t pay my bill at my.t-mobile.com… ‘scheduled maintenance’, my arse…

  19. yg17 says:

    Even if they have a 2nd datacenter somewhere else in the world, that doesn’t help if their internet connectivity out of the datacenter in the northwest. They would have no way to access it.

  20. krztov says:

    @Hanke:
    i’ll assume they still use the same shipping as when i worked there, in which case:

    go to ups.com, click track, track by reference number, enter 10 digit mobile number with the dashes. should work unless theyve changed that (then again havent been a csr there for about a year.)

  21. EmmaC says:

    Uh oh. I tried to pay my bill online yesterday and today and while the site is up, the bill pay portion is down. My bill is due tomorrow. It sounds like bill pay over the phone is down too. I hope they don’t charge me a late fee.

  22. Buran says:

    @Troy F.: Except, consider the result of not processing payments (and therefore cutting people off for “nonpayment”) and customers deciding that they’d rather get service from someone who can help them and help open new accounts and so on, and get service from someone else.

    Cheaping out on your data center costs you customers who aren’t going to come back for a long time. If ever. You don’t shut down your ENTIRE operation for more than a few minutes.

  23. yg17 says:

    @Buran: I don’t think T-Mobile’s going to be cutting off accounts for nonpayment or charging late fees to anyone whose been affected. From what I’ve read on other sites (mainly HowardForums where some T-Mobile employees have commented) T-Mobile’s been taking care of customers. New activations are having fees waived, prepaid users will be given free bonus minutes since they couldn’t refill their accounts and accounts won’t be shut off or charged late fees.

    And I’d hardly call their entire operation “shut down.” I’ve still been able to make and receive calls and texts on my T-Mobile phone the entire time. The calls were still crystal clear with no static or dropouts, and my data connection has been working flawlessly too. And to me, and most of their 20-something million customers, I’m sure that’s all that matters. And hell, some other companies *cough*at&t*cough* can’t provide that level of service even when their infrastructure isn’t underwater.

  24. @Buran: Two words: Shit Happens.

  25. MeOhMy says:

    @Buran: I’m sure the well-paid DRP experts at T-Mobile have already considered the risk. Hell, half of DR planning is risk assessment. Everyone with opposable thumbs has access to the technology to set up real-time replica data centers – it doesn’t really require any magic. What it does require is oodles of money. An amount that, in general, exceeds the anticipated cost of nonprotracted downtime.

    You may think of it as “cheaping out on your data center” but there’s not much point in “not cheaping out on your data center” if “not cheaping out on your data center” costs so much that you “go out of business” keeping it up.

  26. SupTech says:

    @Buran
    “T-Mobile’s been taking care of customers. New activations are having fees waived, prepaid users will be given free bonus minutes since they couldn’t refill their accounts and accounts won’t be shut off or charged late fees. “
    WRONG !
    My T-Mobile TOGO account is dead . I could not add more minutes for the last two days. Can’t make call , can’t receive calls.
    Huge call center here in Bellingham , WA . No flooding . I hope they fire a bunch of people!

  27. Glaven says:

    I would expect some services to be hit, really. But they have major network operating centers in WA state and in NJ. The one in WA state is up in Snoqualmie in the Cascade foothills, so too high for floods. I don’t know where the servers are that hold customer data, though.

  28. Shepherd says:

    @Glaven

    T-Mobile’s data center is located in Bothell, not Snoqualmie, and the flooding came from North Creek which crested well above a dike. Though T-Mobile may regret the decision not to join the exodus, most firms previously located in the stunning hills of Bothell had relocated to Tukwila or in Downtown Seattle because it has much lower risk.