Computer Glitch Renders Thousands Of HSBC Accounts Inaccessible

HSBC’s core banking system has been hosed for almost a week, preventing thousands of customers from knowing how much money is stashed in their accounts. The widespread problem is limiting access to HSBCDirect accounts, and at least 8,000 Catholic Health System employees up in Buffalo are still waiting for their direct deposit payments to materialize.

The bank initially downplayed the issue, calling it an “infrastructure problem” or a “systems issue involving disks in our mainframe computer” that affected “certain applications.” In reality, it was much worse.

“It’s very serious to have your core system go down,” said Bart Narter, senior vice president of the banking group at technology research firm Celent. “It is a major technical failure.”

A bank’s core system is the back-office computer that keeps track of checking and other account balances and transactions. Many other bank systems, such as online banking and ATMs, rely on the information in that system. Other computer systems are unrelated.

“The core system is the system of record, the absolute arbiter of how much money you have and how much money you deposited today,” Narter said.

Most U. S. banks do not operate in “real time,” but rather keep computerized notes or “memos” of all transactions that occur during a given day, Narter said. At night, during a specified window when the bank shuts down, officials run the core program to update the accounts using “batches” of those transaction records. They then bring the system back up in the morning.

If the hard drive fails, the bank has to reboot and restore the system, and then rerun the overnight batches to catch up. That takes time at a big bank.

“This is a bank’s worst nightmare, to have the core banking system goes down,” he said. “That means any channel you go to, they’re running blind.”

The bank is keeping select upstate branches open late so Catholic Health System employees can come in and trade their pay stubs for cash. As for affected HSBCDirect customers, the bank promises to “return full–system functionality to you as soon as possible.”

HSBC’s issues mostly repaired, but cause still sought [The Buffalo News]
HSBC struggles with customer transactions after crash of computer system [The Buffalo News]
(Photo: Getty)

Comments

Edit Your Comment

  1. taking_this_easy says:

    i dunno… is it better news for a glitch to bring down a system 1 week, or a hacker to take off with 10 thousand bank accounts hacked off an insecure system?

    how about those overdraft fees huh… :P

  2. Marshfield says:

    I used to work in the IT dept at Washington Mutual, and this is the nightmare all IT workers dread. Most critical systems are fault tolerant, but it can still take some time to failover to backup sites. Having to restore from backup and re-run transactions makes me think they didn’t have their disaster recovery plans quite up to date.

  3. WickedKoala says:

    Business continuity anyone? It’s becoming increasingly important yet some still don’t get it.

  4. Fredex says:

    Something funny going on here. If they are using Tandem computers this just doesn’t happen. If they are using Digital computers then their setup will have two computers running in parallel at all times (meaning they have three computers so that even if one goes down they still have two computers running).

    HSBC not be running legally.

  5. ViperBorg says:

    @Fredex: HSBC not be running legally.

    My thought exactly. I thought that redundancies were required with such an event in mind. I see the FDIC investigating on this one.

  6. Tank says:

    with technology the way it is these days, why aren’t they real time?

  7. LJKelley says:

    It makes sense to not do it real time, because then if you have an error it can be more problomatic if #1) Customer expects balance to 100% correct all the time and #2) If Transaction gets lost.

    But this is all said and done… The story is last updated the 20th and talks about last Monday and Tuesday. HSBC Customers (such as me) have already gotten an email addressing these issues from HSBC. Also if people still weren’t getting their money as Consumerist claims then who is saying that? Where is the source and why hasn’t local Buffalo News or National News picked up on it?

    I know I cashed a check etc at my HSBC Branch on Saturday. No problems.

  8. krunk4ever says:

    Isn’t this story 4 days too old? I believe their systems are up and running fine already:

    I am pleased to be able to let you know that our systems have now returned to normal operating conditions. As always, we continue to monitor all systems closely and will let you know if there is any change in status. I can confirm to you that all customer information, data and funds were secure at all times. We want to extend our sincerest apologies for any inconvenience you may have experienced as a result of our recent system issue.

    For specific details on how we will refund any fees and ensure you receive proper value for your savings please log in now and click on the banner “Customer Update.”

    Should you have any other questions about your HSBC Direct account you may also contact our Customer Service at 1-888-404-4050.

    As part of HSBC Group, one of the largest and most highly-capitalized financial institutions in the world with over 140 years of experience, we remain committed to meeting and servicing your financial needs.

    Thank you again for your support and patience and for saving with HSBC Direct.

  9. krunk4ever says:

    Forgot to provide a link above:
    [www.hsbcdirect.com]

  10. krunk4ever says:

    @LJKelley: Agreed. The EFTs (electronic fund transfers) which I had scheduled for last week went through without a hitch. I did get the email (even before I knew there was an issue) and when I logged onto the website, the only difference I noticed was the fact that there was an alert on the top of my account that said HSBC is undergoing some technical issues, but should be back as soon as possible.

  11. krunk4ever says:

    Here’s the email I received on 8/20:

    Dear [CUSTOMER NAME],

    I want to update you on our recent systems issue and extend my apologies for any inconvenience you may have experienced in trying to access your HSBC Direct account.

    Our teams have worked diligently to test and re-engage the impacted systems to ensure that we are able to return full-system functionality to you as soon as possible. Most of the impacted systems are now up and running and we continue to work through any remaining issues.

    As part of HSBC Group, one of the largest and most highly-capitalized financial institutions in the world, with over 140 years of experience helping our customers achieve their financial goals, we remain committed to meeting your needs and understand how important continuity of our online services are to you.

    I will continue to keep you updated on our progress, and again I sincerely apologize for any inconvenience this system issue may have caused you. And I thank you for saving with HSBC Direct.

    Sincerely,
    Kevin Martin
    Executive Vice President,
    Head of HSBC Direct U.S.

  12. MPHinPgh says:

    @ViperBorg: My thought exactly. I thought that redundancies were required with such an event in mind. I see the FDIC investigating on this one.

    The hardware and the data may be redundant, but if the data is corrupt, all you’ve got is _redundant corrupt_ data. Still, I have to agree with you to an extent. Surely, they must keep point-in-time copies of data so that they can roll back and re-commit transactions. I can sincerely say I’m glad I’m not working in that particluar data center right now… And to be down for a week, I have to wonder if something is wrong with whatever type of backups they perform. Could it possibly take that long to restore and re-commit? Could there be _that much_ data?

  13. speedwell (propagandist and secular snarkist) says:

    I don’t know much about odds, but I do work in IT as one of the top-level administrators for the system I support. Folks, the first thing that popped into my paranoid little brain was, “If it’s that hard to break, then someone must have broken it deliberately.”

    What?

  14. edrebber says:

    This is inexcusable. The CEO and all direct reports should be fired on the spot.

  15. arielariel says:

    My direct deposit paycheck did not go through as a part of this mess last week.

    I was on vacation and didn’t have easy access to my balance and figured my paycheck was fine, only to get a text alert from mint.com saying that my balance was overdrawn. I was expecting a war with the CSR over the telephone. Instead, logging on to check the status of things once I got home — everything had been fixed, my paycheck had made it into my account, and the overdraft fees I had accumulated had all been reversed. That’s at least one point in their favor.

  16. mac-phisto says:

    ok, everyone take a deep breath. this isn’t a major issue – everything’s back online. if hsbc didn’t have redundancies/ disaster recovery in place – well then, that would suck. but they did (or they never would have recovered this quick).

    i know a few days doesn’t seem very quick, but we’re talking about entire transaction histories/account info for millions of accounts – that’s got to be a few hundred TBs of data (on the light side). you have to expect that rollback, restore, re-batch data for a couple days & then error checking is going to take some time.

    from what krunk says, sounds like they managed everything fine.

  17. narq says:

    And the question is… What is the name of the next bank to fold? Seriously, I hate HSBC and I hope this marks their end. I have a cc through them and I was one day late on a bill just once and they had debt collectors calling me 5 times a day for 3 weeks. Looks like I won’t be keeping that card. If something major like this happens, it’s your duty to report it to your “investors”. I would not keep my money in the bank just from the lack on honesty and the total disregard for integrity.

  18. PunditGuy says:

    Lot of apologists in this thread.

    “A statement from the bank Friday said while most of the impacted systems are now up and running, recovery efforts continue.” [biz.yahoo.com]

    Most != All. Everything is not apparently fine. Plus, there’s the whole “shouldn’t have happened in the first place” aspect.

  19. bohemian says:

    The lack of redundancy or offsite disaster plan to take over when the main one died does not instill confidence in HSBC. All banks make data backups every 24 hrs minimum, at least they are supposed to based on what people who do this in banking IT have told me.

    It should have been a matter of stop, restore, rerun the batch. Something is seriously not right that they were out this long.

  20. LINIS says:

    @narq: You hope this marks their end? HSBC is one of the world’s largest banks. No, this won’t be their end and you should be happy about that, even if you aren’t an HSBC customer. The implications of HSBC folding would hurt everyone.

  21. ViperBorg says:

    @MPHinPgh: That’s why I have set up here a weekly backup, a bi-weekly backup, and a daily backup. All stored separately. And I keep them for 6 months. Now, how easy something like that would be for a bank, I don’t know, but if something did get corrupt, I’d still have a backup of data before the corruption.

    You’d think a bank would at least have someone thinking like that.

  22. vastrightwing says:

    Reason #46 why your mattress is a better place to keep your money.
    46 – Your mattress’ data center will never fail.

  23. mac-phisto says:

    @PunditGuy: if i had to take a guess, i’d say we’re talking about a 15-year old server setup getting completely hosed. now obviously, they’re not going to replace antiquated technology with antiquated technology. a switch to new hardware takes time & invariably, there are networking issues that rear their ugly heads.

    of course, that’s all conjecture, but having experienced the above situation myself, this incident rings eerily familiar (albeit on a scale 1000x larger). maybe that’s why i’m an “apologist”. walk a mile & such.

  24. Mazda Eric says:

    Hell, my direct deposit went through to my HSBC account almost 3 days early ;)

  25. PunditGuy says:

    @mac-phisto: Worked in telecom for more than five years, so I understand that things happen. But critical systems get that designation for a reason, and it says something very fundamental about both the IT department and senior management at HSBC if either 1) this is their SOP, or 2) they didn’t follow or are incapable of following their own SOP.

  26. Indecent says:

    Although I’m not with HSBC banking, I do have a loan from Beneficial, one of their “arms”. I had a weird experience this week with them electronically returning my payment I had made on my loan. I haven’t received any answers from them yet on why this was done; and my account still shows that I paid.

  27. sir_eccles says:

    @vastrightwing: Ever been in a house fire?

  28. SonicPhoenix says:

    @ViperBorg: The amount of data that the bank backs up and would consequently have to restore is probably several orders of magnitude larger than your backups. Not to mention the fact that they may have to coordinate timing between hundreds of backup systems (potentially one for each branch in addition to the ones in the data center). Then after restoring the data, the application folks would have to sort through it and make sure every thing was lined up correctly before they started batch processing again. In all likelihood they have tons of backups but this isn’t as simple as saying, “I accidentally deleted this Word doc, could you please restore it for me?”

    Then there’s the possbility that there was a specific transaction that caused the corruption. If that were the case, they could roll back to the last good backup but after processing all of the restored batches, the corruption would returen and then you’d have to either look through the logs to see what happened (that’s assuming that the logs contain the potential error) or comb through the batch transactions to see exactly what screwed things up.

    I’m not saying that the situation was handled correctly but I think that many posters here may not appreciate the potential complexity of the problem nor the physical, technical or logistical limitations of the IT technologies at work here.

  29. cerbie says:

    HSBC probably has computers from the likes of Tandem all the way up to state-of-the-art rackmount goodness with the latest Windows or REHL. Computers and software from different eras, with vastly different methods of interfacing, must work together on a daily basis to give you a smooth banking experience.

    I can see needing a day or two to get it back up and running. Then they figure out exactly what went wrong and make sure that one thing never happens again (or, at least won’t bring them down again). It’s not like they run all modern hardware and software, and have banks of synchronized, load-balanced, fault-tolerant servers all over. I’m sure they have that…but those then also talk to the best of what could do that work when dinosaurs roamed, and sometimes there are no better tools than shoe lace and duct tape. Then, there’s the FreeBSD box in the closet that performs some critical function, but has never so much as hiccuped, isn’t on the network diagram, and only the Janitor knows about it (that’s much less fictional than you might think!).

    As long as this level of failure remains something that happens once or twice per decade, and they didn’t lose days of transactions (transactions not able to go through are OK, it’s transactions other banks think were fine and then get lost that I’d worry about), I wouldn’t worry about it. Yes, it’s annoying, and I’m sure they’ve got their hands full with people that needed the money then, or had failed deposits and withdrawals, etc., but they clearly have good disaster recovery protocols in place, and it will all smooth over in a bit.

  30. andyfvp says:

    My HSBC account seems to be working fine? So that’s how they pay their high APYS compared to ING – they hold back money…

  31. meefer says:

    My HSBC account has no issues. Of course I’m not looking at it hoping my paycheck will show up so I only log in once in a while.

  32. blackmage439 says:

    This is extremely surprising to me. My father works at Lasalle’s Chicago office as a server administrator. He told me about their disaster recovery plans. This one regional office MUST keep their services operational nearly down to the minute. They even run a warm/hot backup site in the area to move to at a moment’s notice. To have HSBC’s possibly multi-national banking system down for a week is inexcusable.

    Of course, he now technically works for IBM, outsourced to Bank of America, so who knows how good their backups are.