HSBC’s core banking system has been hosed for almost a week, preventing thousands of customers from knowing how much money is stashed in their accounts. The widespread problem is limiting access to HSBCDirect accounts, and at least 8,000 Catholic Health System employees up in Buffalo are still waiting for their direct deposit payments to materialize.
The bank initially downplayed the issue, calling it an “infrastructure problem” or a “systems issue involving disks in our mainframe computer” that affected “certain applications.” In reality, it was much worse.
“It’s very serious to have your core system go down,” said Bart Narter, senior vice president of the banking group at technology research firm Celent. “It is a major technical failure.”
A bank’s core system is the back-office computer that keeps track of checking and other account balances and transactions. Many other bank systems, such as online banking and ATMs, rely on the information in that system. Other computer systems are unrelated.
“The core system is the system of record, the absolute arbiter of how much money you have and how much money you deposited today,” Narter said.
Most U. S. banks do not operate in “real time,” but rather keep computerized notes or “memos” of all transactions that occur during a given day, Narter said. At night, during a specified window when the bank shuts down, officials run the core program to update the accounts using “batches” of those transaction records. They then bring the system back up in the morning.
If the hard drive fails, the bank has to reboot and restore the system, and then rerun the overnight batches to catch up. That takes time at a big bank.
“This is a bank’s worst nightmare, to have the core banking system goes down,” he said. “That means any channel you go to, they’re running blind.”
The bank is keeping select upstate branches open late so Catholic Health System employees can come in and trade their pay stubs for cash. As for affected HSBCDirect customers, the bank promises to “return full–system functionality to you as soon as possible.”
HSBC’s issues mostly repaired, but cause still sought [The Buffalo News]
HSBC struggles with customer transactions after crash of computer system [The Buffalo News]
(Photo: Getty)
More From Consumerist
-
United Airlines Computer Glitch Traps, Strands & Delays Thousands Of Passengers Worldwide
-
No, You Can’t Fly To Hong Kong In First Class For $33 & 4 Frequent Flier Miles
-
It’s The 21st Century, But I Can’t Pick My Seat When Booking A Flight Over The Phone
-
Newegg: No, We’ll Totally Take Returns After You Install Linux
-
AT&T U-Verse Computer Glitch Opens Account For Dead Man
i dunno… is it better news for a glitch to bring down a system 1 week, or a hacker to take off with 10 thousand bank accounts hacked off an insecure system?
how about those overdraft fees huh…
I used to work in the IT dept at Washington Mutual, and this is the nightmare all IT workers dread. Most critical systems are fault tolerant, but it can still take some time to failover to backup sites. Having to restore from backup and re-run transactions makes me think they didn’t have their disaster recovery plans quite up to date.
Business continuity anyone? It’s becoming increasingly important yet some still don’t get it.
Something funny going on here. If they are using Tandem computers this just doesn’t happen. If they are using Digital computers then their setup will have two computers running in parallel at all times (meaning they have three computers so that even if one goes down they still have two computers running).
HSBC not be running legally.
@Fredex: HSBC not be running legally.
My thought exactly. I thought that redundancies were required with such an event in mind. I see the FDIC investigating on this one.
with technology the way it is these days, why aren’t they real time?
It makes sense to not do it real time, because then if you have an error it can be more problomatic if #1) Customer expects balance to 100% correct all the time and #2) If Transaction gets lost.
But this is all said and done… The story is last updated the 20th and talks about last Monday and Tuesday. HSBC Customers (such as me) have already gotten an email addressing these issues from HSBC. Also if people still weren’t getting their money as Consumerist claims then who is saying that? Where is the source and why hasn’t local Buffalo News or National News picked up on it?
I know I cashed a check etc at my HSBC Branch on Saturday. No problems.
Isn’t this story 4 days too old? I believe their systems are up and running fine already:
Forgot to provide a link above:
[www.hsbcdirect.com]
@LJKelley: Agreed. The EFTs (electronic fund transfers) which I had scheduled for last week went through without a hitch. I did get the email (even before I knew there was an issue) and when I logged onto the website, the only difference I noticed was the fact that there was an alert on the top of my account that said HSBC is undergoing some technical issues, but should be back as soon as possible.
Here’s the email I received on 8/20:
@ViperBorg: My thought exactly. I thought that redundancies were required with such an event in mind. I see the FDIC investigating on this one.
The hardware and the data may be redundant, but if the data is corrupt, all you’ve got is _redundant corrupt_ data. Still, I have to agree with you to an extent. Surely, they must keep point-in-time copies of data so that they can roll back and re-commit transactions. I can sincerely say I’m glad I’m not working in that particluar data center right now… And to be down for a week, I have to wonder if something is wrong with whatever type of backups they perform. Could it possibly take that long to restore and re-commit? Could there be _that much_ data?
I don’t know much about odds, but I do work in IT as one of the top-level administrators for the system I support. Folks, the first thing that popped into my paranoid little brain was, “If it’s that hard to break, then someone must have broken it deliberately.”
What?
This is inexcusable. The CEO and all direct reports should be fired on the spot.
My direct deposit paycheck did not go through as a part of this mess last week.
I was on vacation and didn’t have easy access to my balance and figured my paycheck was fine, only to get a text alert from mint.com saying that my balance was overdrawn. I was expecting a war with the CSR over the telephone. Instead, logging on to check the status of things once I got home — everything had been fixed, my paycheck had made it into my account, and the overdraft fees I had accumulated had all been reversed. That’s at least one point in their favor.
ok, everyone take a deep breath. this isn’t a major issue – everything’s back online. if hsbc didn’t have redundancies/ disaster recovery in place – well then, that would suck. but they did (or they never would have recovered this quick).
i know a few days doesn’t seem very quick, but we’re talking about entire transaction histories/account info for millions of accounts – that’s got to be a few hundred TBs of data (on the light side). you have to expect that rollback, restore, re-batch data for a couple days & then error checking is going to take some time.
from what krunk says, sounds like they managed everything fine.
And the question is… What is the name of the next bank to fold? Seriously, I hate HSBC and I hope this marks their end. I have a cc through them and I was one day late on a bill just once and they had debt collectors calling me 5 times a day for 3 weeks. Looks like I won’t be keeping that card. If something major like this happens, it’s your duty to report it to your “investors”. I would not keep my money in the bank just from the lack on honesty and the total disregard for integrity.
Lot of apologists in this thread.
“A statement from the bank Friday said while most of the impacted systems are now up and running, recovery efforts continue.” [biz.yahoo.com]
Most != All. Everything is not apparently fine. Plus, there’s the whole “shouldn’t have happened in the first place” aspect.
The lack of redundancy or offsite disaster plan to take over when the main one died does not instill confidence in HSBC. All banks make data backups every 24 hrs minimum, at least they are supposed to based on what people who do this in banking IT have told me.
It should have been a matter of stop, restore, rerun the batch. Something is seriously not right that they were out this long.
@narq: You hope this marks their end? HSBC is one of the world’s largest banks. No, this won’t be their end and you should be happy about that, even if you aren’t an HSBC customer. The implications of HSBC folding would hurt everyone.
@MPHinPgh: That’s why I have set up here a weekly backup, a bi-weekly backup, and a daily backup. All stored separately. And I keep them for 6 months. Now, how easy something like that would be for a bank, I don’t know, but if something did get corrupt, I’d still have a backup of data before the corruption.
You’d think a bank would at least have someone thinking like that.
Reason #46 why your mattress is a better place to keep your money.
46 – Your mattress’ data center will never fail.
@PunditGuy: if i had to take a guess, i’d say we’re talking about a 15-year old server setup getting completely hosed. now obviously, they’re not going to replace antiquated technology with antiquated technology. a switch to new hardware takes time & invariably, there are networking issues that rear their ugly heads.
of course, that’s all conjecture, but having experienced the above situation myself, this incident rings eerily familiar (albeit on a scale 1000x larger). maybe that’s why i’m an “apologist”. walk a mile & such.
Hell, my direct deposit went through to my HSBC account almost 3 days early
@mac-phisto: Worked in telecom for more than five years, so I understand that things happen. But critical systems get that designation for a reason, and it says something very fundamental about both the IT department and senior management at HSBC if either 1) this is their SOP, or 2) they didn’t follow or are incapable of following their own SOP.
Although I’m not with HSBC banking, I do have a loan from Beneficial, one of their “arms”. I had a weird experience this week with them electronically returning my payment I had made on my loan. I haven’t received any answers from them yet on why this was done; and my account still shows that I paid.
@vastrightwing: Ever been in a house fire?
@ViperBorg: The amount of data that the bank backs up and would consequently have to restore is probably several orders of magnitude larger than your backups. Not to mention the fact that they may have to coordinate timing between hundreds of backup systems (potentially one for each branch in addition to the ones in the data center). Then after restoring the data, the application folks would have to sort through it and make sure every thing was lined up correctly before they started batch processing again. In all likelihood they have tons of backups but this isn’t as simple as saying, “I accidentally deleted this Word doc, could you please restore it for me?”
Then there’s the possbility that there was a specific transaction that caused the corruption. If that were the case, they could roll back to the last good backup but after processing all of the restored batches, the corruption would returen and then you’d have to either look through the logs to see what happened (that’s assuming that the logs contain the potential error) or comb through the batch transactions to see exactly what screwed things up.
I’m not saying that the situation was handled correctly but I think that many posters here may not appreciate the potential complexity of the problem nor the physical, technical or logistical limitations of the IT technologies at work here.
HSBC probably has computers from the likes of Tandem all the way up to state-of-the-art rackmount goodness with the latest Windows or REHL. Computers and software from different eras, with vastly different methods of interfacing, must work together on a daily basis to give you a smooth banking experience.
I can see needing a day or two to get it back up and running. Then they figure out exactly what went wrong and make sure that one thing never happens again (or, at least won’t bring them down again). It’s not like they run all modern hardware and software, and have banks of synchronized, load-balanced, fault-tolerant servers all over. I’m sure they have that…but those then also talk to the best of what could do that work when dinosaurs roamed, and sometimes there are no better tools than shoe lace and duct tape. Then, there’s the FreeBSD box in the closet that performs some critical function, but has never so much as hiccuped, isn’t on the network diagram, and only the Janitor knows about it (that’s much less fictional than you might think!).
As long as this level of failure remains something that happens once or twice per decade, and they didn’t lose days of transactions (transactions not able to go through are OK, it’s transactions other banks think were fine and then get lost that I’d worry about), I wouldn’t worry about it. Yes, it’s annoying, and I’m sure they’ve got their hands full with people that needed the money then, or had failed deposits and withdrawals, etc., but they clearly have good disaster recovery protocols in place, and it will all smooth over in a bit.
My HSBC account seems to be working fine? So that’s how they pay their high APYS compared to ING – they hold back money…
My HSBC account has no issues. Of course I’m not looking at it hoping my paycheck will show up so I only log in once in a while.
This is extremely surprising to me. My father works at Lasalle’s Chicago office as a server administrator. He told me about their disaster recovery plans. This one regional office MUST keep their services operational nearly down to the minute. They even run a warm/hot backup site in the area to move to at a moment’s notice. To have HSBC’s possibly multi-national banking system down for a week is inexcusable.
Of course, he now technically works for IBM, outsourced to Bank of America, so who knows how good their backups are.