Dreamhost Is "Very Very Sorry" For $7.5 Million Billing Error

Dreamhost would like you to know that its very very sorry for accidentally billing its customers $7.5 million it wasn’t actually owed. You see, someone typed 2008 when they really meant 2007 and their billing system decided to charge all of their customers in advance for the entire 2008 calendar year. This included debiting huge amounts of money from people’s checking accounts and all the “worst possible scenario” situations you could possibly imagine.

Tom, friend of the blog, and master of the internet, was among those affected:

Well, this morning I got a billing email from them:

This is just a notice that your DreamHost [redacted] (“zug’s Account”) has a balance of $380.87 (including any charges not due until 2009-01-14), with $340.97 due (since 2008-12-14).

You also have $321.02 past due (owed since 2008-11-14), and if by 2009-01-14 you do not pay at least the $321.02 part, your account will be automatically suspended until payment is received.

Ok, that’s confusing… WTF, I ask myself. Looking farther into things…

WHAT DO YOU OWE MONEY FOR?

We have the last payment on this account to be $12.46 on 2008-01-15 06:48:38.

Since then the following charges have been made to the account:

2008-01-18 – $19.95 for “CODE MONSTER!” through 2008-02-17.

2008-01-18 – $19.95 for “CODE MONSTER!” through 2008-02-17.

2008-02-18 – $19.95 for “CODE MONSTER!” through 2008-03-17.

2008-02-18 – $19.95 for “CODE MONSTER!” through 2008-03-17…..

It turns out that Dreamhost is really, really sorry.

From Dreamhost’s blog:

I’m very very sorry, we’re very very sorry, and I’m sure you’re very very sorry this happened. I really am. I understand the sort of problems that an unexpected large charge to your credit card (or worse yet, your debit card) can cause. If the tone of this blog post seemed a little light, I apologize I don’t mean to offend and I realize how serious an issue this is. I’ve been up since 3:50am trying to undo the damage and maybe I’m a little shell-shocked.

A new service is running right now (in parallel on all the controllers) that fixes all those future charges, re-enables your account if it was erroneously suspended, and if your credit card was automatically rebilled, refunds the payment automatically. You don’t have to contact us or your bank, and you’ll get an email when your account is finished fixing up. It’s going to take several more hours to complete. There are (or were, after this incident) a lot of you these days!

If, because of this billing mistake, you somehow incurred some fees from your bank or credit card company, please let us know after tomorrow (today we are just replying to all 10,000+ billing messages with a generic explanation) and we’ll do our best to make it right for you.

As Tom points out, an apology is probably small consolation for people whose mortgage payments bounced because of this bug. We wish Dreamhost the best of luck compensating their users.

Dreamhost fucks up bigtime! [ScatteredGenius]
Um, Whoops. [Dreamhost Blog]
Billing Issues (846 comments) [Dreamhost Status]

Comments

  1. Craig says:

    There’s no excuse for something like this happening just because somebody entered an incorrect date…Dreamhost’s billing system should have failsafe measures that flag a situation like this before the actual billing occurs. It’s unbelievable to me that people are such Dreamhost fanboys that they would consider something like this to be even remotely acceptable. This has class action lawsuit written all over it.

  2. bkpatt says:

    @Craig: What an insightful post… If you actually read anything about the problem, Dreamhost acknowledged the problem of future-dated billing and has said they will be making a change to not allow that in the future.

    You are another example of the sue-first, ask-questions-later culture that is clogging up court systems and wasting tax dollars by the truckload. The $7.5 million dreamhost screw-up is at least being refunded, the money wasted by morons who demand entitlement for being the least bit inconvenienced is gone for good.

  3. dantsea says:

    @Craig: What would the “class action lawsuit” be based upon, in this instance?

  4. Hogan1 says:

    @Craig: Perhaps you live in some sort of utopia where technology and people never fail. Class action…lol. I’ve had Dreamhost for years and they always admit their mistakes and take action to correct them immediately. I’ve never had any billing issues with them. In any case, if you set up a debit card (Or a credit card close to its limit) for anything that auto bills…Thats just asking for trouble.

  5. UnnamedUser says:

    Heh,heh. Happy Dreamhost customer here. Very happy. I manage several accounts for self and others that are hosted at Dreamhost. I’ve been a Dreamhost customer since 2002 (about).

    Yup! Dreamhost screwed up. They do that from time to time. Caca Pasa, as they say.

    The good news is that Dreamhost owned up to it right away and did what was necessary to fix it.

    A little story, if you please. I have “rescued” more than a few domains from incompetent hosting companies, webmasters (using the term very loosely) on behalf of others in the last few years. Everything from billing nightmares that make Dreamhost’s fubduck look like pennies to webmasters simply walking off the job while holding their customers’ domains in their own name and not paying the hosting company’s and/or registrar bills to hosting companies simply disappearing completely. Some of these “rescues” take weeks to get sorted out.

    Of all the shared hosting companies I’ve dealt with, Dreamhost is the best of the lot. Dreamhost are not better by a little bit either, they are orders of magnitude better. They are transparent with their operations and online status. Their prices are not the cheapest, but very competitive with similar companies and lot cheaper than some bigger companies. Their stuff just works. … When it does break, they fix it quickly, then tell you what went sour and what they will do to keep it from happening again.

    Shared hosting is a very competitive business. Many people have unreasonable expectations for shared hosting. Five-Nines is just not something you can expect for under $10/month. If you need Five-Nines, then expect to pay more than an order of magnitude more per month. Then you can holler.

    Happy Dreamhost customer, indeed.

  6. amccoll says:

    It looks like they took the measures to correct their billing issue. The people on the billing team are, in fact, human. How many people here have already put down 2007 instead of 2008? It was acknowledged, corrected, and people still seem to think that anything they put money to should be fail-safe. Wake up, nothing in life is fail-safe. Good job, Dreamhost!

  7. Craig says:

    @bkpatt: I’ve filed a lawsuit once in my life, for medical expenses as the result of a car accident. (I’ve had plenty of opportunities to file other, frivolous ones.) You are another example of passive consumers who have come to accept substandard company practices as the norm, thereby assuring us all that sites like this will never run out of content.

    I’ll say it again…there is no excuse for such a massive misbilling given standard failsafe procedures that should be in place in any company. One or two customers misbilled is one thing. This is something completely different.

  8. FullFlava says:

    It blows my mind how quick you all are to just roll over and take it and then defend Dreamhost for a fuckup of this magnitude. Regardless of whether or not they’re fixing the problem, there is absolutely no excuse for having a vulnerability like this in their system. I know nothing’s failsafe, but allowing a single missed keystroke to wreak this much havok is unthinkable for any kind of serious business.

    If this were any other company who handled it with a press release instead of a funny picture and a bunch of jokes, you’d be screaming to sue them, out their CEO, and shut the entire company down. Just because they’re internet geeks like you who post funny pictures on their blog doesn’t make them any different from any other large company out there. They’re not your friends, they’re a business.

    I understand that people shouldn’t be auto-debiting payments, that’s common sense, but to imply that the huge problems people have incurred from this error are in any way the customers’ fault is just unbelievable to me. It really is amazing to me how quick nerds will rush to the defense of companies that fuck them over repeatedly; see also: video games and computer hardware manufacturers. “Wacky” ads do not excuse shit like this.

  9. dantsea says:

    @FullFlava: A one-time mistake is forgivable, a pattern is not. If Dreamhost starts autobilling screwups on a regular basis, I promise I’ll start screaming about it.

  10. FullFlava says:

    @DanB:
    Thanks :) I think the thing that really rubbed me the wrong way about this story, which I’ve been following on other sites since yesterday morning, is that there wasn’t even a notification e-mail sent out to customers to let them know about the error.

    From Digg:

    No email, no headline on their front page, just a little red post-it thing on their support contact form (visible only to customers) and the issue on their dreamhoststatus.com blog, which mostly only customers know about.
    [...]
    Absolutely NO mass email to inform subscribers of the error. You have to go to their site to see that they’re gonna fix it. This is INEXCUSABLE. How about folks with email but not internet access (ie, cell phone or something)? They’re gonna be flipping out.

    It’s nice that they have time to update the status page with a picture of the guy “at fault”, but cannot send us all a message saying it’s being taken care of.

    Where’s the checks-and-balances? Weak.

    Wow. Could dreamhost handle this any more poorly? Just like everyone else I received an email this morning saying that I owed money for a years worth of service. I actually thought I had done something wrong since I had been managing domains the day before. It wasn’t until I clicked went to the support feature that I discovered there had been a problem.

    If they were aware of the problem why didn’t they send an email to every customer as soon as they found out? I expect compensation for this total disaster.

    That’s just ridiculous. I’m glad they could find a funny Simpsons picture for their blog, but it’s fucked up when a company is more concerned with being “zany” than actually letting their customers know what’s up. Dreamhost is a bunch of amateurs.

  11. dantsea says:

    Huh, odd, since the first I heard about any of this was an email from the company announcing their screwup and that it was being taken care of. Understandable that people who didn’t see that right away would be upset, though.

  12. dantsea says:

    Email arrived at 2:28pm Pacific on Tuesday, that is.

  13. FullFlava says:

    @DanB:
    Not tryin to rag on you or anything, but their now-infamous Homer Simpson blog post was made at 9:52am. Waiting until that much later in the day to notify customers is not cool. We’re talking most of a work day; lunches, gas fill-ups, Starbucks, and all sorts of other things that no doubt incurred more and more overdraft charges for people who, y’know, didn’t expect to have hundreds (or thousands) of dollars debited from their accounts without their knowledge.

    The bottom line is that they should have notified their customers directly as soon as possible, if only to say “we know there’s a problem and we’re working on it.”

  14. gerasimatos says:

    Hello,

    As a previous employee of DreamHost and shift manager of their Graveyard Team I can advise you that billing errors within the company are nothing out of the ordinary, suffice to say I was not very shocked when I read there was a 7.5 million dollar billing error. The DreamHost billing system is far from “robust and stable” as Josh Jones proclaimed on the companies web blog, the only thing that could be counted on is the constant issues that have plagued DreamHost for the past few years. Billing issues are quite common with their company and the “fat fingers” he refers to having have led to countless mishaps causing customers downtime and other costly consequences.

    As a matter of fact, I am actually the head of a class action lawsuit “Gerasimatos vs New Dream Network” in regards to DreamHost not abiding by the California Department of Labor regulations for paying employees overtime for working greater than 8 hours daily, and also for illegally deducting vested wages for sick and vacation time. I became a whistle blower on DreamHost and shortly after the DLSE contacted the DreamHost owners I was terminated for “doing the right thing”.

    Here is the exact letter the DLSE representative sent DreamHost prior to my termination for engaging in protected activities with the DLSE.The law firms participating in the Class Action suit are as follows.

    [www.coviello-law.com] and [www.duvel-law.com]

    Thank you,

    Nicholas Gerasimatos

    Dallas Kashuba and Josh Jones:

    I am a retired Senior Deputy Labor Commissioner who was asked to return to assist in answering queries that come to the Division of Labor Standards Enforcement from the public concerning California Labor Law. One recent query had to do with alleged practices of your business. I do not know if the query originated from an ex-employee, current employee or friend of somebody who knows your business practice(s). This response is informational in hopes that your business practices do not generate future wage claims and additional liability for your business.

    It is alleged that you offer both sick leave and vacation leave as benefits of employment. It is also alleged that in the case where an employee takes sick leave and does not have enough sick leave time accrued, that you deduct double the excess number of hours from that employee’s vacation leave bank. The example give was that if an employee had two hours of sick leave on the books and took a day off “sick” (a total of 8 hours) then you would deduct the two hours from the available sick leave and 12 hours vacation (double the 6 hours necessary to cover the absence). If this is a factual representation of your policies, you are incurring a great deal of liability.

    Labor Code § 227.3 protects vacation hours as vested wages. The California Supreme Court in the case of Suastez v. Plastic Dress-Up (1982) [31 Cal.3d 774, 647 P.2d 122, 183 Cal.Rptr. 846] unanimously ruled that under the provisions of Labor Code § 227.3, vacations are earned day by day and any unused vacation must be paid on a pro-rata basis to the employee at the time of termination. The California Supreme Court concluded as follows:

    “The right to a paid vacation, when offered in an employer’s policy or contract of employment, constitutes deferred wages for services rendered. Case law from this State and others, as well as principles of equity and justice, compel the conclusion that a proportionate right to a paid vacation vests as the labor is rendered. Once vested, “the right is protected from forfeiture by § 227.3 on termination of employment, therefore, the statute requires that an employee be paid in wages for a pro-rata share of his vacation pay.”

    Vacation wages, being vested as earned, cannot be taken from the employee; they must be paid. Therefore (back to the example) taking an additional 6 hours of vacation hours from the employee’s leave bank is a failure to pay for the hours that were vested. This is cumulative and affects every current and prior employee that has had excess vacation hours deducted from their leave bank. A particular quirk of the protection is that there is no violation of the statute until such time as the employment agreement is severed (termination or quit); when payment of final wages becomes due. If the illegally deducted vacation hours are not paid at the final wage rate in accordance to the applicable statue (either Labor Code §§ 201 or 202), the employee has the basis to file a wage claim against you for not only the vacation hours owed, but for penalty wages under Labor Code § 203 (for up to an additional 30 days of wages at their final wage rate).

    I have advised the party that queried the Division of unlawfulness of the deduction and suggested that they have the basis of a wage claim if they were an employee of your business and had the hours deducted or would have such a claim if they were a current employee and such wages were not paid when the employment relationship ends.

    Please note that business that combine sick leave and vacation leave into some form of “paid time off” or “PTO” such hours would be entitled to the same protection as vacation wages under Labor Code § 227.3. Certainly you do not owe wages to employees for work not performed; you are within your right to deduct (hour for hour) any absence from the wages owed that were not worked (time off without pay). You could deduct (hour for hour) from the vacation hours (so long as the employee is paid for these hours on their regular pay check). I would strongly suggest that such hours are reflected on the wage earning statement as something other than regular wages (either sick leave or vacation hours, as is applicable). The only issue with your policy would be any deduction of vacation hours that were not paid.

    You may access the Labor Code from the left side of our web site at http://www.dir.ca.gov

    The foregoing has been provided for informational purposes only and does not constitute legal advice. Information contained here may not be relied upon or used as an official opinion of the Division of Labor Standards Enforcement ( DLSE ) in any forum. Access to, transmission or receipt of, or reliance upon this information from the DLSE does not create, and is not intended to create, an attorney/client relationship between you or any other person and the DLSE or between you or any other person.

    102