Dell Business Blows Up Servers

If there’s one thing you expect Dell to know, it’s computers. And although that assumption may be baseless when you’re talking to a CSR from the Orient, telling you the only solution to your monitor being broken is to wipe your hard drive and reinstall, if you’re a business-class customer, talking to your Dell contact to set-up a RAID, you at least hope that they have an inkling of what they are doing.

But au contraire! Dell Business’ enthusiastic recommendation of 4 SCSI drives and a RAID controller to Matt B. not only led to him flushing $2000 down the toilet, but his entire business’ network to be fried. As he so expertly puts it, “Dude, we got f’ed in the A!”

Matt’s email, after the jump:

A week and a half ago, my cohort noticed that our small company’s email server was nearly out of space. Not only is the email server a two year old Dell, Dell has also always hooked us up reasonably well on price and support. So their business advisor people suggested to my suggestible coworker that a RAID controller and a few more SCSI drives could handily take our one small SCSI drive and turn it into a bigass RAID 5 array. They assured him it was the easiest thing ever and the whole process would be quick and painless. This was important because we were already aware that our exchange/active directory setup was extremely fragile since we’d installed both programs on the same computer back in the dark ages. We trusted them and bought 1500 bucks worth of recommended hardware (4 drives, one pci-x scsi raid controller) to avoid a long difficult migration to one of our more modern servers.

Cautiously optimistic, he went to our colo last night with the goods in hand and called up Dell to have them walk him through the process (3 times) before he was willing to change anything. Two hours later, I recieved a curt call that “the server is fried, this could take a while. I’ll call you when it’s back up.” This morning at 5:40 I got a text message that “The old server got wiped out, I had to put everything on the new servers.” Turns out Dell’s bliss came from ignorance. Or they just wanted our money. Either way, they lied to us to sell us over a grand in hardware and then calmly walked him through the steps to destroy our working setup and nearly screwed our whole company over. It’s only because he pulled an overnighter to setup a less fragile system that we’re able to communicate with our clients today. Dude, we got f’ed in the a!

Comments

Edit Your Comment

  1. Mike_ says:

    If you don’t know what you’re doing, and your business depends on it being done right, hire a professional. Having Dell walk you through it over the phone is NOT the same as understanding the problem and being able to arrive at a solution. Even small companies can afford an occasional visit from a qualified technician. Next time, don’t be so cheap or arrogant or whatever it is that compels a car mechanic to attempt brain surgery.

    If you ask me, you dudes f’d yourselves.

  2. I agree with Mike. Hire someone who knows what they are doing. Now, many companies swear by Dell business service, but I’ve never been impressed.

    Saying that, the IBM service people who show up are very professional and damned good. Like “let’s back up your data first” comments.

    And, I thought it was Rocket Surgery.

  3. For one thing I’m amazed that their enterprise level server was running the Exchange Information Store without a RAID to begin with. Implying that Dell had manipulated your “suggestible” co-worker is kind of a stretch considering having Exchange on a RAID is rather common practice given how critical the systems tend to be.

    Furthermore I don’t know what exactly you were expecting from the migration to a RAID– copy the files over and be done and walk away for the night? It’s often not quite that simple. Exchange has an entire migration procedure and that’s not even including how you intended to have the OS setup (boot from RAID? boot from the old drive?).

    I distribute blame points equally to both sides for being ill prepared.

  4. Falconfire says:

    From what I am seeing, their recomendation wasnt without merit. Working in the IT world I can tell you without a doubt, computers suck. You have to practically be Mr Scott half the time and multiply your estimates because what ever you think it will take, it ends up much longer.

    In this case I can think of at least 5 different things that could have gone wrong all of which not Dells fault. The biggest peice of advice I would have given you is…. back that shit up.

    For just about every server you run there should be a complete backup of it ready in a moments notice to switch online and take over. It doesnt even have to be a powerful one, just one that you can switch to while repairs are being made. Two years as a email servier is a decent amount of use there. While its kinda a shock for it t just go, its not unheard of especially on a changeover.

    Trust me on this one Dell might not at all be responsible here.

  5. bambino says:

    ^reminds me of an old adage we have in the architecture world:

    Estimate how much time you think a project will take, multiply that by 3 & add 5, & you’ll still be cutting it close.

  6. Mike_ says:

    Hardware RAID5 with a warm spare and hot swap capability is appropriate for some environments. In a lot of cases, it’s overkill. If you tell Dell you need more disk space, and they sell you 4 drives and a controller card, you may have thrown $2,000 at a $200 problem. Their job is to sell you hardware. Your job is to know what you’re buying and why.

    Before you make this kind of purchase, you should probably try to figure out exactly what your requirements are. This guy needed more capacity. He bought more capacity, but he also spent a lot of money on hardware-fault tolerance and improved performance he probably didn’t need.

    This isn’t really the place to argue about who needs RAID and why, but I will say that experience and intuition tell me that many people who buy RAID equipment do so for all the wrong reasons. They’re buying buzz-words (e.g. “bigass RAID5 array”), and not a necessary tool to meet a specific need.

    I think Dell should have known better than to try to walk a novice through this sort of transition over the phone. Ignoring Murphy’s Law and crossing your fingers is no way to ensure a favorable outcome. Even then, if the customer insists on giving it a try, it’s good of them to offer some assistance. It seems to me that Dell’s only error was being too eager to help.

    This guy shouldn’t have messed with his “extremely fragile” server without knowing what he was doing. If you have to have Dell walk you through it 3 times, you’re not ready, grasshopper. On the bright side, we learn from our mistakes, and this guy will be a little better at his job after this experience.

  7. AcilletaM says:

    I’m in the f’d yourselves category. Yeah it sucks that this happened but your computer system is too important to leave to the happenstance and the “lucky we noticed” management style. What would have happened if the server did run out of space? Get help in that area, it pays off in the end.

    That said, I found it’s buyer beware when it comes to support from HP, IBM, Sun, Dell, whoever. I work mainly with UNIX servers and I’ve had good “let’s back up your data first” support and bad “shutdown the production server in the middle of the day because it was missing the latest patch for a process we don’t use” support.

  8. barneyfyfe says:

    Matt didn’t do his due diligence and seek a second opinion. He didn’t set up a mission critical app on a RAID. He didn’t have an experienced onsite tech to do a hardware migration. He treated a serious issue as if he was changing a light bulb. This guy took the word of a sales clerk on how to run his business. No sympathy.

  9. drsmith says:

    I’m with the others in this. The email above wasn’t written by the guy actually doing the repairs, so the information is second hand from the start. We’re really left not knowing what the experience level of the coworker was(which mean he’s a guru or he works at mcdonalds in his spare time and did this as a favor).

    In any case, you’re not supposed to have active directory and exchange on the same server. Yes, some people get away with it, but it’s not a recommended configuration. What’s worse than that is the fact that it didn’t have raid installed from day 1. Maybe a multi-day downtime is ok in your company, but it’s never been an option anywhere I’ve ever worked. The tone of the email suggests that not having raid in the server represented an unacceptable level of service, in any case.

    Finally, you can’t expect Dell, IBM, or anyone else to know anything about your environment. It’s your responsibility to know that you’re buying the right hardware and to know how to get it installed. Only when the hardware you bought is faulty does Dell assume any responsibility. Again – that point is unclear from the email. Did the new adapter fry the server because it was faulty? Or was the technician inexperienced?

    I’m left with more questions, it seems.

  10. tz says:

    When someone says “it is the easisest thing in the world”, ask them if they will guarantee it or pay for your downtime or damage if something does go wrong.

    I’m fairly handy and have an easy to service car, but a few bucks extra to a mechanic who does just that for hours every day is cheaper than a new engine or transmission.

  11. Mike_ says:

    I still disagree with the notion that “mission critical” always means “RAID”. As far as fault-tolerance goes, RAID is insurance for a very specific peril: drive hardware failure. It’s not an invincible disk that you can configure to solve all your problems. You still need reliable backups, and you are still going to have down time if your filesystem gets corrupted.

    You’ve got to look at your needs and your budget, and decide what configuration is appropriate for your implementation. There might be a place for RAID in your design, but the suggestion that there’s no substitute for hardware RAID in a mission critical environment does just one thing: sells more drives and controllers.

    But I digress.

  12. Spiny Norman says:

    Having been in the situation where I have had to resurrect a Dell, I learned the hard lesson that you upgrade the power supply before you toss more hardware in the box. Otherwise, it’s crash, crash, crash and the concomitant heartache. My guess is that you put too much on the buss and burned because of it. Enterprise motherboards aren’t that expensive and most of them support hardware RAID, but I digress. The job of the Dell team is to sell you stuff. Perhaps not all the stuff you need. Perhaps not the right stuff. If you are using Dell tech support as your IT department, you got what you paid for.

  13. joehowe64 says:

    But for God’s sake you did buy the Extended Service Plan they offered!

    And the fuzzy server cozy!

    And the Dell DJ dock!

    Tell us you did!

  14. jlobster says:

    I might also add that the details on what happened are surprisingly scarce. “Fried” and “wiped out” offer no information on what happened and when it happened–akin to saying “The Internet’s broken.” Given that the user had to be led through a transfer process, their inexperience in this type of procedure was evident. We don’t even know what they tried to get it back up and running. We don’t know what part got “fried” or if a rollback was indeed possible in some fashion. Add the fact that there was apparently no readily accessible backup (it’s your Exchange server and even YOU admit it is fragile! C’mon!) and it seems that naivete and inexperience were the culprit here.

    Believe me, I have been on both sides of this problem (freaked user and even-more-freaked IT guy), and it seems that something more was needed, both in preparation (proactive) and solution (reactive).

    I know Dell can be pushy and eager to sell, but aside from “easiest thing ever” statement (which should have sent up a huge red flag on the BS detector), their recommendations were completely in line with a proper storage setup for a small business. Bitch about them all you want, they were not at fault here.

  15. biggeek says:

    Sounds like whoever is playing System Administrator finally got off his ass and did the “difficult migration” Matt mentions…Which he probably should have done as soon as they received those “new servers.”

    The histrionics over $1500 is pointless. Dell will probably replace the motherboard free and you can repurpose it as a file server.

    This whole thing stinks of cheap management and a lazy/inexperienced Sysadmin.

  16. RAID 5 is overkill where RAID-0 + 1 hot spare is always a simpler solution./

  17. madderhatter says:

    RAID 5 is not overkill. Hell, why not RAID 10+1 ? Seriously, mission critical (if e-mail can be considered that – mainframe maybe) should have as much redundancy as possible if your crap is really mission critical. Drives, power supplies, network cards, etc. I’m one of a few sysadmins in a shop with over 150 Dell servers – everything from PE1550s to monster 6800s and Itaniums running Linux, Win2003, FreeBSD, etc. We have guaranteed on-site response times of 4 hours, but that is Rarely used.

    Know what you’re doing if you’re going to work there. If you’re in over your head, and you’ll know if you are, get someone in there that knows what they’re doing.

    Backups are key, so they say. All of our systems are backed (except test and dev) and the only time I’ve had to use a backup it didn’t work. Don’t restore entire systems (OS, programs, and files) just get the data. The rest of it you can recreate in minimal time.

  18. AcilletaM says:

    One further point, one my company’s Wintel server team cannot seem to learn, backups are not the key. The ability to restore a backup is the key. No matter how sophisticated the backup/data recovery scheme is, if you can’t execute it, it’s useless. And don’t assume if you get a “good” backup the resulting restore will be fine as well.

  19. kickslop says:

    Good grief. I haven’t been back here in a long while, but I saw this article linked to from lifehacker.com today.

    You people are as bad as FOX News. A completely incendiary headline, lead-in write-up, and stupid email from an incompetent and ignorant system administrator (or whatever he is).

    I’m all for consumer rights, but you f’d yourselves, period.

  20. Watergirl says:

    I hate DELL….plain and simple. The customer support is lame, exhausting, and tedious. I have had more than several frustrating conversations and experiences with tech support. I was warned when shopping for a laptop and did not listen. If you want a part time job that is aggravating and does not pay, buy a Dell computer. You will spend more time on hold, being forwarded to numbers that are not in service, talking in circles with the “techs” (from India), and reporting problems to the BBB than anything else you have ever invested time in your life.
    Never, never, never again.

  21. tippps says:

    Rule #1 FULL BACKUP before you do ANYTHING on a production box!!!!

    Rule #2 unless you absolutley know what your doing and are an IT pro. Don’t touch your production box!!!

    Rule #3… first 2 rules sum it up. But i absolutely cannot believe “Dell” was actually going to walk a lehman through setting up a RAID as if it were troubleshooting a desktop.