Matt hasn’t been able to upload anything for 10 days now which is a bit of a problem when you have a home business built around uploading stuff for clients.
He’s called, had a tech visit, and escalated, and now, is resorting to emailing Verizon CEO Ivan Seidenberg (and emailing Consumerist):
Dear Mr. Seidenberg,
I understand that your (and your team’s) time is valuable, and that you most likely only get emails from extremely frustrated customers who like myself at the moment are at wits end, but I feel I need to share this story with you. I will detail it below, but to give you a quick summary: it has been ten days since I’ve been able to do my job here at my home office running over Verizon Fios. I have 5 static IP addresses with you, which have run flawlessly for the 3 years since installed. None of these addresses will allow uploads any longer. My business still resides in my home because I’m a bit of a workaholic, and the technology fios provides has made that a possibility – there is a large server rack in my basement which is normally providing MY clients with antivirus updates, backup files, patches, and everything I promised I’d do for them remotely, which is frankly something I’ve been unable to do for a minimum of 10 days.
I’ve tried to detail this as much as possible, because not only has this situation kept me from my friends and family, it’s doing the same with your field technicians who have been ignored just as much as I have. It’s not right, and I can’t imagine it’s how anyone envisioned their company being run. So without further delay…
On October 11, I began noticing some issues with rolling out patches to my clients. I believed it to be an internal issue as our firewall was old and replacement was due. I spent a day in emergency mode building a new firewall. Same issues occurred after, but things still seemed to be working “well enough” to do my job.
By October 13th, our backup server which has for years uploaded all of our changed files to an offsite location at 2am every night began reporting backups are no longer being uploaded, due to a connection reset. I thought maybe it wasn’t the firewall after all, but an Internal switch. We replaced that too.
By October 18th I found that the issue was truly not with us. I went into hardcore tech-troubleshooting mode, and connected a laptop straight to the Verizon ONT. Speed tests would run fine, but uploading anything: a youtube video, facebook picures, an FTP of a new website: fails. Sometimes I’d get lucky and a picture or two would go up, but overall everything fails. Any long connection: Fails. I was testing on a Windows 7 laptop with antivirus, which I knew Verizon support would have issue with, so I then tried a Windows XP laptop, with no antivirus or firewall – only to receive the same result. Short bursts of data such as the speed test report no issues – but any large files (5MB or bigger) fail like clockwork.
October 20th I spoke with an agent at the business support center. I don’t know his name, but boy was he rude and inept. I realize it’s hard to get all A-team players, but this guy was condescending and in left field when it comes to customer satisfaction, let along any technical expertise. When I explained to him I had my laptop connected straight to the ONT, he demanded that an Actiontec router be in line for it to work properly. That’s OK, because I happen to have purchased two from Verizon as spares considering my livelihood depends on that connection functioning properly. When testing with the actiontec in place, the same result occurred. The agent stated it must be something on my edge because everything “looked good” from his end. I then tried the other actiontec router, and other laptop I have here as well, with the same exact result. The agent seems to believe it’d be normal to have 2 bad routers and 2 bad laptops, and nothing could be wrong with his connection. I asked politely to speak with a supervisor, and after being on hold for 5 minutes was told a Verizon technician would be there between 1 and 5 the next day.
On October 21st I rearranged my support schedule so that I was available to a technician between 1 and 5pm. No one showed. No one called. I stayed here until 6pm.
On October 22nd I called in the morning to find I’d been bumped a day, but not notified. I was told two technicians were going to show up within 2 hours. A few hours later, a technician named Josh (who is fantastic) showed up. He acknowledged the problem, and did everything in his power to resolve it. He replaced the ONT. He had central try “test connects” and various new connections, would would change the behavior of uploads, but not actually fix it. He tried another fiber line between my residence and the hub down the street. He tried an even different model of ONT. He tried TWO MORE actiontec routers. He has the same issue, speed tests work, an email without an attachment will work, but as soon as you try to upload a file… FAIULRE. After about 5 hours of him staying there on a Friday night to the point we’re both missing time with friends and family, he escalated the ticket. He was kind enough to give me his cell number if he thought he could be of more assistance, but at that point it was obviously out of his hands and it seems even Verizon’s field technicians need to call into support only to get the run around.
The weekend I expected someone would be fixing the issue somewhere in Verizon, considering at this point it is obvious that the issue is somewhere “inside the system” not at my end.
Today, October 25, 2010 – I found the behavior was the same. I came into the office to find nothing but a list of failures where my servers attempted to send updates to clients, few of which made it out through the connection. I called business support, who was of course very apologetic, and had dispatch send someone who was at my home office within 2 hours. His name was Jay and he was another great tech. Unfortunately, he also had to call into the higher level support who had him jump through the same hoops my time was wasted with the prior work day. We could watch as a youtube video would get to about 4%, then fail. Or watch as I tried to make an FTP connection (to various locations) and watch it fail, then reset, then attempt to continue, then fail again. It was like clockwork – failures. He argue with support on my behalf but again was given the run around. Eventually we convinced them to reset the card I’m connected to at their office. When that happened, uploads failed less soon, and would even reach almost 50% for a 55MB file, but they would fail. The fact that resetting that card (PON card, pawn card?) caused such a drastic change in performance makes it blatantly obvious that the faulty piece of hardware needs to be replaced. It was 4:30pm at that point, and Jay, the field technician at my home office, was told that the woman who he was speaking with’s shift was just about over, and someone would contact him shortly about receiving approval to replace the piece of equipment. No one called. Half an hour later Jay began calling anyone he could think of, his supervisors included – no one answered. Just like that I lost yet another day of productivity in what should be a business I enjoy, which has now left me with a 50 hour back log and lots of stress.
It’s upsetting enough to see this happen to me knowing the effect it has on me, and at this point, my pocketbook as I am literally unable to do my job of remote support right now, but I also find it upsetting that your own field technicians are thrown to the wolves because it’s “quitting time.”
I understand that what has occurred are in a system which is many levels below you and your team, but it is a situation which is about to lose a client who manages networks around here, who would have issue recommending fios as a solution for small and medium businesses in the area at the moment. Honestly other than this letter, I’m not sure what my course of action could be other than to begin calling competitors to find another connection with static IP addresses here.
Epic email is epic. Matt might also want to try giving Lisa Charles in Verizon FiOS Executive Customer Service, 212-321-8463, a shout too.