What’s wrong with software development? My Comcast Xfinity Experience or What having my internet out a week reminded me about software engineering
Here’s a game for you: As you read my account about a terrible experience with Comcast Xfinity, please list on a separate sheet of paper all of the things you learn about how to fix a technical engineering and customer support project. At the end of the account, compare them against mine, and list any ones you thought about which are different from mine in the comments section.
On Apr 1, during a work conference call from my apartment, my internet connection went out. I checked my modem and router, did the basic user debugging [turn off the power and power on, check all connections], then called Comcast Xfinity. That call was pretty much useless. They told me they couldn’t see my modem with their network tools, that there was no outage in the area, and that I’d risk paying for a technician if they sent a tech out and they found out it was my setup. I have an unsupported modem, so I didn’t want to take the risk. I had also recently added an Amazon Echo to my network load. That night, after working from a nearby library, my internet connection was live again.
My next work from home day was Apr 3, and, again, during a morning work conference call from my apartment, my internet connection went out. I went to work from the library after calling Comcast Xfinity and trying to restart my modem/router setup with all other devices removed from the network.
Again: my only option was to call out a tech at my own risk with a desupported modem. Comcast Xfinity has this habit of sending me notices about my desupported modem regularly, of course offering that they can lease me a new one, despite the fact that my modem is DOCSYS compliant to the level they require. They even told me I’d lose internet early in January of 2018 through at least one written notice if I didn’t upgrade. I didn’t want to take the risk again of having a tech arrive and dismiss my issue out of hand.
My first opportunity to buy a modem was on Saturday morning, April 7. I went to a cafe nearby with internet to find out which modems I could consider buying. Comcast Xfinity’s site didn’t work on my tablet, so I tried to chat with customer service about what modems are supported. However, the chat didn’t go well. The person ignored the fact that I had already tried the MyDevices portion of their site, and recommended it again. He continued to tell me he could get me a list, even after I got him to recognize I couldn’t use that part of the site on my tablet. Later, he confessed that he actually couldn’t get the list to me: he couldn’t email it to me or provide it through any other means than the MyDevices site which didn’t work on my device. I had wasted an hour of shopping time chatting on a subject he couldn’t help me with. Actually, he probably could have, by telling me what I found out next, but I ended up going to a big box retail store to buy a modem already frustrated and angry.
At the big box retail store, I found out that modems are labeled for the internet service providers they support. That’s all customer support needed to tell me on chat if they couldn’t trigger an email to be sent to me with all supported modems.
I settled on a cheap modem, and returned home. It didn’t fix the problem. My unsupported modem wasn’t the problem, so I called technical support to send a tech out. They sent one out the next morning, Sunday morning.
On Sunday morning, the tech arrived, promptly told me it wasn’t my setup, told me it related to the cable not being live coming into my apartment, and offered to take a look at the apartment’s main connection box. He didn’t know where that was. I didn’t either. I called my weekend emergency landlord who didn’t pick up. After 30 minutes of searching my old apartment building, the Comcast Xfinity tech left without fixing the problem.
When my weekend landlord finally contacted me, it was clear the weekend landlord didn’t know where the connection box was. The daytime/weekday super did. My complex management blamed Comcast Xfinity since Comcast Xfinity installed the cable, and Comcast Xfinity blamed the apartment complex, since the building owners should know where the box is located.
I got a call from Comcast Xfinity to schedule the next tech visit. The earliest the person could arrive was Monday morning any time from 8am to 10am. That caused me an issue: for me to be present to let the Comcast tech into my building anytime between 8am to 10am, I’d need to be physically present — to buzz them into the building and to help them find the apartment’s internet connection box. However, to be physically present, I couldn’t work from home — I rely on the internet to work from home. My sprint was in a place where I could not be absent, and my employer isn’t very flexible with that sort of thing on a good day, so I felt stuck. The only option was to get my landlord involved.
Me [to Comcast Xfinity dispatch]: Since the problem is not in my apartment, can I give you my super’s phone number and you and he work out a time for you to figure this out?
Comcast Xfinity dispatch: You have to be present to let the tech into your apartment.
Me: I can’t. Anyway, the problem is the cable leading into my apartment, not my apartment so why?
Me: Let me speak to a supervisor.
[long time passes]
Supervisor: You’re right. You don’t need to be present, you just need to get a super to let our tech into the building and show him where the connection box is.
Me: I have a hard time getting my landlord to do anything, and he definitely won’t pick up on the weekend.
Comcast Xfinity Supervisor: Then we’re stuck. I can’t call him or ask the techs to call him.
Me: I have an idea.
[At this point I conference in the weekend super of the building, she talks on another phone to the weekday super].
Comcast Xfinity Supervisor: Does 8am to 10am on Monday work for your landlord?
The weekend super, the weekday super, and I agree the weekday super will be present to let in the tech. I feel hope, even though the landlord must now buzz the tech in within 10 minutes of the tech arriving, or else the tech leaves.
Monday morning, before leaving for work, I tape his number to the front door of the complex, addressed to Comcast in case the text didn’t get the message from the notes on the call, and I go to work.
Monday morning, April 9, is stressful with a lot of conference calls and work, and when I haven’t heard from my landlord at 9:50am, I call him. “Are they still supposed to come out?” he asks me.
I’m incredulous. I call Comcast Xfinity and have to go through their complete phone prompt navigation to get to a person. The person tells me the appointment is 8am to 10pm. I tell him I got an automated text and a verbal confirmation that it was 8am to 10pm, and he tells me he has no ability to see what was texted to me, can’t talk to the technicians at all, can’t put me in touch with them, and that I must be wrong: the system says 8am to 10pm. I’m 1.5 hours by shuttle/train/then bus from my apartment. I need to the internet to prepare a Thursday night Kotlin presentation on Test and Data Driven Development, then finish my taxes. He talks to a supervisor, and they tell me a tech is coming out in 15 minutes. I call my super, and he says he’ll wait at the building an extra 15 minutes after already having waited there for 3 hours.
I call the super back in 30 minutes, and nobody arrived.
I call Comcast Xfinity, navigate their impossibly bad phone prompts, and get someone who tells me that they changed my appointment to a “pole visit” which means the tech will wander around outside my building looking for an internet connection which is not outside of my building and then leave. A pole visit doesn’t even notify me when the tech will arrive. The tech might have already arrived and left. I insist that was never communicated to me, the change, and that I really need help.
They offer to change it to a trouble ticket, which would ensure a tech would arrive at 3pm, wait 10 minutes to get into the building, then leave if nobody lets him in. I agree to take a risk and leave work at 1:15pm to get home to let the tech in. That means dropping my project to get this to work, because I can no longer reach my landlord and Comcast Xfinity is unwilling to reach out to the property owner to arrange a visit.
I was on the shuttle to the train [then to the bus] beginning at 1:15pm.
I arrived home on foot because the bus was late at precisely 3pm. Within 15 minutes, I got a text from the tech. It was the same tech as before. In the meantime the weekend super of my building told me approximately where the boxes where, though that person didn’t know precisely where.
I let the tech into the building, and we walked to the cable connection boxes in the basement. Sure enough, there was juice going up to my apartment, just not entering my apartment.
Now I was imagining trying to get either Comcast or my landlord/property management to fish new cable up into my apartment — I might be a month without internet, at least.
The tech offered, half-heartedly, to check if there was a defective splitter in my apartment, but I think we both knew that was unlikely to be the issue. Because there were no other options, I okayed that. He sat down near the cable box, felt around in his utility belt for the tool to check the splitter, and then: that was where something miraculous happened.
He didn’t have the tool to check the splitter; he had probably left it at his last site.
Because he needed to phone another tech. There are hundreds of techs he could have phoned, but he phoned — let’s call him Fritz.
Fritz drove over with a new tool.
However, once Fritz arrived, he realized something. I heard them discussing it in the stairwell:
Fritz: This is precisely the apartment complex I was working in on April 3!
My Tech: You don’t say.
Fritz: Yes, I was working in apartment 21, directly underneath this guy’s apartment — apartment 31.
My Tech: Oh, really.
Fritz: Yes. Let’s knock on her door and see if she’s home.
The resident of 21 was home — which was also very lucky, considering it was about 4pm on a work day — and she let them in. Within a few minutes, Fritz reconnected some wiring he had disconnected on April 3, and I had internet again.
Fritz explained to me that when he was in apartment 21 on April 3, almost a week ago, he saw what looked like a hack job. He saw in that hack job what looked like extra cable, so he disconnected it. That “extra cable” was my internet.
I was later reimbursed about $3 per day of outage, coming to a grand total of about $21 for losing the ability to work from home, take care of a professional presentation, do my taxes from my home’s secured network, conduct financial transactions from home, and all of the unnecessary hours I put in actively facilitating the solution of Comcast Xfinity’s problem, despite the fact that they were 100% at fault and their agents are being paid to fix this stuff, not me. A person who didn’t have as much work-life-flexibility couldn’t have managed this. Imagine, for example, a single parent of two who has work and school and other responsibilities to balance: completely screwed.
Make sure to record your answers before reading mine below.
- Someone left a hack job behind in apartment 21, which Fritz found. Don’t leave ugly or undocumented code in production.
- Fritz thought he’d fix it by disconnecting a stray cable. In this case, he couldn’t test his change. The code was so badly documented, that he didn’t know to reach out to the person in Apartment 31 who was connected to that cable. However, an ideal situation would allow him to have flagged the situation as “active” for three days, during which any trouble tickets would be associated with his change.
- Hack jobs should be treated with utmost care. Think The Hurt Locker when working on a hack job. Hack jobs are expensive, and messing up a hack job even worse is even more expensive.
- When I called in, my trouble ticket was not connected in any way to work done on the exact same day the outage began. A production support team needs to look at “deployments” which occurred on or around the day that an issue began for a trouble ticket.
- The tech who visited my apartment didn’t know where the connection box was for the complex. Comcast Xfinity needs to store that information at the building level for buildings in which they install cable. According to customer support I spoke with later, they have this capability, but they don’t document the locations clearly or consistently. Your code, your documents, and your comments must point support to the exact location of a problem, including consistent program logging.
- There was a turf war between Comcast Xfinity and my landlord/property management. They couldn’t talk to each other, and both claimed the other one should have known where the cable connection box was for the complex, that the other one owned the cable in the building. Tech leadership, I’m looking at you. This happens too frequently in real world software development. Ownership should be clearly defined, and departments should be structured not to oppose collaboration but to facilitate it. This was more a customer / support relationship, which is more difficult, but these things should be understood, especially, as in this case, where Comcast Xfinity both owned the cable and needed to know where its own connection boxes were.
- There was a turf war within Comcast Xfinity between the techs and support. Why couldn’t customer support put me, the customer, in touch with the techs? Why couldn’t customer service effectively communicate with the techs and dispatch for the techs? Why couldn’t the techs and dispatch reach out directly to the property owner, not go through an intermediary like me? This all seems like the result of bizarre department structuring, and is inconceivably bad.
- The Comcast Xfinity MyDevices web site didn’t work on my iPad. Make sure to regression test all of your development work against supported devices, including those mobile or handheld devices a user might reasonable utilize.
- Support chat was completely useless in trying to provide me a list of supported modems. At the very least, support chat should have been able to tell me that modems are labeled on the packaging as to whether or not they are supported. Ideally, support chat could trigger an email to my email address on file containing helpful information regarding supported devices.
- My modem works, but is unsupported. Okay, I understand you can’t support devices or configurations forever. At some point, Microsoft needs to stop supporting Windows 3.1 on the laptop I had in college. [They did]. However, the reason for being unsupported should be clear [my modem complies with the DOCSYS requirements for support], and at no time should a company send out an email saying my service will end on such and such a date because of using this or that deprecated anything or this or that unsupported anything unless my service will, in fact, end on that date and it has been tested.
- Active tickets should get preference and should not have to navigate the entire help desk system to reach a knowledgeable support agent. This is true for IT support of software, too.
- My ticket got changed from a trouble ticket [8am to 10am with a phone call] to a pole ticket [8am to 10pm with no phone call] without notifying any stakeholder: me, the landlord. It wasn’t even noted adequately on the ticket itself, so customer support ended up arguing with me. I couldn’t even email customer support an image of the text message I got confirming 8am to 10am. I give up. There’s too much failure here for me to derive a useful helpful guidance to tech/IT leadership.
- The penalties of being at fault are asymmetrical, and they don’t respect the true cost of the outage to the customer. Had I wrongly called out a tech to work on my apartment’s internet when my configuration had been at fault, I could have been charged $40 per visit. However, Comcast Xfinity only reimbursed me about $20 for about six full business days worth of outage they caused. This might work for your business if, like Comcast Xfinity, you are a monopoly, but this will cause customers to leave immediately if you are not a monopoly. I will be looking for my next apartment not to have Comcast Xfinity.
- Customer service seemed barely able to read scripts at times, much less analytically address the problems they were confronted with. Invest in all levels of customer support. Hiring bottom-dollar customer support is why your customers are leaving and why your employees who rely on them are, like, a third as effective as they should be.
- It shouldn’t be a miracle when a software/tech issue is resolved. Never let it be a miracle when a problem with software/tech you manage is resolved.
- Require that every level of customer support escalate lessons learned. When I complain that this or that thing is broken in software or tech, I don’t just want to know you or a manager deployed this kind of awesome workaround to make it work. I want to hear you are escalating the problem so the right team can resolve it going forward. Not to do so makes the organization seem not just broken, but incompetent.
Thanks for reading! When I see applause for a piece of mine, I want to write more pieces! Please applaud if you appreciate people writing about topics like this.
If you enjoyed this piece, you might consider reading some of the other stories I wrote for this series. Thanks!
Disclaimer: I am a senior software professional with experience as project lead, team lead, and lead engineer in the greater Boston area. I am actively seeking a job while currently fully employed. The world of software development can be an ugly place, and I hope to expose some of the more bewildering and uncomfortable corners in the hopes of improving the industry and the craft. Out of respect for current and former employers, I will keep the names of employers and/or clients scrubbed and anonymized as best as possible in these reports.