Author Archives: Kirk Schnable

My Negative Review of ARS Rescue Rooter

It’s winter, and the heater is not starting up!

Living near Chicago, it can get quite cold in the winter time. Just last week, the temperatures dropped below 0 degrees Fahrenheit. My furnace was working overtime keeping my house warm.

I’m a pretty warm blooded person, I usually run my winter thermostat around 64 degrees, a temperature that would be too cold for most people, but works well for me. As anyone who reads my blog knows, I have a lot of computer equipment, and as anyone who has owned computer equipment knows, it creates a lot of heat. Keeping my thermostat low helps keep my computer room comfortable and my server rack cool.

Despite this low setpoint, my furnace was struggling last week. Luckily, it made it to the finish line, but this week when temperatures rose to almost 40 degrees Fahrenheit, problems emerged.

I am a night shifter, and usually stay up until at least 4 or 5AM most nights. On one particular night earlier this week, as I was getting ready for bed, I emerged from my warm computer room and noticed the downstairs of my house felt chillier than usual. Sure enough, I checked the thermostat, and I was at 61 degrees, 3 degrees below setpoint. Uh oh.

I went down to the basement to check the furnace, it was running, but blowing cold air out the vents. I noticed a flashing indicator light blinking a 4 blink pattern, and after checking the inside panel, diagnosed that it was a “sensor over limit” error. I assumed this meant there was a bad sensor, probably reading 0 or infinity. I did not assume this would be a major repair.

However, by this time, it’s almost 6AM. I need to get to bed so I can get up for work, even for me, it’s now getting to be past my bedtime. But, although it’s 40 degrees outside, there are 20 degree lows in the forecast, I am not sure how my house will handle without a furnace as temps drop below freezing, and I know that when I get up and go to the office, I will not likely have a lot of time to talk to HVAC companies while they’re open. I think, it’s already almost the start of the business day, I can probably call someone now.

So I began to quickly try and find someone to call, I checked around online, but so many HVAC contractors feel shady or scammy to me from the get-go, I want someone trustworthy. My grandma recently had her system repaired, so although it’s early, I give her a quick call. She can’t recommend her guy because he doesn’t come out to my area, so the search continues.

After some deliberation, we decided to call the phone number on the last service sticker. Evidently the previous owner of the house had the system serviced in 2011 by Anderson Heating (ARS). Grandma thought that they were a good company, and they’d worked on the system in the past, so I decided they were my best bet. I gave them a call, and they were able to schedule an appointment for me that would work fairly well with my work schedule. I’d miss out on a few hours of sleep, but be able to address the issue that afternoon before going into the office.

The ARS Tech Appointment

The next morning, quite a bit earlier than the time we’d agreed upon, I received a call that ARS was on the way. I’d only slept about 2 hours at this point and I was not super thrilled, but at least I’d have this situation behind me soon, or so I thought.

When the truck pulled up, I immediately got some weird vibes. I noticed the name of the company was ARS Rescue Rooter, and the logo was different than the one on my service sticker from 2011. Had I called the right company? Was this one of those scams I had heard about where a company buys up a bunch of out of business contractor’s phone numbers to scoop up leads? (Fear of this type of scam was one reason I was so skeptical of calling someone off of Google.) My suspicion was high. I guess some of this is on me for not researching them ahead of time, but it was late and I was tired, and at the time I was doing what I thought was best.

The technician came inside, and he seemed like a nice guy, things seemed to be off to a reasonably good start. Though, I did notice, he immediately seemed negative about the outcome before even opening the panel, telling me that since the unit was 12 years old, we could be having problems.

Upon arriving at the furnace, he opened the panel up, and told me he would take a look at everything and should have some information in about 15 minutes, somewhat encouraging me to leave the area so he could work. I wish I hadn’t left him alone with the furnace now in hindsight, but that’s a totally normal thing to do when people are working on your appliances, and I still thought this was just an honest service call. So I went and got a drink, waiting around to hear some results.

When he called me back, he showed me some video from a small flexible camera. He pointed out some lines in a metal panel, stating that they were cracks in the heat exchanger. He proceeded to explain that replacing the heat exchanger is usually not worthwhile, because of the labor-intensiveness, and that his company is not legally allowed to turn the furnace back on with a defective heat exchanger.

A Cracked Heat Exchanger, Huh?

I now know that this is a common scam, so common in fact, that it appears on multiple websites about avoiding HVAC scams.

Here’s a quote from another contractor, Hot Point Heating and Air Conditioning LLC’s website, on the subject of avoiding HVAC scams:

Perhaps the most common furnace repair scam is the “cracked heat exchanger” scam. A shady repair technician may try to trick you into believing the heat exchanger in your furnace is cracked and leaking deadly amounts of carbon monoxide into your home. They will push you hard to either get it fixed immediately or replace the unit entirely on the spot.

Remember: it is possible for your heat exchanger to be cracked and it is a dangerous problem needing immediate attention. However, it is also very rare.

If your repairman claims to have found a crack in your heat exchanger, insist on seeing the crack for yourself—particularly if it took very little time for the repair technician to make the diagnosis.

If you cannot see the crack, do not take it on faith that only the expert’s eyes can see it. Get another opinion as soon as possible to make sure you aren’t getting scammed.

So, let’s do a litmus test here. Was I shown the crack?

Well, I was shown something. The technician had a live camera feed (I know for a 100% fact it was a real camera feed, not pre-recorded, because the technician accidentally dropped the camera at one point and I saw stuff going by as he did).

The camera feed did show some kind of black formation that appeared to be along a crease in the metal. Was it a dangerous crack in the heat exchanger, with the potential to release carbon monoxide? I’m not qualified to tell, this is why I have HVAC technicians working for me.

I suppose it might be a small hairline crack, I suppose it might also be soot buildup. Who knows?

But do I have a carbon monoxide leak? My carbon monoxide detector says: No!

ARS definitely provided some evidence of their diagnosis, and I am not saying that this evidence is fabricated. But, I think the diagnosis that the unit is dangerous is dubious at best.

A high pressure sales situation?

So now ARS had me backed into a corner. Obviously, I need to get my furnace running. The cold temperatures are coming back, next week is going to be even colder. They are essentially telling me that repair is not worthwhile.

The technician told me that my unit was unsafe to operate, and that I had to sign a form acknowledging this fact so that they would not be liable. He did not clearly communicate to me that I had the option to not disable the unit, and pointed to a section on the form and told me to sign it.

Later, when I reviewed the form closer, there was a second section I could have signed that said I accept responsibility and do NOT disable my unit. But, thanks to the pressure, and lack of sleep, I’d just authorized the gas to be shut off, the power to be shut off, and the unit to receive a big red sticker declaring it dangerous.

In the end, I feel that my choice to not disable the unit was not made clear, and that I was coerced into agreeing to disable it.

Furnace scare tactics at work.

After signing this waiver, I am provided an enormous quote of around $2,500 (around what my grandma paid for her entire furnace when she had it installed) for all of the repairs that ARS says would be necessary to make my furnace safe again, including:

  • Replace heat exchanger (~$1,500)
  • Replace pressure switch (~$400)
  • Replace run capacitor (~$250)
  • Replace gas union (~$185)
  • Parts acquisition fee (~$125)

Obviously, when these numbers start to add up, purchasing a furnace seems like a pretty good idea. Fortunately, my ARS technician is able to schedule a sales rep to come out that day and spec out my new system.

In the heat of the moment, with 2 hours of sleep under my belt, this felt like a good choice. I could get a quote, then think about it later at work. So I went ahead and scheduled the next visit. We schedule an appointment between 2-4PM, which is when my original tech visit was supposed to be scheduled.

I attempted to go back to sleep, and sent the repair quote to my parents along with a request to investigate if they feel like the company is legitimate while I’m trying to sleep before my second appointment.

The Sales Consultation

Later in the day, I was awoken at around 3:15PM by the phone. I’d set a last-minute alarm for 3:30PM, which would be just enough time for me to get to work, but expected to be awoken by the phone much earlier than that. I was surprised by how late it was.

The sales rep had gotten held up at another job and wanted to know if he could come at 5PM. Since I had to leave almost immediately for work, I told him that this wouldn’t work.

This turned out to be the best scheduling mishap that could ever happen, because had he come to my house that day, it’s quite likely I may have unnecessarily spent over $5,000 on a new furnace and A\C installation (of course, they pushed A\C too).

We agreed to reschedule to 10AM the next morning. This is too early for my liking, but we picked this time because it might allow for them to do an installation the same day if necessary and get my heat back up and running.

Reconsidering, A Second Opinion

In the time that’s passed, both myself and my parents who I’d enlisted to help research had begun to grow skeptical of the diagnosis from ARS. We decided that since I would already have to go a night without heat anyway, we may as well take our time and get another opinion. Afterall, we still had a few days of nearly 40 degree temps on the forecast, and if I could make it one night, surely I could make it two or three if I had to.

We called another local HVAC contractor who my dad had found, and scheduled for an appointment the next day at noon. (Remember, ARS was scheduled at 10AM).

I decided that it was best to not sacrifice sleep, and growing more and more skeptical whether I wanted to buy a furnace from these folks, so I called up the number I’d called originally to schedule my first appointment. I went through the phone prompts to update my appointment status, and spoke to a very kind dispatcher who informed me that as requested, my appointment had been cancelled. I told him thanks, and I’d create a new appointment soon if I decided to move forward.

An Unexpected Visit

After I cancelled my appointment, I assumed I might receive a call the next day to try to follow up. This would be a fairly standard practice I think for any business. But, I knew I needed my phone turned up so I could receive a heads-up call from my other contractor. So as a temporary measure, I blocked the office number of ARS, so their call would go to voicemail. I figured this would allow me to follow up later without my sleep being interrupted by a sales pitch.

Much to my shock, I received an awakening at 10AM in the form of a phone call from the sales rep. I answered the call and my Google Voice call screening confirmed his identity, but I let the call go to voicemail and went into the bathroom. I was considering what to do, I didn’t ask for this appointment to be completed, and I wasn’t sure I wanted to do business with this company after all.

I decided to call my mom quickly and decide how I should proceed. While I was on the phone with her, there was constant pounding on my front door and multiple calls spamming my phone.

I can forgive this excessive attempt at making noise — I do know the sales rep assumed that I was sleeping (I’d told him my schedule), and I assume he was trying to wake me up. But throughout the course of this, it seems like he realized that one of their numbers was going to voicemail, and proceeded to call me from two other numbers.

He’s now called me from both office numbers and his cell phone number, but doesn’t realize that the appointment is cancelled? How does the office not know this? You would think at some point during this situation, he would check the status of the appointment. Maybe something unexpectedly came up, maybe I wasn’t home, the whole situation just felt very off.

It became quite clear I needed to address the sales rep, but my mom told me I should not let him in the house, and she was beginning to get freaked out by all the noise while we were on the phone. I agreed and didn’t want to answer the door, this was starting to feel a bit too aggressive for my comfort. Especially when I looked at some voicemail transcripts, and I saw that he’d said “I can hear you in the house, why aren’t you answering the door?”.

Further, he was implying in the voicemails that he’d set aside his entire day for me and had no other appointments. It had a tone to it that seemed like he meant to make me feel bad. But did I feel bad? No. I cancelled the appointment, I provided notice, so why was this guy here waking me up pounding on my door? Is this not ultimately my decision whether to proceed with the sale?

Finally I got off the phone with my mom, and called the sales rep on his cell phone. I explained that I’d felt pressured into buying a furnace yesterday, and after having time to think, I wanted time to get a second opinion. He seemed OK with this, but reiterated how I was wasting his whole day and blowing him off. I explained that I’d cancelled the appointment, he claimed to know nothing about this, and he asked why I didn’t tell him directly.

Well, I’d never met this guy. He was scheduled to come out the day before, but how am I to know the company is sending the same rep? I spoke to him once on the phone and the conversation was about him being late. I didn’t realize he had personal attachment to this sale, companies send out different people for different jobs all the time, so I notified the company properly through their phone system and speaking to a representative. The fact this guy seemed to think I should feel obligated to tell him personally, and almost offended that I didn’t, felt off to me.

Was this just an attempt to manipulate me with guilt? Was the appointment cancellation genuinely not communicated to him by his company?

Either way, I decided this is not my problem, and this is not a company I want to be doing business with. At worst, maybe they attempted to manipulate me into a purchase I didn’t need. At best, maybe their internal communication between departments sucks. Regardless, why do I want to be involved?

The sales rep and I spent about 15 minutes talking on the phone, mostly he defended his company, asserting that they do things by the book and appealing to authority a little bit regarding how they are affiliated with Home Depot. We did leave things on good terms, at least I felt so, and he seemed to be genuinely surprised that I’d felt the way I did. I was quite up front with him about how I felt coerced and pressured.

A Real Repair

Later in the day, the alternative contractor we hired arrived. I did not tell them directly any details about the previous diagnosis, but they did know that another company had investigated and we were seeking a second opinion.

Before he arrived, I removed the “DO NOT OPERATE” sticker that I’d been coerced to have stuck on there. I wanted to let this tech make his own, unbiased, assessments.

He opened up the furnace, and I went through the same details with him that I did with the ARS rep, about what error code I was getting, the fact the furnace was blowing cold air, and questioning if a sensor could be bad.

Within 5 minutes, he had a jumper on the sensor, and the furnace ignited! We had heat blowing out the ducts.

After his investigation, he informed me that the error code was caused by a bad sensor, and that the sensor may have overheated due to restricted air flow. He believed that the filters I was using were too thick, and suggested using a filter with a lower MERS rating to prevent future sensor burn outs.

This company quoted me a total of under $300 to correct the problem. Of course I jumped on this, and the problem was fixed in under an hour.

As an added bonus, he told me that he could increase my blower motor speed, and that this might offset the thicker filter and prevent a future sensor burnout. I had absolutely no idea that this was something you could do! My air pressure on the second floor has always been lackluster, so I jumped on this offer. Now my air pressure throughout the house is remarkable. I look forward to seeing if this helps the A\C in the summer time.

After he was done repairing the problem, he seemed to do other tasks related to checking sensors and components of the furnace. He spent quite awhile checking things and I felt he was quite thorough. He spent a lot more time than ARS did before they declared the furnace a lost cause.

After he was finished, I asked him what his assessment was of the lifespan. As a 12 year old unit with an expected lifespan of 15-20 years, it’s not a young furnace. He said it looked like things were mostly in good shape, and he’d expect it to last closer to 20 years (so, I might get another 8 years out of it!).

I waited until after the transaction was over and I’d paid to ask him about the heat exchanger. ARS had made me so terrified to even say anything about it, because they told me that, in a nutshell, no legitimate company would repair a furnace with this defect and any company that would was risking serious liability.

He told me that if the heat exchanger were cracked, we should see flame rolling, or see carbon monoxide leaking. He observed neither of these things, and had measured the carbon monoxide himself during the diagnosis.

I told him all about my experience with ARS, and if anything, he spent most of the time rolling his eyes. I told him what they said about liability, and that I’d never find anyone legitimate to fix it. He seemed almost in shock at this.

The Takeaway

For me personally, ARS is not the type of company I want to do further business with, and I will not be calling them again. I will be calling the contractor who did fix my problem for my future needs.

So what’s the deal, is ARS scamming people with heat exchanger ghost stories? There seem to be a lot of reviews by other ARS customers mentioning bad heat exchangers. But, these customers seem mostly satisfied, having the tone of “maybe ARS caught a serious problem I didn’t know about”.

I won’t make a hard claim one way or another. Perhaps there might be a hairline crack in my heat exchanger. But if it is not leaking, is it a serious problem? My furnace is up and running, and a competent HVAC technician gave it a clean bill of health.

I can understand that ARS may be technically correct, while also being morally in the wrong. I feel that they exaggerated a minor, age and wear related, symptom that is common in furnaces and blew it up into a major life or death situation in order to use the cold winter temperatures to force a quick sale.

I also feel that they must do this a lot… not just because of the reviews we read, but also because of how readily the tech whipped out these forms and went through his script.

I think about how someone with less knowledge and experience might react to these tactics. The previous owner of my house was an elderly lady, and I wonder if the furnace that is now in my house was installed as a result of this kind of sales tactic. I hope she was treated fairly by them in 2009.

Could this have been one bad apple, one bad technician? It seems unlikely, because my sales rep, while defending the technician, expressed to me that he is one of their senior technicians, and that he trained the technician personally. So, if their senior technicians are operating this way, it’s likely they are training their subordinates to do so as well.

Conclusion

ARS got $95 of my money for a service call where they did nothing but show up, did not disclose to me what the actual problem was (at no point during that visit did the tech mention the bad sensor), and I feel used scare tactics to try to sell me a new furnace which I clearly do not need.

Still, I saved big by not falling into the trap.

I decided to use my extra money I saved by not buying a furnace to buy a new carbon monoxide detector which I will install in the basement, just in case. My existing detectors were over 5 years old.

I feel much better about this solution than trying to do an emergency replacement of the entire system in the middle of winter.

If you believe the heat exchanger fear hype, perhaps I’m now at a higher risk of a carbon monoxide leak? But, this has caused me to ramp up my protection by buying a new detector, so I feel that I am now MORE safe and LESS likely to be poisoned by carbon monoxide. Any appliance that burns fuel can spring a leak, “hairline crack” or not. Monitoring is the key to not succumbing to carbon monoxide poisoning.

I feel that the other contractor I hired was competent, fixed the problem correctly, and did a thorough inspection of the furnace. My furnace now operates better than before it broke, thanks to the increased air pressure, and the problem is fixed for much less money than ARS quoted me, if I had elected to repair with them.

Ultimately, I think everyone’s takeaway here should be, if you get an astronomical quote from any appliance repair company or if you feel that you are being pressured into a sale, second guess them! Get another opinion! You may come out very far ahead.

10Gbps Home Network Tour Video Follow-up!

Recently, I was contacted by Lon Seidman from Lon.TV on YouTube about my home network and my Gigabit Pro setup. He found me through my article here on Binary Impulse: What It Was Like Getting Comcast Gigabit Pro.

Interviewing with Lon was quite a fun experience, and I enjoyed the opportunity to both show off my setup and talk shop with him, as a prospective Gigabit Pro customer himself who is just at the start of his journey.

Not too long ago, I posted an article entitled 10Gbps Home Network Tour, detailing my setup behind the scenes and going into detail about the equipment I am running on my home network, in my home lab, and how I am taking advantage of Comcast’s multi-gigabit Internet service.

The video with Lon is a great follow-up to this post, as I got to show on video and talk about aspects of my setup that would have been too in-depth for the scope of that post, so I wanted to post it here for anyone who might stumble on my blog and maybe did not see the YouTube video. If you enjoyed my network tour article, are curious about Gigabit Pro, or just like looking at other people’s networks, then this is well worth a watch!

10Gbps Home Network Tour

If you are following my articles here, you may have heard that I recently got Comcast Gigabit Pro and now have 2Gbps Internet at home.

Many have asked me how I am accomplishing this internally and what approach I have taken to getting multi-gigabit networking up and running in my house, so without further ado…

My Edge Network Cabinet

Edge Network Cabinet

This cabinet is where my Internet connection enters the house from Comcast. The top two rack U’s are Comcast-owned equipment.

  • Fiber Patch Panel – Where the Comcast fiber terminates inside my house.
  • Juniper ACX2100 – Comcast’s fiber CPE, acting as just a switch to pass the handoff to me.

My equipment in here consists of:

  • A PDU for nice fancy cable management. (UPS that powers this rack is located elsewhere).
  • Ubiquiti EdgeRouter Infinity (ER-8-XG).

The EdgeRouter Infinity has 4 ports in use. I ran two 10G fiber connections to my core networking cabinet, for reasons I will explain later. There is one 10G fiber connection to the Juniper and one 1G ethernet connection to the Juniper.

This is because the Comcast Gigabit Pro service provides two circuits, a 2Gbps fiber and a 1Gbps ethernet with different IPs. The EdgeRouter Infinity is my termination point for both of these connections and it holds both public IPs.

I use Policy Based Routing to direct traffic to the desired connection. Currently, I route only my DNS resolvers, VoIP, and Twitch RTMP (broadcasting) out my single gigabit ethernet connection. I also have an off-site video recording solution for my security cameras which I have routed out this connection. This way those critical, and fairly “consistent” usage, services will not be degraded by surges of traffic on the fiber line. This is completely unnecessary but it seemed like a good use of the extra capacity to me. At any time I can add an IP to my NAT group that I set up and route it out the connection of my choice.

My Core Network Cabinet

Core Residential Network Cabinet

My core network cabinet is the next step, and main distribution panel for Internet and network connectivity in my house.

It contains the following equipment:

  • 2 Obihai Obi200 VoIP adapters for my two Google Voice lines.
  • Raspberry Pi for Bonjour relay between VLANs. (Helps the Chromecasts work as expected across my multiple VLANs.)
  • 3 CAT6 patch panels for ethernet distribution. The cables on the other sides of these panels go to wall jacks and installed mounted equipment throughout my house.
  • My old EdgeRouter ERPro-8 that I haven’t deracked yet, I used to use this as my core router for my cable Internet lines.
  • 48 Port Ubiquiti EdgeSwitch (ES-48-Lite), for 1Gbps ethernet distribution to general purpose wall jacks.
  • 24 Port Ubiquiti EdgeSwitch (ES24-500W), for 1Gbps POE ethernet distribution to IP cameras and wireless access points.
  • Several small project computers (similar to NUCs) for seasonal projects.
  • Ubiquiti EdgeSwitch 16-XG (ES-16-XG), for 10Gbps network distribution.
  • PDU for fancy cable management.
  • UPS that powers both this and the Edge cabinet next to it.

The bread and butter in this cabinet is the ES-16-XG which acts as a core 10Gbps distribution switch for the rest of my network.

While I do have two separate physical networks for server rack and residential network, separated by VLANs, there is a 2x10Gbps fiber connection going to my server rack from the ES-16-XG, which provides inter-vlan connectivity between my servers and the rest of my network.

Additionally, there is a 2x10Gbps uplink to the 48 port switch. The POE switch does not support 10G, so it has a 2x1Gbps uplink to the 48 port switch. This means the theoretical maximum throughput on my WiFi would be 2Gbps, but that is also the maximum speed of the fiber drop that I am using as my primary Internet connection, so this is acceptable to me.

10Gbps to My Computers

The remaining 10G ports on the ES-16-XG can be used for 10Gbps network connectivity of other devices. I have a total of 3 computers on the second floor of my house that are connected with 10Gbps currently.

  • My main Linux workstation.
  • My gaming PC.
  • My DJ studio computer where I broadcast to Twitch.

This can be accomplished by using the 4 RJ45 ports that come on the ES-16-XG, but I didn’t want to be limited to 4 ports, so I also am using SFP-10G-T transceivers from FiberStore, they are currently selling for $59 new.

I haven’t really encountered any significant difference between this and the onboard 10G ethernet ports. I plan to continue to use these to expand the 10G connectivity as needed.

I have no problems running 10G over my existing CAT6 wiring in my house that I ran 4 years ago. I am able to get about 9.6Gbps on iperf from the second floor computers to the server rack in the basement.

To accomplish this on the computers upstairs, I am using Intel X540-T2 network cards I bought on eBay, they are currently going for about $130 used.

So, it is somewhat costly to get 10G to a computer upstairs via ethernet, but within my price range and working just as reliably as standard 1Gbps ethernet for me so far.

10Gbps to My Servers

For my server rack, I am using a Cisco Nexus 3064-X (N3K-C3064PQ-10GX) as the core switch.

Nexus in my server rack – and a few other spare\unused switches.

This was obtained on eBay for under $400, and actually was cheaper than the EdgeSwitch (because I bought it used), but not ideal for a core residential switch because it is to deep for my wall mount network cabinet. For the server rack though, it works great and provides 48 10G ports (and 4x 40G ports).

As mentioned earlier, it has 2x10G fiber uplinks to my residential network, and also has its own 10G uplink to the EdgeRouter. These are all fiber links.

For the servers themselves, I use direct attach copper cables like this. This seems to be the cheapest way to do 10G over short distance and was ideal for my single server cabinet.

In a number of my servers, I am using Dell Mellanox CX332A cards, but I did run into some motherboard compatibility issues and also have two servers that are using Intel X520-DA2 cards, because for some reason even after BIOS updates and other troubleshooting I could not get the Mellanox cards working in all of my systems.

I do not notice any difference between the different 10G cards or media… DAC, ethernet, fiber – it has not made any difference that I have noticed and all performs well.

My server rack contains several storage servers which I have connected two DAC cables (2x10G) and bonded them.

For my other servers, hypervisors\compute, I have 2x10G as well but I have it set up as “public” and “private”, where the public port has a routable IP that can lead to the Internet, and the private port has a non-routable IP on a separate VLAN that is used for inter-server communications, mostly accessing NFS mounts on the storage servers.

In practice 10G is well beyond what I need, so this is still over-built for my purposes. Prior to upgrading my Internet, I was doing the same with 1G ethernet, and actually did not even have the storage servers bonded.

As you might imagine, my growing rack of server equipment consumes some serious power, so to that end, while I was having some electrical work recently I had a dedicated 120V 20A circuit installed for the rack. I also had a second one installed for future expansion.

What Am I Doing With All This Capacity?

There is no doubt that I still have much more bandwidth and capacity than I could ever use — and I am a heavy power user on a good day.

Having this kind of network and connectivity to the Internet has allowed me to free myself from any worry of excessive bandwidth usage. I no longer need ask if I can attempt a project due to bandwidth limitations.

The greatest limitation to me now tends to be external services. VPNs, servers, and connections outside of my control. Maxing out my Internet is basically impossible, even when downloading games from Steam (which is probably the application that comes the closest to doing it).

The best thing for me to come out of this connection is the freedom to use it however I want without problems. I need not worry about backup scripts running while I’m gaming, or Steam downloads choking out my Twitch streams.

My peak usage amounts that actually show up on my graphs are typically still under 1Gbps, and in many cases under 100Mbps, but the ability to do anything anytime without fear of choking due to node congestion or RF interference of cable service is very freeing.

The service is expensive, but for me it’s money well spent.

Building a More Complete & Full Featured CKEditor5

CKEditor 5 is a WYSIWYG text editor that can be used for a variety of purposes, from creating your own Google Docs type of site to creating your own WordPress knock-off platform.

I recently wanted to use this editor, but was disappointed with the lack of features in the default builds. It doesn’t even have underlining. Evidently the official stance of the developers is that you should build your own and they won’t provide a full build anymore like they did for CKEditor 4.

CKEditor provides some detailed build instructions on their documentation, but as someone who is not a Javascript developer and has never used Node before, I found the process a bit intimidating at first. So I am writing this guide up for my own future reference when I need to update my build, and also to hopefully help someone else who is in the same situation by explaining what I feel isn’t well explained to someone who’s completely new to NPM.

The Basics – Starting Your Build

First you will need NPM installed as well as Yarn. For me on Debian 10 the package name for Yarn was not immediately intuitive and the command was different than just “yarn”.

On Debian 10 my dependencies were:
# apt-get install npm yarnpkg git

Once these are installed you can simply clone the Git repository. I didn’t plan on keeping this server long term so I’m just doing it the lazy way on to a temporary VM I am going to delete when I have my final build. This isn’t the best way if you are developing your own stuff, but if you are like me and you just want a build that CKEditor won’t provide, you can just use a temp environment on a throwaway VM like I did. I had no desire to junk up my live server or even my desktop with all this NPM stuff I will not likely use again anytime soon.

For the purposes of this project I am starting with a “Classic” editor as that’s closest to what I want.

# git clone -b stable https://github.com/ckeditor/ckeditor5-build-classic.git
# cd ckeditor5-build-classic
# git remote add upstream https://github.com/ckeditor/ckeditor5-build-classic.git

Finding The Plugins You Want

The plugins you want can all be located on this page of the official documentation. I simply went through each option on the sidebar to see which plugins I might want.

Some of these are already included in the build, which you can discern by reading the build file located in src/ckeditor.js on your VM.

For the purposes of my build, I am adding Alignment, Strikethrough, Underline, Subscript, Superscript, Code, Highlight, HorizontalLine, RemoveFormat, Base64UploadAdapter, and ImageResize.

Installing Plugins

To install your desired plugin, there are 3 steps.

  1. Install the NPM package. The directions for this are provided on the plugin page on the documentation. Here are the NPM installs I ran to install the plugins I wanted:

# npm install --save @ckeditor/ckeditor5-alignment
# npm install --save @ckeditor/ckeditor5-highlight
# npm install --save @ckeditor/ckeditor5-horizontal-line
# npm install --save @ckeditor/ckeditor5-remove-format
# npm install --save @ckeditor/ckeditor5-upload

At a glance, you might notice these plugins do not match what I stated I wanted to add above. This is because you may see that some plugins contain several features, not all of which must be imported. For example, Base64UploadAdapter is one feature of ckeditor5-upload, there are other features which I haven’t imported such as SimpleUploadAdapter.

You can discern which features are part of which plugin from the plugin’s documentation page. Each one has a link to a page containing more information about the feature.

  1. Edit src/ckeditor.js to contain an import line for each plugin feature that you wish to import. For the purposes of my build, I added these import lines below the default ones.

import Alignment from '@ckeditor/ckeditor5-alignment/src/alignment'
import Strikethrough from '@ckeditor/ckeditor5-basic-styles/src/strikethrough';
import Underline from '@ckeditor/ckeditor5-basic-styles/src/underline';
import Subscript from '@ckeditor/ckeditor5-basic-styles/src/subscript';
import Superscript from '@ckeditor/ckeditor5-basic-styles/src/superscript';
import Code from '@ckeditor/ckeditor5-basic-styles/src/code';
import Highlight from '@ckeditor/ckeditor5-highlight/src/highlight';
import HorizontalLine from '@ckeditor/ckeditor5-horizontal-line/src/horizontalline';
import RemoveFormat from '@ckeditor/ckeditor5-remove-format/src/removeformat';
import Base64UploadAdapter from '@ckeditor/ckeditor5-upload/src/adapters/base64uploadadapter';
import ImageResize from '@ckeditor/ckeditor5-image/src/imageresize';

As a newbie to NPM, I wasn’t 100% sure how to determine exactly what should go here at first. Since I for example ran “npm install –save @ckeditor/ckeditor5-upload” how do I determine the remainder of the string to import the feature?

The best way I found is to click through to the GitHub page for the plugin, and navigate into the “src” folder. There, you will see .js files, and you simply need to put the path to the .js file, minus the extension.

  1. Add a line for each plugin in the ClassicEditor.builtinPlugins section of src/ckeditor.js. Unless you do this the plugin won’t actually be built into your build, which afterall is the goal.

Once again, I came across some ambiguity here. Where do the names of the plugins come from and how can I make sure I have the capitazliation correct and everything?

I copied the exact name as displayed on the documentation page for the plugin, it seemed to work correctly 100% of the time.

Some caution is needed here as some plugins contain dashes in the NPM package but the plugin itself doesn’t contain a dash. For example “@ckeditor/ckeditor5-horizontal-line” the plugin name is “HorizontalLine”.

On the Horizontal Line page of documentation, it says “See the Horizontal line feature guide and the HorizontalLine plugin documentation.” This is where I was sourcing my exact spellings and it was working reliably.

Here’s what my additional lines looked like:

        Alignment,
        Strikethrough,
        Underline,
        Subscript,
        Superscript,
        Code,
        Highlight,
        HorizontalLine,
        RemoveFormat,
        Base64UploadAdapter,
        ImageResize,
  1. Add your desired plugin to the “toolbar:” section of the ClassicEditor.defaultConfig in src/ckeditor.js.

Once again some ambiguity here. I used the lowercase version of the plugin name from the section above in step 3. This seemed to work 100% of the time.

By the way, you can use the pipe ‘|’, in the toolbar section to add spacers to the toolbar.

I moved some stuff around so here is what my whole toolbar section ended up looking like:

        toolbar: {
                items: [
                        'heading',
                        'removeformat',
                        'horizontalline',
                        '|',
                        'alignment',
                        'bold',
                        'italic',
                        'underline',
                        '|',
                        'strikethrough',
                        'subscript',
                        'superscript',
                        '|',
                        'link',
                        'bulletedList',
                        'numberedList',
                        '|',
                        'highlight',
                        'indent',
                        'outdent',
                        '|',
                        'imageUpload',
                        'mediaEmbed',
                        '|',
                        'code',
                        'blockQuote',
                        'insertTable',
                        '|',
                        'undo',
                        'redo'
                ]
        },

A Note About Some Dumb (In My Opinion) Defaults

CKEditor comes bundled with a few things which I removed for the purposes of my build.

Firstly it comes bundled with CKFinder and its associated UploadAdapter. Are far as I can tell, this does not function without a server side script I don’t care to invest time investigating, so I’m removing these from my build.

Additionally I am removing EasyImage because I have no plans to use cloud services.

To remove these items, I’m simply commenting the imports out and the plugin declarations in builtinPlugins in my src/ckeditor.js before finishing my build.

Out of the box, the image upload features of the editor do not work unless you put in some elbow grease. For now I am implementing Base64 image uploading so I don’t have to mess with a server side handler and filesystem permissions issues that can come along with uploading files. We’ll see how this works for me in my use case long term, I may switch to the Simple Upload Adapter and write a server side handler in the future.

I personally think CK Editor should just include the Base64 uploader by default so that the features work out of the box instead of this CKFinder plugin that doesn’t work without additional dependencies.

Finishing Your Build

Once you have added all of the customizations to your build, you can compile it with the Yarn tool you installed.

Although the official documentation suggests the command is “yarn”, on my Debian 10 system, it was “yarnpkg”.

So to finish my build I ran:

# yarnpkg build

Once it’s finished, the completed file is located in build/ckeditor.js. This file can be used as a drop-in replacement for any other downloadable build direct from CKEditor, and it should contain your new features.

I found I could run this build over and over as I refined my source file and I didn’t have any problems, it just overwrote my build file with a new one.

There ya go! I hope this guide simplifies someone’s project. 🙂

What It Was Like Getting Comcast Gigabit Pro

As an excited prospective Gigabit Pro customer, my favorite reading to return to has been James Watt’s Medium.com article about his install experience. To that end, I wanted to create a similar post documenting my own install experiences so that you future Gigabit Pro users can learn from my experience and maybe get some of your questions answered.

The focus of this article is not a technical article about how I prepared my network for 10Gbps. I may create another article soon detailing the setup of my network and how I am handling the bandwidth. The purpose of this article is to document my installation experience. So let’s dive in!

Initial Inquiry

My journey starts in mid-April of 2019. I headed over to the /r/Comcast_Xfinity subreddit to inquire on how I could get an order going. They were actually quite helpful over there, and shortly I was receiving a call from a representative from Comcast Sales Operations.

Talking with the sales representative, he was able to look at his network maps and determine that there was a fiber tap very close to my house, actually only about 5 houses down to the main street. Based on this information, he thought I should be eligible for the service.

For those of you who are curious if you might be eligible, check utility poles near you for something that looks like this:

Comcast’s fiber tap near my house.

I went over all of the initial information with sales, pretty much in line with what I had found out during my research. It would basically cost $1,000 to install and $319.99/month thereafter. This was not unexpected, but was a large increase over the roughly $70/month I was paying for my cable line, so I decided to give it some thought before proceeding.

My old WOW cable Internet continued to be on the decline in quality, and it didn’t seem like there was any resolution in sight. This frustration gave me the push I needed, and about a month later in mid-May, I was reaching back out to sales to tell them to get the ball rolling.

Pre-Installation Process

Although I appeared to be close by to the fiber tap on the maps, the first step in the process still needed to be an on site survey. Comcast informed me that they would get someone on the site to make sure I was really eligible, and then we could move forward from there.

It was made clear to me that this was going to be a multiple-month process, and if eligible, we would “try” to get it installed before the winter months came and the ground would become too frozen to dig.

Someone must have come to the site at some point, but no interaction was needed with me. Comcast didn’t call me back directly, but after a few weeks I called to follow up and was told that my initial site survey had been approved and we were moving forward with the installation.

The next step would be waiting for approvals and permits. Evidently the install process for Comcast works something like this internally:

  • Perform site survey to make sure that the building is serviceable.
  • Create a formal installation design \ plan and submit it for approval.
  • Review of the installation plan by finance team to determine if costs are acceptable to move forward.
  • Submit permit requests to applicable local government.
  • Begin construction once permits are received.

The whole installation was rife with misinformation from my Comcast representative who always seemed to be out of the loop and a few steps behind on the process, which makes me wonder who is really in charge of coordinating the tasks. I don’t hold this against him but I do think there is a ton of room for Comcast to improve communication both internally and with their customer during this long process.

A few examples of such confusion \ misinformation throughout the process:

  • Around a month after being told my site survey was complete, I received a call from the construction contractor asking to schedule a time to meet me for a site survey. Apparently this was when he was out to draw up the design.
  • At one point, I was told by Comcast that we were “waiting on permits” from the government and then told by the cable installer contractor that we were waiting on Comcast for financial approval. That turned out to be correct, and once it was approved then the wait for permits began.
  • When construction began, I called up my Comcast rep to inform him of the progress, but couldn’t reach him. The next day he called me to inform me that the permits had come in and we could start construction soon. (Though it already started a day earlier.)

5 Months Later

Most of what I had read online suggested that it was about a 3 month lead time from inquiry to install, but for me this was not correct.

Finally after months of waiting for permits and approval and seemingly making no progress, the contractor called me to inform me that they were scheduling directional boring. From mid-May to the beginning of October, we were finally ready to get some actual work done.

The installation would happen in 4 stages:

  • Directional Boring – The cable contractor installed an orange conduit several feet underground leading from my back yard to the utility pole where the fiber tap is located.
  • Pulling The Cable – The cable contractor would visit to pull the fiber cable from my basement to the utility pole.
  • Splicing – The cable contractor would visit to add the connections on both ends of the cable and plug it in.
  • Setup – Comcast would send out a representative to install the Juniper router and actually bring the connection up.

Directional Boring

The directional boring process began for me on the last day of September. No appointment was necessary, I simply awoke in the afternoon to the sound of machinery running outside. I had missed the voicemail from the contractor so I wasn’t even aware anyone would be coming – it was a really nice surprise.

Most of the work occurred in the utility easement behind my house. They did have to bore into my backyard, but they only dug a single hole next to the house. It doesn’t appear they even brought the equipment into my yard.

Deep Holes for Directional Boring
It was a bit wet that day!
Fiber’s on the way!

This was such a process, they actually had to leave their equipment overnight and finish the next day. Of course, being as excited as I was, I had to go back and take some pictures to document this part too!

Directional boring equipment — not so boring after all!
North American Fiber-Seeking Backhoe
The legendary orange spool – usually reserved only for large infrastructure installs on the side of the highway.

Once they came back the next day, they finished everything up and planted some grass. They left a pull cord in the conduit so that they could return and fish the cable more easily later.

My conduit emerging from the ground near the utility pole.
Re-planting grass over the holes.

Running The Cable

I eagerly waited for the next step to begin, and for me it took about 10 days. I awoke to a few missed calls, and Comcast’s contractor was telling me they had some crews nearby and asking if they could do the installation that day.

I had work in a few hours, but I’d been waiting for this so eagerly for so long, so I made it work. The crew came out and promptly finished the inside part of the job so I could go to work on time

Fiber coiled up in my basement, so close, yet so far away…

When I returned from work, I was thrilled to see that the fiber was hanging from the utility pole. I went out the next morning to take some pictures.

My fiber coiled up on the utility pole, waiting to be connected.

Now, I was just simply waiting for step 3 of the install – to schedule the guy to come out and splice the connection. The Comcast contractor informed me that we were waiting on final schematics from Comcast to proceed.

Almost 2 weeks later, there was still no update. There were some changes on the pole though, someone had moved my fiber around and made this mess that looked like it was ready to fall down any minute!

The fiber moves on the pole.
Hopefully it stays up there!

At this point, my contact at the contractor stopped answering my calls, and his voicemail box was full. I was so eager to continue the install, but even Comcast was not able to update me on what was going on.

Disaster Strikes… Temporarily.

During the wait, we had a weekend where we got very large amounts of rain. I was getting ready for bed one Saturday night, and ran down to the basement to quickly grab something. There, I was shocked to find my floor flooded.

Water on the floor! Water under the server rack!
I was glad to have my elevated shelving keeping my old computer stuff off the floor today.

At first, I was just very surprised and confused. The areas where this water was located were not areas where there were any pipes or pumps very close by to account for the reason why there was water there.

It hadn’t even dawned on me right away to check the fiber, until I swore I saw it drip while I was standing there trying to figure out what had happened. Sure enough, I gave it a little shake, and:

Leaky fiber conduit.

Fortunately, after a Sunday afternoon call to the cable contractors, they quickly sent someone out the next day to fix the issue (right before a week with lots of rain, too). Fortunately in the end, nothing of any value was damaged. The contractors did some additional sealing where the fiber comes into my house, as well as at the fiber vault, so we are hoping this issue is now solved.

I should note, my house is located at a very low elevation. The fiber cable is at a much higher elevation than my basement for most of its run, and my backyard is very sloped. This did not occur due to a poorly sealed entry point, they did a nice job sealing around the wall penetration, what actually happened was water entered the fiber conduit up at a higher elevation and then came rushing down the conduit into my house.

With this out of the way, this got me back in contact with the contractor and we immediately got the next part of the process moving!

Splicing The Fiber

Finally, I had some forward motion from the contractor again, and we were able to move forward with the splicing part of the installation at the very end of October.

For this part of the installation, two Comcast techs came out with some expensive equipment and completed the splicing of 6 fiber pairs into a patch panel which they installed in my rack.

Fiber splicing!
Another angle on the fiber splicing.

Once they left, they informed me that they still needed to complete some work on the pole before they would be able to pass the job back to Comcast.

This would take another 9 days, but finally, the contractor’s work was done and Comcast was ready to schedule the last appointment for the installation!

Comcast needed some time to order the equipment, so the installation would be scheduled a little more than a week out, for a final install date in mid-November.

The Historic Day — Installation!

Today started out very early for me. I am a night shifter, and for most of the install process I have had a lot of success scheduling things at 2PM. Unfortunately, for this part of the install, they would only schedule it at 8AM. So, last night, I got to bed as early as I could after work and started the day a bit sleep deprived.

A few minutes before 8AM, I popped out of bed and saw a Comcast truck parked outside. I went outside and talked to the technician, who had arrived with an XFi modem for Gigabit cable service. I explained to him that this was a FTTH service and we needed different equipment. He fortunately understood what I was talking about and put in a call to dispatch. At this point, I also tried to reach my account manager via phone, but apparently he hadn’t started his day yet. After finishing his call to dispatch, the tech informed me that someone else would be en route.

I called the 1-800-XFINITY hotline and spoke to their appointment managers to confirm that someone else was on the way. It seemed like everything was being handled correctly. I got off the phone and 30 minutes later the correct technicians with the Juniper equipment arrived!

The install started out pretty smoothly, the Juniper ACX2100 was racked up and powered on and fiber connected to the patch panel. Unfortunately, the head-end had installed the incorrect optics. There was a 1Gbps SFP at the head-end and my 10Gbps SFP at my Juniper would not negotiate a connection (plus the connection is supposed to be above 1Gbps). The install technician was very patient and stuck around for several hours while making phone calls and trying to get the head end issue resolved.

Eventually, after someone was finally dispatched to the head-end, they did not have the correct SFP. I almost thought my installation would be off for the day, but as luck struck, the technician on site at my house found that he did have an extra correct module. So he drove off to the head-end to install it himself.

Upon his return, we finally were getting somewhere. He spent several more hours on the phone with the Metro-E team performing circuit testing to ensure that everything was up to quality specs.

In total, the install which we expected to take 4 hours took just over 6 hours. I was glad that we scheduled it so early in the day, or it might not have even gotten done in a single day.

I really do have to commend the install technician for his persistence and I feel he went way above and beyond the call of duty to make sure my connection got online today. I plan to leave him an outstanding review when his project manager calls to take my feedback survey, which my account manager told me to expect.

The completed install at the pole… I do hope they plan to come back and clean this up more and it doesn’t fall down in the meantime.
The finished install in my network cabinet, and the Juniper in all its glory!

So now that it’s installed, how is it?

I don’t think reality has even quite sunk in for me yet, but from my experience so far, I feel that I now have the best Internet connection money can buy!

How can you beat a stable 2.5ms ping to Google?!

This next part seems to be up for dispute on a lot of Reddit\Internet posts, so let me clear this up for anyone who wants to know.

Yes! Gigabit Pro comes with two circuits. A fiber optic handoff provisioned for 2Gbps/2Gbps and an ethernet handoff provisioned for 1Gbps/1Gbps. Each circuit gets a public IPv4 and a /48 block of IPv6.

Here are my current speed test results on each independent circuit:

Fiber circuit speed test, 2Gbps symmetrical.
Copper circuit speed test, 1Gbps symmetrical.

How am I going to use all of this bandwidth? Well, I consider the copper circuit to be somewhat of a bonus, so I am planning on using Policy Based Routing (PBR) on my Ubiquiti EdgeRouter Infinity to route very quality sensitive traffic, such as VoIP phone traffic and my Twitch RTMP broadcasts, out the ethernet connection. This way it will not be impacted by bursty traffic on my fiber connection and will provide the best overall quality and reliability. I was doing something similar already with my existing dual cable modem dual ISP setup that I had created temporarily while waiting for the fiber.

For those wondering, my preliminary testing shows that it is in fact possible to use all 3Gbps at once:

IPerf Testing

Overall I am extremely pleased with the service so far and I look forward to being a Comcast fiber customer for years to come! 🙂