10Gbps Home Network Tour

If you are following my articles here, you may have heard that I recently got Comcast Gigabit Pro and now have 2Gbps Internet at home.

Many have asked me how I am accomplishing this internally and what approach I have taken to getting multi-gigabit networking up and running in my house, so without further ado…

My Edge Network Cabinet

Edge Network Cabinet

This cabinet is where my Internet connection enters the house from Comcast. The top two rack U’s are Comcast-owned equipment.

  • Fiber Patch Panel – Where the Comcast fiber terminates inside my house.
  • Juniper ACX2100 – Comcast’s fiber CPE, acting as just a switch to pass the handoff to me.

My equipment in here consists of:

  • A PDU for nice fancy cable management. (UPS that powers this rack is located elsewhere).
  • Ubiquiti EdgeRouter Infinity (ER-8-XG).

The EdgeRouter Infinity has 4 ports in use. I ran two 10G fiber connections to my core networking cabinet, for reasons I will explain later. There is one 10G fiber connection to the Juniper and one 1G ethernet connection to the Juniper.

This is because the Comcast Gigabit Pro service provides two circuits, a 2Gbps fiber and a 1Gbps ethernet with different IPs. The EdgeRouter Infinity is my termination point for both of these connections and it holds both public IPs.

I use Policy Based Routing to direct traffic to the desired connection. Currently, I route only my DNS resolvers, VoIP, and Twitch RTMP (broadcasting) out my single gigabit ethernet connection. I also have an off-site video recording solution for my security cameras which I have routed out this connection. This way those critical, and fairly “consistent” usage, services will not be degraded by surges of traffic on the fiber line. This is completely unnecessary but it seemed like a good use of the extra capacity to me. At any time I can add an IP to my NAT group that I set up and route it out the connection of my choice.

My Core Network Cabinet

Core Residential Network Cabinet

My core network cabinet is the next step, and main distribution panel for Internet and network connectivity in my house.

It contains the following equipment:

  • 2 Obihai Obi200 VoIP adapters for my two Google Voice lines.
  • Raspberry Pi for Bonjour relay between VLANs. (Helps the Chromecasts work as expected across my multiple VLANs.)
  • 3 CAT6 patch panels for ethernet distribution. The cables on the other sides of these panels go to wall jacks and installed mounted equipment throughout my house.
  • My old EdgeRouter ERPro-8 that I haven’t deracked yet, I used to use this as my core router for my cable Internet lines.
  • 48 Port Ubiquiti EdgeSwitch (ES-48-Lite), for 1Gbps ethernet distribution to general purpose wall jacks.
  • 24 Port Ubiquiti EdgeSwitch (ES24-500W), for 1Gbps POE ethernet distribution to IP cameras and wireless access points.
  • Several small project computers (similar to NUCs) for seasonal projects.
  • Ubiquiti EdgeSwitch 16-XG (ES-16-XG), for 10Gbps network distribution.
  • PDU for fancy cable management.
  • UPS that powers both this and the Edge cabinet next to it.

The bread and butter in this cabinet is the ES-16-XG which acts as a core 10Gbps distribution switch for the rest of my network.

While I do have two separate physical networks for server rack and residential network, separated by VLANs, there is a 2x10Gbps fiber connection going to my server rack from the ES-16-XG, which provides inter-vlan connectivity between my servers and the rest of my network.

Additionally, there is a 2x10Gbps uplink to the 48 port switch. The POE switch does not support 10G, so it has a 2x1Gbps uplink to the 48 port switch. This means the theoretical maximum throughput on my WiFi would be 2Gbps, but that is also the maximum speed of the fiber drop that I am using as my primary Internet connection, so this is acceptable to me.

10Gbps to My Computers

The remaining 10G ports on the ES-16-XG can be used for 10Gbps network connectivity of other devices. I have a total of 3 computers on the second floor of my house that are connected with 10Gbps currently.

  • My main Linux workstation.
  • My gaming PC.
  • My DJ studio computer where I broadcast to Twitch.

This can be accomplished by using the 4 RJ45 ports that come on the ES-16-XG, but I didn’t want to be limited to 4 ports, so I also am using SFP-10G-T transceivers from FiberStore, they are currently selling for $59 new.

I haven’t really encountered any significant difference between this and the onboard 10G ethernet ports. I plan to continue to use these to expand the 10G connectivity as needed.

I have no problems running 10G over my existing CAT6 wiring in my house that I ran 4 years ago. I am able to get about 9.6Gbps on iperf from the second floor computers to the server rack in the basement.

To accomplish this on the computers upstairs, I am using Intel X540-T2 network cards I bought on eBay, they are currently going for about $130 used.

So, it is somewhat costly to get 10G to a computer upstairs via ethernet, but within my price range and working just as reliably as standard 1Gbps ethernet for me so far.

10Gbps to My Servers

For my server rack, I am using a Cisco Nexus 3064-X (N3K-C3064PQ-10GX) as the core switch.

Nexus in my server rack – and a few other spare\unused switches.

This was obtained on eBay for under $400, and actually was cheaper than the EdgeSwitch (because I bought it used), but not ideal for a core residential switch because it is to deep for my wall mount network cabinet. For the server rack though, it works great and provides 48 10G ports (and 4x 40G ports).

As mentioned earlier, it has 2x10G fiber uplinks to my residential network, and also has its own 10G uplink to the EdgeRouter. These are all fiber links.

For the servers themselves, I use direct attach copper cables like this. This seems to be the cheapest way to do 10G over short distance and was ideal for my single server cabinet.

In a number of my servers, I am using Dell Mellanox CX332A cards, but I did run into some motherboard compatibility issues and also have two servers that are using Intel X520-DA2 cards, because for some reason even after BIOS updates and other troubleshooting I could not get the Mellanox cards working in all of my systems.

I do not notice any difference between the different 10G cards or media… DAC, ethernet, fiber – it has not made any difference that I have noticed and all performs well.

My server rack contains several storage servers which I have connected two DAC cables (2x10G) and bonded them.

For my other servers, hypervisors\compute, I have 2x10G as well but I have it set up as “public” and “private”, where the public port has a routable IP that can lead to the Internet, and the private port has a non-routable IP on a separate VLAN that is used for inter-server communications, mostly accessing NFS mounts on the storage servers.

In practice 10G is well beyond what I need, so this is still over-built for my purposes. Prior to upgrading my Internet, I was doing the same with 1G ethernet, and actually did not even have the storage servers bonded.

As you might imagine, my growing rack of server equipment consumes some serious power, so to that end, while I was having some electrical work recently I had a dedicated 120V 20A circuit installed for the rack. I also had a second one installed for future expansion.

What Am I Doing With All This Capacity?

There is no doubt that I still have much more bandwidth and capacity than I could ever use — and I am a heavy power user on a good day.

Having this kind of network and connectivity to the Internet has allowed me to free myself from any worry of excessive bandwidth usage. I no longer need ask if I can attempt a project due to bandwidth limitations.

The greatest limitation to me now tends to be external services. VPNs, servers, and connections outside of my control. Maxing out my Internet is basically impossible, even when downloading games from Steam (which is probably the application that comes the closest to doing it).

The best thing for me to come out of this connection is the freedom to use it however I want without problems. I need not worry about backup scripts running while I’m gaming, or Steam downloads choking out my Twitch streams.

My peak usage amounts that actually show up on my graphs are typically still under 1Gbps, and in many cases under 100Mbps, but the ability to do anything anytime without fear of choking due to node congestion or RF interference of cable service is very freeing.

The service is expensive, but for me it’s money well spent.

Building a More Complete & Full Featured CKEditor5

CKEditor 5 is a WYSIWYG text editor that can be used for a variety of purposes, from creating your own Google Docs type of site to creating your own WordPress knock-off platform.

I recently wanted to use this editor, but was disappointed with the lack of features in the default builds. It doesn’t even have underlining. Evidently the official stance of the developers is that you should build your own and they won’t provide a full build anymore like they did for CKEditor 4.

CKEditor provides some detailed build instructions on their documentation, but as someone who is not a Javascript developer and has never used Node before, I found the process a bit intimidating at first. So I am writing this guide up for my own future reference when I need to update my build, and also to hopefully help someone else who is in the same situation by explaining what I feel isn’t well explained to someone who’s completely new to NPM.

The Basics – Starting Your Build

First you will need NPM installed as well as Yarn. For me on Debian 10 the package name for Yarn was not immediately intuitive and the command was different than just “yarn”.

On Debian 10 my dependencies were:
# apt-get install npm yarnpkg git

Once these are installed you can simply clone the Git repository. I didn’t plan on keeping this server long term so I’m just doing it the lazy way on to a temporary VM I am going to delete when I have my final build. This isn’t the best way if you are developing your own stuff, but if you are like me and you just want a build that CKEditor won’t provide, you can just use a temp environment on a throwaway VM like I did. I had no desire to junk up my live server or even my desktop with all this NPM stuff I will not likely use again anytime soon.

For the purposes of this project I am starting with a “Classic” editor as that’s closest to what I want.

# git clone -b stable https://github.com/ckeditor/ckeditor5-build-classic.git
# cd ckeditor5-build-classic
# git remote add upstream https://github.com/ckeditor/ckeditor5-build-classic.git

Finding The Plugins You Want

The plugins you want can all be located on this page of the official documentation. I simply went through each option on the sidebar to see which plugins I might want.

Some of these are already included in the build, which you can discern by reading the build file located in src/ckeditor.js on your VM.

For the purposes of my build, I am adding Alignment, Strikethrough, Underline, Subscript, Superscript, Code, Highlight, HorizontalLine, RemoveFormat, Base64UploadAdapter, and ImageResize.

Installing Plugins

To install your desired plugin, there are 3 steps.

  1. Install the NPM package. The directions for this are provided on the plugin page on the documentation. Here are the NPM installs I ran to install the plugins I wanted:

# npm install --save @ckeditor/ckeditor5-alignment
# npm install --save @ckeditor/ckeditor5-highlight
# npm install --save @ckeditor/ckeditor5-horizontal-line
# npm install --save @ckeditor/ckeditor5-remove-format
# npm install --save @ckeditor/ckeditor5-upload

At a glance, you might notice these plugins do not match what I stated I wanted to add above. This is because you may see that some plugins contain several features, not all of which must be imported. For example, Base64UploadAdapter is one feature of ckeditor5-upload, there are other features which I haven’t imported such as SimpleUploadAdapter.

You can discern which features are part of which plugin from the plugin’s documentation page. Each one has a link to a page containing more information about the feature.

  1. Edit src/ckeditor.js to contain an import line for each plugin feature that you wish to import. For the purposes of my build, I added these import lines below the default ones.

import Alignment from '@ckeditor/ckeditor5-alignment/src/alignment'
import Strikethrough from '@ckeditor/ckeditor5-basic-styles/src/strikethrough';
import Underline from '@ckeditor/ckeditor5-basic-styles/src/underline';
import Subscript from '@ckeditor/ckeditor5-basic-styles/src/subscript';
import Superscript from '@ckeditor/ckeditor5-basic-styles/src/superscript';
import Code from '@ckeditor/ckeditor5-basic-styles/src/code';
import Highlight from '@ckeditor/ckeditor5-highlight/src/highlight';
import HorizontalLine from '@ckeditor/ckeditor5-horizontal-line/src/horizontalline';
import RemoveFormat from '@ckeditor/ckeditor5-remove-format/src/removeformat';
import Base64UploadAdapter from '@ckeditor/ckeditor5-upload/src/adapters/base64uploadadapter';
import ImageResize from '@ckeditor/ckeditor5-image/src/imageresize';

As a newbie to NPM, I wasn’t 100% sure how to determine exactly what should go here at first. Since I for example ran “npm install –save @ckeditor/ckeditor5-upload” how do I determine the remainder of the string to import the feature?

The best way I found is to click through to the GitHub page for the plugin, and navigate into the “src” folder. There, you will see .js files, and you simply need to put the path to the .js file, minus the extension.

  1. Add a line for each plugin in the ClassicEditor.builtinPlugins section of src/ckeditor.js. Unless you do this the plugin won’t actually be built into your build, which afterall is the goal.

Once again, I came across some ambiguity here. Where do the names of the plugins come from and how can I make sure I have the capitazliation correct and everything?

I copied the exact name as displayed on the documentation page for the plugin, it seemed to work correctly 100% of the time.

Some caution is needed here as some plugins contain dashes in the NPM package but the plugin itself doesn’t contain a dash. For example “@ckeditor/ckeditor5-horizontal-line” the plugin name is “HorizontalLine”.

On the Horizontal Line page of documentation, it says “See the Horizontal line feature guide and the HorizontalLine plugin documentation.” This is where I was sourcing my exact spellings and it was working reliably.

Here’s what my additional lines looked like:

  1. Add your desired plugin to the “toolbar:” section of the ClassicEditor.defaultConfig in src/ckeditor.js.

Once again some ambiguity here. I used the lowercase version of the plugin name from the section above in step 3. This seemed to work 100% of the time.

By the way, you can use the pipe ‘|’, in the toolbar section to add spacers to the toolbar.

I moved some stuff around so here is what my whole toolbar section ended up looking like:

        toolbar: {
                items: [

A Note About Some Dumb (In My Opinion) Defaults

CKEditor comes bundled with a few things which I removed for the purposes of my build.

Firstly it comes bundled with CKFinder and its associated UploadAdapter. Are far as I can tell, this does not function without a server side script I don’t care to invest time investigating, so I’m removing these from my build.

Additionally I am removing EasyImage because I have no plans to use cloud services.

To remove these items, I’m simply commenting the imports out and the plugin declarations in builtinPlugins in my src/ckeditor.js before finishing my build.

Out of the box, the image upload features of the editor do not work unless you put in some elbow grease. For now I am implementing Base64 image uploading so I don’t have to mess with a server side handler and filesystem permissions issues that can come along with uploading files. We’ll see how this works for me in my use case long term, I may switch to the Simple Upload Adapter and write a server side handler in the future.

I personally think CK Editor should just include the Base64 uploader by default so that the features work out of the box instead of this CKFinder plugin that doesn’t work without additional dependencies.

Finishing Your Build

Once you have added all of the customizations to your build, you can compile it with the Yarn tool you installed.

Although the official documentation suggests the command is “yarn”, on my Debian 10 system, it was “yarnpkg”.

So to finish my build I ran:

# yarnpkg build

Once it’s finished, the completed file is located in build/ckeditor.js. This file can be used as a drop-in replacement for any other downloadable build direct from CKEditor, and it should contain your new features.

I found I could run this build over and over as I refined my source file and I didn’t have any problems, it just overwrote my build file with a new one.

There ya go! I hope this guide simplifies someone’s project. 🙂

What It Was Like Getting Comcast Gigabit Pro

As an excited prospective Gigabit Pro customer, my favorite reading to return to has been James Watt’s Medium.com article about his install experience. To that end, I wanted to create a similar post documenting my own install experiences so that you future Gigabit Pro users can learn from my experience and maybe get some of your questions answered.

The focus of this article is not a technical article about how I prepared my network for 10Gbps. I may create another article soon detailing the setup of my network and how I am handling the bandwidth. The purpose of this article is to document my installation experience. So let’s dive in!

Initial Inquiry

My journey starts in mid-April of 2019. I headed over to the /r/Comcast_Xfinity subreddit to inquire on how I could get an order going. They were actually quite helpful over there, and shortly I was receiving a call from a representative from Comcast Sales Operations.

Talking with the sales representative, he was able to look at his network maps and determine that there was a fiber tap very close to my house, actually only about 5 houses down to the main street. Based on this information, he thought I should be eligible for the service.

For those of you who are curious if you might be eligible, check utility poles near you for something that looks like this:

Comcast’s fiber tap near my house.

I went over all of the initial information with sales, pretty much in line with what I had found out during my research. It would basically cost $1,000 to install and $319.99/month thereafter. This was not unexpected, but was a large increase over the roughly $70/month I was paying for my cable line, so I decided to give it some thought before proceeding.

My old WOW cable Internet continued to be on the decline in quality, and it didn’t seem like there was any resolution in sight. This frustration gave me the push I needed, and about a month later in mid-May, I was reaching back out to sales to tell them to get the ball rolling.

Pre-Installation Process

Although I appeared to be close by to the fiber tap on the maps, the first step in the process still needed to be an on site survey. Comcast informed me that they would get someone on the site to make sure I was really eligible, and then we could move forward from there.

It was made clear to me that this was going to be a multiple-month process, and if eligible, we would “try” to get it installed before the winter months came and the ground would become too frozen to dig.

Someone must have come to the site at some point, but no interaction was needed with me. Comcast didn’t call me back directly, but after a few weeks I called to follow up and was told that my initial site survey had been approved and we were moving forward with the installation.

The next step would be waiting for approvals and permits. Evidently the install process for Comcast works something like this internally:

  • Perform site survey to make sure that the building is serviceable.
  • Create a formal installation design \ plan and submit it for approval.
  • Review of the installation plan by finance team to determine if costs are acceptable to move forward.
  • Submit permit requests to applicable local government.
  • Begin construction once permits are received.

The whole installation was rife with misinformation from my Comcast representative who always seemed to be out of the loop and a few steps behind on the process, which makes me wonder who is really in charge of coordinating the tasks. I don’t hold this against him but I do think there is a ton of room for Comcast to improve communication both internally and with their customer during this long process.

A few examples of such confusion \ misinformation throughout the process:

  • Around a month after being told my site survey was complete, I received a call from the construction contractor asking to schedule a time to meet me for a site survey. Apparently this was when he was out to draw up the design.
  • At one point, I was told by Comcast that we were “waiting on permits” from the government and then told by the cable installer contractor that we were waiting on Comcast for financial approval. That turned out to be correct, and once it was approved then the wait for permits began.
  • When construction began, I called up my Comcast rep to inform him of the progress, but couldn’t reach him. The next day he called me to inform me that the permits had come in and we could start construction soon. (Though it already started a day earlier.)

5 Months Later

Most of what I had read online suggested that it was about a 3 month lead time from inquiry to install, but for me this was not correct.

Finally after months of waiting for permits and approval and seemingly making no progress, the contractor called me to inform me that they were scheduling directional boring. From mid-May to the beginning of October, we were finally ready to get some actual work done.

The installation would happen in 4 stages:

  • Directional Boring – The cable contractor installed an orange conduit several feet underground leading from my back yard to the utility pole where the fiber tap is located.
  • Pulling The Cable – The cable contractor would visit to pull the fiber cable from my basement to the utility pole.
  • Splicing – The cable contractor would visit to add the connections on both ends of the cable and plug it in.
  • Setup – Comcast would send out a representative to install the Juniper router and actually bring the connection up.

Directional Boring

The directional boring process began for me on the last day of September. No appointment was necessary, I simply awoke in the afternoon to the sound of machinery running outside. I had missed the voicemail from the contractor so I wasn’t even aware anyone would be coming – it was a really nice surprise.

Most of the work occurred in the utility easement behind my house. They did have to bore into my backyard, but they only dug a single hole next to the house. It doesn’t appear they even brought the equipment into my yard.

Deep Holes for Directional Boring
It was a bit wet that day!
Fiber’s on the way!

This was such a process, they actually had to leave their equipment overnight and finish the next day. Of course, being as excited as I was, I had to go back and take some pictures to document this part too!

Directional boring equipment — not so boring after all!
North American Fiber-Seeking Backhoe
The legendary orange spool – usually reserved only for large infrastructure installs on the side of the highway.

Once they came back the next day, they finished everything up and planted some grass. They left a pull cord in the conduit so that they could return and fish the cable more easily later.

My conduit emerging from the ground near the utility pole.
Re-planting grass over the holes.

Running The Cable

I eagerly waited for the next step to begin, and for me it took about 10 days. I awoke to a few missed calls, and Comcast’s contractor was telling me they had some crews nearby and asking if they could do the installation that day.

I had work in a few hours, but I’d been waiting for this so eagerly for so long, so I made it work. The crew came out and promptly finished the inside part of the job so I could go to work on time

Fiber coiled up in my basement, so close, yet so far away…

When I returned from work, I was thrilled to see that the fiber was hanging from the utility pole. I went out the next morning to take some pictures.

My fiber coiled up on the utility pole, waiting to be connected.

Now, I was just simply waiting for step 3 of the install – to schedule the guy to come out and splice the connection. The Comcast contractor informed me that we were waiting on final schematics from Comcast to proceed.

Almost 2 weeks later, there was still no update. There were some changes on the pole though, someone had moved my fiber around and made this mess that looked like it was ready to fall down any minute!

The fiber moves on the pole.
Hopefully it stays up there!

At this point, my contact at the contractor stopped answering my calls, and his voicemail box was full. I was so eager to continue the install, but even Comcast was not able to update me on what was going on.

Disaster Strikes… Temporarily.

During the wait, we had a weekend where we got very large amounts of rain. I was getting ready for bed one Saturday night, and ran down to the basement to quickly grab something. There, I was shocked to find my floor flooded.

Water on the floor! Water under the server rack!
I was glad to have my elevated shelving keeping my old computer stuff off the floor today.

At first, I was just very surprised and confused. The areas where this water was located were not areas where there were any pipes or pumps very close by to account for the reason why there was water there.

It hadn’t even dawned on me right away to check the fiber, until I swore I saw it drip while I was standing there trying to figure out what had happened. Sure enough, I gave it a little shake, and:

Leaky fiber conduit.

Fortunately, after a Sunday afternoon call to the cable contractors, they quickly sent someone out the next day to fix the issue (right before a week with lots of rain, too). Fortunately in the end, nothing of any value was damaged. The contractors did some additional sealing where the fiber comes into my house, as well as at the fiber vault, so we are hoping this issue is now solved.

I should note, my house is located at a very low elevation. The fiber cable is at a much higher elevation than my basement for most of its run, and my backyard is very sloped. This did not occur due to a poorly sealed entry point, they did a nice job sealing around the wall penetration, what actually happened was water entered the fiber conduit up at a higher elevation and then came rushing down the conduit into my house.

With this out of the way, this got me back in contact with the contractor and we immediately got the next part of the process moving!

Splicing The Fiber

Finally, I had some forward motion from the contractor again, and we were able to move forward with the splicing part of the installation at the very end of October.

For this part of the installation, two Comcast techs came out with some expensive equipment and completed the splicing of 6 fiber pairs into a patch panel which they installed in my rack.

Fiber splicing!
Another angle on the fiber splicing.

Once they left, they informed me that they still needed to complete some work on the pole before they would be able to pass the job back to Comcast.

This would take another 9 days, but finally, the contractor’s work was done and Comcast was ready to schedule the last appointment for the installation!

Comcast needed some time to order the equipment, so the installation would be scheduled a little more than a week out, for a final install date in mid-November.

The Historic Day — Installation!

Today started out very early for me. I am a night shifter, and for most of the install process I have had a lot of success scheduling things at 2PM. Unfortunately, for this part of the install, they would only schedule it at 8AM. So, last night, I got to bed as early as I could after work and started the day a bit sleep deprived.

A few minutes before 8AM, I popped out of bed and saw a Comcast truck parked outside. I went outside and talked to the technician, who had arrived with an XFi modem for Gigabit cable service. I explained to him that this was a FTTH service and we needed different equipment. He fortunately understood what I was talking about and put in a call to dispatch. At this point, I also tried to reach my account manager via phone, but apparently he hadn’t started his day yet. After finishing his call to dispatch, the tech informed me that someone else would be en route.

I called the 1-800-XFINITY hotline and spoke to their appointment managers to confirm that someone else was on the way. It seemed like everything was being handled correctly. I got off the phone and 30 minutes later the correct technicians with the Juniper equipment arrived!

The install started out pretty smoothly, the Juniper ACX2100 was racked up and powered on and fiber connected to the patch panel. Unfortunately, the head-end had installed the incorrect optics. There was a 1Gbps SFP at the head-end and my 10Gbps SFP at my Juniper would not negotiate a connection (plus the connection is supposed to be above 1Gbps). The install technician was very patient and stuck around for several hours while making phone calls and trying to get the head end issue resolved.

Eventually, after someone was finally dispatched to the head-end, they did not have the correct SFP. I almost thought my installation would be off for the day, but as luck struck, the technician on site at my house found that he did have an extra correct module. So he drove off to the head-end to install it himself.

Upon his return, we finally were getting somewhere. He spent several more hours on the phone with the Metro-E team performing circuit testing to ensure that everything was up to quality specs.

In total, the install which we expected to take 4 hours took just over 6 hours. I was glad that we scheduled it so early in the day, or it might not have even gotten done in a single day.

I really do have to commend the install technician for his persistence and I feel he went way above and beyond the call of duty to make sure my connection got online today. I plan to leave him an outstanding review when his project manager calls to take my feedback survey, which my account manager told me to expect.

The completed install at the pole… I do hope they plan to come back and clean this up more and it doesn’t fall down in the meantime.
The finished install in my network cabinet, and the Juniper in all its glory!

So now that it’s installed, how is it?

I don’t think reality has even quite sunk in for me yet, but from my experience so far, I feel that I now have the best Internet connection money can buy!

How can you beat a stable 2.5ms ping to Google?!

This next part seems to be up for dispute on a lot of Reddit\Internet posts, so let me clear this up for anyone who wants to know.

Yes! Gigabit Pro comes with two circuits. A fiber optic handoff provisioned for 2Gbps/2Gbps and an ethernet handoff provisioned for 1Gbps/1Gbps. Each circuit gets a public IPv4 and a /48 block of IPv6.

Here are my current speed test results on each independent circuit:

Fiber circuit speed test, 2Gbps symmetrical.
Copper circuit speed test, 1Gbps symmetrical.

How am I going to use all of this bandwidth? Well, I consider the copper circuit to be somewhat of a bonus, so I am planning on using Policy Based Routing (PBR) on my Ubiquiti EdgeRouter Infinity to route very quality sensitive traffic, such as VoIP phone traffic and my Twitch RTMP broadcasts, out the ethernet connection. This way it will not be impacted by bursty traffic on my fiber connection and will provide the best overall quality and reliability. I was doing something similar already with my existing dual cable modem dual ISP setup that I had created temporarily while waiting for the fiber.

For those wondering, my preliminary testing shows that it is in fact possible to use all 3Gbps at once:

IPerf Testing

Overall I am extremely pleased with the service so far and I look forward to being a Comcast fiber customer for years to come! 🙂

The Decline of WideOpenWest

A few years ago, I moved from a very rural area into a subdivision that was fortunate to have more options for Internet than most communities in America. Two cable providers and a telecom – WOW, Comcast, and AT&T.

When I first moved in, I was skeptical of WOW. I had prior experience with Comcast and I knew that their network was reliable in the area, so I passed WOW by and jumped on board the Comcast bandwagon.

That is, until one day I was given a reason to give WOW a chance. Comcast rolled out their enforced 1TB data caps in my market. With capped data plans looming on the horizon, I decided to jump ship and give WOW a shot.

When I first became a WOW subscriber, I was coming from a 75/10 plan on Comcast’s network and came on to WOW as a 110/10 subscriber, for roughly the same price. It seemed like a very competitive service – even though Comcast’s 75/10 was overprovisioned to 90/12, and WOW’s 110/10 service was barely overprovisioned at all, delivering maybe 115/10.

Evolution of WOW’s Services

After being a WOW customer for almost a year, they rolled out extremely competitive new speed tiers. They now offered speeds of up to 1Gbps, and the pricing absolutely blew Comcast away. They offered a 500/50 plan for $5 more than I was paying for my 110/10, so I called them up.

Despite the plan being officially for new customers only, I convinced the phone rep to upgrade my tier in exchange for paying a $75 ETF on my old contract. All in all, a very good experience, and once I grabbed a new modem and got provisioned, I was very happy with my speeds.

The whole process of upgrading struck me as a very consumer friendly practice, I felt that Comcast would never have allowed me to do the same as simply as WOW had done, and they had given me exactly what they promised.

The Golden Years

I had signed a 2 year contract when I jumped on the 500/50 service plan. For most of these two years, I had the luxury of not giving much thought to my Internet speeds. It’s one of those things that you don’t really think about while it’s in abundance, I definitely took it for granted. The monthly bill I was paying felt well worth it, like a steal in fact, for such fantastic service.

I did run into one rather advanced problem during the “golden years” of my WOW service. WOW’s network was not properly battery backed up, and during power outages, even a small blip of a few seconds, I would lose my Internet service despite having backup batteries and a backup generator at my house.

I thought that this would be a nearly insurmountable issue to get fixed, but after talking to WOW’s support, I was given direct contacts for some lead techs in my area. They informed me that they found a bad battery backup and it had been replaced. And lo and behold, the next time the power went out, my network stayed up!

At this time, WOW was to me, the ideal that every Internet provider should strive for. They owned up to and resolved issues promptly, and most of the time things just worked great. If people asked me for a recommendation for ISP, I would respond with a rave about WOW and how fantastic they are.

The Decline

I suppose I was lucky – that the decline of WOW fell in line so well with my 2 year contract. Right about the time I was thinking about asking for a new contract rate, as mine was going to be coming up, I started to experience some problems with my service.

I am a heavy user and a bit of an atypical user, as a tech savvy user and a content creator, I do a lot of uploading, and upload tends to be the first thing to go when a cable network has quality problems. Upstream signal is just more sensitive to problems than downstream.

My 50Mbps upload was degrading, to the point where it was more like 40Mbps, and during peak hours it would drop to 30Mbps.

No problem, I assumed, I would just get in touch with WOW’s fantastic support and they would get this issue resolved, just like they always have in the past.

I first called in through the regular phone support, and at first I was hopeful. The representative seemed interested in my problem and said that they would have a look and see what they could find out. But, that it was an advanced issue unlike any they had worked on before and they weren’t sure what they would find. Fine – great, let’s get someone high level involved I’m all for it!

As the weeks dragged on, I grew more and more frustrated with my intermittent upload speeds, and the latency spikes that accompanied them whenever I tried to push a lot of bandwidth outbound. It was especially frustrating for me as a night shift employee, experiencing congestion at 3AM when I know the network is not congested.

After getting nowhere with the phone support at first, I decided to reach out to the techs whose contacts I had been given in the past, to see if they could help with the issue. But I soon found that no one I had contact information on seemed to be with the company anymore.

Out of desperation, I ran a WHOIS on the IP block and emailed all of the WHOIS contacts to try to get in touch with someone at the company with some level of technical knowledge. Every single email bounced back. None of the technical contacts on the IP WHOIS seemed to work for the company anymore either.

So, left with no other options, I realized I had to go the standard route. I would have a tech visit so they could observe the problem and get a solution in motion.

The tech who came was awesome, and again seemed genuinely interested in solving the problem. We ran some speed tests and he did acknowledge that the upload speed was poor. In order to rule out anything on my end so he could get an escalation going, we brought the modem in my back yard, and he plugged it directly into the tap. We still observed the poor speeds from there. He took some pictures, went on his way, and promised to submit a ticket to have someone take a look at the lines.

Another month passed – no end in sight for my upload performance issues. On top of that, my next month’s bill arrives and I am billed for a tech visit. This is the first time in my multiple years of being a customer that I have ever been billed for a tech visit. When I inquire for an explanation, they explain that the tech “found nothing wrong” and so I was billed for the problem. I was outraged. So had anything been escalated? No, apparently not.

I was able to get WOW to waive the technician fee, but they said that in order to get an escalation going, I would need another tech visit because too much time had passed.

At this point, I feel like I am treading water, so I reach out to friends\family who know people within the company to try to get some help. Clearly having another tech out is not going to fix the issue, so what is wrong with this escalation path?

I come to find – basically barely anyone at the company is employed in my state anymore. The local work is all being done by contractors now. Things have changed from the days when I signed on as a WOW customer and a WOW employee came to do my install.

WOW’s employee structure was now beginning to represent Comcast’s but with one distinction – no one seemed to be available to escalate to and no one actually empowered to solve these higher issues. Even with my contacts at the company I could not get anyone to help me, and they didn’t know of anyone who could help me.

Giving Up

So, now, that it seemed I would be stuck in this loop of dispatching technicians who can’t fix the problem and have no one to escalate to, I decided to get out.

I looked at Comcast’s offerings, and I found the only way to get anywhere close to the upload I was accustomed to was to buy their 1000/35 package, take a speed cut, and pay more than twice the price (because I need to also pay Comcast’s $50 unlimited data ransom).

I had been looking at my alternatives and I found out that Comcast had rolled out Gigabit Pro, a very niche fiber service that provides full duplex multi-gig connectivity.

At this point, I had been using WOW’s 500/50 because it was providing the most upload speed available to me. (Their Gigabit plan was also 50 up). So finding out that I had an option that would provide me more upload was very liberating.

The price tag on Gigabit Pro was steep. More than 4x what I was paying WOW for my 500/50, but Comcast would provide me with 2Gbps symmetrical FTTH Internet.

Given that the price for Gigabit Pro was only roughly twice the cost of their 1000/35+unlimited plan, with twice the download speed but orders of magnitude more upload speed, it seemed worthwhile to me. I began to look for ways to justify this to myself and ways I could cut costs to be able to pay Comcast’s hefty price tag to get the Internet of my dreams.

I eventually started the process of signing up for Gigabit Pro. This process has taken up many months of 2019, but an end is finally in sight for me.

In the meantime, I needed a solution to my issues, because WOW was no longer reliable for even broadcasting on Twitch. My streams were interrupted multiple times a night due to connection quality problems. So, I picked up a Comcast 250/20 line to use for streaming and VoIP temporarily. It more than doubled my monthly Internet cost, but cost about the same as getting Comcast + unlimited data, so I kept my unreliable WOW connection around for bulk data use and my Comcast line is now serving as a temporary “premium” lane for my things that need to be reliable.

I look forward to having my Gigabit Pro service active before the end of the year, and finally saying goodbye to coax cable Internet – and all of the RF signal noise problems that can come with it – and saying goodbye to WOW after several years of hailing them as the best ISP in the business.

The Writing On The Wall?

Hindsight, as they say, is 20/20. Now I can look back at the past with clarity.

Around the time WOW rolled out their new speed tiers, around the time I signed my contract with them, they also had their IPO and began offering stock.

At the time, I didn’t see this as a negative, I saw it as the company’s attempt to grow and expand.

Now, I wonder – could the sudden massive boost in speed tiers have been merely an attempt to try to gather up as many new customers as possible to boost the numbers?

What is WOW’s long term plan? Are they looking to pump their subscriber numbers so that they can find someone to buy them out?

There is a huge part of me that suspects that WOW will one day no longer exist, maybe it will be acquired one day by another cable company. I think Charter\Spectrum could be a likely future buyer, just based on what I see of the market position of the two companies.

Maybe it’s not so nefarious, maybe it was simply the transition from being a small “startup” company to being a corporate company with investors that ended up being WOW’s downfall.

I feel that this employee review I found on Glassdoor from someone who says they have worked for WOW for 10 years most likely accurately sums up the state of affairs:

The Future?

As long as WOW remains independent and in business, there is the potential for them to fix their systemic problems. It’s not extremely likely, but management can change.

I would be open to giving WOW a shot in the future, but I will no longer recommend them to people (and I may warn against them in fact). I would be very leery of ever signing another contract with them.

Your service may still be fine, the network by you may have a good tech taking care of it still, but I feel that overall that if you have good service from WOW, your days are probably numbered. The people who made the service good seem to be seeing the writing on the wall and leaving.

I believe that the most likely outcome for them is acquisition by another cable company.

Either way, I most likely won’t be a part of it. I don’t have any plans to go back to cable from my fiber line anytime soon.

Adaptec 6805T Troubleshooting Experience

I recently started building out my third file server, and picked up some Adaptec 6805T RAID cards on eBay to interface with my multiple drive trays. Having built similar servers and also working with numerous Adaptec product lines in my day to day datacenter work, I consider myself to be a subject matter expert, but this experience seemed interesting enough to write a post about since the issue had me scratching my head for a bit, hopefully this will help someone who is in my shoes in the future.

An Unusual Error

One of the cards out of the batch was producing an unusual error.

One or more drives are either missing or not responding.
Please check if the drives are connected and powered on.
<< Correct the problem and Reboot the system >>

On the surface, this error makes a lot of sense, it’s simply warning that the hard drives from the previous configuration are gone. Since I bought this card on eBay, and there’s no telling what the previous owner’s configuration was, this makes perfect sense.

What doesn’t make sense is this error message halting the boot process, and the lack of ability to accept this new configuration state. Normally, you would be prompted to accept or reject the new configuration when drives are removed.

What makes even LESS sense is the complete absence of any documentation or other posts online containing the text of this error. My Google searches returned nothing of use. No mention of this on any Adaptec resources I could find. No one on /r/homelab able to provide a helpful response to my post.

Troubleshooting Efforts

Troubleshooting this problem was difficult, because my computer would not boot with the PCIe card installed. This error message completely halted the boot process. Since PCIe isn’t hot plug, it’s not as if I could turn the computer on and then slide the card in.

I remembered that I have a few server boards which allow me the capability of disabling the PCIe option ROMs on a per-port basis, so my hope was that I could disable the oprom on the card and boot into Linux, then do further troubleshooting with arcconf.

So, I popped the card in and went to go into the BIOS, but of course, thanks to this error, I couldn’t even open the BIOS. The ROM initialization happened before the BIOS opened and thus locked up my system and I wasn’t able to proceed further.

I decided to put a working 6805T card in, disable the option ROM, then swap the card for the non-working one and hope that the ROM would stay disabled in the BIOS.

This worked, and I was able to get booted into Linux!

Trial & Error

Poking around arcconf I had a few ideas.

I tried resetting the controller to factory defaults:
arcconf setconfig 1 default

Curiously, this alone did NOT fix the problem.

Reading the docs, I came across this command, and I was VERY hopeful:
arcconf setbiosparams 1 BIOSHALTONMISSINGDRIVECOUNT <count>

But unfortunately, apparently my controller doesn’t “support” changing this setting?
Setting the BIOS parameters in not supported on this controller.
This was definitely a letdown, as this seemed like the setting I wanted to change.

What Ended Up Working

I decided, since the firmware on the controller was outdated, I would update the firmware, try to reset as much as possible, and then test it.

So I downloaded the latest BIOS ROM image from Adaptec and proceeded:

arcconf romupdate 1 as680T01.ufi
arcconf setconfig 1 default
arcconf resetstatisticscounters 1

This worked! I was now able to boot into the Adaptec BIOS with the CTRL+A prompt, after moving the card to a PCIe slot where the option ROM was enabled.

Curiously, even after resetting everything, I STILL had to “accept” the new configuration the first time. But, after this, everything was working normally! 🙂