The Decline of WideOpenWest

October 11, 2019 10:07 PM / Leave a Comment /

A few years ago, I moved from a very rural area into a subdivision that was fortunate to have more options for Internet than most communities in America. Two cable providers and a telecom – WOW, Comcast, and AT&T.

When I first moved in, I was skeptical of WOW. I had prior experience with Comcast and I knew that their network was reliable in the area, so I passed WOW by and jumped on board the Comcast bandwagon.

That is, until one day I was given a reason to give WOW a chance. Comcast rolled out their enforced 1TB data caps in my market. With capped data plans looming on the horizon, I decided to jump ship and give WOW a shot.

When I first became a WOW subscriber, I was coming from a 75/10 plan on Comcast’s network and came on to WOW as a 110/10 subscriber, for roughly the same price. It seemed like a very competitive service – even though Comcast’s 75/10 was overprovisioned to 90/12, and WOW’s 110/10 service was barely overprovisioned at all, delivering maybe 115/10.

Evolution of WOW’s Services

After being a WOW customer for almost a year, they rolled out extremely competitive new speed tiers. They now offered speeds of up to 1Gbps, and the pricing absolutely blew Comcast away. They offered a 500/50 plan for $5 more than I was paying for my 110/10, so I called them up.

Despite the plan being officially for new customers only, I convinced the phone rep to upgrade my tier in exchange for paying a $75 ETF on my old contract. All in all, a very good experience, and once I grabbed a new modem and got provisioned, I was very happy with my speeds.

The whole process of upgrading struck me as a very consumer friendly practice, I felt that Comcast would never have allowed me to do the same as simply as WOW had done, and they had given me exactly what they promised.

The Golden Years

I had signed a 2 year contract when I jumped on the 500/50 service plan. For most of these two years, I had the luxury of not giving much thought to my Internet speeds. It’s one of those things that you don’t really think about while it’s in abundance, I definitely took it for granted. The monthly bill I was paying felt well worth it, like a steal in fact, for such fantastic service.

I did run into one rather advanced problem during the “golden years” of my WOW service. WOW’s network was not properly battery backed up, and during power outages, even a small blip of a few seconds, I would lose my Internet service despite having backup batteries and a backup generator at my house.

I thought that this would be a nearly insurmountable issue to get fixed, but after talking to WOW’s support, I was given direct contacts for some lead techs in my area. They informed me that they found a bad battery backup and it had been replaced. And lo and behold, the next time the power went out, my network stayed up!

At this time, WOW was to me, the ideal that every Internet provider should strive for. They owned up to and resolved issues promptly, and most of the time things just worked great. If people asked me for a recommendation for ISP, I would respond with a rave about WOW and how fantastic they are.

The Decline

I suppose I was lucky – that the decline of WOW fell in line so well with my 2 year contract. Right about the time I was thinking about asking for a new contract rate, as mine was going to be coming up, I started to experience some problems with my service.

I am a heavy user and a bit of an atypical user, as a tech savvy user and a content creator, I do a lot of uploading, and upload tends to be the first thing to go when a cable network has quality problems. Upstream signal is just more sensitive to problems than downstream.

My 50Mbps upload was degrading, to the point where it was more like 40Mbps, and during peak hours it would drop to 30Mbps.

No problem, I assumed, I would just get in touch with WOW’s fantastic support and they would get this issue resolved, just like they always have in the past.

I first called in through the regular phone support, and at first I was hopeful. The representative seemed interested in my problem and said that they would have a look and see what they could find out. But, that it was an advanced issue unlike any they had worked on before and they weren’t sure what they would find. Fine – great, let’s get someone high level involved I’m all for it!

As the weeks dragged on, I grew more and more frustrated with my intermittent upload speeds, and the latency spikes that accompanied them whenever I tried to push a lot of bandwidth outbound. It was especially frustrating for me as a night shift employee, experiencing congestion at 3AM when I know the network is not congested.

After getting nowhere with the phone support at first, I decided to reach out to the techs whose contacts I had been given in the past, to see if they could help with the issue. But I soon found that no one I had contact information on seemed to be with the company anymore.

Out of desperation, I ran a WHOIS on the IP block and emailed all of the WHOIS contacts to try to get in touch with someone at the company with some level of technical knowledge. Every single email bounced back. None of the technical contacts on the IP WHOIS seemed to work for the company anymore either.

So, left with no other options, I realized I had to go the standard route. I would have a tech visit so they could observe the problem and get a solution in motion.

The tech who came was awesome, and again seemed genuinely interested in solving the problem. We ran some speed tests and he did acknowledge that the upload speed was poor. In order to rule out anything on my end so he could get an escalation going, we brought the modem in my back yard, and he plugged it directly into the tap. We still observed the poor speeds from there. He took some pictures, went on his way, and promised to submit a ticket to have someone take a look at the lines.

Another month passed – no end in sight for my upload performance issues. On top of that, my next month’s bill arrives and I am billed for a tech visit. This is the first time in my multiple years of being a customer that I have ever been billed for a tech visit. When I inquire for an explanation, they explain that the tech “found nothing wrong” and so I was billed for the problem. I was outraged. So had anything been escalated? No, apparently not.

I was able to get WOW to waive the technician fee, but they said that in order to get an escalation going, I would need another tech visit because too much time had passed.

At this point, I feel like I am treading water, so I reach out to friends\family who know people within the company to try to get some help. Clearly having another tech out is not going to fix the issue, so what is wrong with this escalation path?

I come to find – basically barely anyone at the company is employed in my state anymore. The local work is all being done by contractors now. Things have changed from the days when I signed on as a WOW customer and a WOW employee came to do my install.

WOW’s employee structure was now beginning to represent Comcast’s but with one distinction – no one seemed to be available to escalate to and no one actually empowered to solve these higher issues. Even with my contacts at the company I could not get anyone to help me, and they didn’t know of anyone who could help me.

Giving Up

So, now, that it seemed I would be stuck in this loop of dispatching technicians who can’t fix the problem and have no one to escalate to, I decided to get out.

I looked at Comcast’s offerings, and I found the only way to get anywhere close to the upload I was accustomed to was to buy their 1000/35 package, take a speed cut, and pay more than twice the price (because I need to also pay Comcast’s $50 unlimited data ransom).

I had been looking at my alternatives and I found out that Comcast had rolled out Gigabit Pro, a very niche fiber service that provides full duplex multi-gig connectivity.

At this point, I had been using WOW’s 500/50 because it was providing the most upload speed available to me. (Their Gigabit plan was also 50 up). So finding out that I had an option that would provide me more upload was very liberating.

The price tag on Gigabit Pro was steep. More than 4x what I was paying WOW for my 500/50, but Comcast would provide me with 2Gbps symmetrical FTTH Internet.

Given that the price for Gigabit Pro was only roughly twice the cost of their 1000/35+unlimited plan, with twice the download speed but orders of magnitude more upload speed, it seemed worthwhile to me. I began to look for ways to justify this to myself and ways I could cut costs to be able to pay Comcast’s hefty price tag to get the Internet of my dreams.

I eventually started the process of signing up for Gigabit Pro. This process has taken up many months of 2019, but an end is finally in sight for me.

In the meantime, I needed a solution to my issues, because WOW was no longer reliable for even broadcasting on Twitch. My streams were interrupted multiple times a night due to connection quality problems. So, I picked up a Comcast 250/20 line to use for streaming and VoIP temporarily. It more than doubled my monthly Internet cost, but cost about the same as getting Comcast + unlimited data, so I kept my unreliable WOW connection around for bulk data use and my Comcast line is now serving as a temporary “premium” lane for my things that need to be reliable.

I look forward to having my Gigabit Pro service active before the end of the year, and finally saying goodbye to coax cable Internet – and all of the RF signal noise problems that can come with it – and saying goodbye to WOW after several years of hailing them as the best ISP in the business.

The Writing On The Wall?

Hindsight, as they say, is 20/20. Now I can look back at the past with clarity.

Around the time WOW rolled out their new speed tiers, around the time I signed my contract with them, they also had their IPO and began offering stock.

At the time, I didn’t see this as a negative, I saw it as the company’s attempt to grow and expand.

Now, I wonder – could the sudden massive boost in speed tiers have been merely an attempt to try to gather up as many new customers as possible to boost the numbers?

What is WOW’s long term plan? Are they looking to pump their subscriber numbers so that they can find someone to buy them out?

There is a huge part of me that suspects that WOW will one day no longer exist, maybe it will be acquired one day by another cable company. I think Charter\Spectrum could be a likely future buyer, just based on what I see of the market position of the two companies.

Maybe it’s not so nefarious, maybe it was simply the transition from being a small “startup” company to being a corporate company with investors that ended up being WOW’s downfall.

I feel that this employee review I found on Glassdoor from someone who says they have worked for WOW for 10 years most likely accurately sums up the state of affairs:

The Future?

As long as WOW remains independent and in business, there is the potential for them to fix their systemic problems. It’s not extremely likely, but management can change.

I would be open to giving WOW a shot in the future, but I will no longer recommend them to people (and I may warn against them in fact). I would be very leery of ever signing another contract with them.

Your service may still be fine, the network by you may have a good tech taking care of it still, but I feel that overall that if you have good service from WOW, your days are probably numbered. The people who made the service good seem to be seeing the writing on the wall and leaving.

I believe that the most likely outcome for them is acquisition by another cable company.

Either way, I most likely won’t be a part of it. I don’t have any plans to go back to cable from my fiber line anytime soon.

Posted in: Musings

Adaptec 6805T Troubleshooting Experience

August 27, 2019 1:51 AM / Leave a Comment /

I recently started building out my third file server, and picked up some Adaptec 6805T RAID cards on eBay to interface with my multiple drive trays. Having built similar servers and also working with numerous Adaptec product lines in my day to day datacenter work, I consider myself to be a subject matter expert, but this experience seemed interesting enough to write a post about since the issue had me scratching my head for a bit, hopefully this will help someone who is in my shoes in the future.

An Unusual Error

One of the cards out of the batch was producing an unusual error.

One or more drives are either missing or not responding.
Please check if the drives are connected and powered on.
<<<< FATAL CONFIGURATION ERROR DETECTED >>>>
<<<< CANNOT CONTINUE BOOT PROCESS >>>>
<< Correct the problem and Reboot the system >>

On the surface, this error makes a lot of sense, it’s simply warning that the hard drives from the previous configuration are gone. Since I bought this card on eBay, and there’s no telling what the previous owner’s configuration was, this makes perfect sense.

What doesn’t make sense is this error message halting the boot process, and the lack of ability to accept this new configuration state. Normally, you would be prompted to accept or reject the new configuration when drives are removed.

What makes even LESS sense is the complete absence of any documentation or other posts online containing the text of this error. My Google searches returned nothing of use. No mention of this on any Adaptec resources I could find. No one on /r/homelab able to provide a helpful response to my post.

Troubleshooting Efforts

Troubleshooting this problem was difficult, because my computer would not boot with the PCIe card installed. This error message completely halted the boot process. Since PCIe isn’t hot plug, it’s not as if I could turn the computer on and then slide the card in.

I remembered that I have a few server boards which allow me the capability of disabling the PCIe option ROMs on a per-port basis, so my hope was that I could disable the oprom on the card and boot into Linux, then do further troubleshooting with arcconf.

So, I popped the card in and went to go into the BIOS, but of course, thanks to this error, I couldn’t even open the BIOS. The ROM initialization happened before the BIOS opened and thus locked up my system and I wasn’t able to proceed further.

I decided to put a working 6805T card in, disable the option ROM, then swap the card for the non-working one and hope that the ROM would stay disabled in the BIOS.

This worked, and I was able to get booted into Linux!

Trial & Error

Poking around arcconf I had a few ideas.

I tried resetting the controller to factory defaults:
arcconf setconfig 1 default

Curiously, this alone did NOT fix the problem.

Reading the docs, I came across this command, and I was VERY hopeful:
arcconf setbiosparams 1 BIOSHALTONMISSINGDRIVECOUNT <count>

But unfortunately, apparently my controller doesn’t “support” changing this setting?
Setting the BIOS parameters in not supported on this controller.
This was definitely a letdown, as this seemed like the setting I wanted to change.

What Ended Up Working

I decided, since the firmware on the controller was outdated, I would update the firmware, try to reset as much as possible, and then test it.

So I downloaded the latest BIOS ROM image from Adaptec and proceeded:

arcconf romupdate 1 as680T01.ufi
arcconf setconfig 1 default
arcconf resetstatisticscounters 1

This worked! I was now able to boot into the Adaptec BIOS with the CTRL+A prompt, after moving the card to a PCIe slot where the option ROM was enabled.

Curiously, even after resetting everything, I STILL had to “accept” the new configuration the first time. But, after this, everything was working normally! 🙂

Posted in: How-To's, Musings

Monitoring & Graphing Eldewrito Server Statistics In LibreNMS

December 14, 2018 10:13 PM / Leave a Comment /

Introduction
LibreNMS is a fantastic open source NMS which I have been using for some time to monitor statistics on my network devices and servers.  The SNMP-based monitoring it offers is very simple to set up and provides fantastic insights into a variety of devices.

I recently spun up some Halo Online \ Eldewrito servers and wanted to keep track of the performance of the servers, so I could know the capacity of my VM hypervisor which I dedicated to the project.  My first step was configuring LibreNMS and monitoring the basic server statistics, but I really wanted to take this a step further, and correlate that data to the player count on the server.  Since Eldewrito is a community based game, a lot of tools are very limited, so tools such as GameTracker aren’t supportive of the game.

Fortunately, Eldewrito has a very simple query port usage and returns simple JSON data, so I was off to figuring out how to get this done.

The Approach
After consulting with some LibreNMS community members on their Discord, it was pointed out to me that LibreNMS supports Nagios plugins, so I got to work writing my own.

My Nagios Plugin
I have published the code for my Nagios plugin on Github.  The repository can be found here, and a direct link to the plugin file here.

If you are setting this up, download the plugin file, and proceed with this article.

LibreNMS Nagios Plugin Setup
LibreNMS provides excellent documentation on how to set up Nagios plugin support, you can find it here.  I am not going to cover this entire setup here, only the parts necessary to get this plugin working.  If you aren’t going to use other Nagios plugins, you don’t need to follow the rest of their guide’s steps either, but it’s there for your reference if the configuration steps change down the road.

The only steps which were required for the setup I wanted were:

  • Update the LibreNMS config.php, add the following lines:
$config['show_services']           = 1;
$config['nagios_plugins']   = "/usr/lib/nagios/plugins";
  • Create the Nagios plugins directory:
mkdir -p /usr/lib/nagios/plugins/
  • Place my plugin file (found in the above section) in this directory.
  • Make the plugin file executable:
chmod +x /usr/lib/nagios/plugins/check_eldewrito
  • Your LibreNMS cron in /etc/cron.d/librenms needs a line which is checking the services.  Mine had this line already, so I didn’t need to change anything.  If yours doesn’t, you could try adding mine or consult the LibreNMS documentation above.
*/5  *    * * *   librenms    /opt/librenms/check-services.php >> /dev/null 2>&1

Set Up Your Monitor!
Now that you’ve done the above, LibreNMS should display a Services tab at the top.  Navigate there to add a new service monitor.

On this tab, your next step is filling in the details for the plugin to work.

You will need to know the IP address of your Eldewrito server, as well as the query port.  The default query port is 11775.

Enter the following options:

  • Device – Choose the server which you are hosting the Eldewrito server on.
  • Type  – Select check_eldewrito to use our script.  This should be the filename of the file you put in your plugins folder, if you changed it.
  • Description – Enter whatever you please here, I don’t care. 🙂
  • IP Address – Can be left blank and LibreNMS will autofill the device’s IP, or you can type the device’s IP if you enjoy verbosity like I do.
  • Parameters – Just enter the query port of the Eldewrito server here, default is 11775.

Locating The Graph & Data
Once you’ve set up your first service monitor, go grab a cup of coffee or some dinner.  Allow at least 5-10 minutes for the graphing to start.

The graph is located under the Services tab, you can either click on the device and go to its services tab, or you can go to the services tab on the top and click “All Services”.  You’ll need to toggle the tab from Basic to Detailed as well to see the graph.

Unfortunately on my version of LibreNMS there are some styling issues, the last check time overlaps the plugin check name a bit.  Maybe they will correct this in a future update, it is mostly a cosmetic issue anyway.

There you have it!  You can add this plugin on multiple server devices in LibreNMS, or use it on the same server several times across multiple ports.  The possibilities are quite endless!  It’s possible to extend the plugin to gather other statistics from the server as well, but for the moment, player count was of the most interest to me.

Please feel free to leave a comment here or open an issue on the Github if you run into any questions or problems with the plugin, and I’ll be happy to take a look.

Posted in: How-To's, Musings

Reinstall GRUB When Your System Isn’t Bootable

April 4, 2016 4:22 AM / Leave a Comment /

Background
From time to time, I have encountered a system where due to some conditions the GRUB installation has become corrupted.  It is a fairly simple matter to reinstall GRUB, and sometimes this can quickly salvage a broken installation.

This is not meant to be a comprehensive guide, but is a compilation of information I found online ([1], [2]) that was useful to me in repairing a few problematic GRUB loaders.  Please feel free to contribute in the comments section if you have something to add.

Setting Up The Environment
The first thing you will want to do is locate a live CD as close to the version of Linux you have installed as possible.  Boot into the Live CD.

You need to set up a functional chroot.  For the purposes of this guide, I will assume /dev/sda1 is your / partition on your installation.  We will make a chroot in /mnt/chroot/.

mkdir /mnt/chroot/
mount /dev/sda1 /mnt/chroot/

Next, you need to bind the directories that GRUB needs to access.

mount --bind /dev /mnt/chroot/dev
mount --bind /dev/pts /mnt/chroot/dev/pts
mount --bind /proc /mnt/chroot/proc
mount --bind /sys /mnt/chroot/sys

Finally, chroot to your environment.

chroot /mnt/chroot/

Reinstalling GRUB Automatically
You can try a simple reinstall of GRUB automatically first, see if any insightful errors are produced.

grub-install /dev/sda

Failing that, try the following which will rescan the BIOS drives and update information in the GRUB configuration.

grub-install --recheck /dev/sda

Reinstalling GRUB Manually
If the automatic reinstall does not work for you, try reinstalling via the GRUB shell.

grub
grub> root(hd0,0)
grub> setup(hd0)
grub> quit
Posted in: How-To's, Musings

How To Easily Create A DVD From Just About Any Video Format (Linux CLI)

February 25, 2016 4:05 AM / 3 Comments /

Background
Encoding DVDs seems to be a task which is often overcomplicated.  If you have a video file you would like to convert into a DVD, it turns out it is very easy to do it from the Linux command line.

I was looking for a quick way to encode video to DVD, and I found this fantastic post by vmiimu on the ffmpeg forum. His post was fantastic and gave me the basis to create this guide, so props to vmiimu!  I thought the post was a bit crowded and might be difficult for some people to follow, so I have distilled its essence into this post.

This guide will allow you to convert a wide variety of common video formats into a video DVD.  The scope of this guide is to encode 1 video to 1 DVD with no menu.   The source file can be much larger than a DVD, it will be reduced in size automatically.

Prerequisites
For this guide, you will need the following programs \ commands available.  You may need to install the package from your distribution’s package library.

  • ffmpeg (usually available as a package called ‘ffmpeg‘).
  • dvdauthor (usually available as the package ‘dvdauthor‘)
  • genisoimage (usually available as the package ‘genisoimage‘)
  • growisofs (usually available in the package ‘dvd+rw-tools‘)

Installing these programs is beyond the scope of this guide.  You are assumed to already have them installed.

Hint: If you are using a Red Hat based distro, you may need EPEL and NUX Desktop.

Filenames & Paths Used In This Guide
For ease of understanding, I will be using the following filenames in this guide.  Please substitute your own filenames as you choose.
source.mkv – The original source video file you want to burn to DVD.  This is the only file you must supply, the rest will be created.
temp.mpg – The mpeg encoded video file which will be actually burned to DVD.
dvdmovie – The directory which will store your DVD files during encoding (AUDIO_TS, VIDEO_TS)
dvd.iso – The final .ISO image which you will burn to your DVD.
/dev/dvd – The device path for your DVD burner.


The Process:
Convert the source video to an MPEG video file <4.7GB.
(This may take a long time, and is the most CPU intensive part of the process.)

For videos with 4:3 aspect ratio, use this command:

ffmpeg -i source.mkv -filter:v "scale='if(gt(a,720/480),720,-1)':'if(gt(a,720/480),-1,480)',pad=w=720:h=480:x=(ow-iw)/2:y=(oh-ih)/2" -target ntsc-dvd temp.mpg

For videos with 16:9 (widescreen) aspect ratio, use this command:

ffmpeg -i source.mkv -filter:v "scale='if(gt(a,720/480),720,-1)':'if(gt(a,720/480),-1,480)',pad=w=720:h=480:x=(ow-iw)/2:y=(oh-ih)/2" -aspect 16:9 -target ntsc-dvd temp.mpg

Assemble a DVD file structure.

export VIDEO_FORMAT=NTSC
dvdauthor --title -o dvdmovie -f temp.mpg
dvdauthor -T -o dvdmovie

Generate an ISO image file.

genisoimage -dvd-video -o dvd.iso dvdmovie

Burn the ISO to a DVD disk.
Now would be a good time to insert your blank DVD.

growisofs -dvd-compat -Z /dev/dvd=dvd.iso

That’s It!
Hopefully that went smoothly for you.  Please feel free to post in the comments section if you have any questions or suggestions for improvement of this guide.

Posted in: How-To's