What It Was Like Getting Comcast Gigabit Pro

As an excited prospective Gigabit Pro customer, my favorite reading to return to has been James Watt’s Medium.com article about his install experience. To that end, I wanted to create a similar post documenting my own install experiences so that you future Gigabit Pro users can learn from my experience and maybe get some of your questions answered.

The focus of this article is not a technical article about how I prepared my network for 10Gbps. I may create another article soon detailing the setup of my network and how I am handling the bandwidth. The purpose of this article is to document my installation experience. So let’s dive in!

Initial Inquiry

My journey starts in mid-April of 2019. I headed over to the /r/Comcast_Xfinity subreddit to inquire on how I could get an order going. They were actually quite helpful over there, and shortly I was receiving a call from a representative from Comcast Sales Operations.

Talking with the sales representative, he was able to look at his network maps and determine that there was a fiber tap very close to my house, actually only about 5 houses down to the main street. Based on this information, he thought I should be eligible for the service.

For those of you who are curious if you might be eligible, check utility poles near you for something that looks like this:

Comcast’s fiber tap near my house.

I went over all of the initial information with sales, pretty much in line with what I had found out during my research. It would basically cost $1,000 to install and $319.99/month thereafter. This was not unexpected, but was a large increase over the roughly $70/month I was paying for my cable line, so I decided to give it some thought before proceeding.

My old WOW cable Internet continued to be on the decline in quality, and it didn’t seem like there was any resolution in sight. This frustration gave me the push I needed, and about a month later in mid-May, I was reaching back out to sales to tell them to get the ball rolling.

Pre-Installation Process

Although I appeared to be close by to the fiber tap on the maps, the first step in the process still needed to be an on site survey. Comcast informed me that they would get someone on the site to make sure I was really eligible, and then we could move forward from there.

It was made clear to me that this was going to be a multiple-month process, and if eligible, we would “try” to get it installed before the winter months came and the ground would become too frozen to dig.

Someone must have come to the site at some point, but no interaction was needed with me. Comcast didn’t call me back directly, but after a few weeks I called to follow up and was told that my initial site survey had been approved and we were moving forward with the installation.

The next step would be waiting for approvals and permits. Evidently the install process for Comcast works something like this internally:

  • Perform site survey to make sure that the building is serviceable.
  • Create a formal installation design \ plan and submit it for approval.
  • Review of the installation plan by finance team to determine if costs are acceptable to move forward.
  • Submit permit requests to applicable local government.
  • Begin construction once permits are received.

The whole installation was rife with misinformation from my Comcast representative who always seemed to be out of the loop and a few steps behind on the process, which makes me wonder who is really in charge of coordinating the tasks. I don’t hold this against him but I do think there is a ton of room for Comcast to improve communication both internally and with their customer during this long process.

A few examples of such confusion \ misinformation throughout the process:

  • Around a month after being told my site survey was complete, I received a call from the construction contractor asking to schedule a time to meet me for a site survey. Apparently this was when he was out to draw up the design.
  • At one point, I was told by Comcast that we were “waiting on permits” from the government and then told by the cable installer contractor that we were waiting on Comcast for financial approval. That turned out to be correct, and once it was approved then the wait for permits began.
  • When construction began, I called up my Comcast rep to inform him of the progress, but couldn’t reach him. The next day he called me to inform me that the permits had come in and we could start construction soon. (Though it already started a day earlier.)

5 Months Later

Most of what I had read online suggested that it was about a 3 month lead time from inquiry to install, but for me this was not correct.

Finally after months of waiting for permits and approval and seemingly making no progress, the contractor called me to inform me that they were scheduling directional boring. From mid-May to the beginning of October, we were finally ready to get some actual work done.

The installation would happen in 4 stages:

  • Directional Boring – The cable contractor installed an orange conduit several feet underground leading from my back yard to the utility pole where the fiber tap is located.
  • Pulling The Cable – The cable contractor would visit to pull the fiber cable from my basement to the utility pole.
  • Splicing – The cable contractor would visit to add the connections on both ends of the cable and plug it in.
  • Setup – Comcast would send out a representative to install the Juniper router and actually bring the connection up.

Directional Boring

The directional boring process began for me on the last day of September. No appointment was necessary, I simply awoke in the afternoon to the sound of machinery running outside. I had missed the voicemail from the contractor so I wasn’t even aware anyone would be coming – it was a really nice surprise.

Most of the work occurred in the utility easement behind my house. They did have to bore into my backyard, but they only dug a single hole next to the house. It doesn’t appear they even brought the equipment into my yard.

Deep Holes for Directional Boring
It was a bit wet that day!
Fiber’s on the way!

This was such a process, they actually had to leave their equipment overnight and finish the next day. Of course, being as excited as I was, I had to go back and take some pictures to document this part too!

Directional boring equipment — not so boring after all!
North American Fiber-Seeking Backhoe
The legendary orange spool – usually reserved only for large infrastructure installs on the side of the highway.

Once they came back the next day, they finished everything up and planted some grass. They left a pull cord in the conduit so that they could return and fish the cable more easily later.

My conduit emerging from the ground near the utility pole.
Re-planting grass over the holes.

Running The Cable

I eagerly waited for the next step to begin, and for me it took about 10 days. I awoke to a few missed calls, and Comcast’s contractor was telling me they had some crews nearby and asking if they could do the installation that day.

I had work in a few hours, but I’d been waiting for this so eagerly for so long, so I made it work. The crew came out and promptly finished the inside part of the job so I could go to work on time

Fiber coiled up in my basement, so close, yet so far away…

When I returned from work, I was thrilled to see that the fiber was hanging from the utility pole. I went out the next morning to take some pictures.

My fiber coiled up on the utility pole, waiting to be connected.

Now, I was just simply waiting for step 3 of the install – to schedule the guy to come out and splice the connection. The Comcast contractor informed me that we were waiting on final schematics from Comcast to proceed.

Almost 2 weeks later, there was still no update. There were some changes on the pole though, someone had moved my fiber around and made this mess that looked like it was ready to fall down any minute!

The fiber moves on the pole.
Hopefully it stays up there!

At this point, my contact at the contractor stopped answering my calls, and his voicemail box was full. I was so eager to continue the install, but even Comcast was not able to update me on what was going on.

Disaster Strikes… Temporarily.

During the wait, we had a weekend where we got very large amounts of rain. I was getting ready for bed one Saturday night, and ran down to the basement to quickly grab something. There, I was shocked to find my floor flooded.

Water on the floor! Water under the server rack!
I was glad to have my elevated shelving keeping my old computer stuff off the floor today.

At first, I was just very surprised and confused. The areas where this water was located were not areas where there were any pipes or pumps very close by to account for the reason why there was water there.

It hadn’t even dawned on me right away to check the fiber, until I swore I saw it drip while I was standing there trying to figure out what had happened. Sure enough, I gave it a little shake, and:

Leaky fiber conduit.

Fortunately, after a Sunday afternoon call to the cable contractors, they quickly sent someone out the next day to fix the issue (right before a week with lots of rain, too). Fortunately in the end, nothing of any value was damaged. The contractors did some additional sealing where the fiber comes into my house, as well as at the fiber vault, so we are hoping this issue is now solved.

I should note, my house is located at a very low elevation. The fiber cable is at a much higher elevation than my basement for most of its run, and my backyard is very sloped. This did not occur due to a poorly sealed entry point, they did a nice job sealing around the wall penetration, what actually happened was water entered the fiber conduit up at a higher elevation and then came rushing down the conduit into my house.

With this out of the way, this got me back in contact with the contractor and we immediately got the next part of the process moving!

Splicing The Fiber

Finally, I had some forward motion from the contractor again, and we were able to move forward with the splicing part of the installation at the very end of October.

For this part of the installation, two Comcast techs came out with some expensive equipment and completed the splicing of 6 fiber pairs into a patch panel which they installed in my rack.

Fiber splicing!
Another angle on the fiber splicing.

Once they left, they informed me that they still needed to complete some work on the pole before they would be able to pass the job back to Comcast.

This would take another 9 days, but finally, the contractor’s work was done and Comcast was ready to schedule the last appointment for the installation!

Comcast needed some time to order the equipment, so the installation would be scheduled a little more than a week out, for a final install date in mid-November.

The Historic Day — Installation!

Today started out very early for me. I am a night shifter, and for most of the install process I have had a lot of success scheduling things at 2PM. Unfortunately, for this part of the install, they would only schedule it at 8AM. So, last night, I got to bed as early as I could after work and started the day a bit sleep deprived.

A few minutes before 8AM, I popped out of bed and saw a Comcast truck parked outside. I went outside and talked to the technician, who had arrived with an XFi modem for Gigabit cable service. I explained to him that this was a FTTH service and we needed different equipment. He fortunately understood what I was talking about and put in a call to dispatch. At this point, I also tried to reach my account manager via phone, but apparently he hadn’t started his day yet. After finishing his call to dispatch, the tech informed me that someone else would be en route.

I called the 1-800-XFINITY hotline and spoke to their appointment managers to confirm that someone else was on the way. It seemed like everything was being handled correctly. I got off the phone and 30 minutes later the correct technicians with the Juniper equipment arrived!

The install started out pretty smoothly, the Juniper ACX2100 was racked up and powered on and fiber connected to the patch panel. Unfortunately, the head-end had installed the incorrect optics. There was a 1Gbps SFP at the head-end and my 10Gbps SFP at my Juniper would not negotiate a connection (plus the connection is supposed to be above 1Gbps). The install technician was very patient and stuck around for several hours while making phone calls and trying to get the head end issue resolved.

Eventually, after someone was finally dispatched to the head-end, they did not have the correct SFP. I almost thought my installation would be off for the day, but as luck struck, the technician on site at my house found that he did have an extra correct module. So he drove off to the head-end to install it himself.

Upon his return, we finally were getting somewhere. He spent several more hours on the phone with the Metro-E team performing circuit testing to ensure that everything was up to quality specs.

In total, the install which we expected to take 4 hours took just over 6 hours. I was glad that we scheduled it so early in the day, or it might not have even gotten done in a single day.

I really do have to commend the install technician for his persistence and I feel he went way above and beyond the call of duty to make sure my connection got online today. I plan to leave him an outstanding review when his project manager calls to take my feedback survey, which my account manager told me to expect.

The completed install at the pole… I do hope they plan to come back and clean this up more and it doesn’t fall down in the meantime.
The finished install in my network cabinet, and the Juniper in all its glory!

So now that it’s installed, how is it?

I don’t think reality has even quite sunk in for me yet, but from my experience so far, I feel that I now have the best Internet connection money can buy!

How can you beat a stable 2.5ms ping to Google?!

This next part seems to be up for dispute on a lot of Reddit\Internet posts, so let me clear this up for anyone who wants to know.

Yes! Gigabit Pro comes with two circuits. A fiber optic handoff provisioned for 2Gbps/2Gbps and an ethernet handoff provisioned for 1Gbps/1Gbps. Each circuit gets a public IPv4 and a /48 block of IPv6.

Here are my current speed test results on each independent circuit:

Fiber circuit speed test, 2Gbps symmetrical.
Copper circuit speed test, 1Gbps symmetrical.

How am I going to use all of this bandwidth? Well, I consider the copper circuit to be somewhat of a bonus, so I am planning on using Policy Based Routing (PBR) on my Ubiquiti EdgeRouter Infinity to route very quality sensitive traffic, such as VoIP phone traffic and my Twitch RTMP broadcasts, out the ethernet connection. This way it will not be impacted by bursty traffic on my fiber connection and will provide the best overall quality and reliability. I was doing something similar already with my existing dual cable modem dual ISP setup that I had created temporarily while waiting for the fiber.

For those wondering, my preliminary testing shows that it is in fact possible to use all 3Gbps at once:

IPerf Testing

Overall I am extremely pleased with the service so far and I look forward to being a Comcast fiber customer for years to come! 🙂

The Decline of WideOpenWest

A few years ago, I moved from a very rural area into a subdivision that was fortunate to have more options for Internet than most communities in America. Two cable providers and a telecom – WOW, Comcast, and AT&T.

When I first moved in, I was skeptical of WOW. I had prior experience with Comcast and I knew that their network was reliable in the area, so I passed WOW by and jumped on board the Comcast bandwagon.

That is, until one day I was given a reason to give WOW a chance. Comcast rolled out their enforced 1TB data caps in my market. With capped data plans looming on the horizon, I decided to jump ship and give WOW a shot.

When I first became a WOW subscriber, I was coming from a 75/10 plan on Comcast’s network and came on to WOW as a 110/10 subscriber, for roughly the same price. It seemed like a very competitive service – even though Comcast’s 75/10 was overprovisioned to 90/12, and WOW’s 110/10 service was barely overprovisioned at all, delivering maybe 115/10.

Evolution of WOW’s Services

After being a WOW customer for almost a year, they rolled out extremely competitive new speed tiers. They now offered speeds of up to 1Gbps, and the pricing absolutely blew Comcast away. They offered a 500/50 plan for $5 more than I was paying for my 110/10, so I called them up.

Despite the plan being officially for new customers only, I convinced the phone rep to upgrade my tier in exchange for paying a $75 ETF on my old contract. All in all, a very good experience, and once I grabbed a new modem and got provisioned, I was very happy with my speeds.

The whole process of upgrading struck me as a very consumer friendly practice, I felt that Comcast would never have allowed me to do the same as simply as WOW had done, and they had given me exactly what they promised.

The Golden Years

I had signed a 2 year contract when I jumped on the 500/50 service plan. For most of these two years, I had the luxury of not giving much thought to my Internet speeds. It’s one of those things that you don’t really think about while it’s in abundance, I definitely took it for granted. The monthly bill I was paying felt well worth it, like a steal in fact, for such fantastic service.

I did run into one rather advanced problem during the “golden years” of my WOW service. WOW’s network was not properly battery backed up, and during power outages, even a small blip of a few seconds, I would lose my Internet service despite having backup batteries and a backup generator at my house.

I thought that this would be a nearly insurmountable issue to get fixed, but after talking to WOW’s support, I was given direct contacts for some lead techs in my area. They informed me that they found a bad battery backup and it had been replaced. And lo and behold, the next time the power went out, my network stayed up!

At this time, WOW was to me, the ideal that every Internet provider should strive for. They owned up to and resolved issues promptly, and most of the time things just worked great. If people asked me for a recommendation for ISP, I would respond with a rave about WOW and how fantastic they are.

The Decline

I suppose I was lucky – that the decline of WOW fell in line so well with my 2 year contract. Right about the time I was thinking about asking for a new contract rate, as mine was going to be coming up, I started to experience some problems with my service.

I am a heavy user and a bit of an atypical user, as a tech savvy user and a content creator, I do a lot of uploading, and upload tends to be the first thing to go when a cable network has quality problems. Upstream signal is just more sensitive to problems than downstream.

My 50Mbps upload was degrading, to the point where it was more like 40Mbps, and during peak hours it would drop to 30Mbps.

No problem, I assumed, I would just get in touch with WOW’s fantastic support and they would get this issue resolved, just like they always have in the past.

I first called in through the regular phone support, and at first I was hopeful. The representative seemed interested in my problem and said that they would have a look and see what they could find out. But, that it was an advanced issue unlike any they had worked on before and they weren’t sure what they would find. Fine – great, let’s get someone high level involved I’m all for it!

As the weeks dragged on, I grew more and more frustrated with my intermittent upload speeds, and the latency spikes that accompanied them whenever I tried to push a lot of bandwidth outbound. It was especially frustrating for me as a night shift employee, experiencing congestion at 3AM when I know the network is not congested.

After getting nowhere with the phone support at first, I decided to reach out to the techs whose contacts I had been given in the past, to see if they could help with the issue. But I soon found that no one I had contact information on seemed to be with the company anymore.

Out of desperation, I ran a WHOIS on the IP block and emailed all of the WHOIS contacts to try to get in touch with someone at the company with some level of technical knowledge. Every single email bounced back. None of the technical contacts on the IP WHOIS seemed to work for the company anymore either.

So, left with no other options, I realized I had to go the standard route. I would have a tech visit so they could observe the problem and get a solution in motion.

The tech who came was awesome, and again seemed genuinely interested in solving the problem. We ran some speed tests and he did acknowledge that the upload speed was poor. In order to rule out anything on my end so he could get an escalation going, we brought the modem in my back yard, and he plugged it directly into the tap. We still observed the poor speeds from there. He took some pictures, went on his way, and promised to submit a ticket to have someone take a look at the lines.

Another month passed – no end in sight for my upload performance issues. On top of that, my next month’s bill arrives and I am billed for a tech visit. This is the first time in my multiple years of being a customer that I have ever been billed for a tech visit. When I inquire for an explanation, they explain that the tech “found nothing wrong” and so I was billed for the problem. I was outraged. So had anything been escalated? No, apparently not.

I was able to get WOW to waive the technician fee, but they said that in order to get an escalation going, I would need another tech visit because too much time had passed.

At this point, I feel like I am treading water, so I reach out to friends\family who know people within the company to try to get some help. Clearly having another tech out is not going to fix the issue, so what is wrong with this escalation path?

I come to find – basically barely anyone at the company is employed in my state anymore. The local work is all being done by contractors now. Things have changed from the days when I signed on as a WOW customer and a WOW employee came to do my install.

WOW’s employee structure was now beginning to represent Comcast’s but with one distinction – no one seemed to be available to escalate to and no one actually empowered to solve these higher issues. Even with my contacts at the company I could not get anyone to help me, and they didn’t know of anyone who could help me.

Giving Up

So, now, that it seemed I would be stuck in this loop of dispatching technicians who can’t fix the problem and have no one to escalate to, I decided to get out.

I looked at Comcast’s offerings, and I found the only way to get anywhere close to the upload I was accustomed to was to buy their 1000/35 package, take a speed cut, and pay more than twice the price (because I need to also pay Comcast’s $50 unlimited data ransom).

I had been looking at my alternatives and I found out that Comcast had rolled out Gigabit Pro, a very niche fiber service that provides full duplex multi-gig connectivity.

At this point, I had been using WOW’s 500/50 because it was providing the most upload speed available to me. (Their Gigabit plan was also 50 up). So finding out that I had an option that would provide me more upload was very liberating.

The price tag on Gigabit Pro was steep. More than 4x what I was paying WOW for my 500/50, but Comcast would provide me with 2Gbps symmetrical FTTH Internet.

Given that the price for Gigabit Pro was only roughly twice the cost of their 1000/35+unlimited plan, with twice the download speed but orders of magnitude more upload speed, it seemed worthwhile to me. I began to look for ways to justify this to myself and ways I could cut costs to be able to pay Comcast’s hefty price tag to get the Internet of my dreams.

I eventually started the process of signing up for Gigabit Pro. This process has taken up many months of 2019, but an end is finally in sight for me.

In the meantime, I needed a solution to my issues, because WOW was no longer reliable for even broadcasting on Twitch. My streams were interrupted multiple times a night due to connection quality problems. So, I picked up a Comcast 250/20 line to use for streaming and VoIP temporarily. It more than doubled my monthly Internet cost, but cost about the same as getting Comcast + unlimited data, so I kept my unreliable WOW connection around for bulk data use and my Comcast line is now serving as a temporary “premium” lane for my things that need to be reliable.

I look forward to having my Gigabit Pro service active before the end of the year, and finally saying goodbye to coax cable Internet – and all of the RF signal noise problems that can come with it – and saying goodbye to WOW after several years of hailing them as the best ISP in the business.

The Writing On The Wall?

Hindsight, as they say, is 20/20. Now I can look back at the past with clarity.

Around the time WOW rolled out their new speed tiers, around the time I signed my contract with them, they also had their IPO and began offering stock.

At the time, I didn’t see this as a negative, I saw it as the company’s attempt to grow and expand.

Now, I wonder – could the sudden massive boost in speed tiers have been merely an attempt to try to gather up as many new customers as possible to boost the numbers?

What is WOW’s long term plan? Are they looking to pump their subscriber numbers so that they can find someone to buy them out?

There is a huge part of me that suspects that WOW will one day no longer exist, maybe it will be acquired one day by another cable company. I think Charter\Spectrum could be a likely future buyer, just based on what I see of the market position of the two companies.

Maybe it’s not so nefarious, maybe it was simply the transition from being a small “startup” company to being a corporate company with investors that ended up being WOW’s downfall.

I feel that this employee review I found on Glassdoor from someone who says they have worked for WOW for 10 years most likely accurately sums up the state of affairs:

The Future?

As long as WOW remains independent and in business, there is the potential for them to fix their systemic problems. It’s not extremely likely, but management can change.

I would be open to giving WOW a shot in the future, but I will no longer recommend them to people (and I may warn against them in fact). I would be very leery of ever signing another contract with them.

Your service may still be fine, the network by you may have a good tech taking care of it still, but I feel that overall that if you have good service from WOW, your days are probably numbered. The people who made the service good seem to be seeing the writing on the wall and leaving.

I believe that the most likely outcome for them is acquisition by another cable company.

Either way, I most likely won’t be a part of it. I don’t have any plans to go back to cable from my fiber line anytime soon.

Adaptec 6805T Troubleshooting Experience

I recently started building out my third file server, and picked up some Adaptec 6805T RAID cards on eBay to interface with my multiple drive trays. Having built similar servers and also working with numerous Adaptec product lines in my day to day datacenter work, I consider myself to be a subject matter expert, but this experience seemed interesting enough to write a post about since the issue had me scratching my head for a bit, hopefully this will help someone who is in my shoes in the future.

An Unusual Error

One of the cards out of the batch was producing an unusual error.

One or more drives are either missing or not responding.
Please check if the drives are connected and powered on.
<<<< FATAL CONFIGURATION ERROR DETECTED >>>>
<<<< CANNOT CONTINUE BOOT PROCESS >>>>
<< Correct the problem and Reboot the system >>

On the surface, this error makes a lot of sense, it’s simply warning that the hard drives from the previous configuration are gone. Since I bought this card on eBay, and there’s no telling what the previous owner’s configuration was, this makes perfect sense.

What doesn’t make sense is this error message halting the boot process, and the lack of ability to accept this new configuration state. Normally, you would be prompted to accept or reject the new configuration when drives are removed.

What makes even LESS sense is the complete absence of any documentation or other posts online containing the text of this error. My Google searches returned nothing of use. No mention of this on any Adaptec resources I could find. No one on /r/homelab able to provide a helpful response to my post.

Troubleshooting Efforts

Troubleshooting this problem was difficult, because my computer would not boot with the PCIe card installed. This error message completely halted the boot process. Since PCIe isn’t hot plug, it’s not as if I could turn the computer on and then slide the card in.

I remembered that I have a few server boards which allow me the capability of disabling the PCIe option ROMs on a per-port basis, so my hope was that I could disable the oprom on the card and boot into Linux, then do further troubleshooting with arcconf.

So, I popped the card in and went to go into the BIOS, but of course, thanks to this error, I couldn’t even open the BIOS. The ROM initialization happened before the BIOS opened and thus locked up my system and I wasn’t able to proceed further.

I decided to put a working 6805T card in, disable the option ROM, then swap the card for the non-working one and hope that the ROM would stay disabled in the BIOS.

This worked, and I was able to get booted into Linux!

Trial & Error

Poking around arcconf I had a few ideas.

I tried resetting the controller to factory defaults:
arcconf setconfig 1 default

Curiously, this alone did NOT fix the problem.

Reading the docs, I came across this command, and I was VERY hopeful:
arcconf setbiosparams 1 BIOSHALTONMISSINGDRIVECOUNT <count>

But unfortunately, apparently my controller doesn’t “support” changing this setting?
Setting the BIOS parameters in not supported on this controller.
This was definitely a letdown, as this seemed like the setting I wanted to change.

What Ended Up Working

I decided, since the firmware on the controller was outdated, I would update the firmware, try to reset as much as possible, and then test it.

So I downloaded the latest BIOS ROM image from Adaptec and proceeded:

arcconf romupdate 1 as680T01.ufi
arcconf setconfig 1 default
arcconf resetstatisticscounters 1

This worked! I was now able to boot into the Adaptec BIOS with the CTRL+A prompt, after moving the card to a PCIe slot where the option ROM was enabled.

Curiously, even after resetting everything, I STILL had to “accept” the new configuration the first time. But, after this, everything was working normally! 🙂

Monitoring & Graphing Eldewrito Server Statistics In LibreNMS

Introduction
LibreNMS is a fantastic open source NMS which I have been using for some time to monitor statistics on my network devices and servers.  The SNMP-based monitoring it offers is very simple to set up and provides fantastic insights into a variety of devices.

I recently spun up some Halo Online \ Eldewrito servers and wanted to keep track of the performance of the servers, so I could know the capacity of my VM hypervisor which I dedicated to the project.  My first step was configuring LibreNMS and monitoring the basic server statistics, but I really wanted to take this a step further, and correlate that data to the player count on the server.  Since Eldewrito is a community based game, a lot of tools are very limited, so tools such as GameTracker aren’t supportive of the game.

Fortunately, Eldewrito has a very simple query port usage and returns simple JSON data, so I was off to figuring out how to get this done.

The Approach
After consulting with some LibreNMS community members on their Discord, it was pointed out to me that LibreNMS supports Nagios plugins, so I got to work writing my own.

My Nagios Plugin
I have published the code for my Nagios plugin on Github.  The repository can be found here, and a direct link to the plugin file here.

If you are setting this up, download the plugin file, and proceed with this article.

LibreNMS Nagios Plugin Setup
LibreNMS provides excellent documentation on how to set up Nagios plugin support, you can find it here.  I am not going to cover this entire setup here, only the parts necessary to get this plugin working.  If you aren’t going to use other Nagios plugins, you don’t need to follow the rest of their guide’s steps either, but it’s there for your reference if the configuration steps change down the road.

The only steps which were required for the setup I wanted were:

  • Update the LibreNMS config.php, add the following lines:
$config['show_services']           = 1;
$config['nagios_plugins']   = "/usr/lib/nagios/plugins";
  • Create the Nagios plugins directory:
mkdir -p /usr/lib/nagios/plugins/
  • Place my plugin file (found in the above section) in this directory.
  • Make the plugin file executable:
chmod +x /usr/lib/nagios/plugins/check_eldewrito
  • Your LibreNMS cron in /etc/cron.d/librenms needs a line which is checking the services.  Mine had this line already, so I didn’t need to change anything.  If yours doesn’t, you could try adding mine or consult the LibreNMS documentation above.
*/5  *    * * *   librenms    /opt/librenms/check-services.php >> /dev/null 2>&1

Set Up Your Monitor!
Now that you’ve done the above, LibreNMS should display a Services tab at the top.  Navigate there to add a new service monitor.

On this tab, your next step is filling in the details for the plugin to work.

You will need to know the IP address of your Eldewrito server, as well as the query port.  The default query port is 11775.

Enter the following options:

  • Device – Choose the server which you are hosting the Eldewrito server on.
  • Type  – Select check_eldewrito to use our script.  This should be the filename of the file you put in your plugins folder, if you changed it.
  • Description – Enter whatever you please here, I don’t care. 🙂
  • IP Address – Can be left blank and LibreNMS will autofill the device’s IP, or you can type the device’s IP if you enjoy verbosity like I do.
  • Parameters – Just enter the query port of the Eldewrito server here, default is 11775.

Locating The Graph & Data
Once you’ve set up your first service monitor, go grab a cup of coffee or some dinner.  Allow at least 5-10 minutes for the graphing to start.

The graph is located under the Services tab, you can either click on the device and go to its services tab, or you can go to the services tab on the top and click “All Services”.  You’ll need to toggle the tab from Basic to Detailed as well to see the graph.

Unfortunately on my version of LibreNMS there are some styling issues, the last check time overlaps the plugin check name a bit.  Maybe they will correct this in a future update, it is mostly a cosmetic issue anyway.

There you have it!  You can add this plugin on multiple server devices in LibreNMS, or use it on the same server several times across multiple ports.  The possibilities are quite endless!  It’s possible to extend the plugin to gather other statistics from the server as well, but for the moment, player count was of the most interest to me.

Please feel free to leave a comment here or open an issue on the Github if you run into any questions or problems with the plugin, and I’ll be happy to take a look.

Reinstall GRUB When Your System Isn’t Bootable

Background
From time to time, I have encountered a system where due to some conditions the GRUB installation has become corrupted.  It is a fairly simple matter to reinstall GRUB, and sometimes this can quickly salvage a broken installation.

This is not meant to be a comprehensive guide, but is a compilation of information I found online ([1], [2]) that was useful to me in repairing a few problematic GRUB loaders.  Please feel free to contribute in the comments section if you have something to add.

Setting Up The Environment
The first thing you will want to do is locate a live CD as close to the version of Linux you have installed as possible.  Boot into the Live CD.

You need to set up a functional chroot.  For the purposes of this guide, I will assume /dev/sda1 is your / partition on your installation.  We will make a chroot in /mnt/chroot/.

mkdir /mnt/chroot/
mount /dev/sda1 /mnt/chroot/

Next, you need to bind the directories that GRUB needs to access.

mount --bind /dev /mnt/chroot/dev
mount --bind /dev/pts /mnt/chroot/dev/pts
mount --bind /proc /mnt/chroot/proc
mount --bind /sys /mnt/chroot/sys

Finally, chroot to your environment.

chroot /mnt/chroot/

Reinstalling GRUB Automatically
You can try a simple reinstall of GRUB automatically first, see if any insightful errors are produced.

grub-install /dev/sda

Failing that, try the following which will rescan the BIOS drives and update information in the GRUB configuration.

grub-install --recheck /dev/sda

Reinstalling GRUB Manually
If the automatic reinstall does not work for you, try reinstalling via the GRUB shell.

grub
grub> root(hd0,0)
grub> setup(hd0)
grub> quit