Category Archives: Musings

Just the musings of technical minds.

Adaptec 6805T Troubleshooting Experience

I recently started building out my third file server, and picked up some Adaptec 6805T RAID cards on eBay to interface with my multiple drive trays. Having built similar servers and also working with numerous Adaptec product lines in my day to day datacenter work, I consider myself to be a subject matter expert, but this experience seemed interesting enough to write a post about since the issue had me scratching my head for a bit, hopefully this will help someone who is in my shoes in the future.

An Unusual Error

One of the cards out of the batch was producing an unusual error.

One or more drives are either missing or not responding.
Please check if the drives are connected and powered on.
<<<< FATAL CONFIGURATION ERROR DETECTED >>>>
<<<< CANNOT CONTINUE BOOT PROCESS >>>>
<< Correct the problem and Reboot the system >>

On the surface, this error makes a lot of sense, it’s simply warning that the hard drives from the previous configuration are gone. Since I bought this card on eBay, and there’s no telling what the previous owner’s configuration was, this makes perfect sense.

What doesn’t make sense is this error message halting the boot process, and the lack of ability to accept this new configuration state. Normally, you would be prompted to accept or reject the new configuration when drives are removed.

What makes even LESS sense is the complete absence of any documentation or other posts online containing the text of this error. My Google searches returned nothing of use. No mention of this on any Adaptec resources I could find. No one on /r/homelab able to provide a helpful response to my post.

Troubleshooting Efforts

Troubleshooting this problem was difficult, because my computer would not boot with the PCIe card installed. This error message completely halted the boot process. Since PCIe isn’t hot plug, it’s not as if I could turn the computer on and then slide the card in.

I remembered that I have a few server boards which allow me the capability of disabling the PCIe option ROMs on a per-port basis, so my hope was that I could disable the oprom on the card and boot into Linux, then do further troubleshooting with arcconf.

So, I popped the card in and went to go into the BIOS, but of course, thanks to this error, I couldn’t even open the BIOS. The ROM initialization happened before the BIOS opened and thus locked up my system and I wasn’t able to proceed further.

I decided to put a working 6805T card in, disable the option ROM, then swap the card for the non-working one and hope that the ROM would stay disabled in the BIOS.

This worked, and I was able to get booted into Linux!

Trial & Error

Poking around arcconf I had a few ideas.

I tried resetting the controller to factory defaults:
arcconf setconfig 1 default

Curiously, this alone did NOT fix the problem.

Reading the docs, I came across this command, and I was VERY hopeful:
arcconf setbiosparams 1 BIOSHALTONMISSINGDRIVECOUNT <count>

But unfortunately, apparently my controller doesn’t “support” changing this setting?
Setting the BIOS parameters in not supported on this controller.
This was definitely a letdown, as this seemed like the setting I wanted to change.

What Ended Up Working

I decided, since the firmware on the controller was outdated, I would update the firmware, try to reset as much as possible, and then test it.

So I downloaded the latest BIOS ROM image from Adaptec and proceeded:

arcconf romupdate 1 as680T01.ufi
arcconf setconfig 1 default
arcconf resetstatisticscounters 1

This worked! I was now able to boot into the Adaptec BIOS with the CTRL+A prompt, after moving the card to a PCIe slot where the option ROM was enabled.

Curiously, even after resetting everything, I STILL had to “accept” the new configuration the first time. But, after this, everything was working normally! ūüôā

Monitoring & Graphing Eldewrito Server Statistics In LibreNMS

Introduction
LibreNMS is a fantastic open source NMS which I have been using for some time to monitor statistics on my network devices and servers.  The SNMP-based monitoring it offers is very simple to set up and provides fantastic insights into a variety of devices.

I recently spun up some Halo Online \ Eldewrito servers and wanted to keep track of the performance of the servers, so I could know the capacity of my VM hypervisor which I dedicated to the project.¬† My first step was configuring LibreNMS and monitoring the basic server statistics, but I really wanted to take this a step further, and correlate that data to the player count on the server.¬† Since Eldewrito is a community based game, a lot of tools are very limited, so tools such as GameTracker aren’t supportive of the game.

Fortunately, Eldewrito has a very simple query port usage and returns simple JSON data, so I was off to figuring out how to get this done.

The Approach
After consulting with some LibreNMS community members on their Discord, it was pointed out to me that LibreNMS supports Nagios plugins, so I got to work writing my own.

My Nagios Plugin
I have published the code for my Nagios plugin on Github.  The repository can be found here, and a direct link to the plugin file here.

If you are setting this up, download the plugin file, and proceed with this article.

LibreNMS Nagios Plugin Setup
LibreNMS provides excellent documentation on how to set up Nagios plugin support, you can find it here.¬† I am not going to cover this entire setup here, only the parts necessary to get this plugin working.¬† If you aren’t going to use other Nagios plugins, you don’t need to follow the rest of their guide’s steps either, but it’s there for your reference if the configuration steps change down the road.

The only steps which were required for the setup I wanted were:

  • Update the LibreNMS config.php, add the following lines:

  • Create the Nagios plugins directory:

  • Place my plugin file (found in the above section) in this directory.
  • Make the plugin file executable:

  • Your LibreNMS cron in /etc/cron.d/librenms needs a line which is checking the services.¬† Mine had this line already, so I didn’t need to change anything.¬† If yours doesn’t, you could try adding mine or consult the LibreNMS documentation above.

Set Up Your Monitor!
Now that you’ve done the above, LibreNMS should display a Services tab at the top.¬† Navigate there to add a new service monitor.

On this tab, your next step is filling in the details for the plugin to work.

You will need to know the IP address of your Eldewrito server, as well as the query port.  The default query port is 11775.

Enter the following options:

  • Device –¬†Choose the server which you are hosting the Eldewrito server on.
  • Type¬† –¬†Select check_eldewrito to use our script.¬† This should be the filename of the file you put in your plugins folder, if you changed it.
  • Description –¬†Enter whatever you please here, I don’t care. ūüôā
  • IP Address –¬†Can be left blank and LibreNMS will autofill the device’s IP, or you can type the device’s IP if you enjoy verbosity like I do.
  • Parameters –¬†Just enter the query port of the Eldewrito server here, default is 11775.

Locating The Graph & Data
Once you’ve set up your first service monitor, go grab a cup of coffee or some dinner.¬† Allow at least 5-10 minutes for the graphing to start.

The graph is located under the Services tab, you can either click on the device and go to its services tab, or you can go to the services tab on the top and click “All Services”.¬† You’ll need to toggle the tab from Basic to Detailed as well to see the graph.

Unfortunately on my version of LibreNMS there are some styling issues, the last check time overlaps the plugin check name a bit.  Maybe they will correct this in a future update, it is mostly a cosmetic issue anyway.

There you have it!¬† You can add this plugin on multiple server devices in LibreNMS, or use it on the same server several times across multiple ports.¬† The possibilities are quite endless!¬† It’s possible to extend the plugin to gather other statistics from the server as well, but for the moment, player count was of the most interest to me.

Please feel free to leave a comment here or open an issue on the Github if you run into any questions or problems with the plugin, and I’ll be happy to take a look.

Reinstall GRUB When Your System Isn’t Bootable

Background
From time to time, I have encountered a system where due to some conditions the GRUB installation has become corrupted.  It is a fairly simple matter to reinstall GRUB, and sometimes this can quickly salvage a broken installation.

This is not meant to be a comprehensive guide, but is a compilation of information I found online ([1], [2]) that was useful to me in repairing a few problematic GRUB loaders.  Please feel free to contribute in the comments section if you have something to add.

Setting Up The Environment
The first thing you will want to do is locate a live CD as close to the version of Linux you have installed as possible.  Boot into the Live CD.

You need to set up a functional chroot.  For the purposes of this guide, I will assume /dev/sda1 is your / partition on your installation.  We will make a chroot in /mnt/chroot/.

Next, you need to bind the directories that GRUB needs to access.

Finally, chroot to your environment.

Reinstalling GRUB Automatically
You can try a simple reinstall of GRUB automatically first, see if any insightful errors are produced.

Failing that, try the following which will rescan the BIOS drives and update information in the GRUB configuration.

Reinstalling GRUB Manually
If the automatic reinstall does not work for you, try reinstalling via the GRUB shell.

Comcast’s Awesomely Accurate Progress Bar

One of my favorite parts of being a Comcast customer is getting to login to my client portal and see my data usage.  (Of course I will enjoy this much less if they eventually decide to enforce their 250GB cap in my area, that will require me to find another ISP or pay their unlimited usage charge or a business class tier).

What I do get a good laugh out of is their ridiculously arbitrary progress bar.

Here is my data usage for January 2016:
comcast-meter-january

As you can see, I used 560GB of data… which although their (non-enforced) cap is 250GB, this appears to be roughly 60% of some arbitrary¬†maximum?

 

I was quite surprised to login today on February 2nd and see this fantastic usage meter appear:
comcast-meter-4gb-off

It would seem that 4GB is almost as much usage as 560GB, who’d have thought?!

I checked on the other page as well and found the same ridiculousness:
comcast-meter-february

I would love to know what’s going on in Comcast’s heads sometimes!

Configuration of OVH IP Failover on VMs

Background
I have had a server with OVH for the past several years, and one of the things that took me many months of research, as well as trial and error, to get right was my IP address configuration for virtual machines.  The documentation available from OVH has been consistently lacking these past few years on how to properly configure it.

Debian \ Ubuntu systems will add to the complications, because they do not agree with the necessary configuration. ¬†If you try to add it traditionally, just by editing /etc/network/interfaces, you will most likely end up with no default gateway when you’re all said and done, and your connection still won’t work.

For the first few months I was with OVH, I was launching my networking through a shell script in a cron job, because configuring¬†it with /etc/network/interfaces just didn’t seem possible.

 

What You Need For This Guide \ Assumptions About Your Setup
1. A dedicated server on the OVH network.  This guide has been tested in the Roubaix datacenter, Gravelines datacenter, and the BHS datacenter.
2. A virtualization platform installed, and a virtual switch bridged with your network interface.  I have been using VMWare, and a vSwitch bridged with my eth0 interface.  This guide should be applicable to any virtualization suite (Xen, OpenVZ, etc) as long as your networking is bridged.
3. A Debian \ Ubuntu VM.  You can apply the knowledge gained from this guide to other operating systems (eg. CentOS) but the configuration will be different as the files are located in different places and are parsed differently.
4. A purchased IP range.  You can purchase IP addresses from OVH in your Manager Panel.

 

Gather Your Information \ Initial Setup On OVH Side
1. Your dedicated server’s Main IP will be needed for this guide. ¬† Say, for example, your main IP is 123.4.5.67.
2. Your dedicated server’s¬†Gateway IP¬†will be needed for this guide. ¬†On the OVH network, this is the first three octets of your main IP, ending in 254. ¬†For 123.4.5.67, the gateway is 123.4.5.254.
3. A Fail Over IP Range registered with OVH.  You can register any size IP block, and you can pick any IP from the block.  Say for example, you bought 5.6.77.8/29.  All of the Fail Over IPs are usable at OVH, so you have available all 8 IPs from 5.6.77.8 to 5.6.77.15 for use for your VMs.  You can purchase an IP block from your Manager Panel.
4. A¬†Registered VM MAC Address¬†associated on OVH’s routers. ¬†Without this step, your connection will not work. ¬†You can register VM MACs in your Manager Panel.

 

How To Register A VM MAC Address
In the new OVH Manager panel, you can register a VM MAC address by going to the “Manage IPs” page, clicking the gear icon next to the IP(s) you want to assign to the VM, and clicking “Add A Virtual MAC”. ¬†You can choose an existing MAC address if you are assigning more than 1 IP, otherwise¬†OVH will assign a new MAC address to you, and you will edit your VM preferences¬†so your network interface has this MAC address. ¬†Make sure you select the MAC address type “VMWare” if you are using a VMWare server, as this will generate MAC addresses which VMWare will accept. ¬†You can use type “OVH” for any other virtualization platform. ¬†The “name of virtual machine” field is just for your notes, and can be anything.

 

Operating System Installation
Install Debian \ Ubuntu to the VM, from a full install media.  Do not plan on doing a network install, as this guide does not cover how to make the OVH networking functional during installation; only after installation.  Skip the step for setting up networking during the installer, leave it unconfigured.

 

Setting Up /etc/network/interfaces
Now that you have your operating system installed, you are ready to configure your /etc/network/interfaces file. ¬†This guide assumes that eth0 is your Internet interface. I will use IP addresses from the example above under “Gather Your Information”. ¬†Do not use these addresses, substitute your own IP addresses.

A few things to note, for anyone asking any technical questions about what I just did above:

  • You should choose 1 IP address from your IP block. ¬†If you want to add more, see the further instructions below. ¬†This IP is the Main IP of your VM.
  • Regardless of the size of your block, you will treat the IP address as a /32 with a netmask 255.255.255.255.
  • Because you are using a /32, the network\broadcast IP is your IP.
  • Your default gateway should be¬†the same as your dedicated server’s default gateway. ¬†DO NOT¬†try to use what should logically be the gateway IP for your IP block, and DO NOT try to set your dedicated server itself as the gateway. ¬†The IP block is irrelevant, the gateway is always the dedicated server’s gateway IP.
  • The routes have to be added this way, because the configuration is not valid. ¬†The gateway should not be reachable from this network range, because it’s a /32. ¬†However, the gateway will be reachable anyway, and adding the routes this way should work.

 

DNS Resolution
You will also need to edit your /etc/resolv.conf to put a DNS name server in there, otherwise your server will be able to ping other IPs, but it will not be able to resolve DNS.  You can just put a line in there, like this line which uses the public OpenDNS resolver:

 

Want To Add More IPs?
Sometimes it is desirable to have more than one IP address on your server. ¬†You can easily add additional IPs from¬†any IP block¬†as an alias IP. ¬†You don’t need to configure any additional gateways, because the gateway is always your dedicated server gateway. ¬†You can purchase as many IP blocks as you want, and assign them all this way, it doesn’t matter if they are totally different IP ranges. ¬†In the example below, I will add 2 additional IPs from the IP¬†range I used as an example, 5.6.77.8/29.

 

Having Problems?
If you are having any problems, please review these items:

  • First,¬†reboot the VM¬†to ensure that all of the networking changes you made are fully applied.
  • Can you ping your default gateway? ¬†If not, did your route persist? ¬†Run “route -n” and see if the route is there. ¬†If it isn’t there, try adding it manually by running the commands in the post-up in /etc/network/interfaces. ¬†See if you get any errors.
  • Did you configure DNS? ¬†Try pinging just an IP address, (eg. ping 4.2.2.1) and see if you get any response.
  • Did you remember to register a MAC address, and did you change the MAC address for the VM? ¬†Run “ifconfig eth0” and make sure the MAC address shown matches what’s registered in your OVH Manager Panel.

 

Need Further Assistance?
As always, feel free to leave a comment below if you need any assistance.  I will be happy to help!