Debian and Ubuntu Serial Console with Puppet

As most of the servers running actual software are these days virtual machines, in our case running under KVM, having a serial console accessible from the hypervisor is always a good idea in case troubleshooting is needed without network access.

The majority of our virtual servers are running either Debian GNU/Linux or Ubuntu. Both of these operating systems share major parts - including the bootloader GRUB . The configuration of this bootloader is almost identical in both systems as well.

The high-level GRUB configuration can be found in /etc/default/grub and after parsing whatever settings happen to be in this file all the files /etc/default/grub.d/*.cfg are processed with the latest instance of any configuration variable overiding previous settings.

In order to make both GRUB and the initial Linux kernel console use the first virtual serial port which can be accessed via libvirt's virsh console on the hypervisor, we create the file /etc/default/grub/99-virsh-console.cfg, providing the following configuration variables:

GRUB_TERMINAL="console serial"
GRUB_TIMEOUT_STYLE=menu
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,19200n8"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_SERIAL_COMMAND="serial"
GRUB_TIMEOUT=10

The reason for re-setting the GRUB_CMDLINE_LINUX_DEFAULT and storing our desired configuration solely in GRUB_CMDLINE_LINUX is quite arbitrary, however it unifies Debian and Ubuntu settings and we do not need different configurations for these systems that way. Note that we also keep both grub and initial console also accessible on the emulated VGA interface reachable using VNC by putting both console and serial in the GRUB_TERMINAL configuration variable. The GRUB_TIMEOUT_STYLE setting to menu ensures that during the initial timeout - which is 10 seconds on both operating systems by default - is the GRUB menu with available operating systems shown.

The last line of the file ensures the command for setting the serial line up is without any arguments. We have found that on some servers, migrated eons ago from physical hardware, there are sometimes settings for actual physical serial port interface - which do not work with the virtual serial console at all. The same applies for explicit GRUB_TIMEOUT which unifies how long the GRUB menu remains visible on all servers.

Although this is useful on its own, we try to keep as much as possible of our servers' configuration in Puppet configuration management system. That is why we have decided to create a custom Puppet class that will create this file resource on all of our managed servers automatically. The class is very simple:

class role::trustica::serialconsole () {
  $config_filename = "/etc/default/grub.d/99-virsh-console.cfg"

  file{
    "${config_filename}":
      ensure => "file",
      mode => '0755',
      owner => 'root',
      group => 'root',
      content => '
      GRUB_TERMINAL="console serial"
      GRUB_TIMEOUT_STYLE=menu
      GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,19200n8"
      GRUB_CMDLINE_LINUX_DEFAULT=""
      GRUB_SERIAL_COMMAND="serial"
      GRUB_TIMEOUT=10
      '
  }
}

As we can see, the class ensures the configuration file exists, it is owned by root with group root, its mode is 0755 and the contents are exactly what we have prepared before. However after creating this file, we must not forget to run update-grub to update the actual bootloader configuration based on these high-level settings.

It comes as no surprise the Puppet class can be extended to include dependency on this file that will trigger an action whenever the file is created or changed. And yes, the action in question will be running update-grub. This can be accomplished by adding the following exec resource after the aforementioned file resource:

  exec{
    "serialconsole_grub_update":
      path => $::path,
      subscribe => File[$config_filename],
      command => "update-grub",
      logoutput => true,
      refreshonly => true
  }

To use this class, all you need is to add it to the list of classes in your hiera YAML file for given server like:

---
classes:
  - role::trustica::serialconsole

Yes, that is all. The next Puppet run will ensure the server provides access to GRUB and initial Linux console through the virtual serial line and you can access it using simple virsh console domainname. See ya next time!