[Ubutu 19.04] [Docker] [Docker-compose] Stack trace when running docker build from golang-docker-credential-helpers.

I had been seeing some issues with the docker build command on my new T480 with Ubuntu 19.04. There was a stack trace when running the command but it didn’t seem to impact the build process as the image was still properly built at the end.

It looked like a go package might be the cause and digging around, it seems I wasn’t the only one with the issue.



Removing the package seems to be enough the fix the issue : sudo apt-get remove golang-docker-credential-helpers. I am not using any Docker credentials so it’s not a big loss for the moment.

The fix is coming in 0.6.2! :





[Debian / Ubuntu] Scrolling input is being duplicated through minimized Electron applications.

If you have been using Slack/Chrome and VSCode for the past few weeks, you might have been experiencing the issue. Basically, when VSCode is minimized and you are scrolling in Chrome with the mouse wheel, the input is “stacked” and applied when VSCode is brought back to the forefront. When you click inside the application window, the input is immediately processed.

This causes the cursor to jump around and is a pretty big annoyance.

Ubuntu 19.10 fixes the issue but I’ve reinstalled a Debian 9.9 which still faces the issue.

You can track the following Github issue for more details : https://github.com/Microsoft/vscode/issues/28795



[Foreman – Debian – Infra as Code] RedHat Canada meetup presentation.

I was fortunate to be able to participate to the 2018 RedHat meetup in Montreal. There, I presented our effort at Lan ETS to implement and design a IaC pipeline built around Foreman, Proxmox and Ansible.

Here is the PDF, in french only for the moment.

Ansible Meetup – Lan ETS


[Debian 9] – Installing Racktables

Installing the DCIM Software Racktables on Debian 9 with Apache2 and PHP 7.0


Cisco CCNA – ICDN1 – Study notes


  • Global Unicast
    • Public IPV6 address – equivalent to public IPV4
  • Link-Local
    • Local network only IPV6 address (within one subnet) – FE80::/10 to FEBF::00/10
    • if not given an address, device will autogenerate one based on the MAC address of the interface
    • Generally used as the def-gw for most devices.
  • Unique local
    • Like private IPv4 addresses. Will not be forwarded by routers
    • FC00::/7 to FDFF::/7
  • IPv6 Multicast
    • Assigned
      • FF02::1 – All nodes multicast – all devices with ipv6 enabled will join the group. Used for RA and RS messages.
      • FF02::2 – All routers multicast –  All routers with “ipv6 unicast-routing” will join the group.
    • Sollicited node
      • Based on ethernet MAC NIC to be received by all hosts but only processed by the one with the matching MAC address.
  • SLAAC – RA message option 1
    • Stateless Address Autoconfiguration.
    • Router RA – Router Advertisement –  message broadcast – every 200 seconds – Also when receiving Router Solicitation message.
    • Contains network prefix and length, Def GW, DNS address and domain name.
  • SLAAC and DHCPv6 – RA message option 2
    • Same as just SLAAC but suggests a DHCPv6 server that feeds the DNS server and domain name information only.
  • DHCPv6 only – RA message option 3
    • Link-local for def GW.
    • DHCPv6 server for Global Unicast, DNS and domain name and everything else.
    • Cannot provide def-gw.
  • EUI64
    • Option 1 and option 2 RA.
    • Uses the MAC address to create a interface address.
    • FF:FE in the middle portion of the IPv6 address.
    • DAD – Duplicate Address Detection – Send NS if no NA received, the address is free to be used.
  • IPv6 Neighbor Solicitation – NS
    • When you know the ipv6 unicast but don’t know the MAC address of the end device.
  • Ipv6 Neighbor Advertisement – NA
    • Message replied when receiving a NS. Contains the MAC address of the sending device.

Ansible & Cisco – Automating configuration management.

Configuring network equipment has always been somewhat of a tedious affair. Copy and pasting a configuration file through the console port doesn’t scale (or you need a lot of interns!) and other solutions like Cisco Prime are slightly overkill if you only want to change a few lines of configuration.

This is where Ansible comes to the rescue. Originally built as a Linux Configuration Management Tool, in the vein of Chef/Puppet/Salt, it’s built around an SSH agent-less push model. Directly using the SSH connection, it’s remotely executing command you define in “Playbooks” This is why it’s a great fit when it comes to network devices as those are still telnet/SSH based. API/Netconf are starting to be more and more common, but SSH is still present. Especially on older network equipment.

Recently, Ansible has been augmented with a series of module that allow a network operator to leverage Ansible to deploy configuration to remote equipment. For example, if you just realized that your template is mysteriously missing NTP, Syslog and SNMP configuration and there are about 40 pieces of equipment deployed – I’m not saying this totally happened to us – Ansible is here to help.

The following playbook does the following :

  • Define a role : cisco-ios-common – which holds all the default configuration that is used by our devices.
  • In our case, the SSH connection is initiated from the sysadmin computer running the playbook – but it can be adapted to run from a bastion host.
  • No need to gather facts.
  • Create the variable holding the credentials necessary for accessing the equipment.

The actual file with the tasks is fairly simple.

This is literally lines that are parsed from the “show running-config” command and compared to each task. If it’s not there, it’s added or changed . In this case, we configure the RO SNMP community and the NTP server. To be extra careful, we also take a backup of the running-config before applying the changes. Using the “provider” statement, we are referencing the variables defined in the playbook file. All other private variables can be stored in an Ansible vault file and encrypted.

With the appropriate inventory file I was able to quickly fix our deployment mistake without manually connecting to every single switch.


Vagrant and Ansible – Building a test cluster from scratch.

Sometimes, you don’t have the luxury of a dedicated lab to test your new shiny tools. Over the past few weeks, I’ve had the need to test multiple Ansible playbooks that I’m writing. I could spin up new VM manually in VirtualBox, but it’s a tedious process and doesn’t scale when you are building a large environment. Here is where Ansible and Vagrant can be useful.

Vagrant is basically an abstraction between your virtualization provider and you. It’s designed so that you can interact with a wide variety of Hypervisors from a single tool. You can use it as a quick way to tell VirtualBox to spin up any number of VMs. It’s much faster than using the command line.

Ansible is a orchestration and configuration management tool. In the line of tools like Puppet, Chef and Salt, it’s designed as a single source of truth for server configuration and resource management. In a perfect world, nothing is managed manually and no one should have to log into a server. Everything is defined and templated within Ansible and then applied to the servers.

Here is an example of a Vagrantfile used by Vagrant in order to know what it must do.


At it’s core, it’s a simple list of instructions for Vagrant.

  1. You select a base image with the config.vm.box line. In our case, Debian 9 was selected.
  2. You set the CPU/Memory resources set per VM. You also set DNS/NAT options as well as the networking for the cluster of VMs.
  3. The interesting part is during the “Master” configuration section where I call Ansible in order to actually configure the cluster of machines. Usually, Ansible is called by the System Administrator for either a Bastion host or his own machine. In this case, the VM called “dhmtl1” will act as the Ansible master machine and actively connect to the other hosts and use the playbooks I provide.

It basically goes like this

  1. From within the directory where the Vagrantfile is located, I call vagrant up –provision
  2. Reading the instructions from the Vagrantfile, Vagrant will create three machines and set any specific option I have selected.
  3. Using the machine labeled as “dhmtl1” as the Ansible master, it’s going to install Ansible on itself and use my playbooks to dynamically apply the configuration to itself and all the other nodes. This means that once the Playbooks have finished running, I have a fully functioning cluster of machines with all of my roles deployed.

Since it’s actually running a real Debian image, I can actually fix what doesn’t work. I have since actually deployed all the Playbooks to a real infrastructure and I can report that it deployed on the first try. The playbooks were originally written for Debian 8 and had to be extensively modified to work for Debian 9. Vagrant allowed me to easily migrate the image from Debian 8 to 9, and see what broke so that I could fix it.

One other bonus, is that the testing platform is very portable. I can send the Vagrantfile and the playbooks to a coworker and he can immediately have his own environment up and running. You could even couple everything with a self-provisioning Ansible playbook that installs VirtualBox and Vagrant.

This is just a small taste of what using Vagrant and Virtualbox can alllow you to do.


Layer 2 security @ Lan ETS

The network team @ Lan ETS helps design and build the infrastructure for one of the biggest LAN event in North America. Over a weekend, over 2000 players will stress our infrastructure to it’s limits. This is some of our observations and lessons.

Ask anyone working in network security if allowing 4000 unknown and untrusted devices on your network is a good idea and you’ll get a chuckle and a “Wait, did you do that?”

This a very common challenge in LAN events, where we host over 2000 players over a 72 hours gaming session. Add wired and wifi devices and you get a large amount of strange devices entering your network. We have very little control over the type of devices that players bring and we need to have mechanisms in place to protect our network.

Here are a few of the tools and configs we use during the event.

DHCP Snooping

DHCP is the process by which a computer asks for a IP address and is given one by a DHCP server. Without a functioning DHCP server, computers connected to the network would have to be configured manually which is impossible with our type of event.

While extremely useful, DHCP can also be used by malicious attackers. The DHCP snooping mechanism allows us to prevent attackers from hosting their own DHCP server while on our network. Basically, we configure specific ports as “trusted”, usually our uplinks, as a source of DHCP OFFERS message type. Ports used by players become “untrusted” after enabling DHCP snooping. This means that an attacker hosting his own rogue DHCP server will never see his fake OFFERS reaching any other players. At the same time, OFFER DHCP messages will safely transit from our network core to the device.

DHCP Snooping does a bit more than just deal with the request portion of the DHCP protocol, but it’s a must-have configuration when it comes to any network. With DHCP being based mostly around broadcast message types, DHCP snooping allows you to cut down on traffic leaking to other ports. The less an attacker knows, the better.

Port-Security – Mac address limit

We also limit the number of MAC addresses per port. While we don’t necessarily mind players bringing several devices, we provide a single port per player for a console or a computer. This is mostly to prevent extreme cases where players bring their own switch (or hubs!) and connect all their friends. While not technically wrong, we offer up to a Gbit port to each player, we see no reason for players to not enjoy the full speed of the network.

This configuration will also prevent attackers from requesting too many IPs from our DHCP server by hosting dozens of Virtual Machines on his machine. This could technically prevent other legitimate players from obtaining leases from the DHCP server.

Here is a configuration snippet. We limit the MAC addresses per port to 2 and any extra MAC address traffic is dropped while the initial 2 can still pass through. This allows players to experience the network without any interruption even if they trip over the limit. We can then take action as we log all events and review throughout the event.

BPDU Guard

There are things you never want to hear when running a LAN event. “They cut the fiber in the ceiling” – “Someone dropped his lunch on the core router” and “I think we have a loop somewhere”. Loops in a network are messy. Due to the somewhat spam-ish tendency of modern networks with broadcast traffic, a single loop can sometimes bring the entire network to it’s knees as traffic is forwarded in a back and forth.

With over 30 kilometers of optical fiber and Ethernet cables and a setup spanning 20 or so hours, it’s easy to lose sight of where cables go. It takes a single moment of inattention to plug a cable back into the same switch it’s from or to create a loop by pulling a second uplink from the same switch.

Fortunately, there is something called “BPDU Guard” that can be used as a first line of defense against loops. A “BPDU”, or a Bridge Protocol Data Unit, is a packet used in the creation of a “Spanning Tree” topology which provides a loop free network when properly designed.

Our network features a topology where end-users are provided a single port for a single device which allows us to enable BPDU Guard on ports used by players. This way, there can be no mistakes when plugging in the equipment as a port receiving a BPDU packet will automatically shutdown. It also protects us against a player bringing his own Spanning-Tree aware equipment. This is also addressed with the MAC addresses limit per port.

More Tools and Techniques – (and the future for us!)

Here are other features that you should look into to protect your network.

  • ARP Inspection – Prevents ARP Spoofing and MITM attacks.
  • Storm-Control – Sets a level of a allowed Multicast and Broadcast traffic and shutdown a port of the limit is exceeded.
  • 802.1x – The holy grail of network security and NAC (Network Access Control). When enabled, all devices on your network must authenticate with a NAC device (Packetfence, Radius, NPS or any other flavor) before being granted access to the network. All devices must go through a guest VLAN for authentication and if successful, the port is switched into your internal VLAN. It was previously used for our event and it’s something we are looking into bringing back.
  • Private VLAN – Allows to further restrict traffic between VLANs by using secondary tags between ports.
  • IP Source Guard – Allows matching of IP per port. This means that another user cannot impersonate another IP while connected to another port.

Monitoring @ Lan ETS

Monitoring at an event of our scale is a critical part of our design process. The different monitoring services are our eyes and ears and allow us to quickly understand the state of our network. Since our event is over a very short period of time, our goal is to have a smooth experience for the player, from the moment he plugs his computer into our network. There is no worst feeling than looking at a row of players getting off their seats and being unsure of it’s a network issue!

Here is a brief overview of the different services we use to monitor our network.


A open source fork of Observium, it’s designed as a plug and play monitoring platform. You simply point it to a SNMP aware device and it will do it’s best to associate common OIDS and MIBS to “automagically” monitor the target. It then polls the OIDS every 5 minutes and updates the corresponding RRD files. It is also possible to export the data into a more modern timeseries database named InfluxDB. This allows us to access the data in a more flexible way using other services that are not compatible with the RRD format.

That said, it’s not a perfect platform. Since it’s designed around auto discovering devices, it can only detect whatever OIDS the developers configured for each platform. This means that if you are running very recent gear or equipment from obscure manufacturers, you might be missing a lot of information from the auto discovery process. There is currently no easy way to add extra OIDS to your installation.

Another issue is the polling interval as the tool was initially built around a 5 minutes window. This is acceptable in most environments but for our event, we need something that gives us almost real-time data on the state of the network. This is why we used Shinken to provide the actual Up/Down status. We rely on LibreNMS for graph and performance data.


Shinken is built as a Nagios drop-in replacement. We use it with a very short polling rate of 10 seconds with ICMP checks. We monitor only our switches, routers and servers. With around 100 or so pieces of equipment, it does tax our server resources, but it allows us for near real-time awareness of the state of our network. Within 10 seconds, we can know if a player kicked the power cord from a switch and if any other equipment is unreachable.


Nagvis was initially built as a mapping tool for Nagios. But it supports a wide variety of different back-ends as ways to fetch data. We use it in conjunction with the “Livestatus” module for Shinken. We upload a map of the venue with all of our equipment displayed. This allows us to quickly know if everything is up and running and at the same time, react if anything turns red.


Oxidized is a modern replacement for Rancid. It’s a network equipment configuration backup tool. It logs into a piece of equipment using credentials you provided and runs any commands you want. In the case of Cisco based gear, a “enable” followed by a “show run”. It then stores the results in a git based structure and provides a web interface to compare different backups. It allows us to backup our configurations throughout our design process. We usually don’t run it during our event as we do not change (hopefully…) any configurations during the weekend.


We have just recently started using Grafana as a visualization tool for our performance data. It uses a InfluxDB data-source to display data in a much more friendly and dynamic way than the RRD format. It’s an excellent tool to display interface stats and traffic.


Splunk is our log aggregation tool. We use it as a central collection point for network equipment and server logs. We use it to create dashboards to highlight important events such as port-security events or unauthorized login attempts against our network. By default, most equipment logging can be very verbose and it’s important to properly filter events to pinpoint what is important.


[Ubuntu 16.04] [kubeadm] An easier way to spin up Kubernetes clusters

Over the past few days, I’ve been experimenting with Kubernetes – also known as k8s – as a way to scale applications containers. Kubernetes is a cloud-ready platform developed by Google and is seen mostly in providers clouds like GCE and AWS. There are ways to deploy Bare-Metals cluster but it’s a mix and match of different tools without a specific one being in the lead. For hosting on AWS, “kops” is a well maintained and production ready tool. For GCE, google already offers the option to deploy K8s clusters.

For bare-metal clusters, a tool named “Kubeadm” is currently being worked on by the community/Google in order to provide a quick and secure way to deploy a K8s cluster. It’s currently in alpha and is not quite ready for production as it cannot deploy HA clusters just yet. But it’s still is an awesome way to spin up a quick test cluster to play around with.

I’ve recently built a few custom Ansible playbooks in order to leverage the automation platform to remotely build my clusters without ever logging into a server.


To use it, simply follow the following steps!

  • Install Ansible
  • Install Make
  • Edit the k8s-inventory file to reflect your machines.
  • From the root folder – use the following syntax : make k8s-*TAB for auto completion*
  • I recommend deploying the “k8s-common” role first as it ensures is setup to support the kubernetes components.