AD-in-a-Box for PoCs and IoCs on the cheap: DetectionLab on LibVirt

Why?

• You.team == blue || red || purple || rainbow;
• You.Need_Ad_Lab == true;
• You.Tired_Of_Rebuilding_Said_Lab == true;
• You.Ran_DCPromo_Count >= 9000;

What?

TL;DR Build DetectionLab on LibVirt/KVM. Readme here. Blabla & guide below.

Blabla

I’m tired of rebuilding my AD lab. Everytime, I plan it out so this time, I’ll do it right. Snapshots in the right place, a “barebone” version, a version with some extra software, snapshots per projects, etc.

Then comes an all-nighter, a project that’s a little bit tight in time, or anything that pushes me to a “one-time” exception to my self-imposed standards. I forget about it for a few weeks and here comes the pain. Snapshot hell, expired images, my dev box is now fully cross-contaminated, etc…  All in all, starting fresh looks better than cleaning up. And THIS TIME, I’ll have some self-discipline and not end up with that mess where the only sensible solution is starting from scratch.

From the day I started pentesting I dreamed about a reusable AD lab I could just throw away and rebuild by the snap of my fingers. A few years ago I experimented with Packer.io to roll out my own Windows images and learned the hard way just how much DevOps can eat your time. It started great then turned into a blackhole devoring my will to live as I watched my Windows boxes failing provisioning one after the others. It was better to just bite the bullet and ++You.Ran_DCPromo_Count… 

Then came DetectionLabs!

DetectionLab in a Nutshell

The concept is simple and elegant. Following a modern DevOps pipeline, it builds a reusable AD lab. But not any AD ab! One packed with pre-configured goodies such as OSQuery, Splunk, Sysmon, Microsoft ATA, and many more tools to help drill down the latest TTPs to the nits and bits or making sure your implant can go through an all-seeing Sauron-Eye style blue team. Type vagrant up, snap your finger, press enter, et voila! AD Lab!

There’s a tremendous amount of work put in this project. I’ve been keeping an eye on it for a while now, and it has come a very long way from what it used to be. DevOps is hard, frustrating, and time consuming. The first time I tried it, I was facing the same provisioning issues I had back then, but when I dusted it back in the spring of 2019, I was so happy that everything just worked! There’s pre-built images too so you can just vagrant-up without going through packer.

I’ve been developing a branch that supports LibVirt/KVM in order to make some use of an old desktop with good specs that gathers dust. The result is a minimal Linux computer running DetectionLabs. No VirtualBox, no VMware, all headless, using official packages and KVM. I’ve also found that while the learning curve of LibVirt much steeper than, say, VirtualBox, you have much more control over your VM. You can use and fine-tune all the features of QEMU-KVM under the hood. I’m sharing this in the hopes all these hours put into the LibVirt provider for DetectionLab will benefit someone else.

DetectionLab-LibVirt OverView

This branch uses the same pipeline as the master branch. However, there’s no build.sh script and no pre-built images. The first run is quite time consuming as you have to provision your Windows golden image with Packer, but you can do this only once every few months or when there’s a new release.

Moreover, you need a somewhat beefy system. The official spec requirement for DetectionLab is barely enough. More cores is better than few powerful cores. More RAM will help too. Have an old desktop with these AMD-FX CPUs? Buy some DDR3 for cheap on Ebay. On my end, 24Gb and an AMD FX8350 worked like a charm. Now, I’m using a server with low-powered Xeon chips that I built for cheap buying second-hand components on Ebay. /r/homelab is your friend.

Also, keep in mind that this is not officially supported by the DetectionLab project. It’s something that fills a very niche need using FOSS v.s. proprietary solutions like ESXi & such. Again, FOSS is as free as your time is. The following guide is somewhat general, and although it’s for Ubuntu 18.04, I had a much more streamlined experience setting the whole thing up on ArchLinux.

Building from Scratch

LibVirt Hypervisor

Full disclaimer: This is not intended to be a secure configuration. Do not expose this on the internet.

This section is a walkthrough on how to get LibVirt setup properly on a  Ubuntu host.

At the end, you’ll have a remote Hypervisor you can access with Virt-Manager, which is available on MacOS, Windows & Linux.

Starting clean with a Ubuntu Bionic image:

1. Create a  detectionlab user (useradd -m detectionlab).
2. Add the user to to sudo & libvirt group (ex.usermod -a -G sudo detectionlab; usermod -a -G libvirt detectionlab)
3. Ensure you have the proper public key in (~detectionlab/.ssh/authorized_keys). This will be used later for remote Virt-Manager connection
4. Install LibVirt. On Ubuntu Bionic, this set of packages is sufficient: apt install -yqq bridge-utils libvirt-bin qemu-kvm fish netcat-openbsd. IMPORTANT: Use the netcat-openbsd version. Not the GNU one!
   
5. Check that everything looks good for virtualization:
   #virt-host-validate
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking if device /dev/kvm exists : PASS
QEMU: Checking if device /dev/kvm is accessible : PASS
QEMU: Checking if device /dev/vhost-net exists : PASS
QEMU: Checking if device /dev/net/tun exists : PASS
QEMU: Checking for cgroup 'memory' controller support : PASS
QEMU: Checking for cgroup 'memory' controller mount-point : PASS
QEMU: Checking for cgroup 'cpu' controller support : PASS
QEMU: Checking for cgroup 'cpu' controller mount-point : PASS
QEMU: Checking for cgroup 'cpuacct' controller support : PASS
QEMU: Checking for cgroup 'cpuacct' controller mount-point : PASS
QEMU: Checking for cgroup 'cpuset' controller support : PASS
QEMU: Checking for cgroup 'cpuset' controller mount-point : PASS
QEMU: Checking for cgroup 'devices' controller support : PASS
QEMU: Checking for cgroup 'devices' controller mount-point : PASS
QEMU: Checking for cgroup 'blkio' controller support : PASS
QEMU: Checking for cgroup 'blkio' controller mount-point : PASS
QEMU: Checking for device assignment IOMMU support : PASS
QEMU: Checking if IOMMU is enabled by kernel : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
6. Start the libvirt daemon systemctl enable libvirtd ; systemctl start libvirtd
7. On the computer with Virt-Manager installed, check that you can remotely connect to your LibVirt hypervisor. Here’s how it looks on my end (replace with the real IP): virt-manager -c 'qemu+ssh://detectionlab@*IP_HERE*/system?socket=/var/run/libvirt/libvirt-sock'

You should have something like this: 

Side note: There’s a quote that goes like “Most computer problems can be solved with another layer of abstraction”. In order to have a clean setup each time, and a somewhat “generic” set of hardware, I run the LibVirt daemon in an LXD container. This is subject to a future post as there was some minor hurdles along the way, but hey! I got an on-demand,  clonable, portable hypervisor in a CoW container, which I think is pretty cool. It’s hardly more work to setup once you are familiar with LXD, but I ran into a couple roadblocks with AppArmor.

Building Windows Golden Images Using Packer

1. Checkout the LibVirt branch of DetectionLab.
2. Download the latest stable VirtIO drivers to ./DetectionLab/Packer/virtio-win.iso
3. Install packer using your distro’s package manager
4. Modify the packer_build_dir variable in ./DetectionLab/Packer/windows_10.json and ./DetectionLab/Packer/windows_2016.json. The prefix path should point to a valid directory with Read/Write access  (ex. sudo mkdir /media/packer_build_dir; sudo chown detectionlab:detectionlab /media/packer_build_dir -R)
5. If you are running libvirt in a headless server (no Xorg, Wayland etc.), make sure that the headless option is set in both .json files for the LibVirt provider. Otherwise Packer will fail. See below on how to use a VNC client in headless mode.
6. On Ubuntu, the user running packer needs to be in the KVM group: usermod -a -G kvm detectionlab. This allows FULL CONTROL of the /dev/kvm device to the user. There’s probably a good privesc or two there. On Arch, I think any user in the libvirt group can use the KVM interface.
7. Build! As the detectionlab user: env TMPDIR=/media/packer_build_dir PACKER_LOG=1 PACKER_LOG_PATH="packer_build_2016.log" packer build --only=qemu windows_2016.json; env TMPDIR=/media/packer_build_dir PACKER_LOG=1 PACKER_LOG_PATH="packer_build_10.log" packer build --only=qemu windows_10.jso.

If everything goes according to plan, you should end up with two .box files in the Packer directory. This WILL take forever. On my beefier FX8350 it’s about an hour. On my low-power Xeon, about twice as long.

Note: If you are using BTRFS for the hypervisor’s filesystem, make sure you disable CoW on directories containing the QCOW Images! See this for more info. (sudo chattr +C on /var/lib/libvirt/images, ./DetectionLab/Boxes, /media/packer_build_dir)

Move them to ./DetectionLab/Boxes

Note: If you want to see what’s going on during the build process, but want to keep things minimal and not install Xorg/Wayland/etc., you can use a VNC client to connect to QEMU. Just forward port 5997 to a computer with a VNC client: ssh -fNT 5997:localhost:5997 detectionlab@<libvirt_server_ip>

Setting Up Vagrant

All commands below, except when requirement packages, should be ran as the detectionlab user.

Install Vagrant plugins:

vagrant plugin install vagrant-reload vagrant-libvirt vagrant-winrm-syncedfolders

If on Ubuntu 18.04:

1. Do NOT install vagrant from the official repository. It comes with a whole lot of outdated plugins that will cause problems futher down. Grab the .deb from Hashicorp’s website.
2. Install theruby-dev and libvirt-dev packages
3. You should be good to go to copy-paste the line above.

Note: If you are using BTRFS for the hypervisor’s filesystem, make sure you disable CoW on the Vagrant local directories (sudo chattr +C on ~/.vagrant.d/boxes, ~/.vagrant.d/tmp)

Import the Vagrant boxes:

vagrant box add windows_10_libvirt.box --name windows_10_libvirt
vagrant box add windows_2016_libvirt.box --name windows_2016_libvirt

Creating a Lab From Scratch

Go in the ./DetectionLab/Vagrant directory, and get some snacks, a coffee, Netflix ready, and run:

vagrant up --provider libvirt --no-parallel --provision

This is how you get a new lab, any time you want. As long as you have the vagrant plugins installed, LibVirt configured, and the Windows Golden Images .box imported, you can run this, almost worry free, and get a fresh Windows lab.

Here’s the hurdles that will happen when provisioning a new lab, because the LibVirt provider is not perfect.

Hurdle #1, Provisioning Stuck

You will probably encounter a stuck provisionning and/or some errors during the first vagrant up. The TL;DR is to shutdown all VMs, reboot them manually, shut them down and vagrant up --provision --provider libvirt --no-parallel again.

These issues seems to happens because of the unique and peculiar way the LibVirt provider works. It needs two network adapters. I’ve done quite a bit of mental gymnastics to get around this, and if anyone have a solution I’m a taker, but this solution seems the lesser evil and, most importantly, doesn’t break DetectionLab. The creation of the two network adapters is handled by the VagrantFile, which you can validate by either going in the network properties of your QEMU/KVM connection in Virt-Manager or using Virsh on the Hypervisor host:

virsh # net-dumpxml Vagrant0
<network connections='3' ipv6='yes'>
  <name>Vagrant0</name>
(...)
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
(...)
  <ip address='192.168.38.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.38.1' end='192.168.38.254'/>
    </dhcp>

virsh # net-dumpxml VagrantMgmt
<network connections='3' ipv6='yes'>
  <name>VagrantMgmt</name>
  <uuid>550cddb6-8f30-45fc-8927-b3c9f21a6b7f</uuid>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:86:1d:a6'/>
  <ip address='192.168.123.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.123.1' end='192.168.123.254'/>
    </dhcp>
  </ip>
</network>

The LibVirt provider is quite unique in that it requires a “Management” network interface to work. Under the hood, it uses DHCP leases to know which hosts to contact when using Vagrant. DetectionLab is designed to have a single NIC, but fortunately having a second one doesn’t break anything.

When Vagrant is stuck during provisioning, it means there was an issue with the NIC static IP configuration. You don’t have to do anything, just shutdown the VM, boot it (from Virt-Manager), check that the IP is properly assigned (list of IPs on the 192.168.38.0/24 network is on the DetectionLab official Readme). Then shutdown again and re-run the provisioning.

DetectionLab-LibVirt related issues

Please report any issues to the unofficial LibVirt fork until it’s merged upstream.

Advertisement