災害の̴女王dreamspace

LXC on Arch Linux

i’m no stranger to containers, but LXC isn’t something i’ve used for production workloads yet. its about time i changed that. you see, LX containers are much closer to LX zones than their docker-based counterparts. i’ve done a bunch of work to migrate sunshine gardens from docker to more traditional unix service admin using openrc. the next step is to consolidate everything on my home server, with the final goal being to replace the linux host with illumos.

so.. to start with let’s get some linux containers up and running! there are a lot of moving of moving parts involved in the infrastructure i run so i’m not going to worry about changing the host system just yet. i’m already running zfs so it won’t be much extra work to move some data around once i have everything copied off of vultr. for now, i’m just going to focus on setting up some LX containers running alpine linux 3.11.

first thing’s first, let’s compile lxd so we don’t have to interface with lxc as directly:

yay -s lxd

if you come from docker land you might expect the next couple of commands will be all she wrote setup-wise.

lxd init
systemctl enable lxd
systemctl start lxd

sadly, i did not find this to be the case. launching containers fails with fun cryptic errors like:

[nexus ~]# lxc launch images:alpine/3.11
creating the container
container name is: internal-badger
starting internal-badger
error: failed to run: /usr/bin/lxd forkstart internal-badger
/var/lib/lxd/containers /var/log/lxd/internal-badger/lxc.conf:
try `lxc info --show-log local:internal-badger` for more info
[nexus ~]# lxc info --show-log local:internal-badger
name: internal-badger
location: none
remote: unix://
architecture: x86_64
created: 2019/12/30 01:29 utc
status: stopped
type: persistent
profiles: default

log:

lxc internal-badger 20191230012924.418 error    start - start.c:proc_pidfd_open:1644 - function not implemented - failed to send signal through pidfd
lxc internal-badger 20191230012924.423 error    conf - conf.c:lxc_map_ids:3008 - newuidmap failed to write mapping "": newuidmap 26939 0 1000000 1000000000
lxc internal-badger 20191230012924.423 error    start - start.c:lxc_spawn:1798 - failed to set up id mapping.
lxc internal-badger 20191230012924.424 error    lxccontainer - lxccontainer.c:wait_on_daemonized_start:872 - received container state "aborting" instead of "running"
lxc internal-badger 20191230012924.424 error    start - start.c:__lxc_start:2036 - failed to spawn container "internal-badger"
lxc internal-badger 20191230012924.507 error    conf - conf.c:lxc_map_ids:3008 - newuidmap failed to write mapping "": newuidmap 26953 0 1000000 1000000000 1000000000 0 1
lxc internal-badger 20191230012924.507 error    conf - conf.c:userns_exec_1:4410 - error setting up {g,u}id mappings for child process "26953"
lxc internal-badger 20191230012924.508 warn     cgfsng - cgroups/cgfsng.c:cgfsng_payload_destroy:1112 - failed to destroy cgroups
lxc 20191230012924.509 warn     commands - commands.c:lxc_cmd_rsp_recv:134 - connection reset by peer - failed to receive response for command "get_state"

clearly there is some more setup we need to do. if you note the line that says newuid failed to write mapping it sounds like lxc is trying to use CONFIG_USER_NS which we haven’t enabled yet. there are some security concerns with user namespaces so we’re going to make some changes to the default profile insead.

linux containers are not security boundaries. there are security concerns on both sides, but the security we are getting is flimsy at best. in this deployment we are using containers as administrative units only. hardening the system will happen as part of the host migration to illumos which provides secure containers in addition to administrative separation.

lxc profile edit default

using $EDITOR modify the default profile to include:

config:
  security.privileged: "true"

now we can set up a simple pastebin!

lxc launch images:alpine/3.11 paste
lxc exec paste -t /bin/ash
paste# apk update
paste# apk upgrade
paste# apk add chicken
paste# chicken-install awful pastiche

you’ll get a bunch of compiler output. wait for that to finish and then write this small scheme program:

; demo.scm
(import awful pastiche)
(pastiche "/" "/srv/paste/paste.db"
   vandusen-host: #f
   awful-settings:
   (lambda (handler)
    (parameterize
      ((debug-file "/tmp/paste")
       (page-css "http://wiki.call-cc.org/chicken.css")
       (page-charset "utf-8")
       (page-doctype "<!doctype html>"))
      (handler))))

we could compile this program, but there’s no need for the purposes of this demo. next, let’s whip up an open-rc script to run this service for us:

#!/sbin/openrc-run
depend() {
    use dns
}
command=/usr/bin/awful
command_args=/srv/paste/app.scm
command_background=true
command_user="nobody:daemon"
pidfile="/run/${rc_svcname}.pid"

maybe talk about nginx forwarding bullshit, dnsmasq setup, setting that no stub thing on systemd-resolved, updating resolve.conf and protecting it.

make sure /srv/paste belongs to nobody:daemon set +x on /etc/init.d/paste and run rc-service paste start. now you can ^D out of the container console and run lxc list to get the ip address you should visit to see your pastebin in action.