Edits

April 17: Fixed an error with the nginx config for use with LiveViews

Motiviation

So recently I’ve wanted to give containers for development another go, but this time instead of using docker for Mac and it’s very slow file sharing to the containers give podman a try, since we are also using podman in production.

While the default mechanism using
podman machine to run containers on Mac did in fact work out fine, and sharing my local development folder with the containers worked fine, it also proved to be a rather slow experience. That wasn’t a realy surprise given that podman machine uses QEMU in the background.

However what got me thinking is that podman remote allows us to use any arbitrary Linux machine as the backend, so shouldn’t it be possible to use a VM in Parallels instead?

This turned out to be possible, but a little bit complicated to setup, so what follows is a (hopefully) complete guide to setting this up.

Install VM

For this guide I am assuming that you already have Parallels installed on your local development machine.

So as a first step install podman on your machine, personally I prefer to use brew for this:

brew install podman

Next get a Linux to install as the VM in Parallels. You should be able to use any Linux, personally I’ve decided to use Fedora, simply because podman machine uses this as well. However I’ve decided to use the server version and not CoreOS, which is used by podman machine. The reason for this is that setting up CoreOS is a bit more involved, and that we need to install and run nginx in a later step as well, and I don’t know how much effort it is to get that running in CoreOS.

Now create a new VM in Paralles and choose to install from the image you’ve just downloaded. Give the VM an easy name, I personally went with podmanhost, and check the “Adjust settings before installation” checkmark before continuing with the dialog.

Settings for VM

In the settings dialog that pops up after clicking the Next button, again make sure that the name is easy. Under settings I prefer the following settings (my Parallels is not running in english, so I’ll try to translate the labels of the settings to show the correct meaning):

Setings -> Start and turn off -> Automatically start in background

Setings -> File Sharing -> Share defined folders with Linux -> Select either your home folder or the folder where your projects live 
Setings -> File Sharing -> turn off "Share Cloud folders"
Setings -> File Sharing -> turn off "Share Mac-Volumes"

Setings -> Programs -> turn off all checkboxes 

Hardware -> 4 CPU`s, 4 GB RAM

Hardware -> Printer -> turn off all checkboxes

Hardware -> Network -> Shared network (as recommended)

Hardware -> Audio and Camera -> Do not share Camera with Linux

Then close the settings dialog and continue with the installation. Once the VM boots up, choose the option Install Fedora. Once the installer of Fedora has launched, open the menu for selecting the required software. In there, choose the minimal installation option on the left, and add container managament on the right side. Alternatively you can install podman later once the server is installed. Next choose the installation target, I’ve simply chosen to let the installer setup everything automatically here. For the root account I’ve decided to leave it deactived, since we do not need it (I plan to run the containers in rootless mode), and finally we need to add normal user. I’ve named the user podman so that it is easier to remember later 😀.

Now we can start the actual installation and wait for it to complete, and then reboot the VM. Next I usually like make sure that all packages are up to date and that vim is installed, so run:

VM ># sudo dnf upgrade
VM ># sudo dnf install -y vim 

Configure ssh with public key authentication from the host

Now in order to work more comfortably with the VM, I like to install a ssh key into the VM. In order to be able to copy it to the VM, allow password logins in ssh for now (if you do not know how to use vi or vim, just use any other editor):

VM ># sudo vim /etc/ssh/sshd_config

# and enable both of these lines:
PubkeyAuthentication yes
PasswordAuthentication yes

Afterwards reload sshd and try to login. You should be able to use the name you’ve given the VM in Parallels to connect, since Parallels should add an entry in your /etc/hosts file with the name of the machine.

VM ># sudo systemctl reload sshd

# from your Mac
MAC ># cat /etc/hosts

# note: options should only be required if you are doing this for the 
# n-th time to write a blog post on how to setup everything you've 
# done like 5 times already :grinning:
MAC ># ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password podman@podmanhost

Next generate a key, setup the .ssh directory on the VM and copy your public key to the VM.

# generate key and copy to clipboard
MAC ># ssh-keygen -t ed25519 -f ~/.ssh/podmanhost.shared
MAC ># pbcopy < ~/.ssh/podmanhost.shared.pub

# create ssh directory on VM and copy your public key into it (assuming you 
# are connected using ssh)
VM ># mkdir ~/.ssh
VM ># chmod 700 ~/.ssh
VM ># vim ~/.ssh/authorized_keys

Then check if you can now connect to the VM from your Mac using the ssh key:

MAC ># ssh -i ~/.ssh/podmanhost.shared podman@podmanhost

# if you run into the problem that you get a "Too many authentication failures"
# disconnect, try this (again, if you've been doing this for 5 times 
# already and are writing a blog post :roll_eyes:)
MAC ># ssh -i ~/.ssh/podmanhost.shared -o IdentitiesOnly=yes podman@podmanhost

If this works as expected, add an entry to your ~/.ssh/config file so that you can reconnect more easily later again:

Host podmanhost
PubkeyAuthentication yes
IdentityFile ~/.ssh/podmanhost.shared
User podman

And finally log into the VM again and disable password authentication again. One could argue if this is really needed since this is a just a local VM, however I prefer to never allow to let this setting slip so that this does not happen on a production machine by accident (I never allow password authentication on production machines!).

VM ># sudo vim /etc/ssh/sshd_config

# and disable this line:
PasswordAuthentication no

Again verify that you can still log into the VM from your host machine, and if that is all running fine we are finally done with the first steps.

Disable firewall

Next we are going to disable the firewall. Again this is something one should never ever do in production, but it is fine for a local development machine and makes our live easier later on with all the port forwarding that is going to be required.

VM ># sudo systemctl stop firewalld
VM ># sudo systemctl disable firewalld
VM ># sudo systemctl status firewalld

Install Parallels guest additions

Next we need to install the Parallels guest additions, so that we can use the folder sharing from the Mac to the VM, and later on into the container. The first thing we need to do for this is to insert the guest additions image into the VM.

At the top of the VM window click on the CD button -> Add image. The image we are looking for is located under Applications -> Parallels Desktop -> Contents -> Resources -> Tools -> prl-tools-lin-arm.iso. Check that the image has been inserted by clicking on the CD button again.

Next we need to switch to the VM and install a couple of libs and mount the image:

VM ># sudo mkdir -p /media/cdrom
VM ># sudo mount -o exec /dev/sr0 /media/cdrom
VM ># sudo dnf install -y gcc kernel-devel.aarch64 kernel-headers.aarch64 make checkpolicy selinux-policy-devel
VM ># sudo /media/cdrom/install

Follow the dialogs of the installer and wait for it to complete. Afterwards we need to reboot the VM:

VM ># sudo reboot

Now we should be able to verify the installation of the guest additions by checking if the shared folders have been mounted into the VM. If so, there should be a folder in /media called pfs, containing all the shared folders.

VM ># ls /media/psf

If the install was successfull, we can eject the image from the VM and continue with the next step.

VM ># sudo umount /media/cdrom

Optional: Disable SELinux

By default Fedora uses SELinux, and containers aren’t able to access the volumes mounted into them. You can either pass the option --security-opt label=disable to your containers when mounting them, or alternatively you can simply turn off SELinux altogether. Again this is a no go on any production machine, but fine on your local development machine. Just don’t ever mix them up 😎.

VM ># sudo vim /etc/selinux/config

# Change this line to SELINUX=disabled
SELINUX=disabled

VM ># sudo reboot

Setup podman from Mac to VM

For this we basically follow the manual from redhat.

First we need to setup podman on the VM so that it can be remoted controlled. Log into the VM and run these commands:

# __DO NOT__ run this as sudo
VM ># systemctl --user enable --now podman.socket

# now run as sudo again
VM ># sudo loginctl enable-linger $USER

Now we should be able to setup the podman remote connection from our Mac host:

MAC ># podman system connection add podmanhost --default --identity ~/.ssh/podmanhost.shared ssh://podman@podmanhost
MAC ># podman system connection list

Now it’s time to test if the remote connection and podman is working:

MAC ># podman info 
MAC ># podman pull docker.io/hello-world
MAC ># podman run --rm hello-world

If this prints out a hello world message, everything is working and we can remove the image again:

MAC ># podman image rm hello-world

Setting up Elixir development using shared folders

Configure port forwarding in Parallels

We’ll use the default ports for developing Elixir applications, so open the Parallels settings and in Network -> Shared add a rule to forward port 4000 to podmanhost to port 4000.

Later on we’ll need to modify this forwarding, but we’ll have a closer look at what to do when we then make the necessary additional setup to serve static files from the Elixir project.

Get a development container

Now it’s time to load your development container into podman. This will probably contain come additional tools like a gcc compiler etc, I might get back to how I prefer to setup my development container in a later blog post. For now we’ll simply use the (at the time of writing) current Elixir container. Note that we are not using otp-26 here, at the time of writing this there is a bug where something fails, but I can’t remember what it was.

MAC ># podman pull docker.io/elixir:1.16-otp-25

# Note that we are forwading from port 80 to 4000 here, that will change in a later setup
# --security-opt label=disable is required so that the container can access the volume with SELinux present
MAC ># podman run --name elixir_dev -p4000:4000 --security-opt label=disable --volume=/media/psf/podman/testing:/src -it elixir:1.16-otp-25 bash

Some note on the run command, first off the volume must point to the location of the shared folder in the VM, not on your Mac, since the container is running inside the VM. Also note that instead of using the --security-opt label=disable you can also simply turn off SELinux completely, as described above.

After running the command you should be dropped into a bash and check the contents of the /src directory, which should in our case be the contents of the shared folder /podman/testing.

For testing we will now create a new phoenix project without ecto here (more on how to setup a postgres connection later in this post), and try to compile and run it.

CONTAINER ># cd /src
CONTAINER ># mix local.hex --force
CONTAINER ># mix archive.install hex phx_new 1.7.11 --force
CONTAINER ># mix phx.new --no-ecto .

If you want you can time the download of the dependencies and the compilation time, and compare it to your native perfomance.

CONTAINER ># cd /src
CONTAINER ># time mix deps.get
CONTAINER ># time mix compile

Now make sure to change the config of the Endpoint so that it listens to external connetions as well. Open /src/config/dev.exs and set the ip address to listen at to {0, 0, 0, 0}, or what is required depending on the http server backend you are using.

Now we should be able to run the server and make a connection. If you cannot connect, try to debug it by executing another bash on the container, then try to make the connection from the VM and finally from your Mac.

MAC ># podman exec -it elixir_dev bash
CONTAINER ># wget http://localhost:4000
VM ># curl http://localhost:4000
MAC ># curl http://localhost:4000

Unable to serve static files

Next you can try to download a static file like app.js from the Elixir server. If this works in your case, great! You are done and can either start developing or skip down to the section where we setup postgres in a container as well. However in my case this did not work because for some reason the connection is always closed prematurely.

MAC ># curl -O http://loaclhost:4000/assets/app.js

# output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  764k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (18) transfer closed with 783282 bytes remaining to read

Investigating this closer revealed a couple of interesting findings:

1. When we serve the static files from a location on the containers own filesystem, the problem goes away and the files are received. However this would require us to have our working directory inside the container where it would not be accessible from the Mac, which is precisely what I do not want for my development setup. Here I’d like to use the editor etc installed on my Mac, and only use Elixir from the container.
2. Trying the same from within the VM does not help either, the same error occurs.
3. Using a different Linux variant like Ubuntu does not solve the problem.
4. Installing another http server, in this case nginx, and trying to serve the files from the shared directory gives the same error.

However after some more trying around I did find out that setting the sendfile option of nginx to off does solve the problem, and I can download the files stored on the shared folder from the container.

Digging in deeper, if sendfile is set to off in nginx, the files are written directly from the file system into the outgoing device on the container, instead of first being buffered in memory by nginx. That set me off on a long quest through the source of Erlang to try to find if there is a similar option, however I was unable to locate anything that would solve the problem.

Close to giving up, I got an idea on how we can still make this work, which I’ll describe in the next section.

Using nginx to serve static files and Elixir for everything else

So the solution I came up with is this: In the build environment, the static files are served directly from the priv/static folder, by creating a symlink in the _build/dev/lib/myapp/ folder back to priv/static/.

Now we know the location of those folders in the VM, so instead of directly serving them from Elixir, we can setup a reverse proxy html server in the VM. That server can then serve all the static files, and forward all other calls to the container for processing by Elixir.

So in order to do this, a couple of more steps are required. Also note that I only tested this so far using the static files. If you need to serve more files from a different location that are stored on a shared folder from your host Mac, you will need to adjust the nginx configuration again for this. Also if you add such a mechanism in the future, you’ll need to adjust the config as well. This is not optimal, but should be manageable.

Installing and configuring nginx

Change port forwarding

Since we are going to use a reverse proxy, we’ll need to change the way we handle the port forwarding. Instead of forwarding from port 4000 from the Mac to port 4000 of the VM, and then again from port 4000 from the VM to port 4000 of the container, we’ll instead use the default http port 80 on the VM.

So change the Parallels port forwarding configuration in Parallels settings -> -> Network -> Shared. Change the forwarding rule to forward from port 4000 to port 80 on the podmanhost.

Install nginx

Next we are going to intall nginx on the server.

VM ># sudo dnf install nginx
VM ># sudo systemctl enable --now nginx

# check if install is successfull 
MAC ># curl http://localhost:4000

The later check should now fetch the default nginx landing page.

Disable SELinux

In order for nginx to be able to read the files from the shared folder we are going to disable SELinux like described in the section above.

As described above this is fine for our local development VM, but if you feel unsure about this and know more about how to work with SELinux feel free to leave it enabled and to grant nginx read permissions on the shared folders.

Configure nginx

Now we need to configure nginx as a reverse proxy and to serve the static files of our project. For simplicity, given that this is only our development server, we are going to configure everything directly inside of the main nginx.conf file.

VM ># sudo vim /etc/nginx/nginx.con

First and most importantly, we need to swicht off sendfile, so that nginx can serve the static files at all.

# switch this line to off
sendfile off

Then in the server section, update the contents so that they look like this:

EDIT April 17: There was an error in the nginx config below where forwarding from the try_files was not working with LiveViews. You must use a named location, and in the named location the proxy_pass line must not end with a dash.

server {
    listen       80;
    listen       [::]:80;
    server_name  _;

    root         /media/psf/podman/testing/priv/static;

    location / {
            try_files $uri @elixir;
    }

    location @elixir {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwared-For $remote_addr;
            proxy_set_header Host $host;
            proxy_pass http://127.0.0.1:4000;

            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
    }

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;
}

And reload the config into nginx, after making sure that it is valid:

VM ># nginx -t
VM ># sudo systemctl reload nginx

Now we should be able to fetch a file from our Mac:

MAC ># curl http://localhost:4000/robots.txt 

Afterwards make sure that your develpment container is running, and check that the forwarding is working as well.

MAC ># podman start elixir_dev --attach

CONTAINER ># cd /src
CONTAINER ># iex -S mix phx.server

MAC ># curl http://localhost:4000

And finally I would recommend to nagivate to http://localhost:4000 in your browser to make sure that the html, Javascript and css files are all loaded correctly. I.e. check that the website renders as intended.

Congratulations

Finally setting up the development environment has been fully setup 💯.

If you are interested in how to use postgres in this setup as well, continue on reading 🧐.

Setting up Postgres in a container for development

Basically in this section we will start the standard postgres container, and that’s it. Again I personally prefer if the DB is stored on a shared folder, so that I can more easily move the files around or switch to a different development setup in the future.

Optional: Shut down your local postgres instance

First we’ll dump any of our old databases from the local instance:

MAC ># mkdir -p ~/tmp/db 
MAC ># cd ~/tmp/db
MAC ># pg_dump your_db > your_db.sql

Next we can shutdown postgres:

MAC ># brew services stop postgresql 

Prepare the database directory

Create a directory where you can store your development databases:

MAC ># mkdir -p ~/tmp/podman/postgres
MAC ># chmod 700 ~/tmp/podman/postgres

Setup port forwarding for postgres

Since we are going to run our only instance of postgres in the container, we’ll use the default postgres port.

Parallels -> Network -> Shared add a rule to forward port 5432 to podmanhost to port 5432.

Create a named podman network for dns lookup between containers

I prefer to be able to use DNS names for communicating between the containers, so we’ll create a new network for our containers. As an alternative you can also consider enabling dns in the default network.

MAC ># podman network create development
MAC ># podman network inspect development

Pull the desired postgres image and initialize it

We’ll use one of the official postgres images from docker:

MAC ># podman pull docker.io/postgres:16

Next we run it for the first time, initializing the database. Note that in the command below I’ve ommitted the security-opt label=disable option, because we’ve disabled SELinux altogether when setting up the nginx server. If you did not do that, just add the option in the commands below.

For the database folder, we’ll simply mount that to the default location of the postgres data directory.

Notice that we need to pass the userns=keep-id:uid=999,gid=999 option in order for postgres to be able to access the folder.

Finally we’ll also pass the -it flag for interactive mode, so that we can more easily debug any issues that might appear during setup of the database, and instruct the container to use our new network and give it a descriptive DNS name.

podman run --name postgres -e POSTGRES_PASSWORD=password --volume=/media/psf/podman/postgres:/var/lib/postgresql/data --userns=keep-id:uid=999,gid=999 --network=development --hostname=postgres.localhost -p5432:5432 -it postgres:16

If there are no erros, stop the container again using Ctrl-C, and then start it again. Also we can check if we can access the server.

MAC ># podman start postgres
MAC ># psql -h postgres.localhost 5432 -U postgres

If this works, we can continue with the further setup steps.

Create development user and allow connections

In my setup I prefer to use a separate user for connecting from the Elixir apps, instead of using postgres. We’ll use the createuser tool for that.

MAC ># podman exec -it postgres bash

# in our example we'll use "phoenix" as the password as well
CONTAINER ># createuser -d -P phoenix

Next we’ll trust all connections inside our local development network. Note that this again is a step you should not be doing in production, but instead use a more restrictive setup there.

MAC ># podman stop postres
MAC ># vim ~/tmp/podman/postgres/pg_hba.conf

In the pg_hba.conf file, add this line:

# podman network connections
host    all             all             10.0.0.1/8            trust

Restart the postgres container afterwards, and check if we can connect as the phoenix user:

MAC ># podman start postgres
MAC ># psql -h postgres.localhost -U phoenix postgres

Create a new Phoenix project with Ecto

For this we’ll remove our current container for development of Elixir projects, and recreate it using the new development network. Afterwards we’ll delete the old testing project and create a new one for testing.

MAC ># podman rm elixir_dev

# Note we've left out the security-opt label=disable option, since we've turned off SELinux
MAC ># podman run --name elixir_dev -p4000:4000 --volume=/media/psf/podman/testing:/src --network=development --hostname=elixir_dev.localhost -it elixir:1.16-otp-25 bash

CONTAINER ># cd /src
CONTAINER ># rm -rf *
CONTAINER ># rm -rf .*
CONTAINER ># mix local.hex --force
CONTAINER ># mix archive.install hex phx_new 1.7.11 --force
CONTAINER ># mix phx.new .

Again, make sure to adjust these settings in the config/dev.exs file:

config :src, Src.Repo,
  username: "phoenix",
  password: "phoenix",
  hostname: "postgres.localhost",
  ...

config :src, SrcWeb.Endpoint,
  http: [ip: {0, 0, 0, 0}, port: 4000],
  ...

Then compile the project and run it:

CONTAINER ># mix deps.get
CONTAINER ># mix compile
CONTAINER ># mix ecto.setup
CONTAINER ># iex -S mix phx.server

And now to make sure everything has actually worked, we can try to connect to our new database from our Mac.

MAC ># psql -h postgres.localhost -U phoenix src_dev

Optional: Reimport your local databases

Finally we can reimport any of the exported databases as well:

MAC ># psql -h postgres.localhost -U phoenix postgres -c "CREATE DATABASE your_db;"
MAC ># psql -h postgres.localhost -U postgres your_db < your_db.sql

Conclusion and Outlook

So this was definitely the longest blog post I’ve written so far, and probably the most detailed 😎.

Up next I plan to add a post about how I prefer to create my Elixir development containers from a Dockerfile, and another one how I handle multiple different projects using only the single nginx server on the VM.