Recently, I deployed a Go-powered app backed by an SQLite database on Fly.io for the first time. Here’s the process that I went through, along with some of the issues that I encountered along the way.
After about four days of reading, configuring, experimenting, and (more than a little) frustration, I deployed my first Go app to Fly.io; a URL shortener backed by SQLite.
The screenshot above of the app doesn’t look like a lot. And, in so many ways, it isn’t. But, the achievement that it represents gives me a very strong sense of accomplishment.
It’s because (to be perfectly honest), up until now I’ve always struggled with deploying apps I’ve built for myself. Across a range of approaches, such as bare git repositories, deployment tools such as Deployer or Capistrano, and a range of other approaches, it’s often been quite an arduous process; Google Cloud Run excepted. However, after coming up to speed with Fly.io, that’s likely a thing of the past.
I only found out about Fly.io around a week ago, as I was looking for an easier way to deploy apps. Since that time, I believe I’ve found a platform and approach that will let me take my existing Docker and containerisation knowledge to both quickly and efficiently deploy apps.
If you’re not familiar with them, to quote their website:
Fly.io transforms containers into (Linux) micro-VMs that run on our hardware in 30+ regions on six continents.
Essentially, all you need to do is package up your app in a Docker image. Their platform and tooling handle (most of) the rest of the work of deploying and running the application.
They support all of the things that you’d likely expect, such as:
They also handle a wealth of other tasks including protecting the app with SSL/TLS, adding load balancing, and logging, etc.
They don’t do everything for you. But the fact that these tasks are taken care of for me is huge! It means that I don’t need to mess around with doing it manually, such as by configuring Traefik to talk to Let’s Encrypt, etc.
Up until now, I’ve been deploying (PHP and Go) apps using Docker Compose. The Compose configuration would always have at least two containers:
The reason for this combination was to secure the app with SSL/TLS through Let’s Encrypt and to add security headers. Traefik is excellent! But I’m not all that experienced with it yet. Because of that, I find it a challenge to set up both quickly and confidently.
What’s more, I have to provision the remote machine as well, which can be a little tedious. Yes, I should spend time learn about Infrastructure as Code, such as by learning tools like Terraform or Ansible. But I’m not there, yet. And, I wanted to get this application up and running with a minimum of fuss.
So, let’s dive into the deployment process for this app. There’s not a lot to it. The essential steps are:
On the surface of it, there’s nothing too challenging about these steps. However, as I mentioned at the top of the article, the app uses SQLite, instead of a client/server database such as PostgreSQL or MySQL.
SQLite is a flat-file or single-file database, meaning that the database file must be on the same filesystem as the Go binary. Let’s consider the implications of that requirement in the context of the app being containerised.
If the database is provisioned as part of the image creation process, every time a container’s started from the image, the database will be in the same state as when the image was created. This means that, after each deployment or app restart, no schema or data changes will be persisted.
So, how do you get around this issue? It comes down to two steps:
A volume is a location in your local filesystem, automatically managed by Docker Desktop. By using one, any changes to the database will survive container restarts and deployments.
This is the easiest of the two steps. As the app was already configured to look for the database file in /data/db, I mounted a volume onto the root filesystem at /data. This was done by adding the following to the end of fly.toml:
[mounts]
source = "app_data"
destination = "/data"
This configuration auto-creates a persistable volume named app_data and, when the micro-VM starts, mounts it to the VM’s filesystem at /data.
It’s the same as adding the following configuration to a Docker Compose configuration:
volumes:
- "app_data:/data"
Or, you could use the following argument with docker run:
--mount source=app_data,target=/data
How could I both provision the SQLite database initially and when changes were made in future releases? This step was the real challenge, and the aspect that took longest to get right.
At this point, I figured that I needed a migration tool. If you’re not familiar with them, you use them to create indempotent migration files which describe the changes that will be made to a database over the course of time.
These can include the inital schema and any changes to it, such as adding/dropping tables, adding/dropping indexes, adding/dropping columns, and so on.
After a bit of searching, I found dbmate; written in Go. It had all of the functionality that I was looking for, and used migration files containing pure DDL statements. Here’s an example from the app.
-- migrate:up
CREATE index IF NOT EXISTS idx_dates ON urls (created, updated);
-- migrate:down
DROP INDEX IF EXISTS idx_dates;
To simplify using it, I wrote a small shell script, which you can see below.
#!/bin/sh
set -u
if [ -z "${DATABASE_URL:-}" ]; then
echo "Database URL is not available. Please set it. Exiting...."
exit 1
fi
db_url_prefix="sqlite:"
db_dir=$(dirname ${DATABASE_URL#"$db_url_prefix"})
if [ ! -d $db_dir ]; then
echo "Database directory [ ${db_dir} ] does not exist. Creating...."
mkdir -p "$db_dir"
fi
npx dbmate up
echo "....Finished provisioning the database."
exit 0
It starts off by checking if $DATABASE_URL is both set and not empty.
This variable contains a DSN string with the database type and path to the SQLite database file; in my case it was sqlite:/data/db/database.sqlite3.
If available, the script extracts the database directory from it and checks if the directory already exists.
If not, it creates it.
Finally, it calls dbmate up to run any pending migrations.
My first thought was to add it to the GitHub Workflow after the app was deployed, which you can see in the snippet below.
jobs:
code-deploy:
name: Deploy app to production on Fly.io
runs-on: ubuntu-latest
concurrency: deploy-group
steps:
- uses: actions/checkout@v3
- uses: superfly/flyctl-actions/setup-flyctl@master
- run: flyctl deploy --remote-only
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
- run: flyctl ssh console --command 'sh /opt/bin/run-migrations.sh'
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
Unfortunately, on launch, the app attempts to open the database file. So, if the file isn’t available or doesn’t contain a SQLite database, the app exits. This leads to the mini-VM its on restarting and starting the Go binary again, until the max number of retries is reached. At that point, the mini-VM shuts down.
Well, I read in the documentation that you can run one-off commands before a deployment is released. However, there is a catch:
To run a command in a temporary VM—using the app’s successfully built Docker image—before the release is deployed, define a release_command…The temporary VM has full access to the network, environment variables and secrets, but not to persistent volumes. Changes made to the file system on the temporary VM will not be retained or deployed.
I imagine that scripts run via this mechanism are intended to run migrations on database hosted on an external provider, such as Kafka or PostgreSQL hosted on DigitalOcean’s Managed Database platform, along with other forms of release tooling. But, because it doesn’t have access to the external volume, then the command isn’t an option.
So, how could I run the migrations? Could I run the migrations? I was running out of options, but was just determined to find a way.
After some concerted effort, I felt that there was one option left to try.
I remembered that at the end of the Dockerfile generated by flyctl launch --no-deploy was the following:
CMD [gourlshortener]
If you’re not familiar with Dockerfile syntax, it is calling the Go binary compiled from the application’s source code.
Note: technically, CMD is for providing arguments to the ENTRYPOINT directive, but that’s a story for another time.
I remembered that I could run a command or a shell script using either the CMD or ENTRYPOINT directives.
So, why not write a small script to run the database migrations, then launch the Go binary afterward?
It could work. After all, at this point, the deployment’s completed, so the environment is ready to use; meaning that the volume would be mounted and accessible to the running container.
#!/bin/sh
set -Cu
/opt/bin/run-migrations.sh
gourlshortener
It was worth a shot. So, I wrote the script above which calls the existing database migrations script, then launches the Go binary, gourlshortener, in the foreground where it listens for and respond to requests.
After that, I refactored the end of the Dockerfile, as follows, to call it instead of the Go binary:
ENTRYPOINT ["/opt/bin/launch.sh"]
From my reading of Docker’s Dockerfile best practices, I don’t think this approach will be shunned upon. Anyway, after committing and pushing the changes to the GitHub repository, I was super excited to see that the deployment worked exactly as expected!
While, in the end, the process was pretty straightforward, getting to that point was a bit of a round about journey. If you’re considering trying out Fly.io, I highly recommend them; no, this is not a sponsored post.
If you’re planning to deploy a Go (or PHP) application backed by an SQLite database, this is my approach for doing so when working with Fly.io. That said, I’d love your input about it in the comments below.
Sometimes, you need to override an image’s command when launching a container with Docker Compose. If you need to do that, in this tutorial I’m going to show you how — without the need to update an image’s Dockerfile or shell scripts.
Sometimes, you need to override an image’s command when launching a container with Docker Compose. If you need to do that, in this tutorial I’m going to show you how — without the need to update an image’s Dockerfile or shell scripts.
There are loads of ways to deploy container-based applications. In this tutorial, you’ll learn how to deploy a small application using Google’s Cloud Run. It’s a powerful, yet not-too-imposing service that helps you deploy applications pretty quickly.
Do you need to have a SQLite database available in a Docker image? I did, and here’s the solution I came up with.
Please consider buying me a coffee. It really helps me to keep producing new tutorials.
Join the discussion
comments powered by Disqus