How I Deployed a Go app With a SQLite Database on Fly.io

DevOps 
How I Deployed a Go app With a SQLite Database on Fly.io

Recently, I deployed a Go-powered app backed by an SQLite database on Fly.io for the first time. Here’s the process that I went through, along with some of the issues that I encountered along the way.


Screenshot of the URL shortener app/website

After about four days of reading, configuring, experimenting, and (more than a little) frustration, I deployed my first Go app to Fly.io; a URL shortener backed by SQLite.

The screenshot above of the app doesn’t look like a lot. And, in so many ways, it isn’t. But, the achievement that it represents gives me a very strong sense of accomplishment.

It’s because (to be perfectly honest), up until now I’ve always struggled with deploying apps I’ve built for myself. Across a range of approaches, such as bare git repositories, deployment tools such as Deployer or Capistrano, and a range of other approaches, it’s often been quite an arduous process; Google Cloud Run excepted. However, after coming up to speed with Fly.io, that’s likely a thing of the past.

I only found out about Fly.io around a week ago, as I was looking for an easier way to deploy apps. Since that time, I believe I’ve found a platform and approach that will let me take my existing Docker and containerisation knowledge to both quickly and efficiently deploy apps.

What is Fly.io?

If you’re not familiar with them, to quote their website:

Fly.io transforms containers into (Linux) micro-VMs that run on our hardware in 30+ regions on six continents.

Essentially, all you need to do is package up your app in a Docker image. Their platform and tooling handle (most of) the rest of the work of deploying and running the application.

The Fly.io Dashboard

They support all of the things that you’d likely expect, such as:

  • A command line tool to manage tasks such as project setup, deployment, check logs, deploy and undeploy; and
  • A well-documented configuration file where you can set restart policies, and scaling policy
  • A user dashboard that collates all of the information you need to know

They also handle a wealth of other tasks including protecting the app with SSL/TLS, adding load balancing, and logging, etc.

They don’t do everything for you. But the fact that these tasks are taken care of for me is huge! It means that I don’t need to mess around with doing it manually, such as by configuring Traefik to talk to Let’s Encrypt, etc.

Why are they such a good choice?

Up until now, I’ve been deploying (PHP and Go) apps using Docker Compose. The Compose configuration would always have at least two containers:

  • One containing the app and its static assets
  • One containing Traefik; a reverse proxy and load balancer

The reason for this combination was to secure the app with SSL/TLS through Let’s Encrypt and to add security headers. Traefik is excellent! But I’m not all that experienced with it yet. Because of that, I find it a challenge to set up both quickly and confidently.

What’s more, I have to provision the remote machine as well, which can be a little tedious. Yes, I should spend time learn about Infrastructure as Code, such as by learning tools like Terraform or Ansible. But I’m not there, yet. And, I wanted to get this application up and running with a minimum of fuss.

What is the deployment process?

So, let’s dive into the deployment process for this app. There’s not a lot to it. The essential steps are:

  1. Package up the app in a Docker image using a Dockerfile
  2. Add the Fly.io configuration file, fly.toml, so that Fly.io knows how to deploy the app
  3. Set up a GitHub Workflow to deploy the app to Fly.io, including provisioning the database, whenever a push is made to a given branch

SQLite presents some unique challenges, though

On the surface of it, there’s nothing too challenging about these steps. However, as I mentioned at the top of the article, the app uses SQLite, instead of a client/server database such as PostgreSQL or MySQL.

SQLite is a flat-file or single-file database, meaning that the database file must be on the same filesystem as the Go binary. Let’s consider the implications of that requirement in the context of the app being containerised.

If the database is provisioned as part of the image creation process, every time a container’s started from the image, the database will be in the same state as when the image was created. This means that, after each deployment or app restart, no schema or data changes will be persisted.

So, how do you get around this issue? It comes down to two steps:

  1. Ensure that the database is stored on a persistable volume accessible to the container. Slightly paraphrasing the Docker documentation:

A volume is a location in your local filesystem, automatically managed by Docker Desktop. By using one, any changes to the database will survive container restarts and deployments.

  1. Provision the database as part of the deployment process, not the image creation process

How do you put the database on a persistable volume with Fly.io?

This is the easiest of the two steps. As the app was already configured to look for the database file in /data/db, I mounted a volume onto the root filesystem at /data. This was done by adding the following to the end of fly.toml:

[mounts]
  source = "app_data"
  destination = "/data"

This configuration auto-creates a persistable volume named app_data and, when the micro-VM starts, mounts it to the VM’s filesystem at /data. It’s the same as adding the following configuration to a Docker Compose configuration:

volumes:
    - "app_data:/data"

Or, you could use the following argument with docker run:

--mount source=app_data,target=/data

How do you provision the database?

How could I both provision the SQLite database initially and when changes were made in future releases? This step was the real challenge, and the aspect that took longest to get right.

At this point, I figured that I needed a migration tool. If you’re not familiar with them, you use them to create indempotent migration files which describe the changes that will be made to a database over the course of time.

These can include the inital schema and any changes to it, such as adding/dropping tables, adding/dropping indexes, adding/dropping columns, and so on.

After a bit of searching, I found dbmate; written in Go. It had all of the functionality that I was looking for, and used migration files containing pure DDL statements. Here’s an example from the app.

-- migrate:up
CREATE index IF NOT EXISTS idx_dates ON urls (created, updated);

-- migrate:down
DROP INDEX IF EXISTS idx_dates;

To simplify using it, I wrote a small shell script, which you can see below.

#!/bin/sh

set -u

if [ -z "${DATABASE_URL:-}" ]; then
    echo "Database URL is not available. Please set it. Exiting...."
    exit 1
fi

db_url_prefix="sqlite:"
db_dir=$(dirname ${DATABASE_URL#"$db_url_prefix"})

if [ ! -d $db_dir ]; then
    echo "Database directory [ ${db_dir} ] does not exist. Creating...."
    mkdir -p "$db_dir"
fi

npx dbmate up

echo "....Finished provisioning the database."

exit 0

It starts off by checking if $DATABASE_URL is both set and not empty. This variable contains a DSN string with the database type and path to the SQLite database file; in my case it was sqlite:/data/db/database.sqlite3.

If available, the script extracts the database directory from it and checks if the directory already exists. If not, it creates it. Finally, it calls dbmate up to run any pending migrations.

My first thought was to add it to the GitHub Workflow after the app was deployed, which you can see in the snippet below.

  jobs:
    code-deploy:
      name: Deploy app to production on Fly.io
      runs-on: ubuntu-latest
      concurrency: deploy-group
      steps:
        - uses: actions/checkout@v3
        - uses: superfly/flyctl-actions/setup-flyctl@master
        - run: flyctl deploy --remote-only
          env:
            FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
          - run: flyctl ssh console --command 'sh /opt/bin/run-migrations.sh'
          env:
            FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

Unfortunately, on launch, the app attempts to open the database file. So, if the file isn’t available or doesn’t contain a SQLite database, the app exits. This leads to the mini-VM its on restarting and starting the Go binary again, until the max number of retries is reached. At that point, the mini-VM shuts down.

Well, I read in the documentation that you can run one-off commands before a deployment is released. However, there is a catch:

To run a command in a temporary VM—using the app’s successfully built Docker image—before the release is deployed, define a release_command…The temporary VM has full access to the network, environment variables and secrets, but not to persistent volumes. Changes made to the file system on the temporary VM will not be retained or deployed.

I imagine that scripts run via this mechanism are intended to run migrations on database hosted on an external provider, such as Kafka or PostgreSQL hosted on DigitalOcean’s Managed Database platform, along with other forms of release tooling. But, because it doesn’t have access to the external volume, then the command isn’t an option.

So, how could I run the migrations? Could I run the migrations? I was running out of options, but was just determined to find a way.

After some concerted effort, I felt that there was one option left to try. I remembered that at the end of the Dockerfile generated by flyctl launch --no-deploy was the following:

CMD [gourlshortener]

If you’re not familiar with Dockerfile syntax, it is calling the Go binary compiled from the application’s source code.

Note: technically, CMD is for providing arguments to the ENTRYPOINT directive, but that’s a story for another time.

I remembered that I could run a command or a shell script using either the CMD or ENTRYPOINT directives. So, why not write a small script to run the database migrations, then launch the Go binary afterward?

It could work. After all, at this point, the deployment’s completed, so the environment is ready to use; meaning that the volume would be mounted and accessible to the running container.

#!/bin/sh

set -Cu

/opt/bin/run-migrations.sh

gourlshortener

It was worth a shot. So, I wrote the script above which calls the existing database migrations script, then launches the Go binary, gourlshortener, in the foreground where it listens for and respond to requests.

After that, I refactored the end of the Dockerfile, as follows, to call it instead of the Go binary:

ENTRYPOINT ["/opt/bin/launch.sh"]

From my reading of Docker’s Dockerfile best practices, I don’t think this approach will be shunned upon. Anyway, after committing and pushing the changes to the GitHub repository, I was super excited to see that the deployment worked exactly as expected!

That’s how I deployed a Go app with a SQLite database to Fly.io

While, in the end, the process was pretty straightforward, getting to that point was a bit of a round about journey. If you’re considering trying out Fly.io, I highly recommend them; no, this is not a sponsored post.

If you’re planning to deploy a Go (or PHP) application backed by an SQLite database, this is my approach for doing so when working with Fly.io. That said, I’d love your input about it in the comments below.

Do you need to get your head around Docker Compose quickly?

What about needing to dockerize existing applications to make them easier to deploy, reducing the time required for develwpers to get started on projects, or learning how to debug an existing Docker Compose-based app? Then this free book is for you!

You might also be interested in these tutorials too...

Override an Image's Command with Docker Compose
Thu, Oct 26, 2023

Override an Image's Command with Docker Compose

Sometimes, you need to override an image’s command when launching a container with Docker Compose. If you need to do that, in this tutorial I’m going to show you how — without the need to update an image’s Dockerfile or shell scripts.

Override an Image's Command with Docker Compose
Thu, Oct 26, 2023

Override an Image's Command with Docker Compose

Sometimes, you need to override an image’s command when launching a container with Docker Compose. If you need to do that, in this tutorial I’m going to show you how — without the need to update an image’s Dockerfile or shell scripts.

How to Deploy a Container Image With Google Cloud Run
Sun, May 28, 2023

How to Deploy a Container Image With Google Cloud Run

There are loads of ways to deploy container-based applications. In this tutorial, you’ll learn how to deploy a small application using Google’s Cloud Run. It’s a powerful, yet not-too-imposing service that helps you deploy applications pretty quickly.


Want more tutorials like this?

If so, enter your email address in the field below and click subscribe.

You can unsubscribe at any time by clicking the link in the footer of the emails you'll receive. Here's my privacy policy, if you'd like to know more. I use Mailchimp to send emails. You can learn more about their privacy practices here.

Join the discussion

comments powered by Disqus