Background Job Processing in Elixir with Oban

The Elixir language has amazing built-in support for concurrency and executing asynchronous tasks as part of the Erlang OTP.  It's true: you can generally just toss tasks into the background of your Elixir app without being overly concerned about concurrency, performance, locking, etc.

This works well for some kinds of tasks, but for others, simply tossing a job into the background can have some shortcomings: what happens when a job fails?  What happens if a server crashes?  How do you monitor and manage load and concurrency?  

If these kinds of scenarios matter for your application, you'll need some kind of system for managing background jobs.  In this post, we'll explore the Oban background job processing system.  In particular, we'll cover:

• what Oban is and why you might need it
• how to setup and install Oban in an Elixir app
• how to define and enqueue background jobs
• how to run jobs and manage queues

Let's get started!

What is Oban?

Oban is a high-performance background job processing system for the Elixir language.  It lets us define jobs, put them into queues, and process them in the background as our application does other work.  Jobs are persistent so if a server or process crashes, it isn't lost.  Oban will handle retry and failure logic in those scenarios.  Additioanlly, it lets us monitor and manage the job queues so we can keep an eye on how many jobs are being processed, if jobs are failing, etc.

Oban uses a standard Postgres database to persist and manage jobs.  This is in contrast to some other job processing systems which might use other infrastructure tools such as Redis or RabbitMQ.  Without getting into a nitty gritty performance comparison between these different tools, it's safe to say that adding additional services to a tech stack can add a great deal of complexity, so using Postgres is a nice feature.

If you're already using Postgres with Ecto (which is the default in Phoenix), you can use the existing application database, though you can configure an alternative database as well if you prefer.  Unless you're operating at a very large scale using the default application database is usually a good place to start.

While Oban works with any kind of Elixir app, it's a natural choice for web apps built with the Phoenix framework.  Web applications are all about real-time interactions and quick responses, so it often makes sense to background certain tasks.  For example, if an application sends a welcome email to a new user on signup, it doesn't make sense to wait for the delivery to complete in the middle of a web request.  Better to render the page immediately and drop the email delivery into a background queue.

While Oban is open-source and free to use,  they do offer optional Pro functionality including an extremely slick web interface for monitoring and managing job queues among other fetures.  

Running Background Jobs with Oban

Now that we have the, uh, background on Oban (terrible pun intended), let's see how to install and use Oban in our Elixir app.

Installing Oban

Most of this section is a recap of what's described in the Oban documentation, but is included here to help you get started.  See the full documentation to learn more about additional configuration options.

First, you'll need to install Oban in your application by adding oban in your mix.exs file and running mix deps.get:

# mix.exs
def deps do
  [
    {:oban, "~> 2.13"}
  ]
end

Oban will use our existing Ecto repo (typically Postgres) to persist jobs.  We'll need to create a migration which creates and configures the required table:

mix ecto.gen.migration add_oban_jobs_table

And then, in the generated migration:

defmodule MyApp.Repo.Migrations.AddObanJobsTable do
  use Ecto.Migration

  def up do
    Oban.Migrations.up(version: 11)
  end

  def down do
    Oban.Migrations.down(version: 1)
  end
end

Finally, we'll configure Oban in our config/config.exs:

config :my_app, Oban,
  repo: MyApp.Repo,
  plugins: [Oban.Plugins.Pruner],
  queues: [default: 10]

To make our background jobs execute immediately when running tests, we'll also add configuration in config/test.exs:

# config/test.exs
config :my_app, Oban, testing: :inline

The full list of configuration options is available in the Oban.Config documentation.

Defining a Job

Defining our background jobs with Oban is simple.  We simply create a module which uses Oban.Worker, specifies some options, and defines a perform function to execute:

defmodule MyApp.MyImportantJob do
  use Oban.Worker, queue: :events

  @impl Oban.Worker
  def perform(%Oban.Job{args: args}) do
    IO.inspect(args)
    :ok
  end
end

In the example above we specify only which queue the job should be placed in, but there are other options that can be specified as well:

  use Oban.Worker,
    queue: :events,
    priority: 3,
    max_attempts: 3,
    tags: ["user"],
    unique: [period: 30]

See the Worker documentation for full details on available options.

Enqueuing and Executing

Creating a job is as simple as creating a job with the desired parameters, and calling Oban.Insert() to enqueue it.

MyApp.MyImportantJob.new(%{id: 1, params: []})
|> Oban.insert()

Now our job is enqueued–but it won't execute yet since we haven't started our queue.  If you're curious, at this point you can pause and inspect your Postgres database to see the enqueued job.  In our oban_jobs table, you should find the job we just enqueued in the available state:

 1 | available | events  | MyApp.MyImportantJob | {"x": 10} | ...

When we're ready to run our jobs, we all we have to do is start the queue!  We do so with a call to Oban.start_queue with the queue name and the concurrency limit:

Oban.start_queue(queue: :events, limit: 4)

Note that if we have multiple queues defined, we can start and stop them independently and with different options.

In this simple example, our job simply inspects the contents of the provided args, so when we start the queue, we should see the logging messages output to the console. In a "real" job, we'd likely perform some action and store the result back in the database.

Next Steps

Now that we've covered the basics of Oban, we know enough to get started using it in our Elixir application for running simple background jobs–but there's a lot of other things Oban can do that we haven't covered, for example:

• Scheduled jobs (similar to cron)
• Job de-duplication
• Clustering support for multiple job processors

Whatever kind of Elixir app you're building, using Oban to manage background jobs is a great addition to the toolkit.