# XactJobs

> Background jobs for EF Core that commit in the same transaction as your data, on SQL Server, PostgreSQL, MySQL, Oracle and SQLite (.NET 8, 9 and 10). A job enqueued with JobEnqueue is a row saved by your own SaveChanges: it runs only if your transaction commits. The worker saves a job's database changes in the transaction that completes it, so they happen exactly once.

XactJobs is pre-1.0: the API can still change between minor versions. Source: https://github.com/XactJobs/XactJobs

# Getting started

Source: https://xactjobs.com/docs/getting-started/

XactJobs lets you schedule and run background jobs **with full database transaction support** using Entity Framework Core.
Jobs are saved alongside your business data and only execute if your transaction commits: no "ghost" jobs on rollback.

It supports **SQL Server, PostgreSQL, MySQL, Oracle and SQLite**, on .NET 8, 9 and 10. XactJobs uses EF Core 9 on .NET 8 and 9,
and EF Core 10 on .NET 10, so your database provider has to support that version. For MySQL, see [which provider to use](https://xactjobs.com/docs/databases/#mysql).

## 1. Install the package

```bash
dotnet add package XactJobs
```

Your app also needs the EF Core provider of its database, registered as usual. The provider isn't part of XactJobs:

**SQL Server**

```csharp
// Microsoft.EntityFrameworkCore.SqlServer
builder.Services.AddDbContext<UserDbContext>(options =>
    options.UseSqlServer(connectionString));
```

**PostgreSQL**

```csharp
// Npgsql.EntityFrameworkCore.PostgreSQL
builder.Services.AddDbContext<UserDbContext>(options =>
    options.UseNpgsql(connectionString));
```

**MySQL**

```csharp
// .NET 9 and 10: MySql.EntityFrameworkCore
builder.Services.AddDbContext<UserDbContext>(options =>
    options.UseMySQL(connectionString));

// .NET 8 and 9: Pomelo.EntityFrameworkCore.MySql
builder.Services.AddDbContext<UserDbContext>(options =>
    options.UseMySql(connectionString, ServerVersion.AutoDetect(connectionString)));
```

With the official provider, `SaveChanges` doesn't start a transaction on its own. See [MySQL](https://xactjobs.com/docs/databases/#mysql) before you go further.

**Oracle**

```csharp
// Oracle.EntityFrameworkCore
builder.Services.AddDbContext<UserDbContext>(options =>
    options.UseOracle(connectionString));
```

**SQLite**

```csharp
// Microsoft.EntityFrameworkCore.Sqlite
builder.Services.AddDbContext<UserDbContext>(options =>
    options.UseSqlite(connectionString));
```

## 2. Create the tables

For a new installation, run the install script for your database. It creates the latest schema: the job table, the history and the periodic jobs.

**SQL Server**

[sql/install/xact-jobs-sqlserver.sql](https://github.com/XactJobs/XactJobs/blob/main/sql/install/xact-jobs-sqlserver.sql) creates the tables in the `XactJobs` schema.

Run it with `QUOTED_IDENTIFIER` and `ANSI_NULLS` on, which are SqlClient's and SSMS's defaults (`sqlcmd` needs `-I`): the job table has a filtered index.

**PostgreSQL**

[sql/install/xact-jobs-postgresql.sql](https://github.com/XactJobs/XactJobs/blob/main/sql/install/xact-jobs-postgresql.sql) creates the tables in the `xact_jobs` schema.

The job table is a high-churn queue: see the [autovacuum settings](https://xactjobs.com/docs/databases/#postgresql) recommended for it.

**MySQL**

[sql/install/xact-jobs-mysql.sql](https://github.com/XactJobs/XactJobs/blob/main/sql/install/xact-jobs-mysql.sql) creates the `xact_jobs_` tables. XactJobs needs MySQL 8.0 or later.

**Oracle**

[sql/install/xact-jobs-oracle.sql](https://github.com/XactJobs/XactJobs/blob/main/sql/install/xact-jobs-oracle.sql) creates the `XACT_JOBS_` tables. XactJobs needs Oracle 12c or later.

**SQLite**

[sql/install/xact-jobs-sqlite.sql](https://github.com/XactJobs/XactJobs/blob/main/sql/install/xact-jobs-sqlite.sql) creates the `xact_jobs_` tables.

All the scripts are in [sql/](https://github.com/XactJobs/XactJobs/tree/main/sql). To upgrade an existing installation, run the versioned scripts instead: see [Upgrading](https://xactjobs.com/docs/upgrading/).

### Or use EF migrations

Instead of the scripts, you can include the XactJobs entities in your EF migrations:

```csharp
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.ApplyXactJobsConfigurations(Database.ProviderName,
        excludeFromMigrations: false); // <--- key part here

    // Apply your entity configurations...
}
```

And then create the migration like you normally would:

```bash
dotnet ef migrations add XactJobsTables
```

## 3. Configure startup

Add the entity configurations to your `DbContext`:

```csharp
public class UserDbContext : DbContext
{
    public UserDbContext(DbContextOptions<UserDbContext> options) : base(options)
    {
    }

    // Your DbSets...

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.ApplyXactJobsConfigurations(Database.ProviderName);

        // Apply your entity configurations...
    }
}
```

Register XactJobs in `Program.cs`. The workers run as hosted services in your app:

```csharp
// This registers 2 workers for the default queue
builder.Services.AddXactJobs<UserDbContext>(options =>
{
    options.WithPriorityQueue();     // optional (2 additional workers)
    options.WithLongRunningQueue();  // optional (2 additional workers)
});
```

All the options are in the [configuration reference](https://xactjobs.com/docs/configuration/).

## 4. Enqueue a job

A job is a public method. Enqueue it on your `DbContext`, next to your business data, and save. `JobEnqueue` and the other methods
are extension methods in the `XactJobs` namespace (`using XactJobs;`):

```csharp
public static class Greeter
{
    public static void Say(string message) => Console.WriteLine(message);
}
```

```csharp
// ... other business logic
dbContext.JobEnqueue(
    () => Greeter.Say("Fire-and-forget!"));

dbContext.SaveChanges();
```

The job is a row in the same transaction as your changes. If the transaction commits, a worker runs it; if it rolls back, the job doesn't exist.

1. Read [Jobs](https://xactjobs.com/docs/jobs/) for what a job method can be, and how instance methods get their dependencies.
2. Read [A job is a unit of work](https://xactjobs.com/docs/unit-of-work/): **the worker saves a job's database changes, not the job.**
3. Add the [dashboard](https://xactjobs.com/docs/dashboard/) to see and manage the jobs.

---

# Jobs

Source: https://xactjobs.com/docs/jobs/

A job is simply a public method. It can be either a static or an instance method, but it must be public and:

- it must not be a generic method,
- it cannot have `out` or `ref` parameters,
- it cannot have parameters that are anonymous functions, delegates, or lambda expressions,
- if it is an async method, it cannot be `async void` (it must be `async Task`).

## Parameters

Job method parameters are evaluated immediately, when the job is enqueued, and serialized using `System.Text.Json.JsonSerializer`.
This means parameters can be both primitives or complex objects, although large objects should be avoided, as they require more storage
and take longer to serialize and deserialize.

Because the arguments are evaluated when the job is enqueued, a key the database generates on `SaveChanges` isn't known yet: pass a
value you already have, like a business key or a `Guid` you create.

The method is stored with its parameter types, so overloaded methods can be jobs.

## Static methods

```csharp
class EmailHelper
{
    // example 1
    public static void SendEmail(string from, string to, string body)
    {
        // ... mail sending logic
    }

    // example 2
    public static async Task SendEmailAsync(string from, string to, string body, CancellationToken cancellationToken)
    {
        // ... mail sending logic
    }
}
```

These jobs can be scheduled with:

```csharp
// example 1
dbContext.JobEnqueue(
    () => EmailHelper.SendEmail("bob@acme.com", "alice@acme.com", "Hello Alice"));

// example 2
dbContext.JobEnqueue(
    () => EmailHelper.SendEmailAsync("bob@acme.com", "alice@acme.com", "Hello Alice", CancellationToken.None));

dbContext.SaveChanges();
```

## Instance methods and dependency injection

```csharp
class EmailSender
{
    private readonly IEmailService _emailService;

    // Constructor with dependencies.
    public EmailSender(IEmailService emailService)
    {
        _emailService = emailService;
    }

    // example 1
    public void SendEmail(string from, string to, string body)
    {
        // ... mail sending logic
    }

    // example 2
    public async Task SendEmailAsync(string from, string to, string body, CancellationToken cancellationToken)
    {
        // ... mail sending logic
    }
}
```

These jobs can be scheduled with:

```csharp
// example 1
dbContext.JobEnqueue<EmailSender>(
    x => x.SendEmail("bob@acme.com", "alice@acme.com", "Hello Alice"));

// example 2
dbContext.JobEnqueue<EmailSender>(
    x => x.SendEmailAsync("bob@acme.com", "alice@acme.com", "Hello Alice", CancellationToken.None));

dbContext.SaveChanges();
```

When a worker is processing an **instance** job method, it uses the first public constructor to create the instance, and then calls the
method on that instance. If the constructor has parameters, the worker uses `IServiceScopeFactory` to create a scope and then resolves
each type with `scope.ServiceProvider.GetRequiredService`.

This means the dependencies (`IEmailService` in the example) of the owning class must be registered in the DI service collection.
The owning class itself (`EmailSender` in the example) does not have to be a registered service: it just has to have a public constructor.

A job that takes your `TDbContext` in its constructor gets the scoped instance the worker saves when the job finishes. That's what makes
a job a [unit of work](https://xactjobs.com/docs/unit-of-work/).

## CancellationToken

If a method accepts a cancellation token, then `null` or `CancellationToken.None` should be passed when the job is enqueued.
When the worker executes the job, it passes the correct stopping token. See [Stopping](https://xactjobs.com/docs/failures-and-retries/#stopping) for when it's cancelled.

## XactJobContext

In the same way, a method can take an `XactJobContext` parameter, with `null` passed when the job is enqueued. The worker passes the
context of the run. For now it has `OriginallyScheduledAt`, the time (UTC) the job was originally scheduled for, which stays the same
across its retries:

```csharp
public async Task SendDailyReportAsync(XactJobContext context, CancellationToken cancellationToken)
{
    var idempotencyKey = $"daily-report-{context.OriginallyScheduledAt:yyyy-MM-dd}";
    // ...
}

dbContext.JobEnsurePeriodic<ReportSender>(
    x => x.SendDailyReportAsync(null!, CancellationToken.None),
    "daily_report",
    Cron.Daily());
```

- For a periodic job it's the cron slot of the run.
- For a continuation, it's the time its first attempt was claimed, once its batch was finished. The time is saved when the first
  attempt is claimed, so a second run of that attempt, after its lease expired, gets the same value.
- The dashboard's *Run now*, *Trigger* and *Retry* keep it too.

The cron slot and `OriginallyScheduledAt` are in UTC, because cron expressions are evaluated in UTC. So a daily run at midnight in a time
zone east of UTC, written as its UTC time (e.g. `0 0 22 * * *` for UTC+2), has the previous day's UTC date. If that's confusing, format
the key with the full time, e.g. `:yyyyMMddHHmmss`.

[Idempotent external calls](https://xactjobs.com/docs/idempotent-external-calls/) shows how to use it.

---

# A job is a unit of work

Source: https://xactjobs.com/docs/unit-of-work/

**The worker saves a job's database changes, not the job.** Changes a job makes through the injected `TDbContext` are saved when the
job finishes, in the same transaction that marks it as completed:

- **Exactly once.** If a lease expires and another worker runs the job again, only one of the runs can commit, so the changes happen once.
- **No transaction stays open while the job runs**, unless the job opens one. The changes wait in EF's change tracker, and are written at
  the end in one short transaction.
- **Don't call `SaveChanges` in a job**, or call it only inside a transaction you open and don't commit: the worker commits it. Changes
  saved any other way are permanent right away, even if the job then fails.
- **Returning saves.** Returning normally, even early, saves the pending changes and records the job as completed. A job that wants to
  stop without saving must throw instead.
- **Throwing discards.** Any exception discards the pending changes, and the job is retried. To stop a job that can never succeed, e.g.
  because the data it works on is gone, throw `XactJobCancelException`: its changes are discarded too, but it isn't retried, and it moves
  to the history as cancelled, with the exception's message.

```csharp
public class InvoiceJob(UserDbContext db)
{
    public async Task CreateInvoiceAsync(int orderId)
    {
        var order = await db.Orders.FindAsync(orderId)
            ?? throw new XactJobCancelException($"Order {orderId} doesn't exist anymore");

        db.Invoices.Add(new Invoice(order.Id)); // no SaveChanges: the worker saves it when the job finishes
    }
}
```

## Exactly-once database effects

A lease can expire while a job is still running, for example if a node loses its database connection for a while. Another worker can
then run the same job. For the database writes this is safe, as long as the job uses the scoped `TDbContext` injected into its class and
does one of these two things:

- **Doesn't save its changes.** The worker saves them when the job finishes, as in the example above.
- **Saves its changes inside a transaction it doesn't commit.** The worker commits it when the job finishes:

```csharp
public async Task CreateInvoiceAsync(int orderId, CancellationToken cancellationToken)
{
    await db.Database.BeginTransactionAsync(cancellationToken);

    db.Invoices.Add(new Invoice(orderId));
    await db.SaveChangesAsync(cancellationToken);

    // ... more work that needs the saved invoice
}
```

Either way the job's writes, the removal of the job row and the new history row all commit in one transaction. If two workers run the
same job, only one of them can commit: the other one's delete affects 0 rows, EF throws a concurrency exception, and its writes roll back.
The same happens if the job fails.

**This doesn't apply to** changes the job saves without a transaction, or commits itself, because they are already permanent when the job
finishes. It also doesn't apply to other DbContext instances, or to external side effects like sending emails or calling HTTP APIs. These
can happen more than once, so they should be idempotent: see [Idempotent external calls](https://xactjobs.com/docs/idempotent-external-calls/).

## The save check

**The worker warns when a job saves without a transaction.** While a job runs, the worker checks the saves on the scoped `TDbContext` it
created for the job. If the job calls `SaveChanges` without a transaction, the worker logs a warning that names the job, once per job
method, and the save goes ahead.

`WithSaveOutsideTransaction` changes this, for all the queues:

```csharp
builder.Services.AddXactJobs<UserDbContext>(options =>
{
    options.WithSaveOutsideTransaction(XactJobSaveCheck.Throw);
});
```

| Value | When a job saves without a transaction |
|---|---|
| `XactJobSaveCheck.Warn` (default) | a warning is logged, once per job method, and the save goes ahead |
| `XactJobSaveCheck.Throw` | `SaveChanges` throws instead, so nothing is saved, and the job fails and is retried as after any other exception |
| `XactJobSaveCheck.Ignore` | no check |

Only `SaveChanges` on that `TDbContext` is checked: not other DbContext instances, not `ExecuteUpdate`, `ExecuteDelete` or raw SQL, which
also run right away without a transaction, and not a transaction the job commits itself.

---

# Scheduling

Source: https://xactjobs.com/docs/scheduling/

The examples use this simple job:

```csharp
public static class Greeter
{
    public static void Say(string message) => Console.WriteLine(message);
}
```

Every method below only adds rows to the `DbContext`: nothing is scheduled until your `SaveChanges` commits.

## Fire and forget

Fire-and-forget jobs are executed once by a background worker, in its next poll. Each worker polls every 6 seconds by default
(`WithPollingInterval`), and the workers of a queue are started at staggered times, so the default 2 workers take turns every 3 seconds.
[QuickPoll](https://xactjobs.com/docs/queues-and-polling/#quickpoll) starts a job without waiting for the next poll.

```csharp
// ... other business logic
dbContext.JobEnqueue(
    () => Greeter.Say("Fire-and-forget!"));

dbContext.SaveChanges();
```

## Delayed jobs

Delayed jobs are executed once after the specified time interval, or at the specified time:

```csharp
dbContext.JobScheduleIn(
    () => Greeter.Say("Delayed!"),
    TimeSpan.FromDays(7));

dbContext.JobScheduleAt(
    () => Greeter.Say("At a time!"),
    new DateTime(2027, 1, 1, 9, 0, 0, DateTimeKind.Utc));

dbContext.SaveChanges();
```

`JobEnqueue`, `JobScheduleIn` and `JobScheduleAt` also take a queue as their last argument: see [Queues](https://xactjobs.com/docs/queues-and-polling/#isolated-queues).

## Several jobs at once

Adding multiple jobs before saving changes enqueues all of them in a single transaction:

```csharp
dbContext.JobEnqueue(() => Greeter.Say("Job 1"));
dbContext.JobEnqueue(() => Greeter.Say("Job 2"));

dbContext.SaveChanges();
```

## Transactions

Jobs participate in DbContext transactions, just like the regular entities in EF Core:

```csharp
using var tx = dbContext.Database.BeginTransaction();

// some business logic
dbContext.JobEnqueue(() => Greeter.Say("Job 1"));
dbContext.SaveChanges();

// some more business logic
dbContext.JobEnqueue(() => Greeter.Say("Job 2"));
dbContext.SaveChanges();

tx.Commit();
```

If the transaction rolls back, neither the business data nor the jobs exist.

## Periodic jobs

Periodic (recurring) jobs run according to a cron schedule:

```csharp
dbContext.JobEnsurePeriodic(
    () => Greeter.Say("Recurring!"),
    "my_recurring_job",
    Cron.Daily());

dbContext.SaveChanges();
```

The id (`my_recurring_job`) names the periodic job: ensuring it again with another definition updates it. `JobEnsurePeriodicAsync`
does the same with an async lookup, and both have an overload with a queue before the cron expression.

If the periodic jobs are known at startup, define them in the options instead:

```csharp
builder.Services.AddXactJobs<UserDbContext>(options =>
{
    options.WithPeriodicJob(
        () => Greeter.Say("Recurring!"),
        "my_recurring_job",
        Cron.Daily());
});
```

A periodic job defined on an isolated queue's options runs on that queue, e.g.
`options.WithLongRunningQueue(queue => queue.WithPeriodicJob(...))`.

### Cron expressions

Cron expressions have 6 fields, starting with the seconds: `second minute hour day-of-month month day-of-week`, evaluated in UTC.
The `Cron` class builds the common ones:

| Helper | Expression |
|---|---|
| `Cron.SecondInterval(10)` | `*/10 * * * * *` |
| `Cron.MinuteInterval(5)` | `0 */5 * * * *` |
| `Cron.HourInterval(2)` | `0 0 */2 * * *` |
| `Cron.Daily()` | `0 0 0 * * *`, every day at midnight UTC |
| `Cron.Daily(new TimeSpan(9, 30, 0))` | `0 30 9 * * *` |
| `Cron.Weekly(DayOfWeek.Monday)` | `0 0 0 * * 1` |
| `Cron.Weekly(DayOfWeek.Monday, new TimeSpan(8, 0, 0), DayOfWeek.Thursday)` | `0 0 8 * * 1,4` |

### Several nodes

Defining periodic jobs in the options is the safe way to do it with several nodes starting at the same time, because the nodes take
turns under a lock. `JobEnsurePeriodic` can't take that lock, since the changes are only saved later by your `SaveChanges`. If several
nodes call it at the same time, it's still safe for an existing job (only the definition saved last runs), but for a new job all of them
except the first get a duplicate key error from `SaveChanges`.

### Reconciliation

Each run of a periodic job schedules the next one. Each node also checks every 5 minutes (`WithPeriodicJobReconciliationInterval`)
for active periodic jobs without a pending run, and schedules their next run. So a periodic job recovers if its pending row gets lost,
and rows added directly to the periodic job table are picked up. They need a valid cron expression, and the same type name, method name
and arguments format that XactJobs writes. The run happens at the next time on the cron schedule: missed runs are not caught up.

After a periodic job is deleted, or its definition changes, the pending run of the old definition stays in the table until its scheduled
time. A worker then skips it, adding a `Skipped` row to the history.

### Pause, resume and delete

Pausing and resuming are available in code, followed by `SaveChanges`:

```csharp
await dbContext.JobSetPeriodicActiveAsync("my_recurring_job", false, cancellationToken); // pause
await dbContext.SaveChangesAsync(cancellationToken);
```

- Pausing removes the pending runs that are not running at the moment, so a paused job doesn't fill the history with skipped runs. A run
  in progress finishes, and doesn't schedule the next one if the job is still paused by then.
- Resuming schedules the next run, unless there already is one.
- A paused job stays paused when the application starts again, even if `JobEnsurePeriodic` or `WithPeriodicJob` defines it.
- `JobDeletePeriodicAsync(id, cancellationToken)` deletes the definition, also followed by `SaveChanges`. A periodic job defined in code
  is added again when the application starts.

Both return `false` if the periodic job doesn't exist. The [dashboard](https://xactjobs.com/docs/dashboard/) has the same actions, and *Trigger*.

---

# Batches and continuations

Source: https://xactjobs.com/docs/batches-and-continuations/

## Continuations

A continuation is a job that starts when another job is finished. It can have continuations of its own:

```csharp
dbContext.JobEnqueue<Importer>(x => x.ImportAsync(fileId))
    .ContinueWith<Reports>(x => x.RebuildAsync())
    .ContinueWith<Mailer>(x => x.SendReportAsync());

dbContext.SaveChanges();
```

By default a continuation runs if the job succeeded. The condition can also be `OnFailure` or `OnFinish`, and a continuation can go to another queue:

```csharp
var import = dbContext.JobEnqueue<Importer>(x => x.ImportAsync(fileId));

import.ContinueWith<Reports>(x => x.RebuildAsync());
import.ContinueWith<Ops>(x => x.AlertAsync(fileId), XactJobContinuation.OnFailure, QueueNames.Priority);
```

| Condition | Runs when |
|---|---|
| `OnSuccess` (default) | all the jobs it waits for succeeded |
| `OnFailure` | at least one of them failed |
| `OnFinish` | they are all finished, whether they succeeded or not |

A job failed if it ran out of retries, threw `XactJobCancelException`, was cancelled from the dashboard, or was a continuation that was
skipped, so in a chain of `OnSuccess` continuations, a failure skips the rest of the chain. A continuation whose condition isn't met moves
to the history as `Skipped`, with the reason as its message. Attempts that were retried don't count: a job of a batch that succeeds on its
second attempt succeeded.

A job can only be continued before it's saved, because its row can't be changed afterwards. To continue jobs that are saved separately,
put them in a batch.

## Batches

A batch is a group of jobs whose continuations start when all of them are finished:

```csharp
var batch = dbContext.JobBatch()
    .Enqueue<InventorySync>(x => x.SyncAsync("leeds"))
    .Enqueue<InventorySync>(x => x.SyncAsync("reno"))
    .ScheduleIn<InventorySync>(x => x.SyncAsync("osaka"), TimeSpan.FromHours(2));

batch.ContinueWith<Reports>(x => x.RebuildAsync())
    .ContinueWith<Mailer>(x => x.SendReportAsync());

batch.ContinueWith<Ops>(x => x.AlertAsync(), XactJobContinuation.OnFailure);

dbContext.SaveChanges();
```

A batch has `Enqueue`, `ScheduleIn` and `ScheduleAt`, each with an optional queue. The jobs of a batch are saved together, with one
`SaveChanges`: once it's saved, a batch can't get more jobs. Continuations can be added at any time, also later by the batch's id
(`batch.Id`). If the batch is already finished, they start right away:

```csharp
dbContext.JobBatch(batchId)
    .ContinueWith<Audit>(x => x.LogAsync(batchId), XactJobContinuation.OnFinish);

dbContext.SaveChanges();
```

A batch id that doesn't exist is a finished batch without jobs, so its continuations run as if all of its jobs succeeded.

## Details

- A retry of a job stays in its batch, and a continuation waits for it. A job retried from the dashboard is a new job, outside of the batch.
- A job of a batch can't be continued on its own: continue the batch. Periodic jobs can't be in a batch, nor be continued.
- A skipped continuation can be retried from the dashboard: it runs as a new job, outside of any batch.
- The worker that finishes the last job of a batch wakes up the workers of the continuations' queues, if they run in the same process,
  and the continuations start as soon as a worker is free. Besides that, each worker checks for continuations of finished batches once
  per polling interval. The check is one query, which looks at every continuation of the queue that's still waiting, so keep it in mind
  with many thousands of them.
- A continuation that waits for its batch is stored with the scheduled time 9999-12-31, so the poll for due jobs never reaches it.
- With [QuickPoll](https://xactjobs.com/docs/queues-and-polling/#quickpoll), `JobBatch()` registers the jobs of the batch for the notification. Their
  continuations don't need it.
- The history cleanup, if you turn it on (see [Configuration](https://xactjobs.com/docs/configuration/)), keeps the history of the batches that continuations
  still wait for, since it tells if their jobs failed.

The dashboard's Batches page shows the active and finished batches, with their progress: see [Dashboard](https://xactjobs.com/docs/dashboard/).

---

# Queues and polling

Source: https://xactjobs.com/docs/queues-and-polling/

## Isolated queues

XactJobs can run isolated workers for named queues. A job goes to a queue when it's enqueued:

```csharp
dbContext.JobEnqueue(() => Greeter.Say("Job 1"), QueueNames.Priority);
dbContext.JobEnqueue(() => Greeter.Say("Job 2"), QueueNames.LongRunning);

dbContext.SaveChanges();
```

Each queue has its own workers, configured at startup:

```csharp
builder.Services.AddXactJobs<UserDbContext>(options =>
{
    options.WithPriorityQueue();     // 2 workers, polling every 4 seconds
    options.WithLongRunningQueue(queue => queue.WithMaxDegreeOfParallelism(2)); // 2 workers, polling every 10 seconds
    options.WithIsolatedQueue("reports", queue => queue.WithWorkerCount(1));
});
```

| Queue | Name | Configured with |
|---|---|---|
| Default | `QueueNames.Default` (`*`) | the options themselves: 2 workers, polling every 6 seconds |
| Priority | `QueueNames.Priority` (`p`) | `WithPriorityQueue()`: 2 workers, polling every 4 seconds |
| Long-running | `QueueNames.LongRunning` (`l`) | `WithLongRunningQueue()`: 2 workers, polling every 10 seconds |
| Your own | any name | `WithIsolatedQueue(name)` |

An isolated queue has its own settings (workers, parallelism, batch size, polling, lease, shutdown timeout, retry strategy and periodic
jobs), set in its configure action. It doesn't take them from the default queue. Long-running jobs are best kept in their own queue (see below).

## Polling

A worker runs up to `MaxDegreeOfParallelism` jobs at a time, and holds a batch of claimed (leased) jobs: by default twice as many, so that
each slot has a job waiting for it. A free slot takes one of the jobs the worker holds, without waiting for a claim. A slow job only
occupies its own slot.

If a poll fills the batch there is likely a backlog, so the worker claims again once fewer jobs wait than it can run, instead of once for
each job that finishes: when the jobs it holds are done, or after a second if some are still running. Otherwise it waits for the polling
interval or a QuickPoll notification. Polling can stop only if all the slots are occupied, so long-running jobs are best kept in their own
queue.

The workers of a queue start at staggered times: with 2 workers polling every 6 seconds, the second one starts 3 seconds after the first.

While a job runs, its lease is extended automatically, every three quarters of `WithLeaseDuration` (default 120 seconds).

## Batch size

`WithBatchSize` changes the batch. A larger one saves claims, which helps short jobs: on PostgreSQL, 2 workers running 4 no-op jobs at a
time each drained a backlog about twice as fast with a batch of 100 as with no batch (see [Throughput](https://xactjobs.com/docs/throughput/)).

The cost is that the jobs a worker holds wait for its slots, also when another worker is idle, and wait for their leases to expire if the
worker crashes. At or below `MaxDegreeOfParallelism`, there's no batch: a worker only claims jobs it can start immediately, so they never
wait behind a slow job while another worker is free. When the worker stops, the jobs it holds that haven't started are released right
away, for the other workers.

```csharp
builder.Services.AddXactJobs<UserDbContext>(options =>
{
    options
        .WithWorkerCount(2)
        .WithMaxDegreeOfParallelism(4)
        .WithBatchSize(100);
});
```

## QuickPoll

QuickPoll can be used if a job should be executed immediately, without waiting for the next poll interval. `QuickPoll<TDbContext>` is
registered as a scoped service, so it can be injected in class constructors, just like the DbContext instances.

```csharp
public class YourWorkerOrController
{
    private readonly QuickPoll<UserDbContext> _quickPoll;
    private readonly UserDbContext _dbContext;

    public YourWorkerOrController(QuickPoll<UserDbContext> quickPoll)
    {
        _quickPoll = quickPoll;
        _dbContext = quickPoll.DbContext;
    }

    public async Task YourBusinessLogic()
    {
        // Your business logic uses _dbContext, as usual.

        // Enqueue the job through _quickPoll (instead of DbContext)
        _quickPoll.JobEnqueue(() => Greeter.Say("QuickPoll"));

        // Save changes through _quickPoll
        // (this calls DbContext.SaveChanges and notifies the workers)
        _quickPoll.SaveChangesAndNotify();
    }
}
```

- `SaveChangesAndNotifyAsync(cancellationToken)` is the async version.
- Inside a transaction, the workers are notified only when it commits: commit with `CommitAndNotify()` or `CommitAndNotifyAsync(cancellationToken)`.
- `JobBatch()` on QuickPoll registers the jobs of the batch for the notification.
- If you save or commit directly through the DbContext, call `Notify()` afterwards.

This can only work if the workers are running in the same process where the jobs are enqueued (which is the default).

A job enqueued this way is scheduled for the current time of the application. If that's still in the future for the database when the
worker polls (the clocks differ, or on SQLite within the same millisecond), the job runs at the next regular poll.

---

# Failures, retries and stopping

Source: https://xactjobs.com/docs/failures-and-retries/

## Failures and retries

When a job throws, its pending database changes are discarded, the attempt is recorded in the history as failed, and the job is scheduled
to run again by the retry strategy. A job that runs out of retries is recorded as cancelled.

By default (`DefaultRetryStrategy`) a job runs up to 10 times, waiting longer after each failed attempt:

| After failed attempt | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| The next attempt runs in | 2 s | 2 s | 5 s | 10 s | 30 s | 1 min | 5 min | 15 min | 30 min | none: cancelled |

`WithRetryStrategy` sets another strategy, per queue. `DefaultRetryStrategy` takes the number of attempts and the waits in seconds, and
uses the last wait for the attempts after them:

```csharp
builder.Services.AddXactJobs<UserDbContext>(options =>
{
    options.WithRetryStrategy(new DefaultRetryStrategy(maxAttempts: 5, retrySeconds: [10, 60, 300]));
});
```

Or implement `IRetryStrategy`, whose `GetRetryTimeUtc(job, newErrorCount)` returns the time of the next attempt in UTC, or `null` for no
more attempts. A custom retry strategy gets the job that failed. For the first attempt of a continuation, its `ScheduledAt` is the reserved
time 9999-12-31, so base the retry time on the current time.

## Stopping a job without retries

A job that finds it can never succeed, e.g. because the data it works on is gone, throws `XactJobCancelException` to stop without retries:

```csharp
public async Task SendInvoiceAsync(int orderId)
{
    var order = await db.Orders.FindAsync(orderId)
        ?? throw new XactJobCancelException($"Order {orderId} doesn't exist anymore");

    // ...
}
```

The job moves to the history as cancelled right away, with the exception's message as the error. As with any failure, its pending
database changes are discarded. The exception is also found when it's wrapped in other exceptions. For a periodic job, only the current
run is cancelled: the next one is scheduled as usual.

A job that has nothing to do should just return: it's recorded as completed.

## Stopping

When the application stops, the workers stop polling and running jobs get `ShutdownTimeoutInSeconds` (default 10) to finish. Then the
cancellation token passed to the jobs is cancelled, and they get 5 more seconds to react. Finally, the leases of the unfinished jobs are
cleared so they can run again right away, which takes up to `ClearLeaseTimeoutInSeconds` (default 10). A cancelled job doesn't count as a
failed attempt.

With the default settings this fits in the default 30 second shutdown timeout of the .NET host. If you lower that timeout, lower these as
well. `WithShutdownTimeout(0)` cancels the running jobs immediately.

A job that ignores the cancellation token may still be running when its lease is cleared, so it can end up running on another worker at
the same time. Its database changes are still saved [only once](https://xactjobs.com/docs/unit-of-work/#exactly-once-database-effects).

---

# Idempotent external calls

Source: https://xactjobs.com/docs/idempotent-external-calls/

Calls to other systems, like a payment provider or an email API, are at least once: a job that fails after the call, or whose lease
expired, runs the call again. To make them safe, send an idempotency key with the call, and pass it to the job as a method argument:

- a business key, like `$"charge-order-{orderId}"`,
- or `Guid.NewGuid()`: method arguments are evaluated once, when the job is enqueued, so every retry gets the same key.

```csharp
dbContext.JobEnqueue<PaymentJob>(x => x.ChargeAsync(orderId, Guid.NewGuid(), CancellationToken.None));
```

## Periodic jobs

A periodic job gets the same arguments on every run, so build its key from the run's cron slot instead, with
[`XactJobContext.OriginallyScheduledAt`](https://xactjobs.com/docs/jobs/#xactjobcontext), e.g. `$"daily-report-{context.OriginallyScheduledAt:yyyy-MM-dd}"`:

```csharp
public async Task SendDailyReportAsync(XactJobContext context, CancellationToken cancellationToken)
{
    var idempotencyKey = $"daily-report-{context.OriginallyScheduledAt:yyyy-MM-dd}";
    // ...
}
```

`OriginallyScheduledAt` doesn't change when a job is retried, and it's in UTC: see [XactJobContext](https://xactjobs.com/docs/jobs/#xactjobcontext).

## The receiving side

The receiving system has to support idempotency keys: it does the work for the first call with a key, and answers the same for the next ones.

The job's own database changes don't need a key: the worker saves them [exactly once](https://xactjobs.com/docs/unit-of-work/#exactly-once-database-effects).

---

# Dashboard and API

Source: https://xactjobs.com/docs/dashboard/

The `XactJobs.UI` package adds a web dashboard and a REST API to see and manage the jobs.

```bash
dotnet add package XactJobs.UI
```

![The dashboard: due, running, succeeded and failed jobs, and the jobs processed per hour](https://raw.githubusercontent.com/XactJobs/XactJobs/main/docs/images/dashboard.png)

```csharp
using XactJobs.UI.DependencyInjection;

builder.Services.AddXactJobsUI<UserDbContext>(options =>
{
    options
        .RequireAuthorizationPolicy("JobAdmins") // who can use it, see below
        .WithBasePath("/XactJobs")               // the default
        .WithApiBasePath("/api/XactJobs");       // the default
});

var app = builder.Build();

app.UseAuthentication();
app.UseAuthorization();

app.MapXactJobsUI();       // maps the pages (Razor Pages), the API, and the dashboard's scripts and styles
```

The dashboard's scripts, styles and fonts are embedded in the package, and `MapXactJobsUI` serves them under the dashboard's path, with the
same access as the pages. So the dashboard works without internet access, without the static files middleware, and in any environment,
whether the application is published or not.

The dashboard follows the light or dark setting of the operating system, and the switch in its navbar chooses light, dark or auto (the
operating system's setting) in each browser.

## Access

By default the dashboard only accepts requests from the local machine. Choose who can use it:

- `RequireAuthorization()`: any signed-in user (the default authorization policy)
- `RequireAuthorizationPolicy("name")`: users that satisfy the policy
- `AllowAnonymous()`: anyone. Only for development, or when the access is restricted in another way
- `AllowLocalRequestsOnly()`: the default. Behind a reverse proxy on the same machine, every request looks local, unless the forwarded
  headers middleware is configured

The API endpoints that change data require an antiforgery token, which the dashboard sends. If the API is called by clients that don't
authenticate with cookies, `DisableAntiforgery()` turns the check off.

The other options (page size, title, refresh interval) are in the [configuration reference](https://xactjobs.com/docs/configuration/#dashboard).

## Pages and actions

- **Dashboard**: due, scheduled and running jobs, continuations waiting for their batch, succeeded and failed jobs in the last 24 hours,
  and a chart per hour
- **Scheduled**: the pending and running jobs. A continuation that waits for its batch shows how many of the batch's jobs are left, and
  jobs of batches link to their batch
  - *Run now* makes a pending job due now, and the job keeps the time it was scheduled for as `XactJobContext.OriginallyScheduledAt`. A
    continuation that waits for its batch can't: it starts when the batch is finished
  - *Cancel* moves a pending job to the history as cancelled. Running jobs can't be cancelled, and neither can runs of periodic jobs: pause
    the periodic job instead. A cancelled job of a batch counts as a failed job of the batch, and if it was the last one, the batch's
    continuations start
- **History**: succeeded, failed (out of retries, or cancelled by the job with `XactJobCancelException`), cancelled from the dashboard,
  retried attempts, and skipped runs and continuations, with the reason
  - *Retry* enqueues a failed or cancelled job, or a skipped continuation, again as a new job. A run of a periodic job is enqueued as a
    one-off job, and a job of a batch or a continuation runs outside of any batch, so it doesn't start continuations. Like a retry by a
    worker, it keeps the time the job was originally scheduled for
- **Batches**: the active batches, with jobs left, and the finished ones, as long as their history is kept, with their progress
  - A batch's page has its jobs left, its continuations and its history, and links to the batch it continues
- **Periodic**: the periodic jobs, with their next run
  - *Pause* removes the pending run, and no runs are scheduled until it's resumed. A paused job stays paused when the application starts
    again, even if `JobEnsurePeriodic` or `WithPeriodicJob` defines it
  - *Resume* schedules the next run
  - *Trigger* makes the pending run due now, so the job keeps a single schedule
  - *Delete* deletes the definition. A periodic job defined in code is added again when the application starts

A job that is running can't be changed: the dashboard never touches a job a worker has leased.

![Scheduled jobs: running, scheduled and retrying jobs, with their queues](https://raw.githubusercontent.com/XactJobs/XactJobs/main/docs/images/scheduled.png)

![A failed job's error message and stack trace](https://raw.githubusercontent.com/XactJobs/XactJobs/main/docs/images/failed-job.png)

![Periodic jobs: their cron expressions and next runs, one of them paused](https://raw.githubusercontent.com/XactJobs/XactJobs/main/docs/images/periodic.png)

![Active batches: their progress, failed jobs and waiting continuations](https://raw.githubusercontent.com/XactJobs/XactJobs/main/docs/images/batches.png)

![A batch: its jobs left, the continuations that wait for it, and its history](https://raw.githubusercontent.com/XactJobs/XactJobs/main/docs/images/batch.png)

## API

All under the API base path (`/api/XactJobs` by default):

| Method | Path | Description |
|---|---|---|
| GET | `/stats` | Dashboard counts |
| GET | `/chart-data?from=&to=&intervalMinutes=` | Counts per interval (default: the last 24 hours) |
| GET | `/jobs?page=&pageSize=&queue=` | Pending and running jobs |
| GET | `/jobs/{id}?queue=&scheduledAt=` | A pending or running job |
| POST | `/jobs/{id}/run-now?queue=&scheduledAt=` | Make a pending job due now |
| POST | `/jobs/{id}/cancel?queue=&scheduledAt=` | Cancel a pending job |
| GET | `/history?category=&page=&pageSize=&queue=` | History: `succeeded`, `failed`, `cancelled`, `retried` or `skipped` |
| GET | `/history/{id}` | A history row |
| POST | `/history/{id}/retry` | Enqueue a failed or cancelled job, or a skipped continuation, again |
| GET | `/batches?state=&page=&pageSize=` | Batches: `active` or `finished` |
| GET | `/batches/{id}` | A batch with its jobs, history and continuations |
| GET | `/periodic?page=&pageSize=` | Periodic jobs |
| GET | `/periodic/{id}` | A periodic job |
| POST | `/periodic/{id}/pause` | Pause a periodic job |
| POST | `/periodic/{id}/resume` | Resume a periodic job |
| POST | `/periodic/{id}/trigger` | Run a periodic job now |
| DELETE | `/periodic/{id}` | Delete a periodic job |

The actions answer `200` with a message, `404` if the job doesn't exist, or `409` if its state doesn't allow the action (e.g. it's running).

A pending job is identified by its primary key: its `id`, `queue` and `scheduledAt`, as `/jobs` returns them, with `scheduledAt` in UTC
(e.g. `2026-11-07T10:00:00.1234567Z`). On most databases there's no index on the id alone, so a lookup by the id would read the whole job
table. Without the queue and the scheduled time, these routes answer `400`.

---

# Metrics and logs

Source: https://xactjobs.com/docs/observability/

The workers publish metrics on the meter `XactJobs` (`System.Diagnostics.Metrics`, part of .NET), so XactJobs needs no OpenTelemetry
package. Collect them with OpenTelemetry, `dotnet-counters` or a `MeterListener`. Nothing is recorded while nothing listens. With the
OpenTelemetry packages:

```csharp
builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics => metrics.AddMeter("XactJobs"));
```

| Metric | Type | Unit | Description |
|---|---|---|---|
| `xactjobs.job.attempts` | counter | `{attempt}` | Attempts that ended, by `xactjobs.outcome`: `completed`; `skipped` (an inactive periodic job, or a continuation whose condition isn't met); `retried` (failed, and scheduled again); `failed` (out of retries, or cancelled itself); `interrupted` (stopped by the shutdown, it runs again) |
| `xactjobs.job.duration` | histogram | s | How long an attempt took, until its outcome was saved, by `xactjobs.outcome` |
| `xactjobs.job.start_delay` | histogram | s | How long a job waited from the time it was due until it started: a queue falling behind shows here first. A continuation is due when it's claimed |
| `xactjobs.claim.duration` | histogram | s | How long a claim took |
| `xactjobs.job.claimed` | counter | `{job}` | Jobs claimed |
| `xactjobs.job.running` | up-down counter | `{job}` | Jobs running now |
| `xactjobs.job.held` | up-down counter | `{job}` | Jobs the workers hold: running, and waiting for a slot |

They're all tagged with `xactjobs.queue`. `options.WithJobTagsInMetrics()` also tags the attempts, their duration and the start delay
with `xactjobs.job.type` and `xactjobs.job.method`: one series per job method instead of per queue.

Each attempt runs in a log scope with the job's `JobId`, `JobType`, `JobMethod`, `JobQueue` and `JobAttempt`. What the job logs, and what
EF logs meanwhile, carries them, with a logging provider that records scopes (the console's `IncludeScopes`, Serilog, OpenTelemetry's
`IncludeScopes`).

---

# Using XactJobs with AI agents

Source: https://xactjobs.com/docs/ai-agents/

Coding agents write jobs the way they've seen background jobs written elsewhere, and the one habit that breaks XactJobs' guarantees is
calling `SaveChanges` inside a job. Paste this into your project's `CLAUDE.md`, `AGENTS.md` or similar instructions file:

```markdown
## Background jobs (XactJobs)

- A job is a unit of work. Changes a job makes through the injected `TDbContext` are saved by the worker
  when the job finishes, in the same transaction that removes the job and records its history. This is
  what makes them exactly once.
- Don't call `SaveChanges` in a job, or call it only inside a transaction the job opens
  (`Database.BeginTransactionAsync`) and doesn't commit: the worker commits it. Anything saved otherwise
  (also `ExecuteUpdate`, `ExecuteDelete`, raw SQL, or another DbContext) is permanent right away, even if
  the job then fails, and happens again if the job runs again.
- Returning normally saves the pending changes. To stop without saving, throw. Any exception discards
  the changes and retries the job. If the job can never succeed (e.g. its data is gone), throw
  `XactJobCancelException`: nothing is saved and the job isn't retried.
- Calls to other systems (payments, email, HTTP APIs) are at least once. Send an idempotency key: pass it
  as a job argument (`$"charge-order-{orderId}"`, or `Guid.NewGuid()`, which is evaluated once, when the
  job is enqueued), or for periodic jobs build it from `XactJobContext.OriginallyScheduledAt`.
- Enqueue jobs on the same DbContext as the business data they belong to (`JobEnqueue`, `JobScheduleIn`,
  `JobScheduleAt`), and let the app's own `SaveChanges` or transaction commit both together.
- Job methods are public, not generic, not `async void`, without `out`/`ref` or delegate parameters.
  Pass `CancellationToken.None` and `null` for `CancellationToken` and `XactJobContext` parameters when
  enqueueing: the worker fills them in.
- Put long-running jobs in their own queue (`QueueNames.LongRunning` with `WithLongRunningQueue()`, or
  `WithIsolatedQueue`), so they don't occupy the slots of short jobs.
```

The rules come from [A job is a unit of work](https://xactjobs.com/docs/unit-of-work/), [Failures and retries](https://xactjobs.com/docs/failures-and-retries/),
[Idempotent external calls](https://xactjobs.com/docs/idempotent-external-calls/) and [Queues](https://xactjobs.com/docs/queues-and-polling/).

## llms.txt

The site publishes its docs for language models, generated from these pages at build time:

- [llms.txt](https://xactjobs.com/llms.txt): a short index of the docs pages, with a line on each
- [llms-full.txt](https://xactjobs.com/llms-full.txt): all the docs pages as plain markdown, in one file

---

# Configuration

Source: https://xactjobs.com/docs/configuration/

```csharp
builder.Services.AddXactJobs<UserDbContext>(options =>
{
    options
        .WithWorkerCount(2)                         // workers polling the queue
        .WithMaxDegreeOfParallelism(8)              // jobs running at the same time, per worker (default: processor count)
     // .WithBatchSize(16)                          // jobs a worker claims and holds at a time (default: twice MaxDegreeOfParallelism)
        .WithPollingInterval(6)                     // seconds between polls, per worker
        .WithLeaseDuration(120)                     // seconds, extended automatically while a job is running
        .WithShutdownTimeout(10)                    // seconds running jobs get to finish on stop, before they are cancelled
        .WithPeriodicJobReconciliationInterval(300) // seconds between the checks for periodic jobs without a pending run
        .WithSaveOutsideTransaction(XactJobSaveCheck.Warn); // when a job saves without a transaction: Warn, Throw or Ignore (all the queues)
     // .WithHistoryRetentionPeriodInDays(30)       // deletes the older history every hour (default: the history is kept)
     // .WithJobTagsInMetrics()                     // tags the jobs' metrics with the job's type and method (all the queues)

    // isolated queues have their own settings
    options.WithLongRunningQueue(queue => queue.WithMaxDegreeOfParallelism(2));
});
```

The values above are the defaults, except for `WithMaxDegreeOfParallelism`.

## Each queue

These are set on the options for the default queue, and in the configure action of each isolated queue. An isolated queue starts from
these defaults, not from the default queue's settings.

| Option | Default | Description |
|---|---|---|
| `WithWorkerCount(n)` | 2 | Workers polling the queue, started at staggered times |
| `WithMaxDegreeOfParallelism(n)` | processor count | Jobs running at the same time, per worker |
| `WithBatchSize(n)` | twice `MaxDegreeOfParallelism` | Jobs a worker claims and holds at a time. At or below `MaxDegreeOfParallelism`, no batch: see [Batch size](https://xactjobs.com/docs/queues-and-polling/#batch-size) |
| `WithPollingInterval(seconds)` | 6 | Seconds between polls, per worker |
| `WithLeaseDuration(seconds)` | 120 | How long a claimed job is leased, extended every three quarters of it while the job runs |
| `WithShutdownTimeout(seconds)` | 10 | Seconds running jobs get to finish on stop, before they are cancelled. 0 cancels them immediately |
| `ClearLeaseTimeoutInSeconds` | 10 | How long the workers wait to clear the leases on stop. No builder method: set `options.Options.ClearLeaseTimeoutInSeconds` |
| `WithRetryStrategy(strategy)` | `DefaultRetryStrategy` | When a failed job runs again: see [Failures and retries](https://xactjobs.com/docs/failures-and-retries/) |
| `WithPeriodicJob(job, id, cron)` | none | A periodic job ensured at startup, on this queue: see [Periodic jobs](https://xactjobs.com/docs/scheduling/#periodic-jobs) |

## All the queues

| Option | Default | Description |
|---|---|---|
| `WithHistoryRetentionPeriodInDays(days)` | none: the history is kept | Adds the periodic job `xj_history_cleanup`, which deletes the older history every hour, except the history of the batches that continuations still wait for |
| `WithPeriodicJobReconciliationInterval(seconds)` | 300 | Seconds between each node's checks for active periodic jobs without a pending run |
| `WithSaveOutsideTransaction(check)` | `XactJobSaveCheck.Warn` | When a job saves without a transaction: `Warn`, `Throw` or `Ignore`. See [the save check](https://xactjobs.com/docs/unit-of-work/#the-save-check) |
| `WithJobTagsInMetrics(enabled?)` | off | Also tags the jobs' metrics with the job's type and method. See [Metrics and logs](https://xactjobs.com/docs/observability/) |
| `WithPriorityQueue(configure?)` | not started | Workers for `QueueNames.Priority`: 2, polling every 4 seconds |
| `WithLongRunningQueue(configure?)` | not started | Workers for `QueueNames.LongRunning`: 2, polling every 10 seconds |
| `WithIsolatedQueue(name, configure?)` | none | Workers for a queue of your own |

By default the history is kept, and XactJobs deletes nothing. Set `WithHistoryRetentionPeriodInDays` the same on every node: a node without
it removes the cleanup job when it starts. Versions before 0.3.0 always added it, with 30 days: to keep that, call
`WithHistoryRetentionPeriodInDays(30)`.

## The model

```csharp
modelBuilder.ApplyXactJobsConfigurations(Database.ProviderName, excludeFromMigrations: true, schema: null);
```

| Parameter | Default | Description |
|---|---|---|
| `providerName` | | `Database.ProviderName`: the entity configurations depend on the database |
| `excludeFromMigrations` | `true` | `false` includes the XactJobs tables in your EF migrations |
| `schema` | `xact_jobs` on PostgreSQL, `XactJobs` on SQL Server | The schema of XactJobs' tables, for [several apps in one database](https://xactjobs.com/docs/databases/#several-apps-in-one-database). PostgreSQL and SQL Server only |

## Dashboard

`AddXactJobsUI<TDbContext>(options => ...)`, from the `XactJobs.UI` package:

| Option | Default | Description |
|---|---|---|
| `WithBasePath(path)` | `/XactJobs` | The dashboard's pages |
| `WithApiBasePath(path)` | `/api/XactJobs` | The REST API |
| `WithPageSize(n)` | 20 | Rows per page in the lists |
| `WithDashboardTitle(title)` | `XactJobs Dashboard` | The title shown in the dashboard |
| `WithRefreshInterval(seconds)` | 5 | How often the pages refresh, at least 5 |
| `AllowLocalRequestsOnly()` | the default | Only requests from the local machine |
| `RequireAuthorization()` | | Any signed-in user |
| `RequireAuthorizationPolicy(name)` | | Users that satisfy the policy |
| `AllowAnonymous()` | | Anyone: only for development, or when the access is restricted in another way |
| `DisableAntiforgery()` | antiforgery on | The API's actions don't require an antiforgery token, for clients that don't use cookies |

---

# Databases

Source: https://xactjobs.com/docs/databases/

XactJobs supports SQL Server, PostgreSQL, MySQL, Oracle and SQLite, with EF Core 9 on .NET 8 and 9, and EF Core 10 on .NET 10. Your
database provider has to support that version. The core is tested on all five databases, with an integration test suite running against
each one (SQLite in-process, the others in containers).

| Database | XactJobs' tables | Install script |
|---|---|---|
| SQL Server | `[XactJobs].[Job]`, `[XactJobs].[JobHistory]`, `[XactJobs].[JobPeriodic]` | [xact-jobs-sqlserver.sql](https://github.com/XactJobs/XactJobs/blob/main/sql/install/xact-jobs-sqlserver.sql) |
| PostgreSQL | `xact_jobs.job`, `xact_jobs.job_history`, `xact_jobs.job_periodic` | [xact-jobs-postgresql.sql](https://github.com/XactJobs/XactJobs/blob/main/sql/install/xact-jobs-postgresql.sql) |
| MySQL | `xact_jobs_job`, `xact_jobs_job_history`, `xact_jobs_job_periodic` | [xact-jobs-mysql.sql](https://github.com/XactJobs/XactJobs/blob/main/sql/install/xact-jobs-mysql.sql) |
| Oracle | `XACT_JOBS_JOB`, `XACT_JOBS_JOB_HISTORY`, `XACT_JOBS_JOB_PERIODIC` | [xact-jobs-oracle.sql](https://github.com/XactJobs/XactJobs/blob/main/sql/install/xact-jobs-oracle.sql) |
| SQLite | `xact_jobs_job`, `xact_jobs_job_history`, `xact_jobs_job_periodic` | [xact-jobs-sqlite.sql](https://github.com/XactJobs/XactJobs/blob/main/sql/install/xact-jobs-sqlite.sql) |

## SQL Server

Run the scripts with `QUOTED_IDENTIFIER` and `ANSI_NULLS` on, which are SqlClient's and SSMS's defaults (`sqlcmd` needs `-I`): the job
table has a filtered index. Writing to the table needs them on as well.

## PostgreSQL

The job table is a high-churn queue: every job inserts, updates and deletes a row. Aggressive autovacuum settings for this table are
recommended, for example:

```sql
ALTER TABLE xact_jobs.job SET (
  autovacuum_vacuum_scale_factor = 0,
  autovacuum_vacuum_threshold = 1000,
  autovacuum_vacuum_cost_limit = 2000
);
```

Long-running transactions **anywhere in the cluster** hold back the cleanup of dead rows, not just the ones using this table. Dead rows
pile up meanwhile, which slows down fetching jobs.

## MySQL

XactJobs needs **MySQL 8.0 or later**, because claiming jobs uses `SKIP LOCKED` and `FOR UPDATE OF`. MariaDB isn't supported: it doesn't
support `FOR UPDATE OF`, and only supports `SKIP LOCKED` from 10.6.

The workers claim jobs at `READ COMMITTED`, not at InnoDB's default `REPEATABLE READ`. At `REPEATABLE READ`, a claim would keep the running
jobs it scans locked, and the gaps between rows too, so finished jobs couldn't be deleted, and new jobs (`JobEnqueue` and `SaveChanges`)
couldn't be saved until the claim commits. Only the claims use it: your own transactions, and the jobs, keep their isolation level.

The provider depends on the version of .NET:

| .NET | MySQL provider |
|---|---|
| .NET 8 | `Pomelo.EntityFrameworkCore.MySql` |
| .NET 9 | `Pomelo.EntityFrameworkCore.MySql` or `MySql.EntityFrameworkCore` |
| .NET 10 | `MySql.EntityFrameworkCore` |

Pomelo doesn't support EF Core 10 yet, and on .NET 8 the official provider only supports EF Core 8. These combinations build, but fail at runtime.

**With the official provider (`MySql.EntityFrameworkCore`), `SaveChanges` doesn't start a transaction on its own.** If a statement fails,
the ones before it stay committed. This can break the main promise of XactJobs in your own code: for example, a job can be saved without
the business data it belongs to.

To make your saves all-or-nothing, either use a transaction, as in [Transactions](https://xactjobs.com/docs/scheduling/#transactions), or set this on your `DbContext`:

```csharp
public class UserDbContext : DbContext
{
    public UserDbContext(DbContextOptions<UserDbContext> options) : base(options)
    {
        Database.AutoTransactionBehavior = AutoTransactionBehavior.Always;
    }
}
```

The workers already do this for their own saves, so running jobs are not affected. Pomelo starts a transaction when a save has more than one statement.

## Oracle

XactJobs needs **Oracle 12c or later**. A claim is one PL/SQL block, which returns the claimed jobs with `DBMS_SQL.RETURN_RESULT`.

Jobs are claimed oldest first. The block locks the due jobs with `FOR UPDATE SKIP LOCKED`, fetches only as many as the worker has free
slots, and leases them. Oracle locks a row only when it's fetched, so workers that poll at once each get different jobs from a backlog.
`ROWNUM` can't be used to limit the rows, because it's applied before locked rows are skipped: all the workers would try the same first
rows, and all but one would get none.

The claims read the table's primary key in order (with an index hint), so a claim doesn't sort the whole backlog, even if Oracle cached a
plan from when the table was nearly empty.

## SQLite

On SQLite the primary key of the job table is only the id, so the claims use an index on the queue and the scheduled time. Without it,
every claim would read and sort the whole job table. The install script creates it; for an existing installation, the v0.3.0 script adds it.

A job enqueued with [QuickPoll](https://xactjobs.com/docs/queues-and-polling/#quickpoll) within the same millisecond as the poll can wait for the next regular poll.

## Several apps in one database

On PostgreSQL and SQL Server, XactJobs' tables are in their own schema: `xact_jobs` on PostgreSQL, `XactJobs` on SQL Server. Apps that
share a database each need their own tables, or their workers would run each other's jobs. Give each app a schema of its own:

```csharp
modelBuilder.ApplyXactJobsConfigurations(Database.ProviderName, schema: "app_b");
```

The workers take the schema from the model. With the SQL scripts, replace `xact_jobs` (or `XactJobs`) with your schema before you run
them; EF migrations use it on their own. On PostgreSQL the name must be lowercase.

On MySQL, Oracle and SQLite there's no schema option: give each app its own database (on Oracle, its own user), which already has its own tables.

---

# Upgrading

Source: https://xactjobs.com/docs/upgrading/

Schema changes ship as SQL scripts, one per version and database. Existing data is always kept. To upgrade an existing installation, run
the versioned scripts newer than its version, in version order, with the notes below. If you include the XactJobs entities in your
[EF migrations](https://xactjobs.com/docs/getting-started/#or-use-ef-migrations), add a migration instead.

For a new installation, run the install script instead: it creates the same schema as all the versioned scripts.

| Database | v0.1.0 | v0.1.4 | v0.1.5 | v0.2.0 | v0.3.0 | v0.4.0 |
|---|---|---|---|---|---|---|
| SQL Server | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.0/xact-jobs-sqlserver-v0.1.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.4/xact-jobs-sqlserver-v0.1.4.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.5/xact-jobs-sqlserver-v0.1.5.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.2.0/xact-jobs-sqlserver-v0.2.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.3.0/xact-jobs-sqlserver-v0.3.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.4.0/xact-jobs-sqlserver-v0.4.0.sql) |
| PostgreSQL | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.0/xact-jobs-postgresql-v0.1.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.4/xact-jobs-postgresql-v0.1.4.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.5/xact-jobs-postgresql-v0.1.5.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.2.0/xact-jobs-postgresql-v0.2.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.3.0/xact-jobs-postgresql-v0.3.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.4.0/xact-jobs-postgresql-v0.4.0.sql) |
| MySQL | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.0/xact-jobs-mysql-v0.1.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.4/xact-jobs-mysql-v0.1.4.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.5/xact-jobs-mysql-v0.1.5.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.2.0/xact-jobs-mysql-v0.2.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.3.0/xact-jobs-mysql-v0.3.0.sql) | none |
| Oracle | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.0/xact-jobs-oracle-v0.1.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.4/xact-jobs-oracle-v0.1.4.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.5/xact-jobs-oracle-v0.1.5.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.2.0/xact-jobs-oracle-v0.2.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.3.0/xact-jobs-oracle-v0.3.0.sql) | none |
| SQLite | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.0/xact-jobs-sqlite-v0.1.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.4/xact-jobs-sqlite-v0.1.4.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.1.5/xact-jobs-sqlite-v0.1.5.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.2.0/xact-jobs-sqlite-v0.2.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.3.0/xact-jobs-sqlite-v0.3.0.sql) | [script](https://github.com/XactJobs/XactJobs/blob/main/sql/v0.4.0/xact-jobs-sqlite-v0.4.0.sql) |

On SQL Server, run the scripts with `QUOTED_IDENTIFIER` and `ANSI_NULLS` on, which are SqlClient's and SSMS's defaults (`sqlcmd` needs `-I`).

Until 1.0 the API can still change between minor versions (0.x). Breaking changes are listed in the
[release notes](https://github.com/XactJobs/XactJobs/releases), with how to migrate.

## v0.4.0

The script can run before or after the workers are upgraded. On PostgreSQL, SQL Server and SQLite, it rebuilds the indexes of the batch
columns, in the job table and the history, so that they hold only the rows of batches and continuations: the other jobs' inserts and
deletes don't write to them. There's no script for MySQL, which has no filtered indexes, nor for Oracle, whose indexes already leave out
those rows.

## v0.3.0

Run the script first: it adds the column for the time a job was originally scheduled for, which v0.3.0 needs. Then upgrade the workers.
A worker of an older version that retries a job leaves the column empty, so that retry sees its own retry time as
`XactJobContext.OriginallyScheduledAt`.

- On SQLite, the script also adds an index on the queue and the scheduled time, for the claims: without it, every claim reads and sorts
  the whole job table.
- Except on MySQL, it also adds an index of the periodic job column, with only the runs of periodic jobs, so that pausing a periodic job,
  resuming it, the reconciliation and the dashboard's Periodic page don't read the whole table.
- On SQL Server it changes the column from `nvarchar(max)`, which can't be indexed, to `nvarchar(450)`, the type of the periodic job's
  id, and it's a filtered index: creating it, and writing to the table, needs `QUOTED_IDENTIFIER` and `ANSI_NULLS` on, which are
  SqlClient's defaults (`sqlcmd` needs `-I`).
- The dashboard's API identifies a pending job by its primary key now: see [API](https://xactjobs.com/docs/dashboard/#api).

## v0.2.0

Run the script first: it adds the columns for batches and continuations, which v0.2.0 needs. Start using batches and continuations only
once all the workers run v0.2.0: a worker of an older version that retries a job of a batch removes the job from its batch, so the batch's
continuations could start too early.

## v0.1.5

Upgrade all the workers first and then run the script, because it drops an index that older versions need.

---

# Throughput and benchmarks

Source: https://xactjobs.com/docs/throughput/

## Throughput

On a Hetzner Cloud CX33 (cost-optimized, 4 shared vCPUs, AMD EPYC Rome, 8 GB of RAM), with PostgreSQL 16 in a container on the same
machine (its default settings, so every commit waits for the disk), .NET 10 and EF Core 10, XactJobs drained a backlog of 20,000 no-op
jobs at **2,300 to 2,600 jobs per second**. The best setting was 2 workers, `MaxDegreeOfParallelism` 4 and `WithBatchSize(100)`.

Each job is still its own transaction: its changes, the delete of its row and its history row commit together.

With real jobs, their own work sets the pace. A larger batch saves claims, which helps short jobs, at a cost for slow ones: see
[Batch size](https://xactjobs.com/docs/queues-and-polling/#batch-size).

## Benchmarks

The benchmarks are in [src/XactJobs.Benchmarks](https://github.com/XactJobs/XactJobs/tree/main/src/XactJobs.Benchmarks), to measure on
your own machine. They measure XactJobs' own code, to catch regressions between commits: compare two commits on the same machine, one run
at a time, with nothing else running. Always build in Release.

```bash
dotnet build -c Release src/XactJobs.Benchmarks/XactJobs.Benchmarks.csproj
```

### micro: without a database

BenchmarkDotNet, with the allocations per call: enqueueing (the expression tree, evaluating the arguments and serializing them), running a
job (reading the arguments, finding the compiled delegate and calling it), the next run of a periodic job, and the save check.

```bash
dotnet run --no-build -c Release -f net10.0 --project src/XactJobs.Benchmarks -- micro --filter "*"
```

### drain: a backlog of no-op jobs

Drains a backlog of jobs that do nothing, with XactJobs and with plain EF doing the least database work for the same result (the same
claim, then for each job its delete and its history row, saved together). The difference per job is XactJobs' own cost. It runs on SQLite
in memory by default, or on a PostgreSQL or SQL Server server:

```bash
dotnet run --no-build -c Release -f net10.0 --project src/XactJobs.Benchmarks -- drain --jobs 3000 --runs 5

dotnet run --no-build -c Release -f net10.0 --project src/XactJobs.Benchmarks -- drain --jobs 10000 --runs 3 --workers 2 --parallelism 4 \
    --batch-size 100 --postgres "Host=127.0.0.1;Port=55432;Username=postgres;Password=bench;Database=postgres"
```

`--sqlserver <connection string>` runs it on SQL Server. Each drain gets a new database on that server, with the schema from the install
script, dropped afterwards. The [benchmark README](https://github.com/XactJobs/XactJobs/tree/main/src/XactJobs.Benchmarks) has all the
options, and how to start the database in a container.