Queue Management

Control concurrent execution and resource usage with queues.

Overview

Boltbase's queue system helps you:

• Limit concurrent workflow executions
• Manage resource usage
• Prioritize critical workflows
• Prevent system overload

How Queues Work:

• Global queues (defined in config.yaml) control concurrency via max_concurrency
• Local queues (per-DAG, used when no queue field is set) always process FIFO with concurrency of 1
• To control concurrent execution, define global queues and assign DAGs using the queue field

Basic Queue Configuration

Assign to Queue

queue: "batch"              # Assign to global queue for concurrency control
schedule: "*/10 * * * *"    # Every 10 minutes

steps:
  - command: echo "Processing batch"

The queue's max_concurrency (defined in config.yaml) controls how many DAGs assigned to this queue can run concurrently.

Global Queue Configuration

Configure queues in server config:

# ~/.config/boltbase/config.yaml
queues:
  enabled: true
  config:
    - name: "critical"
      max_concurrency: 5      # 5 critical jobs concurrently
    - name: "batch"
      max_concurrency: 1      # One batch job at a time
    - name: "reporting"
      max_concurrency: 3      # 3 reports concurrently

Default Queue via Base Config

Set a default queue for all workflows in your base config:

# ~/.config/boltbase/base.yaml
queue: "default"

# All DAGs inherit this queue assignment
# Define the "default" queue in config.yaml with desired max_concurrency

Manual Queue Management

Enqueue Workflows

# Basic enqueue
boltbase enqueue workflow.yaml

# With custom run ID
boltbase enqueue workflow.yaml --run-id=batch-2024-01-15

# With parameters
boltbase enqueue process.yaml -- DATE=2024-01-15 TYPE=daily

# Override the queue at enqueue-time
boltbase enqueue --queue=high-priority workflow.yaml

Remove from Queue

# Remove the next item in a queue
boltbase dequeue default

# Remove a specific run from a queue
boltbase dequeue default --dag-run=workflow:batch-2024-01-15