Skip to content

Meerschaum 5

5.0.2

  • The console's Performance page keeps the window and page you chose when a Refresh (or the 30-second update) answers after you changed them; a late reply used to bring back the previous selection.
  • The first use of DuckDB retries its library download after a temporary failure (a 5xx, 408 or 429 status, or a dropped connection) twice, 3 and then 10 seconds later; a single server error from GitHub used to fail the install.
  • The API server warns at startup about a public page (@web_page(login_required=False)) whose plugin registered Dash callbacks but made none of them public: visitors without a console session would see the page but get no updates. Register the callbacks in a @dash_plugin(public=True) function or with @web_callback(..., public=True).
  • Arrow 60.1 validates RunEndEncoded arrays (RUSTSEC-2026-0442: a malformed one could read out of bounds), and the other Rust dependencies take their latest compatible releases.

5.0.1

  • An API server restarts its root's stopped --restart jobs again, as in 4.x: jobs a container restart, a recreated container or a reboot stopped come back once the API starts, every second while it runs. Jobs stopped by hand stay stopped.
  • Jobs created by Meerschaum 4 keep their commands. 4.x saved the command of a job created through the API or in a container only in its pickle.pkl; 5.0.0 showed such a job with no command and could not restart it. The command is now read from that file as data (nothing in it is executed), also for jobs 5.0.0 already started without one.
  • TLS connections no longer trust certificates from the distrusted emSign intermediates (rustls-webpki 0.103.16, webpki-roots 1.0.11), and the bundled zstd (0.13.4) and ctutils (0.4.8) fix memory-safety and constant-time issues.
  • The api, dash and full extras require PyJWT>=2.15.0.

5.0.0

The data engine, database connectors, CLI, HTTP API, jobs, and Compose now run in Rust. Keep your Pipe definitions, configuration, Python plugins, and existing data. Start with the migration guide.

Pipelines that know the dependencies

Save a group of Pipes, preview its plan, then run parents before children. The engine records run history, retries failures, and tracks late data for backfills.

Lookup dependencies preserve ordering without capping event progress at an old lookup timestamp. New tail rows stay incremental; unchanged UPSERTs leave dependents quiet. Approved UPSERT rebuilds replay history in chunks while retaining stored rows. A run warns when a child syncs nothing while its parents advance, as when it shifts or scales their time axis; verify pipes syncs the gap. show run names parallel branches on the critical path once (+3 in parallel).

mrsm register pipelines daily -i sql:local -c sql:local -m daily --parents
mrsm plan pipelines daily -i sql:local
mrsm sync pipelines daily -i sql:local
mrsm show runs -i sql:local

See Pipelines for selection, schedules, and recovery.

Compose can declare named Pipelines under pipelines:. Jobs reference their names; compose explain previews exact members and compose up --dry applies metadata. See Pipelines in Compose.

A fuller web console

Overview, Pipes, Pipelines, Runs, and Jobs put current work and history in one place. The Overview summarizes Pipes by sync status, connector and latest sync, Pipelines by last-run status with a 24-hour chart of every run, and Jobs by state with the next runs of the jobs waiting for their next lap, beside Recent runs, which shows each Pipeline's whole name. Performance charts show sync duration, failures, and rows affected. Inspect plugin source without running it, and manage roles, teams, access, and locks.

Performance totals and charts cover the selected SQL history window; recent records are paged separately. Search and rank results, with cards on phones. Refreshes keep charts visible, and unchanged chart bundles reuse the browser cache. Leaving the Pipes table discards queued statistics refreshes.

SQL connection settings are preserved for Python queries and Pipelines.

Pipelines have a searchable, sortable runbook with member details and recent runs loaded on demand. The canvas keeps View options in a compact popup and supports pinch zoom on phones.

Show related Pipes as dimmed context, review shared membership, and add them through an editable draft. Runs share one history timeline; switch between a run's Gantt chart and table. Integer timestamp axes show readable UTC sync times using their declared precision.

Jobs are grouped with running jobs first, show a one-word status and a Last Run time, and refresh in place. Job listings read every systemd job's state with one systemctl call. Connectors are grouped by type. Create a plugin from the console with the same choices as bootstrap plugins, then edit its source in place (administrators who may use the web terminal). Edit opens a code editor with highlighting, indentation, search, undo, and optional Vim keybindings (:w saves); it loads only when you choose Edit and never refreshes while you type. Fullscreen sits beside Copy and hides the Source heading. A plugin's connectors link to its source with an icon. Publish an installed plugin to a configured API repository, signing in with a username and password, client credentials, or an API key. Keep credentials temporarily in the browser session or save them to the selected connector. Copy token details includes the API key and every displayed field. The web terminal shows its tmux windows as tabs, and an API run as a job keeps its terminal's shell interactive, even when an unattended script started it with MRSM_NOASK: python opens a REPL and bootstrap asks its questions.

Relationship views read graph metadata without planning a sync. Plans bound the total chunk list per Pipe before allocating it, including disjoint dirty ranges.

python -m pip install 'meerschaum[dash]'
mrsm start api -i sql:local

Open /dash. A new server offers a first administrator account. See the console guide.

Python stays familiar

Pipe.sync() still unpacks into (success, message). Its SyncResult also provides row counts, duration, and changed intervals. Read plain dictionaries with the base package, or install a DataFrame extra.

import meerschaum as mrsm

pipe = mrsm.Pipe('plugin:manual', 'readings', instance='sql:local', columns={'id': 'id'})
result = pipe.sync([{'id': 1, 'temperature': 21.5}])
print(result.success, result.inserted, result.duration)
print(pipe.get_data(as_docs=True))

New roots use Polars; existing roots keep pandas. Choose the DataFrame library explicitly when upgrading.

Teams, permissions, and locks

Viewer, Editor, and Operator roles provide starting permissions. Teams control shared Pipe and Job access. Tokens stay within their owner's current permissions, and revocations take effect on subsequent requests. Pipe locks protect operations across the CLI, Python, jobs, HTTP, MCP, and the console.

Self-registration closes after the first administrator unless configured otherwise. Editors no longer receive unrestricted action or terminal access. Review access and locks and API permissions.

Reliable syncs and operations

  • Replace an authoritative time window with pipe.sync(rows, begin=..., end=..., replace=True).
  • Persist numeric sync metrics across processes, with current access checks before aggregation.
  • Save run events before marking completion, keeping mrsm watch run output complete.
  • Keep private history private when a Pipe or Pipeline is deleted, recreated, or its access changes.
  • Bound SQL parent reads before window functions, keeping incremental syncs fast.
  • Sync small SQL Pipes faster than 4.1: a SQL connector keeps what the database catalog said about a table for pipes:dtypes:columns_types_cache_seconds (60 by default) and forgets it when it changes the table or a statement on it fails, a static Pipe's in-place sync reads no columns, Pipeline bookkeeping takes the initialization lock only on its first write, and a sync skips the Python worker when no plugin has a pre-sync hook. On PostgreSQL a small in-place sync takes about 25 ms instead of 90 (4.1: 50), and a five-Pipe lap 230 ms instead of 950 (4.1: 275).
  • The first sync of a plugin Pipe on a SQL instance no longer installs SQLAlchemy and the database's Python drivers into the root's virtual environment.
  • Use 30-day TimescaleDB metrics partitions and correct undersized RC partitions.
  • Initialize internal tables and indexes safely when workers start together.
  • Keep redirects fast on consoles with many plugin pages.
  • Isolate Python hosts when API and MCP servers share a configuration root.
  • Bound scheduling estimates to recent history for the selected Pipes.
  • Release completed telemetry connectors, bounding memory for repeated Pipeline calls.
  • Publish complete Job properties atomically, and finish metadata writes before deleting completed Jobs.
  • Release a Pipe lease even when acquisition completes after cancellation.
  • Back up, restore, and upgrade Stack PostgreSQL through guided commands.
  • Keep long-running jobs' memory bounded: Python workers are replaced after a call or memory limit and stop when idle, and jobs return freed memory between laps.
  • Reduce memory use when syncing and reading JSON columns through pandas.
  • Return freed memory after each fetched chunk and sync: a 1,000,000-row SQLite fetch sync peaks at about 300 MiB instead of 460.
  • Stream SQL fetches into the sync chunk by chunk instead of reading the whole result first: a 2,000,000-row fetch sync peaks at about 240 MiB on SQLite (was 300) and 170 MiB on PostgreSQL (was 260), and finishes sooner.
  • List a Pipeline's scheduled jobs without querying every job's status.
  • Drain the API on shutdown: it stops accepting connections, finishes requests in flight within api.shutdown.grace_seconds (8 by default, an upper bound), and ends streams at once. GET /healthcheck/ready returns 503 while it starts or drains, and while the Python API host it needs is not serving (no timeout makes it 200); a 503 names its reason.
  • Report a job killed by a signal as failed and name the signal, instead of showing a stale or successful result.
  • Run a scheduled job once at its first fire after a restart between two fires, instead of twice in a row, and keep an interval such as daily counted from that fire, not from the restart.
  • A plugin action that loops by itself reports its laps with mrsm.jobs.report_lap(success, message, next_in=...), so its job shows health and its next run.
  • Ask a console tab opened before an upgrade to reload when it requests a callback the server no longer has, instead of answering every poll with an error.
  • Report an error raised inside an MCP tool as that error and where it was raised, not as invalid arguments.
  • Stop the Python API host when its server is killed, and let the next API take over and stop a killed API's web terminal tmux server.
  • Find sql clients outside a service's reduced PATH (~/.cargo/bin, ~/.local/bin, Meerschaum's virtual environments), and offer to install the flavor's Python client when none is found; a Python client whose interpreter cannot import it (a script left by an older Python) is skipped with a warning.
  • The Stack provisions two Grafana dashboards: Syncs replaces the three performance dashboards (overview, Pipes and Pipelines rows), and Engine charts runs, failed and retried nodes, Pipe freshness, lag behind parents and the dirty-range backlog. Unmodified copies of the old dashboards are removed; customized ones are kept.
  • The console's Performance page has an Engine section, like the Grafana Engine dashboard: runs by outcome and duration, Pipe freshness with lag behind parents, the dirty-range backlog by child, and failed or retried nodes with the planner's reason, for the pipes the viewer may read.

See Stack upgrades and sync metrics.

Rust plugins and applications

A plugin can be one .rs file, or a crate using the native SDK. Applications can embed the Pipe and engine APIs without compiling the CLI. The developer guides cover both Python and Rust.

  • A native route's caller carries its token's expiration and a signed-in user's user_type, so a plugin can rotate tokens before they expire and check administrators.
  • Raw bytes a native plugin syncs keep their exact bytes in a bytes column (they were stored as the base64 reading of their Python repr).
  • A native plugin's routes mint and revoke API tokens with the plugin's own scopes for their caller (host::mint_token, host::revoke_token), so a device can be provisioned and rotate its own token, and a long poll ends as soon as the server drains (host::draining()). See Tokens and draining.

Before upgrading

Check the migration checklist for supported platforms, optional dependencies, DataFrame defaults, permissions, API behavior, and installation ownership. Keep a backup of configuration and your database before changing a deployed installation.