Meerschaum 4¶

v4.1.7¶

• Fix starting the API with FastAPI 0.145 or newer. The API passed its tag descriptions to FastAPI() under the misspelled keyword open_api_tags. Older FastAPI versions silently ignored unknown keywords, but FastAPI 0.145 removed that catch-all, so start api failed with "FastAPI.init() got an unexpected keyword argument 'open_api_tags'". The keyword is now openapi_tags, which also restores the tag descriptions on the API documentation pages.

v4.1.6¶

• Let tokens with jobs:stop stop jobs over the API. The stop route required the misspelled scope josb:stop, which is not a scope, so only * tokens and administrators could stop a job through the API. POST /jobs/{name}/stop now requires jobs:execute and jobs:stop, and a token which explicitly holds the old josb:stop is still accepted in place of jobs:stop. A refused request names jobs:stop.

• Store * literally so it grants every scope. edit tokens and edit users replaced a typed * with the built-in scope list, so an "all scopes" token or user lacked custom (plugin) scopes and any scope added later. * is now stored as *, which every scope check already honored. The prompts say "* grants all scopes, including custom and future scopes", and the "Grant all scopes" switch in the web console's token form registers *. register tokens --scopes '*' and the token registration route already kept a literal *. A token or user created without scopes still gets the built-in list.

• Let * execute actions and jobs for non-admins. The check for permission to run actions and jobs looked for actions:execute and jobs:execute in the granted scopes, so a non-admin whose scopes were * was refused with "Missing scope 'actions:execute'". A literal * now satisfies it.

• Keep * when editing a token in the web console. The edit form listed only the built-in scopes, so a token holding * showed an empty checklist and saving it replaced * with nothing. The form now has the same "Grant all scopes" switch as the registration form, turned on (with the checklist hidden) for a token which holds *, and saving keeps * unless the switch is turned off.

• Fix register tokens --nopretty. Printing the new token as JSON failed with "Object of type UUID is not JSON serializable". The token's ID and timestamps are now serialized, so the command prints the token and its secret as JSON.

• Match detached jobs only within their own root directory. When a job's PID file is lost, its process is found by the job's name in the command line. A job with the same name under another MRSM_ROOT_DIR matched too, so stopping or killing a job without a PID file could end another root's job. Processes are now matched only if their MRSM_ROOT_DIR is this root (unset means the default root). If a process's environment cannot be read, or its root is a relative path, it is still matched as before.

v4.1.5¶

• Stop scanning the process table for completed jobs. When a job has no process.pid file, reading its PID searched every running process's command line for the job's name, in case a detached job had lost its PID file. Completed jobs never have a PID file, and a single supervisor pass read the PID of every job many times (while building jobs, checking status and restarts, and formatting), so a production API with retained completed jobs spent most of a CPU core walking the process table every second. A job which recorded that its last run ended is now known to be stopped without any search, constructing a job and formatting it never search the process table, and a search that is needed (a job which began but never recorded an end) is shared by all jobs for two seconds and not repeated for a job which was not found for a minute. Starting, stopping, pausing, resuming, or deleting a job (or recording a new launch from another process) discards the remembered results. A running job which lost its PID file is still found, reported as running, and stoppable: its PID file is written again so later reads are cheap, and processes which started after the job recorded that it began (a reused PID) are ignored. With 20 completed jobs, five supervisor passes with status reads dropped from 3000 process-table scans (108 seconds) to none (under 1 second).

v4.1.4¶

• Don't restart running jobs with a bare compose start jobs. v4.1.3 scoped Compose job actions by expanding a bare compose start jobs into the project's job names, but naming a running job asks to restart it, so compose start jobs -y restarted every running job in the project. Bare job actions now expand only to the jobs the bare command would act on: start to stopped and paused jobs, restart to jobs created with --restart or --loop, and stop / pause to running jobs. compose stop jobs and compose pause jobs also keep their confirmation prompt.

v4.1.3¶

• Scope compose start jobs (and other job actions) to the project. Job actions without job names were passed straight through to mrsm, so mrsm compose start jobs started every stopped job in the root directory, not just the project's. Projects sharing a root directory then had their jobs started with the calling project's environment and config, losing their own connectors. compose start, stop, pause, restart, and delete jobs with no names now act only on the project's jobs, and naming a job owned by another Compose project is refused.

v4.1.2¶

• Serve already-imported plugins instantly from from_plugin_import(). Every call built a new Plugin, activated its virtual environment (re-parsing its dependencies), and re-ran import_plugins(), which builds a Plugin for every installed plugin and creates resource directories along the way: about 4 ms per call, or 16 ms with thirty plugins installed. A fully-imported submodule is now returned straight from sys.modules (about 0.01 ms per call), and Plugin.module reuses an already-imported module. Reloading or unloading plugins still picks up new code.

• Return one None per name when from_plugin_import() fails. A failed submodule import returned a single None even when several attributes were requested, so a, b = from_plugin_import(...) raised a confusing unpacking error far from the cause. It now returns one None per requested attribute, after the existing warning and stacktrace.

• Cut the fixed overhead of Pipe.get_data(). attempt_import() activated the mrsm virtual environment on every call, even for modules already imported, and get_data() makes dozens of these calls. Loaded modules are now returned directly. get_data() also skips re-sorting in Pandas when the datetime column is already unique and in order (as returned by the SQL ORDER BY); ties and nulls are still fully sorted. A warm get_data() on a small SQLite pipe dropped from 24 ms to 13 ms.

• Create one PostgreSQL unique structure for upsert pipes. PostgreSQL-family connectors created a unique index and then added an equivalent unique constraint, which built a second backing index over the same columns. New upsert pipes now create the constraint only, recognize its UQ_ backing index on later index passes, and drop either the new single structure or a legacy IX_ + UQ_ pair correctly.

• Serialize first reads of shared pipe metadata. Concurrent readers could observe an unfinished attribute load and cache a guessed table name, producing errors such as relation "public.acme_clients_sites" does not exist despite a registered clients_sites target. Attribute loads now serialize per pipe and mark metadata fresh only after success. Failed SQL or API reads warn and remain retryable, registered pipes with empty parameters do not cache guessed targets, and an explicit parameter refresh invalidates the target and freshness caches.

• Record clean daemon stops as successful. The daemon's own shutdown handler raises SystemExit(0), but the result recorder classified it as a failure and stored a red SystemExit: 0 traceback. Exit codes None and 0 now record a successful stop; nonzero exits, KeyboardInterrupt, and BrokenPipeError remain failures. A Failed to gracefully quit warning now includes the underlying quit reason.

• Keep full tracebacks for failed daemon actions. Non-debug daemon actions previously discarded the stack trace and suggested rerunning with --debug, losing the evidence needed to diagnose intermittent background failures. Daemons now log the full traceback while retaining a concise failure result; foreground non-debug errors remain concise.

v4.1.1¶

• Honor an explicitly empty routes allowlist. api:permissions:routes:allowlist: [] (the documented way to serve nothing but /login and plugin routes) was treated as unset ([] or ['*']) and silently served every core route group. An empty list now registers no core route groups; only a missing (or null) allowlist falls back to ['*'].

• Boot with a restricted allowlist. With no authenticated core routes registered, the generated OpenAPI schema has no components.securitySchemes, and the startup event crashed with KeyError: 'securitySchemes', a crash loop, on exactly the locked-down deployments the allowlist exists for. The schema sections are now created when absent.

v4.1.0¶

• Filter Polars DataFrames against a pipe without converting to Pandas. Pipe.filter_existing() raised AttributeError: 'DataFrame' object has no attribute 'empty' when handed a polars.DataFrame, because the guard which detects non-DataFrame input matches on the string 'dataframe' and a Polars frame's type name contains it. A Polars DataFrame or LazyFrame is now accepted, and the three returned frames (unseen, update, delta) are Polars frames rather than Pandas ones.

The comparison itself also runs natively: the existing rows are read with Pipe.get_data(as_polars=True), and both the delta anti-join and the unseen / update split are performed by Polars, so no frame is round-tripped through Pandas. On a 150k-row sync against SQLite this cuts filter_existing() from 1.49s to 0.94s. Frames which Polars cannot compare (uuid, object, and geometry columns, or pipes using autoincrement / autotime) transparently fall back to the Pandas implementation and are converted back to Polars on the way out. Where the incoming and stored frames disagree on a dtype within the same family (Pandas never downcasts, so an incoming int64 meets Oracle's stored int32), both are widened to their common dtype; anything else, such as an int against a float, falls back to Pandas, which coerces mixed numerics into numeric values. Pandas remains the default: a Pandas frame in is still filtered by Pandas and returned as Pandas.

import polars as pl
import meerschaum as mrsm

pipe = mrsm.Pipe('demo', 'polars', instance='sql:main')
unseen_df, update_df, delta_df = pipe.filter_existing(pl.DataFrame(docs))

• Speed up shell autocompletion. Typing a flag in the shell was laggy: -c (and any other incomplete flag) re-parsed the line through the pretty-printed error path on every keystroke, and completing a connector re-imported every installed plugin. argparse failures now raise directly instead of routing through meerschaum.utils.warnings.error() (whose Rich traceback was built and then discarded), and get_data_plugins() caches its result against the installed plugins' names. With 21 plugins installed, a keystroke after -c fell from 275 ms to 8 ms.

• Preserve dtypes when splitting new rows from updated rows. filter_existing() separated the two with joined_df.where(mask).dropna(how='all'), which masks the non-matching rows to null before dropping them. That upcast integer and boolean columns to float or object on the way through, and silently discarded any row whose values were all null. Both frames are now selected with a boolean mask, which keeps the dtypes and the rows.

• Grant admins every scope at /login. The password grant decided which requested scopes to grant from '*' in allowed_scopes alone, so an admin whose stored scope list was narrow (or merely lacked '*') received a token missing scopes their admin type already implies (admins short-circuit scope enforcement everywhere else (is_user_allowed_to_execute()). An admin-typed user is now treated as holding '*' at grant time: requesting specific scopes grants them, and requesting none grants ['*']. API keys are unchanged) a deliberately narrowed key stays narrow.

• Gate the web console's pages behind the routes allowlist. api:permissions:routes:allowlist restricted the core FastAPI route groups, but the web console registered its pages regardless, and the only recourse was --no-dash, unusable when plugin pages are served through the Dash app. The console's own pages (/dash, /dash/pipes, /dash/users, /dash/tokens, /dash/jobs, /dash/plugins, /dash/register) now register only when the new dash route group matches the allowlist. The login page is always served (it is the auth entrypoint for plugin pages requiring a login), pages registered by plugins via @web_page are unaffected, and the default allowlist (['*']) serves everything as before. The index redirect points at the login page when the console is stripped.

# Serve only /version, /login, and plugin-registered routes and pages:
api:
  permissions:
    routes:
      allowlist: ['version']

• Set a session cookie at login. The web console kept its session ID only in a dcc.Store (browser local storage), which Dash callbacks can read but plain HTTP routes cannot, so a route added by an API plugin could never see that the browser was already signed in. Logging in now also mirrors the session ID into an HttpOnly, SameSite=Lax cookie (mrsm-session-id, scoped to /, Secure over HTTPS), and signing out clears it. Read it server-side with meerschaum.api.dash.sessions.get_session_id_from_request() and validate with is_session_authenticated(). Dash callbacks still authenticate through the session store, as before.

• Tolerate comments and nested CTEs when wrapping SQL definitions. wrap_query_with_cte() scanned for a definition's top-level SELECT character-by-character, without skipping comments or string literals. A -- comment before the final SELECT swallowed the CTE-joining comma and spliced the comment's text into the generated SQL; a comment before a leading WITH skipped flattening entirely and nested the CTE illegally (on SQL Server); and a parent query with CTEs of its own (the parents pushdown path) always emitted an invalid second WITH. All three produced broken SQL silently, surfacing only as "No data were fetched". The scan now skips -- and /* */ comments, string literals, and quoted identifiers; parent CTEs are hoisted into the enclosing WITH list; and a definition whose top-level SELECT cannot be located raises a clear error instead of emitting invalid SQL. The same comment-aware scan replaces the rfind('select') in SELECT INTO creation on SQL Server, which could match inside a comment or an identifier like selected_at.

• Keep subscription connectors alive with sync pipes --listen. A connector whose sync() subscribes to a stream (e.g. MQTT) and syncs via callbacks had to keep its process alive with --loop --min-seconds <huge>, which misrepresents the job (it is not resyncing) and picks an arbitrary interval. sync pipes --listen runs the initial sync pass and then blocks until a stop is requested (stop jobs, SIGTERM, or CTRL-C), so the subscription keeps syncing for the life of the job.

mrsm sync pipes -c mqtt:main --listen

• Provision plugin venvs on the first compose init. compose init decided whether to run install required / setup plugins from a snapshot of installed plugins taken before the project's plugins were installed. In a fresh checkout or git worktree that snapshot is empty, so the first init skipped dependency provisioning entirely and a second run was required. The check now re-discovers installed plugins after the install pass.

• Warn when a pipe declares both upsert and compress. Compressing a chunk invalidates the composite unique index the upsert's ON CONFLICT clause needs, so an upsert which re-syncs history older than compress:after fails with "there is no unique or exclusion constraint matching the ON CONFLICT specification", long after the configuration looked accepted. Applying a columnstore policy to an upsert pipe now warns (once per pipe per process) naming the failure and the condition under which the combination is safe.

• Surface plugin import failures on Plugin.import_error. mrsm.Plugin('name').module returned None when the plugin's import raised, with the exception swallowed, indistinguishable from the plugin not being installed. The most recent import failure (including a missing venv) is now retained and exposed as Plugin.import_error, and cleared when the plugin imports successfully or is unloaded.

• Cast strings to dates and read lazy schemas without deprecated Polars calls. Enforcing a date dtype on a string column cast straight to polars.Date, which Polars deprecates and removes in 2.0, and enforce_dtypes() read LazyFrame.columns, which silently resolves the frame's schema. String columns now parse with str.to_date() (matching how datetime columns are already handled), and a lazy frame's column names come from collect_schema().names().

• Scan for stale temporary tables once a minute, not once a sync. _drop_old_temporary_tables() throttles its stale-table scan to once every 60 seconds, but recorded the timestamp only on the branch that skips the scan. The attribute therefore never advanced past its initial 0, perf_counter() - 0 never fell under the threshold, and every single sync_pipe() ran the full scan.

The scan then read every stale row out of mrsm_temp_tables and inserted a fresh duplicate row for each one, issued as one INSERT per row, while _drop_temporary_tables() only deletes rows for the tables in its in-memory set, non-empty during a sync. The duplicates written for every other table survived, so the count doubled on each pass. A test instance left running for three days reached 12.7 million rows across 1,472 distinct table names, roughly 8,500 copies each, and every subsequent sync replayed millions of single-row inserts against it.

The timestamp is now recorded when the scan actually runs, the stale tables are flagged in place with one idempotent UPDATE ... SET ready_to_drop WHERE date_created < end AND ready_to_drop IS NULL, and both bookkeeping selects are DISTINCT. Oracle felt this worst (it has no DROP TABLE IF EXISTS, so each drop also pays an existence check and a dedicated connection) where the integration suite had slowed from 412s to 8,083s for that flavor alone.

v4.0.3¶

• Adopt Compose jobs which run compound commands. Meerschaum rewrites a compound job command (... + ... : --loop) into the start pipeline invocation it records on the daemon, so it never matched the command configured in the compose file. A pipeline job started before v4.0.2 stamped its project onto it therefore read as foreign, and compose up refused to replace it until the job was deleted by hand. Both commands are now compared in their condensed form.

jobs:
  maps: >
    sync pipes -c plugin:osm -t maps
    + sync pipes -c sql:main -m segments -t maps
    : --loop --min-seconds 604800

v4.0.2¶

• Recognize Compose jobs which declare their own tags. compose up proved a job belonged to the project by the -t <project_name> tag it injects, and it injects that tag only when the job's command carries no tags of its own. A job which sets -t to select the pipes it syncs therefore read as foreign, and up aborted with "Refusing to replace jobs not owned by this Compose project", after the pipe changes had already been applied. Adding the project name as a second tag isn't a fix: tags are an OR when selecting pipes, so the job would sync every pipe in the project.

A job started inside a project now records that project's name (MRSM__COMPOSE_PROJECT) in its properties, which proves ownership no matter how the job's command later changes. Only the name is persisted; MRSM__COMPOSE_CONFIG carries the project's config: block and never reaches a job. Jobs started before this release are adopted by their command: a job running exactly what the project would start is the project's job. Editing such a job's command in the YAML replaces it, as it did under the Compose plugin, because Job keeps an existing daemon's command and only a delete-and-recreate applies new arguments. A job running a different command under a configured name, with no stamp, is still refused, and compose down reaps the project's jobs instead of orphaning them.

jobs:
  recompute: >
    sync pipes -c sql:main -t recompute --loop --min-seconds 300

v4.0.1¶

• Start the configured jobs instead of syncing pipes in compose up. A project with a jobs key registered its pipes and then ran an initial sync pass over them before starting anything, so a failing sync aborted up and the configured jobs never started. When jobs is defined it is the project's workload: up registers the pipes and starts the jobs. mrsm compose up --presync still forces a sync pass.

• Install Meerschaum without any dependencies. fasteners was the last hard requirement, needed before the mrsm venv exists because runtime package installation is serialized with an inter-process lock. That lock is now meerschaum.utils.locks.InterProcessLock, built on fcntl (POSIX) and msvcrt (Windows), so pip install meerschaum pulls nothing and every dependency is installed into the mrsm venv on demand (or up front via an extra, e.g. meerschaum[api]). Locking behavior is unchanged for package installs, plugin symlinks, job restart checks, the SQL metadata cache, and daemon PID files.

v4.0.0¶

• Build Meerschaum Compose into the core package. compose is now a built-in action, so existing mrsm-compose.yaml projects run without installing the compose plugin. The subactions (up, down, run, ps, logs, explain, init), their flags, and arbitrary nested actions behave as before, as do project plugin discovery and per-project root and plugins directories. Each command loads the project's config, environment, and plugins for the duration of the subaction and restores the host process state afterward, so chaining commands does not leak project state. The code lives at meerschaum.compose, so plugin-era imports still resolve. If the old plugin is still installed, the built-in action takes precedence and warns once to uninstall it. New projects no longer copy the plugin into the project directory. Running mrsm compose without a subaction opens a shell in the project environment, and the shell prompt and bottom toolbar name the active project. The executor keys moved out of the prompt into that slot; they are still shown in the bottom toolbar.

mrsm compose up

 [ mrsm@sql:awesome | awesome-sauce ] ➤

• Add opt-in Polars support. Pipes accept polars.DataFrame and LazyFrame inputs, including chunk generators, and Pipe.get_data() may return Polars frames. Geometry columns use GeoArrow WKB and JSON columns use Arrow JSON. Pandas remains the default output and the plugin boundary.

pip install 'meerschaum[polars]'

df = pipe.get_data(as_polars=True)
pipe.sync(df)

• Replace the APScheduler fork with a built-in schedule engine. Interval, calendar, cron, starting, and and / or schedule strings keep their behavior and the trigger.next() interface, now with correct DST handling and no overlapping runs of the same callback.

mrsm sync pipes --schedule 'every 30 minutes starting 00:15'

• Make plugin registration and locking deterministic. Decorators bind to the root plugin module even when defined in submodules, unloading a plugin clears every registry it wrote to, and source and dependency installs are guarded by an inter-process lock. Failed setup hooks and failed upgrades restore the previous state instead of leaving a partial install.

• Control runtime package installation. Set MRSM_NO_AUTO_INSTALL=1 to stop attempt_import() from downloading missing packages, or preview an install with --dry-run. Automatic installation stays on by default and now prefers uv, falling back to pip.

mrsm install packages polars --dry-run

• Fix Dask, geometry, and Docker regressions. Dask frames are detected during syncs and SQL reads, SQL geometry dtype filtering skips columns absent from the frame, and Docker images build with the maintained buildx script.