Meerschaum 3¶

v3.5.2¶

• Fix a stale schema cache surviving DELETE /pipes/{ck}/{mk}/{lk}/cache. A pipe's column types are cached in memory only, but Pipe._invalidate_cache(hard=True) discovered the keys to clear by listing what had been written to disk (or to the cache connector). For a pipe with cache=False (every pipe the API server holds) that list was empty, so a hard invalidation cleared nothing and the route which exists to refresh the schema after a schema-changing sync was a no-op. An API client which added a column and read the pipe back could receive the previous column list for up to a minute. The in-memory schema keys are now cleared explicitly.

• Scale an explicit chunk interval to an integer datetime axis. Pipe.get_chunk_interval() applied the pipe's precision only when resolving the interval from the verify parameters, never when given one directly. repartition (and partition pipes --chunk-minutes) passes its width through that path, so asking for 30-day chunks on a millisecond axis set a chunk interval of 43,200 milliseconds (about 43 seconds) while reporting success and persisting verify.chunk_minutes as 43200. An int argument is still taken verbatim as the axis's own epoch units; a timedelta is now scaled by precision, and the callers which mean minutes pass a timedelta. repartition reports an integer axis's new width in epoch units.

• Register an integer_now function for integer-axis hypertables. TimescaleDB cannot evaluate "older than N" for a columnstore policy on a hypertable partitioned by an integer column until such a function exists. Meerschaum never called set_integer_now_func, so the policy was accepted at creation and then failed on every background run with integer_now function not set on hypertable, compression silently never happened, on exactly the tables which grow largest. A mrsm_integer_now_<unit>() function derived from the pipe's precision is now registered before any columnstore policy is installed, including the one TimescaleDB auto-creates for a Hypercore table at CREATE TABLE.

• Honor a duration compress:after on an integer datetime axis. compress: {after: '30 days'} (the documented spelling on a datetime axis) was discarded on an integer axis and replaced with a hard-coded 7-day equivalent, with no warning and with the configured value still visible in the pipe's parameters. Durations are now parsed and scaled by the axis precision, so the same spelling means the same thing on both axis types. A bare number is still used verbatim as epoch units, and a value which cannot be parsed warns and names the fallback rather than silently substituting a different retention.

• Fix /login dropping a user's custom scopes. The password grant read scopes from a freshly constructed User without refreshing its attributes from the instance, so any scopes stored in a user's attributes were ignored and the issued JWT silently carried the default scope set, routes gated on a custom scope returned 403 with valid credentials. Separately, an omitted scope form field was treated as an explicit request for every default scope, which filtered out anything beyond that set; per RFC 6749 § 3.3, an omitted scope now grants the principal's allowed scopes (and ['*'] for all-scope tokens).

• Probe the cache connector before using it. get_cache_connector() memoized the configured connector without ever contacting the server, so a misconfigured or unreachable cache looked exactly like "no cache" while every cache operation failed at the call site. The connector is now pinged when first resolved; on failure the API warns, continues without a cache (the documented None path), and probes again after 60 seconds so a cache which is merely slow to start isn't disabled for the life of the process.

• Restrict the served routes with api:permissions:routes:allowlist. Serve a minimal API surface (e.g. only the routes added by your API plugins) by listing which groups of core routes to register (fnmatch patterns against actions, connectors, index, jobs, misc, pipes, plugins, tokens, users, version, mcp, and webterm; the default ['*'] serves everything). Disallowed groups are never registered, rather than removed from the app after the fact. /login is always served so that issued tokens keep authenticating plugin routes, and the web console registers its routes independently (start with --no-dash for a restricted surface).

# Under api:permissions, serve only `/version` alongside plugin routes.
routes:
  allowlist:
    - 'version'

v3.5.1¶

• Fix upserting rows which null out json, numeric, or uuid columns. Patching a row so that a special-dtype column becomes NULL failed on PostgreSQL-like flavors with an error such as column "j" is of type jsonb but expression is of type text. An all-null column carries no values to inspect, so the temporary staging table created it as TEXT instead of the column's real type. Staged columns now fall back to the pipe's registered dtype when every incoming value is null.

import meerschaum as mrsm

pipe = mrsm.Pipe(
    'test', 'upsert', 'nulls',
    instance='sql:main',
    columns={'primary': 'id'},
    upsert=True,
    dtypes={'id': 'int', 'j': 'json'},
)
pipe.sync([{'id': 1, 'j': [1, 2]}])
pipe.sync([{'id': 1, 'j': None}])

v3.5.0¶

• Add a first-class MCP server. AI assistants can now discover your pipes, read their data, and run actions, 15 tools, 5 resources, and 3 prompts. Run it locally over stdio with mrsm start mcp, or remotely at an API instance's /mcp endpoint (where each tool requires the same scope as its REST equivalent, and tools your token can't call are hidden).

{
  "mcpServers": {
    "meerschaum": {"command": "mrsm", "args": ["start", "mcp"]}
  }
}

This replaces the community mcp plugin, uninstall it with mrsm uninstall plugin mcp. Read MCP security before exposing /mcp: api:mcp:read_only hides every modifying tool, execute_action denies the code-executing actions by default, and read_sql accepts only a single read-only statement under its own sql:read scope.

The MCP tools enforce the same guardrails as the REST routes: pipes targeting the protected instance tables (mrsm_users, mrsm_tokens, mrsm_pipes, mrsm_plugins) can't be registered, read, or written; clear_pipe and deduplicate_pipe require pipes:delete; instance_keys is checked against the API's allowed instances; and execute_action applies the server's action permissions, pins the instance, and refuses a denied action passed as another action's subaction. Tool calls run off the event loop, so a long sync no longer stalls other requests.

• Reject json columns used as index columns. JSON values can't be sorted or deduplicated, so such a pipe synced once and then failed with TypeError: unhashable type: 'dict'. Registering, editing, or syncing now fails immediately instead. Use indices if you only want a database index on the column. See Data Types.

• Require authentication for web console actions which read or write data. Twelve Dash callbacks (pipe data, parameter and SQL edits, syncing documents, the users listing, job logs, and tokens) could be triggered without a session by posting to the callback endpoint directly.

• Fix --no-auth rejecting every authenticated route. mrsm start api --no-auth returned 401 Not authenticated for all endpoints instead of skipping authentication.

• Require authentication to list API tokens in the web console. The tokens page's refresh callback could be triggered without a session, returning every token's client ID, label, and expiration.

• Fix the web console minting tokens with all scopes. Deselecting "Grant all scopes" and choosing individual scopes had no effect, the registered token always carried every scope.

• Enforce the registration permission in the web console. With api:permissions:registration:users disabled, the web console's register form still created accounts; only the REST route checked it.

• Add Plotly Dash's MCP server for plugin authors. Set api:dash:mcp:enabled (requires dash>=4.3.0) to expose the web console's layout at /dash/_mcp, so an assistant can inspect the component tree while you write a web page plugin. Off by default because it's served without authentication. See Web Console.

v3.4.7¶

• Reduce CPU usage while watching job logs. Rich console formatting is now reused across log lines instead of being rebuilt for every line.

• Clear the plugins update cache after upgrading plugins. upgrade plugins now removes its cached remote-version results after a successful install, so the next shell does not show an outdated upgrade notice.

• Re-sync plugins symlinks after clear cache. The action clear cache now delete and rebuilds internal plugins symlinks (clearing out symlinked plugins from other plugins dirs).

• Move plugins update cache to .internal/cache. The plugins update checker files now reside within .internal/cache.

v3.4.6¶

• Fix venvs path not following set_root(). When MRSM_VENVS_DIR is unset, VIRTENV_RESOURCES_PATH now derives from ROOT_DIR_PATH dynamically, so in-process root swaps (e.g. compose running custom actions under replace_root_dir()) activate venvs from {root}/venvs, matching where subprocess-mode commands like install required install packages. Previously the path was frozen at import time, so custom plugin actions run through compose could not import their own required dependencies.

v3.4.5¶

• Fix a breaking bug in bootstrap plugin. A bug in package_venv() has been fixed, unblocking bootstrap plugin.

• Fix warning bug on shell launch. The plugin updater thread was throwing a login warning for api:mrsm; this warning has been surpressed.

• Require importlib_metadata when starting the API. Some dash dependencies require the legacy importlib_metadata, so this behavior has been accounted for.

• Fix plugins environment scope for compose. The plugins namespace scope has been fixed to harden the isolation of compose.

v3.4.3 – v3.4.4¶

• Add a plugin version API endpoint. The endpoint /plugins/{name}/version now returns a plugin's version string.

• Hide login warning on plugin updater thread. The plugins updater thread no longer will raise a login warning.

• Add precision kwarg aliases to get_current_timestamp(). You may now treat Pipe.precision as kwargs for get_current_timestamp():

import meerschaum as mrsm
from meerschaum.utils.dtypes import get_current_timestamp

pipe = mrsm.Pipe(
    'demo', 'precision', 'timestamp', instance='sql:local',
    precision={'unit': 's', 'interval': 30},
)
now = get_current_timestamp(**pipe.precision)
print(now)
# 2026-06-23 04:30:30+00:00

• Sync plugins symlinks on daemon fork. To mitigate isolation issues with Meerschaum Compose, Daemons now sync plugins' symlinks before forking.

v3.4.2¶

• Run inline Python with mrsm python -c. The python action now accepts Python's own -c CODE idiom (e.g. mrsm python -c "print(mrsm.__version__)"), executing the code with meerschaum already imported and then exiting. Globally -c means --connector-keys, so this rewrite is scoped to the python action only; every other action's -c is unchanged. The previous forms (mrsm python "code" --noask and the bracketed mrsm python [-c "code"]) still work.

• Fix truncated output for actions run on a remote api: executor. Running a long-output action (e.g. show pipes) on an api: executor only showed the last 30 lines, the beginning was silently dropped. The remote job's log monitor defaults to emitting only the last lines_to_show (30) lines, which is the right behavior when attaching to a long-running job but wrong for a one-shot action whose entire output matters. Temporary action jobs created by APIConnector.do_action_async now set logs.lines_to_show = 0 (emit everything), mirroring the local CLI worker.

• Add a Users page to the web console. A new /dash/users page lists the instance's registered users (paginated and searchable, mirroring the Plugins page), with a shareable per-user detail page at /dash/users/<username> that lists the plugins they've published. A plugin card's author now links to its owner's user page. A user's type (admin vs. normal) is intentionally never exposed.

• Format show data output as a table. mrsm show data now renders each pipe's DataFrame as a Markdown table (the same pprint_df() formatter used by mrsm sql) instead of the raw Pandas repr. The --nopretty JSON output is unchanged.

• Add a plugins update checker. Plugins installed from a repository now record their origin (e.g. api:mrsm) in a per-directory .mrsm_origins.json manifest that travels with the plugins directory (so it survives being mounted into another environment). On shell launch a background thread caches the latest versions from each plugin's origin repo, and the interactive prompt prints a banner listing any plugins that have fallen behind, suggesting upgrade plugins. Locally-authored plugins (no recorded origin) are skipped. Reuses the existing shell:updates config (check_remote, refresh_minutes).

• Silence the starlette.middleware.wsgi deprecation warning during start api. The Dash app is now mounted with a2wsgi's WSGIMiddleware instead of the deprecated fastapi.middleware.wsgi.WSGIMiddleware.

v3.4.1¶

• Scope the dbc_dark theme so it no longer leaks onto every Dash page. Several rule blocks in dbc_dark.css (.form-control, the .dash-dropdown-* family, and .dash-options-list) were unscoped, so their background-color: var(--bs-dark) !important declarations applied to every Dash app served by the API, forcing plugins to fight each one with higher-specificity !important overrides. These blocks are now scoped under .dbc_dark, making the dark theme opt-in.

To keep the web console (including its body-portaled dropdown menus) unchanged, the Dash app now sets class="dbc_dark" on <body> by default.

• Add a light theme for plugin pages via @web_page(dark_theme=False). Plugin pages render with the console's dark theme by default. A page can now opt into a light theme with @web_page(dark_theme=False): the Web Console loads both the dark (Bootswatch Darkly) and light (Flatly) Bootstrap themes and enables exactly one per route, switching to light (and dropping the dbc_dark class) while an opted-out page is active, and back to dark for the console and every other page, including on in-app navigation. See Styling Your Page.

• Fix starting jobs whose target reaches unpicklable state (e.g. an asyncio.Task). Starting a job (e.g. sync pipes ... --loop --name foo) could fail with TypeError: cannot pickle '_asyncio.Task' object when a connector cached a live async task. The daemon pickled its target function by value, dragging in the target's global graph; that walk reached the unpicklable state. Importable, top-level targets are now pickled by reference (re-imported in the daemon process), so the global graph is never serialized. Closures still pickle by value, unchanged.

• Add a Termux-style on-screen key row to the web console terminal (mobile). On small screens the Webterm now shows a row of keys above the terminal, ESC, CTRL, SHIFT, TAB, and arrow keys laid out like a physical keyboard. CTRL/SHIFT are sticky modifiers that apply to the next keypress (tap again to toggle off), and tapping a key keeps the soft keyboard open. The row is hidden on larger screens with a physical keyboard.

• Web console UX and accessibility improvements.

  • Loading spinners while pipes, jobs, and tokens load.
  • A confirmation dialog before deleting a job.
  • The tokens table is now horizontally scrollable on small screens.
  • Tooltips/labels on icon-only buttons and alt text on the logo and banner images.
  • Errors that were previously swallowed (CSV download, a pipe's columns / recent-data accordion) now surface their message.

• Fix the web console fullscreen terminal rendering at half width. Toggling the Webterm's fullscreen button clobbered the column's responsive classes and left the terminal narrow without refitting. Fullscreen now preserves the responsive layout and the terminal refits to fill the width.

• Fix the web console "Plugins" button raising a ValueError. Clicking Plugins in the dashboard raised too many values to unpack after the plugins view started returning pagination metadata. The dashboard now ignores the extra trailing values.

v3.4.0¶

• Accept datetime bounds on epoch integer-axis pipes. Actions like show data, sync pipes, clear pipes, and deduplicate pipes now accept a datetime --begin / --end for pipes whose datetime axis is an integer epoch. The bound is translated to the axis's integer value using the pipe's precision, so a single command can target both datetime- and integer-axis pipes at once.

# The datetime is translated to an epoch (e.g. milliseconds) for the integer 'ts' axis.
mrsm show data --begin 2026-05-30 -c mqtt:acme -m test -l backup -i sql:main

Integer bounds (--begin 1780099200000) still pass through unchanged. A datetime bound on a non-epoch integer axis (no precision set) now raises a clear error instead of producing an invalid query.

• Add datetime_to_int() to meerschaum.utils.dtypes. Convert a datetime to an integer epoch at a given precision unit (e.g. 'millisecond'). Naive datetimes are treated as UTC and integers pass through unchanged. get_current_timestamp(..., as_int=True) now reuses this helper.

• Add disk usage and compression tools. New tooling helps manage how much space pipes occupy on disk:

  • Add the show sizes action. Lists pipes with each target table's on-disk size, sorted largest first. Sizes are fetched in parallel when the instance connector is thread-safe (otherwise serially). show rowcounts now also sorts largest first.

  • Add Pipe.get_size(). Returns a target table's on-disk size in bytes (or None if unavailable). Backed by the new instance-connector method get_pipe_size(), with per-flavor support for SQL connectors, TimescaleDB hypertables use hypertable_size(), PostgreSQL-like flavors use pg_total_relation_size(), and MySQL/MariaDB, MSSQL, and SQLite use their respective size queries.

  • Add the compress pipes action and Pipe.compress(). Compress pipes' target tables to reclaim disk space. For TimescaleDB this enables native compression, installs a compression policy, and compresses existing chunks; MySQL/MariaDB and MSSQL use their native table compression. Backed by the new instance-connector method compress_pipe().

  • Add the decompress pipes action and Pipe.decompress(). The inverse of compress pipes. For TimescaleDB this removes the compression policy, converts compressed chunks back to row-store, and disables the columnstore so future synced chunks stay uncompressed; MySQL/MariaDB and MSSQL revert their native table compression. Pass --no-policy to decompress existing chunks now while leaving the policy in place (e.g. for a bulk backfill, after which chunks are recompressed on schedule). Backed by the new instance-connector method decompress_pipe().

  • Add the vacuum pipes and analyze pipes actions (and Pipe.vacuum() / Pipe.analyze()). vacuum pipes reclaims disk space from dead rows (--full runs VACUUM FULL on PostgreSQL-family flavors; MySQL/MariaDB use OPTIMIZE TABLE, MSSQL rebuilds indices, SQLite runs VACUUM). analyze pipes refreshes the query planner's statistics without reclaiming space. Backed by the new instance-connector methods vacuum_pipe() and analyze_pipe(). See Maintenance.

  • Add the show sizes companion show partitions action. Lists each partitioned pipe's partition (or TimescaleDB chunk) count and physical width. Backed by the new instance-connector method get_partition_info().

  • Add the compress pipe parameter. Set compress (a bool or a dictionary of after/segmentby/orderby settings) to mark a pipe for compression. For TimescaleDB hypertables, a compression policy is installed automatically on sync.

  import meerschaum as mrsm
  
  pipe = mrsm.Pipe(
      'demo', 'compress',
      instance='sql:main',
      columns={'datetime': 'ts', 'id': 'station'},
      compress={'after': '7 days'},
  )
  

  compress pipes --no-policy also accepts --no-policy, to compress existing chunks immediately without installing an ongoing policy.

  • Add the hypercore pipe parameter (TimescaleDB). hypercore (default True) enables the Hypercore columnstore at CREATE TABLE (declaring tsdb.segmentby / tsdb.orderby), so TimescaleDB auto-creates a columnstore policy that converts old chunks in the background. Set hypercore to False for a plain row-store hypertable. The columnstore policy is the compression policy, add_columnstore_policy is the modern equivalent of the legacy add_compression_policy.

• Add native range partitioning for non-TimescaleDB flavors. TimescaleDB auto-creates chunks on insert; PostgreSQL / PostGIS, MySQL / MariaDB, and MSSQL do not. The hypertable parameter (which defaults to True) now drives declarative range partitioning on the pipe's datetime column for these flavors too (so datetime-axis pipes are partitioned by default; set hypertable to False to opt out. Only newly created tables are affected (pre-existing plain tables are never retroactively partitioned). The partition width reuses the chunk interval (verify.chunk_minutes, default 43200) 30 days), and boundaries are epoch-aligned so the same value always maps to the same partition (deterministic, value-independent). A single sync creates at most system.connectors.sql.instance.max_partitions_per_sync partitions (default 10,000) as a runaway guard.

import meerschaum as mrsm

pipe = mrsm.Pipe(
    'demo', 'partition',
    instance='sql:main',  # a PostgreSQL/MySQL/MSSQL connector
    columns={'datetime': 'ts', 'id': 'station'},
    parameters={
        'hypertable': True,
        'verify': {'chunk_minutes': 43200},  # 30-day partitions (default)
    },
)

The parent table is created with the flavor's range-partitioning DDL (PARTITION BY RANGE for PostgreSQL, PARTITION BY RANGE COLUMNS for MySQL / MariaDB, a partition function + scheme for MSSQL), and the interval-aligned child partitions a dataframe needs are auto-created in sync_pipe before its rows are inserted. The partition column is folded into the primary key as required by each flavor. Pipe.verify() aligns its chunk edges to the same epoch-anchored grid (get_chunk_bounds(align=True)) so verification chunks coincide with partition boundaries. See SQL Connectors → Native Range Partitioning for the per-flavor details.

verify.chunk_minutes is the authoritative partition width, it is read at sync time, so editing it for a populated table can create misaligned, overlapping partitions. Use the new partition pipes action (below) to change an existing table's width.

• Expand the verify chunk-interval keys. The chunk interval (used both for verification syncs and as the native partition width) can now be set with whichever unit reads most naturally, instead of always converting to minutes.

  • Unit aliases for verify.chunk_minutes. chunk_hours, chunk_days, chunk_weeks, chunk_years, and chunk_seconds join chunk_minutes. If several are set, the first on this priority list wins (mirroring the existing bound_* keys): chunk_minutes, chunk_hours, chunk_days, chunk_weeks, chunk_years, chunk_seconds. The default is unchanged (chunk_minutes: 43200, 30 days).

  import meerschaum as mrsm
  
  # These two pipes use the same one-week chunk interval.
  mrsm.Pipe('demo', 'minutes', parameters={'verify': {'chunk_minutes': 10080}})
  mrsm.Pipe('demo', 'days',    parameters={'verify': {'chunk_days': 7}})
  

  • Add verify.chunk_range for integer datetime axes. For a pipe whose datetime axis is an integer epoch, chunk_range sets the chunk size directly in the axis's own units (used verbatim), the integer-axis counterpart to chunk_minutes.

  import meerschaum as mrsm
  
  pipe = mrsm.Pipe(
      'demo', 'int_axis',
      columns={'datetime': 'ts'},
      dtypes={'ts': 'int'},
      parameters={'verify': {'chunk_range': 1000}},  # 1000 units per chunk
  )
  

  When chunk_range is unset, the legacy behavior is preserved: the time-based size is converted to the axis's units via the pipe's precision, or (when no precision is set) its value in minutes is used verbatim.

The --chunk-minutes flag and an explicit chunk_interval argument to Pipe.get_chunk_interval() still override the configured keys.

• Add the partition pipes action and Pipe.repartition(). Rebuild a partitioned pipe's target table to a new chunk width. Pass --chunk-minutes (defaults to the pipe's verify.chunk_minutes). On TimescaleDB this calls set_chunk_time_interval() (applies to future chunks; existing chunks keep their size); on PostgreSQL / PostGIS, MySQL / MariaDB, and MSSQL the table is rebuilt at the new width (read → drop → re-sync) and verify.chunk_minutes is updated. Backed by the new instance-connector method partition_pipe().

# Repartition a pipe's table to 7-day chunks.
mrsm partition pipes -i sql:main -m weather --chunk-minutes 10080

!!! warning The non-TimescaleDB rebuild reads the whole table into memory and briefly drops it before re-syncing. Run it during a maintenance window for large tables.

• Wire the maintenance actions through APIConnector. APIConnector now implements get_pipe_size, compress_pipe, decompress_pipe, vacuum_pipe, analyze_pipe, and partition_pipe (with matching /size, /compress, /decompress, /vacuum, /analyze, and /partition routes), so show sizes, compress / decompress, vacuum / analyze, and partition all work against api: instances (previously these reported "not supported" over the API).

• Print full, untruncated results from mrsm sql. Reading a table or query with sql ... read now prints the entire DataFrame as a Markdown table, no more ... column or cell truncation. The format is friendly to both users and LLMs and ends with a [rows x columns] shape footer. The --nopretty JSON output is unchanged.

• Back up configuration files before editing. Running edit config now snapshots the existing file before opening your editor. Backups are kept under MRSM_ROOT_DIR/.internal/config_backups/<key>/ (the ten most recent per key).

• Add edit config --rollback. Restore a configuration key from its most recent backup instead of opening the editor. The current file is itself backed up first, so a rollback can always be undone.

# Roll back the 'stack' configuration to its previous state.
mrsm edit config stack --rollback

• Stop overwriting config files with defaults. A malformed configuration file (e.g. from a bad manual edit) is no longer silently overwritten with default values. Meerschaum now falls back to defaults in memory, leaves the file on disk untouched, and points you to edit config <key> --rollback. Brand-new config files are still populated on upgrade.

• Fix stop jobs leaving zombie sync threads. Stopping a job now reliably terminates sync pipes worker threads. Previously the daemon's main thread could exit while non-daemon worker threads kept running and appending to the log, requiring a container restart. Workers now honor a process-wide stop signal and are interrupted when a job is stopped.

• Fix get_sync_time() with params on partitioned MariaDB tables. Because datetime-axis pipes are now range-partitioned by default, get_sync_time(params=...) could return None for a populated MariaDB pipe, a MariaDB optimizer quirk where ORDER BY <dt> DESC LIMIT 1 over a RANGE COLUMNS table with a WHERE clause stops scanning early. Partitioned MariaDB pipes now compute the bound with MAX() / MIN(), which prunes partitions correctly. Other flavors are unaffected.

• Fix native partitioning for date-axis and distant-datetime pipes. Two partition-boundary edge cases no longer error on sync. (1) A pipe whose datetime column has a date dtype (rather than a timestamp) raised TypeError: int() argument must be ... not 'datetime.date'; the boundary helpers now promote a date to a midnight datetime so it follows the timestamp grid instead of the integer-epoch path. (2) A very wide datetime span combined with a large chunk interval could push a computed boundary past datetime.min / datetime.max and raise OverflowError: date value out of range; boundaries are now clamped to the representable range. Affects PostgreSQL / PostGIS, MySQL / MariaDB, and MSSQL.

v3.3.1¶

• Add --targets / --target. Similar to --tags, the --target flag filters pipes based on the target table name. Values may be negated.

• Add show targets. The action show targets prints a table of pipes and their target tables and schemata.

• Fix --dtype filtering on symlinked pipes. Pipes which symlink their dtypes are now correctly handled by the --dtype filter.

• Fix stale plugins bug. A bug with plugin loading and unloading has been fixed.

• Update plugins page. The Web Console plugins page has been improved.

v3.3.0¶

• Add --dtype flag. The flag --dtype (explicitly --datetime-dtypes or --datetime-dtype) allows filtering pipes based on the data type of the datetime axis. Accepted values are datetime, int, None. This behaves similarly to keys selectors: multiple values may be provided, and values may be negated:

mrsm show pipes --dtype int
mrsm show pipes --dtype int None
mrsm show pipes --dtype _datetime

• Improve copy pipes flow. The copy pipes action has been significantly improved for both interactive and scripted use cases.

Non-interactive batch copying: Specify the destination instance as the first positional argument to skip all prompts. Combined with --force and --sync-data, pipes can now be fully scripted:

# Copy pipe definitions only (skip existing pipes)
mrsm copy pipes sql:dest -i sql:src

# Overwrite existing pipes and copy all data
mrsm copy pipes sql:dest -i sql:src --force --sync-data all

# Overwrite and sync with a date filter (--begin/--end imply filter mode automatically)
mrsm copy pipes sql:dest -i sql:src --force --begin 2024-01-01

New --sync-data flag: Controls what data is synced when copying destination pipes. Accepted values are nothing (default), backtrack, all, and filter. When --begin, --end, or --params are provided without --sync-data, filter mode is selected automatically.

Reduced interactive prompts for batches: When copying many pipes to the same destination instance, all choices (conflict resolution, data sync mode, filter flags) are now collected upfront before any copying begins.

Granular conflict resolution: When destination pipes already exist, the interactive prompt now offers three options rather than a simple yes/no: - Skip, leave the destination pipe entirely unchanged. - Data only, sync data without updating stored attributes. - Attributes only, update attributes without syncing data. - Attributes and data, full overwrite (equivalent to --force).

Copy summary table: After copying data, a table is printed showing inserted, updated, and upserted row counts alongside the elapsed time per pipe and totals.

• Improve caching performance. Better cache handling now reduces round-trips when fetching pipes' metadata.

• Allow dictionary return value from fetch_pipes_keys(). The InstanceConnector method fetch_pipes_keys() may now return a dictionary, where the indices are pipes' IDs and the values are keys tuples (and possibly parameters or tags).

import json
import meerschaum as mrsm
from meerschaum.utils import fetch_pipes_keys

conn = mrsm.get_connector('sql:main')
keys_dict = fetch_pipes_keys('registered', mrsm.get_connector(), connector_keys=['sql:main'], metric_keys=['test'])
print(json.dumps(keys_dict, indent=4))
# {
#     "24": [
#         "sql:main",
#         "test",
#         null,
#         {
#             "sql": "SELECT *\nFROM {{ Pipe('plugin:noaa', 'weather') }}\nWHERE 1 = 1",
#             "tags": [
#                 "test"
#             ],
#             "fetch": {
#                 "backtrack_minutes": 1440
#             },
#             "verify": {
#                 "bound_days": 366,
#                 "chunk_minutes": 43200
#             },
#             "columns": {
#                 "id": null,
#                 "datetime": null
#             }
#         }
#     ]
# }

• Add Pipe.get_docs(). The convenience method Pipe.get_docs() returns a pipe's data as a list of dictionaries, bypassing pandas overhead. Combine with as_chunks=True to get an iterator of List[Dict] chunked by time bounds:

import meerschaum as mrsm

pipe = mrsm.Pipe(
    'demo', 'docs',
    instance='sql:memory',
    temporary=True,
    columns={'primary': 'id'},
)
pipe.sync([
    {'id': 1, 'name': 'Alice'},
    {'id': 2, 'name': 'Bob'},
])

# All rows as a list of dicts (no pandas overhead):
docs = pipe.get_docs()
print(docs)
# [{'id': 1, 'name': 'Alice'}, {'id': 2, 'name': 'Bob'}]

# Single row as a dict:
doc = pipe.get_doc(params={'id': 2})
print(doc)
# {'id': 2, 'name': 'Bob'}

# Chunked iterator of lists of dicts:
for chunk in pipe.get_docs(as_chunks=True):
    print(chunk)

• Fix shell suggestions. A bug has been fixed where shell actions were truncated when chaining actions. Additionally, actions suggestions have been updated with highlight colors.

v3.2.4¶

• Improve default chunksize for integer pipes. Pipes with integer datetime columns now default to appropriately sized chunks based on the precision unit.

import meerschaum as mrsm
pipe = mrsm.Pipe(
    'demo', 'chunksize', 'int',
    instance='sql:memory',
    parameters={
        "columns": {
            "datetime": "ts",
            "id": "id",
        },
        "dtypes": {
            "ts": "int",
        },
        "precision": "ms", # millisecond
        "fetch": {
            "backtrack_minutes": 1440,
        },
        "verify": {
            "chunk_minutes": 1440 * 30, # 30 days
        }
    },
)

print(f"{pipe.get_chunk_interval()=}")
# pipe.get_chunk_interval()=2592000000
print(f"{pipe.get_backtrack_interval()=}")
# pipe.get_backtrack_interval()=86400000

pipe.precision = 's'
print(f"{pipe.get_chunk_interval()=}")
# pipe.get_chunk_interval()=2592000
print(f"{pipe.get_backtrack_interval()=}")
# pipe.get_backtrack_interval()=86400

simple_pipe = mrsm.Pipe(
    'demo', 'chunksize', 'int_no_precision',
    instance='sql:memory',
    parameters={
        'columns': {
            'datetime': 'id',
        },
        'dtypes': {
            'id': 'int',
        },
        "precision": None, # omitted precision defaults to literal chunksize value (previous behavior)
        "fetch": {
            "backtrack_minutes": 100,
        },
        'verify': {
            'chunk_minutes': 10000,
        },
    },
)
print(f"{simple_pipe.get_chunk_interval()=}")
# simple_pipe.get_chunk_interval()=10000
print(f"{simple_pipe.get_backtrack_interval()=}")
# simple_pipe.get_backtrack_interval()=100

• Add flush pipes and flush indices. For convenience, the action flush is equivalent to drop + sync without arguments and clear + sync with filters (--begin, --end, --params).

• Fix parametrized sync times. Passing params to get_sync_time() correctly returns the appropriate sync time value for multiplexed pipes.

v3.2.3¶

• Fix environment isolation for compose projects. Internal configuration isolation has been reworked to better handle dynamic environments (i.e. compose projects).

• Prompt user to restart stack when upgrading. When running mrsm upgrade mrsm, the prompt now asks whether to bring up the stack with the new Docker images after pulling, allowing users to detach from a session when upgrading.

• Add schema to target table on pipe card. The web dashboard's pipe card now shows the fully qualified table name for SQL pipes, including its schema and any truncation.

• Sort by descending for query data results. The "Query Data" option the web dashboard pipe card now sorts results in descending order.

• Fix primary key issues with SQLite. Failed primary key creation on SQLite has been fixed.

v3.2.1 – v3.2.2¶

• Improve webterm performance. The webterm loads more quickly due to changes made to accomodate xterm.js v6.0.0.

• Allow for multiple reference pipes. Pipes may inherit parameters from multiple pipes via references (which now behaves the same as parents and children):

import meerschaum as mrsm
import json

base_pipe_1 = mrsm.Pipe(
    'demo', 'base', '1',
    instance='sql:local',
    dtypes={'a': 'int', 'b': 'int'},
)
base_pipe_2 = mrsm.Pipe(
    'demo', 'base', '2',
    instance='sql:local',
    dtypes={'b': 'bool'},
    columns={'primary': 'id'},
)

pipe = mrsm.Pipe(
    'demo', 'inherit',
    instance='sql:local',
    references=[
        base_pipe_1,
        base_pipe_2,
    ],
)

print(json.dumps(pipe.get_parameters(), indent=4))
# {
#     "dtypes": {
#         "a": "int",
#         "b": "bool"
#     },
#     "columns": {
#         "primary": "id"
#     },
#     "references": [
#         {
#             "connector_keys": "demo",
#             "metric_key": "base",
#             "location_key": "1",
#             "instance_keys": "sql:local"
#         },
#         {
#             "connector_keys": "demo",
#             "metric_key": "base",
#             "location_key": "2",
#             "instance_keys": "sql:local"
#         }
#     ]
# }

• Add "Resolve symlinks" toggle to pipe parameters editor. The parameters editor on the web dashboard includes a "Resolve symlinks" switch, which corresponds to the apply_symlinks parameter for Pipe.get_parameters().

v3.2.0¶

• Allow for strings for reference, parents, and children. In addition to keys, reference other pipes with strings of the Pipe constructor:

connector: foo
metric: bar
parameters:
  reference: "Pipe('spam', 'eggs')"

• Default to the pipe's instance keys for reference pipes. When referencing other pipes in parameters (reference, parents, children, etc.), the base pipe's instance keys will be used as the default keys (overriding the environment) to preserve behavior across environments.

### Default instance is 'sql:main'

pipe = mrsm.Pipe('foo', 'bar', columns={'datetime': 'ts'}, instance='sql:local')
downstream_pipe = mrsm.Pipe(
    'foo', 'bar', 'baz',
    instance='sql:local',
    parameters={
        'reference': "Pipe('foo', 'bar')",
    },
)
print(downstream_pipe.columns)
# {'datetime': 'ts'}

• Add reference, parents / parent, and children / child to Pipe constructor. For convenience, reference pipes or keys are now able to passed directly into the Pipe constructor. The accepted values may either be dictionaries, strings, or other pipes (or a list of each for parents / children).

pipe = mrsm.Pipe('foo', 'bar', instance='sql:local')
child_pipe = mrsm.Pipe('foo', 'bar', 'baz', parent=pipe, instance='sql:memory')
print(child_pipe.parent)
# Pipe('foo', 'bar', instance='sql:local')
pipe.child = child_pipe
print(pipe.child)
# Pipe('foo', 'bar', 'baz', instance='sql:memory')

• Reference parent pipes when syncing SQL pipes. If a SQL pipe includes a parent reference, then the date bounding logic injected into the SQL query definition will be more efficiently applied the parent table. This is especially useful for converting between integer and datetime dtypes via SQL.

import meerschaum as mrsm

base_pipe = mrsm.Pipe(
    'demo', 'reference', 'parent',
    instance='sql:local',
    columns={'datetime': 'ts',},
    dtypes={'ts': 'int'},
    precision='s',
)
base_pipe.sync([{'ts': 1775063535, 'color': 'red'}])

pipe = mrsm.Pipe(
    'sql:local', 'reference', 'child',
    instance='sql:local',
    columns={'datetime': 'ts'},
    dtypes={'ts': 'datetime'},
    parent=base_pipe, # Reference the parent pipe
    parameters={
        'sql': (
          "SELECT datetime(ts, 'unixepoch') AS ts, color\n"
          "FROM {{ Pipe('demo', 'reference', 'parent') }}"
        ),
    },
)
metadef = pipe.instance_connector.get_pipe_metadef(pipe, begin=base_pipe.get_sync_time())
print(metadef)
# WITH "_mrsm_pushdown" AS (
#     SELECT *
#     FROM "demo_reference_parent"
#     WHERE
#         "ts" >= 1775063535
# ),
# "definition" AS (
#      SELECT datetime(ts, 'unixepoch') AS ts, color
#      FROM "_mrsm_pushdown"
# )
# SELECT *
# FROM "definition"


pipe.sync()
base_pipe.sync([{'ts': 1775063837, 'color': 'blue'}])
pipe.sync()
df = pipe.get_data()
print(df)
#                          ts color
# 0 2026-04-01 17:12:15+00:00   red
# 1 2026-04-01 17:17:17+00:00  blue

• Allow for changing datetime dtypes in SQL pipes. When syncing SQL pipes with a parent pipe, the parameters for the parent are considered when building the metadefinition, allowing for changing from integer to timestamp dtypes.

• Add run-time information to pipeline steps. When chaining actions into pipelines, the output steps show the run-time for each step.

• Improve config patch performance. The config patching logic has been refactored to improve performance.

• Improve uv and virtual environment support. Meerschaum now installs correctly with uv tool install and enables uv when run from a virtual environment.

• Fix deduplicate pipes for custom schemata.

• Fixed a bug with the sql:local CLI.

• Fix pickling issue when editing jobs via the Shell.

v3.1.8¶

• Fix issue with CTEs (WITH keyword) and UNION ALL subqueries. A bug has been fixed where queries containing both CTEs (WITH) and UNION ALL queries would previously break the query parser.

• Add the schema to the drop pipes confirmation prompt. To enhance clarification, a SQL pipe's table schema will be included the confirmation prompt.

• Upgrade xterm.js. The latest xterm.js (v6.0.0) is now included for the webterm. This also includes refactoring work to address scrollbar issues on Chromium-based browsers.

• Upgrade dash to 4.0. This release upgrades to web dashboard to Plotly Dash 4, which improves the multi-select dropdowns.

v3.1.7¶

• Fix is_systemd_available() for older versions of systemd. Older versions of systemd (e.g. on the Raspberry Pi) are now correctly detected.

• Handle explicitly None environments for run_python_package(). This resolves a subprocess issue with compose.

• Fix default database configuration for Grafana. New deployments correctly configure the default Meerschaum

• Properly cancel jobs when CTRL+C is pressed on confirmation.

v3.1.4 – v3.1.6¶

• Add support for ESRI CRS for geometry columns. In addition to EPSG projections, pipes with geometry data may now sync shapefiles with ESRI (or other authorities, default EPSG) projections.

import meerschaum as mrsm

pipe = mrsm.Pipe(
    'demo', 'geometry', 'esri',
    columns={'primary': 'id'},
    dtypes={'geo': 'geometry[esri:102003]'}
)
pipe.sync([{'id': 1, 'geo': 'POINT (0 0)'}])
gdf = pipe.get_data()
print(gdf.crs.to_authority())
# ('ESRI', '102003')

• Add Pipe.metric and Pipe.location aliases. Like Pipe.connector_keys, the keys metric_key and location_key may now be accessed via metric and location:

import meerschaum as mrsm

pipe = mrsm.Pipe('demo', 'aliases', target='{{ self.metric }}')
print(pipe.target)
# aliases

• Remove None return for Pipe.connector with invalid keys. Rather than returning None for pipes with invalid connectors, return the verbatim connector keys string instead.

import meerschaum as mrsm

pipe = mrsm.Pipe('demo', 'keys')
print(f"{pipe.connector=}")
# pipe.connector='demo'

• Fix api_key login schemes for api connectors. An issue with api connectors using the api_key login scheme has been fixed.

• Fix client credentials for remote jobs. When executing a remote action or job using an api connector with client credentials (client_id and client_secret), scopes are now correctly evaluated.

v3.1.3¶

• Fix syncing pipes with integer datetimes. An issue with dtype detection has been fixed, improving performance for pipes with integer datetimes.

• Fix recursive parameters references. A single pipe may now reference itself when symlinking parameters.

• Add self syntax for parameters symlinking. Pipes may now use self in parameters rather than requiring the fully qualified name:

import meerschaum as mrsm

pipe = mrsm.Pipe(
    'sql:memory', 'symlink', 'self',
    instance='sql:memory',
    parameters={
        "custom": {
          "value": 123,
        },
        "sql": "SELECT * FROM foo WHERE val = {{ self.parameters['custom']['value'] }}",
    },
)

print(pipe.parameters['sql'])
# SELECT * FROM foo WHERE val = 123

v3.1.2¶

• Improve filtering logic for non-datetime pipes. The Pipe.fiter_existing() method has been fixed to correctly apply non-datetime filters when syncing dataframes.

• Handle mixed nulls when filtering dataframes. Dataframes with different nulls (e.g. empty strings vs None vs pd.NA, etc.) are now correctly coerced to reduce excessive updates.

• Fix minor cache race condition. When syncing pipes with many concurrent chunks, additional safeguards have been added to ensure cached values are ready correctly.

• Bump Docker base to Python 3.14. The Docker image has been rebased to fedora:43 which ships with Python 3.14.

v3.1.0 – v3.1.1¶

• Improve geometry performance. Enforcing dtypes for pipes with geometry columns has been significantly improved.

• Add safeguards to in-place syncs. Attempting to sync an in-place SQL pipe without index columns configured will now gracefully fail.

• Add action delete cache. Users may manually purge local or Valkey cache with delete cache or clear cache.

• Clean up stray cache directories. Deleting pipes automatically removes empty cache directories.

• Deprecate the api Docker tag. The latest Docker tag now represents full. The api tag only exists to retain backwards compatability.

• Drop citus from the test suite. Like cockroachdb, the citus flavor has been deprecated; please use the postgresql flavor instead.

v3.0.9 – v3.0.10¶

• Disable CLI daemon by default. New installations will now disable the CLI daemon, which may be enabled by setting system.experimental.cli_daemon to true.

• Support geopackage SQLConnectors. The flavor geopackage (extension of sqlite) enables syncing geometry data as GeoPackage-WKB (as well as the necessary gpkg tables if missing).

• Check for missing plugins paths for bootstrap plugin. The bootstrap plugin wizard now ensures that configured plugins paths (i.e. MRSM_PLUGINS_DIR) exist before proceeding.

• Add FontAwesome icons to the Dash app. Plugins using the @web_page feature will now have access to FontAwesome icons, which are now statically hosted.

• Fix the Valkey stack data directory. The move from bitnami/valkey to valkey/valkey incorrectly pointed the Valkey volume to /valkey/data instead of /data. This error has been rectified.

v3.0.6 – v3.0.8¶

• Fix stack config for new deployments. New stack deployments behave as expected when symlinks fail to resolve.

• Handle other null-like values when syncing to PostgreSQL. The null detection has been improved when syncing in bulk to PostgreSQL.

• Fix web dashboard SQL editor. The SQL editor on the web dashboard now correctly updates SQL queries.

• Add "Resolve symlinks" to web dashboard SQL Editor. The SQL Editor on the pipes cards now includes a toggle to resolve symlinks in the SQL query, allowing you to make changes to the query without overwriting the symlinks.

• Improve performance when fetching pipes' tags on the web dashboard. Cache misses are now better handled when fetching tags and other keys on the web dashboard.

• Preserve symlinks when copying pipes. The action copy pipes now preserves symlinks for new pipes.

• Fix edge case for unload_plugins(). When an empty list is explicitly provided, the behavior now does not unload all plugins.

v3.0.3 – v3.0.5¶

• Fix pipe page routing. Links to specific pipes in the dashboard are now routed correctly.

• Fix cache invalidation for pipes using Valkey connectors. Pipes using a valkey connector to store cache (rather than on-disk) will now correctly invalidate cache on demand.

• Fix Python 3.9 compatability. An issue breaking Python 3.9 has been fixed.

• Prevent resolving symlinks for pipe's parameters card. The pipe card in dash will no longer resolve symlinks.

• Handle pd.NA values in JSON columns for filter_unseen_df(). Non-string values (e.g. pd.NA) are now correctly handled for JSON columns in filter_unseen_df().

• Resolve dependencies when running the sql CLI. Running with sql action will now resolve expected dependencies.

• Refine unloading plugins. Rather than always popping the root plugins package from sys.modules, unloaded plugins are instead deleted directly from the root package. The root plugins package is popped only if all plugins are unloaded at once.

• Disable uv when running inside a virtual environment.

v3.0.1 – v3.0.2¶

• Change the working directory for each action exected by the CLI daemon. The CLI daemon now changes working directory to match the context of the calling client. This handles relative file paths in environment variables (e.g. MRSM_PLUGINS_DIR).

• Fix environment variables handling within the CLI daemon. Certain environment variables interfered with the shell and the Daemon, and this case has been handled.

• Reload the CLI daemon after upgrading packages. Installing or upgrading packages now reloads the CLI daemon.

• Invalidate symlinks check cache when unloading plugins. Unloading plugins now reverts the internal _synced_symlinks check.

• Unload the root plugins package when unloading plugins. The root plugins package is now unloaded when unload_plugins() is called. This invalidates lingering cache from previously loaded plugins.

v3.0.0¶

• Inherit another pipe's base parameters with reference. Pipes may inherit the base parameters of other pipes by setting the key reference:

import meerschaum as mrsm

base_pipe = mrsm.Pipe(
    'demo', 'reference', 'parent',
    instance='sql:memory',
    columns={
        'datetime': 'ts',
        'id': 'id',
    },
    parameters={
        'custom': {
            'foo': 'bar',
            'color': 'red',
        },
    },
)
base_pipe.register()

pipe = mrsm.Pipe(
    'demo', 'reference', 'child',
    instance='sql:memory',
    parameters={
        'reference': {
            'connector': 'demo',
            'metric': 'reference',
            'location': 'parent',
            'instance': 'sql:memory',
        },
        'custom': {
            'color': 'blue',
        },
    },
)

print(pipe.parameters)
# {'custom': {'foo': 'bar', 'color': 'blue'}, 'columns': {'datetime': 'ts', 'id': 'id'}}

• Dynamically symlink to other pipes' attributes. Reference attributes of other pipes using the {{ Pipe(...) }} syntax. These references are resolved at run-time when Pipe.parameters is accessed:

import json
import meerschaum as mrsm

some_pipe = mrsm.Pipe(
    'demo', 'symlink',
    instance='sql:memory',
    parameters={
        'target': 'some_table',
        'columns': {
            'datetime': 'id',
            'id': 'id',
        },
        'upsert': True,
        'custom': {
            'stations': ['KATL', 'KGMU', 'KCEU'],
        },
    },
)
some_pipe.register()

pipe = mrsm.Pipe(
    'demo', 'symlink', 'child',
    instance='sql:memory',
    parameters={
        'target': "{{ Pipe('demo', 'symlink', instance='sql:memory').target }}",
        'columns': "{{ Pipe('demo', 'symlink', instance='sql:memory').columns }}",
        'upsert': "{{ Pipe('demo', 'symlink', instance='sql:memory').upsert }}",
        'custom': "{{ Pipe('demo', 'symlink', instance='sql:memory').parameters['custom'] }}",
        'parent_pipe_id': "{{ Pipe('demo', 'symlink', instance='sql:memory').id }}",
    },
)

print(json.dumps(pipe.parameters, indent=4))
# {
#     "target": "some_table",
#     "columns": {
#         "datetime": "id",
#         "id": "id"
#     },
#     "upsert": true,
#     "custom": {
#         "stations": [
#             "KATL",
#             "KGMU",
#             "KCEU"
#         ]
#     },
#     "parent_pipe_id": 1
# }

• Add MRSM config symlinks to pipe parameters. Similar to the {{ Pipe(...) }} syntax, you can now reference your Meerschaum configuration from within a pipe's parameters:

meerschaum:
  connectors:
    sql:
      main:
        username: 'user'
        password: 'password'
        ...

import meerschaum as mrsm
pipe = mrsm.Pipe(
    'demo', 'config_symlinks',
    parameters = {
        'username': 'MRSM{meerschaum:connectors:sql:main:username}',
        'password': 'MRSM{meerschaum:connectors:sql:main:password}',
    }
)
print(pipe.parameters['username'])
# 'user'

• Add Pipe.update_parameters(). Due to the symlinking features, the method Pipe.update_parameters() will now appropriately handle updating the parameters within the pipe's attributes. Therefore, mutating Pipe.parameters no longer affects the state of the pipe directly.

• Add InstanceConnector base class. Custom connectors which implement the instance connectors interface should now inherit from InstanceConnector as the base class:

# example.py

from meerschaum.connectors import InstanceConnector, make_connector

@make_connector
class ExampleConnector(InstanceConnector):

    def register_pipe(self, pipe: mrsm.Pipe, **kwargs):
        ...

    def get_pipe_attributes(self, pipe: mrsm.Pipe, **kwargs):
        ...

    ...

• Add long-lived authentication tokens. You may now register tokens to programmatically authenticate your applications to a Meerschaum API instance. This is ideal for use cases such as CI/CD, IoT, and other automated workloads. Tokens are restricted by scopes, may expire or be invalidated, and are owned by a user account. Tokens may be managed via the CLI or web console (at /dash/tokens, under Settings > Tokens).

mrsm register token

• Add scopes to user accounts. Similar to tokens, users may be restricted by scopes. Run edit user to edit the scopes attribute for a given user (more comprehensive editing to come).

• Set coerce_types to True in query_df() if any exclude parameters are provided. Prefacing a value with the negation prefix in params for query_df() will now force coerce_types to be True.

import pandas as pd
from meerschaum.utils.dataframe import query_df

df = pd.DataFrame([
    {'id': 1, 'color': 'red', 'count': 3},
    {'id': 2, 'color': 'blue', 'count': 2},
    {'id': 3, 'color': 'red', 'count': 1},
    {'id': 4, 'color': 'green', 'count': 3},
])
result_df = query_df(df, {'count': '_3'})
print(result_df)
#    id color  count
# 1   2  blue      2
# 2   3   red      1

• Project geometry data to WGS84 (EPSG:4326) when serializing as GeoJSON. Setting geometry_format to geojson for to_json() (default) (and serialize_geometry(), though not default) will project to WGS84 if a CRS is provided (to meet the 2016 GeoJSON specification).

import meerschaum as mrsm
from meerschaum.utils.dataframe import to_json

pipe = mrsm.Pipe(
    'demo', 'geojson',
    instance='sql:memory',
    temporary=True,
    columns={'primary': 'id'},
    dtypes={
        'geometry': 'geometry[point, 6570]',
    },
)
pipe.sync([{
    'id': 1,
    'geometry': 'POINT (1583385.033568 1111981.990433)',
}])

df = pipe.get_data()
json_data = to_json(df)
print(json_data)
# [{"id":1,"geometry":{"type":"Point","coordinates":[-82.389048539797542,34.881760860334396]}}]

• Upgrade sql:main to PostgreSQL 17. The Meerschaum stack database now ships as timescale/timescaledb-ha:pg17, which includes additional extensions such as PostGIS. The flavor for sql:main is now timescaledb-ha (see below).

• Add the SQL flavor timescaledb-ha. The default instance connector sql:main now has the flavor timescaledb-ha, corresponding to the timescale/timescaledb-ha Docker image. This image includes PostGIS, timescaledb_toolkit, and pg_stat_statements.

• Add support for sets and Series in query_df(). Sets and Pandas Series within params will now be treated as lists.

• Allow for spaces and an optional mrsm. prefix for templated SQL query definitions. The template format {{Pipe(...)}} will now match leading and trailing spaces around the Pipe declaration, and an optional mrsm. prefix is accepted.

• Add Pipe.autotime. Similar to Pipe.autoincrement, setting autotime will capture the current timestamp for the value of the datetime axis (as datetimes or integers, depending on the dtype):

import time
import meerschaum as mrsm

pipe = mrsm.Pipe(
    'demo', 'autotime',
    instance='sql:local',
    autotime=True,
    precision='ms',
    temporary=True,
    columns={
        'datetime': 'ts',
        'id': 'id',
    },
    dtypes={
        'ts': 'datetime64[ms, UTC]',
        'id': 'int',
    },
)

pipe.sync([
    {'id': 1, 'val': 100.1},
    {'id': 2, 'val': 200.2},
])

time.sleep(3)

pipe.sync([
    {'id': 1, 'val': 300.3},
    {'id': 2, 'val': 400.4},
])

df = pipe.get_data()
print(df)

#    id    val                               ts
# 0   1  100.1 2025-07-22 00:59:19.914000+00:00
# 1   2  200.2 2025-07-22 00:59:19.914000+00:00
# 2   1  300.3 2025-07-22 00:59:23.674000+00:00
# 3   2  400.4 2025-07-22 00:59:23.674000+00:00

• Add Pipe.precision. The parameter precision determines the precision (and therefore the size) of the timestamp captured by autotime. Accepted values are nanosecond, microsecond, millisecond, second, minute, hour, and day. The default value of precision is derived from the dtype of the datetime axis (i.e. datetime64[ns] is nanosecond precision).

import meerschaum as mrsm

day_pipe = mrsm.Pipe(
    'demo', 'precision', 'day',
    instance='sql:memory',
    columns={'datetime': 'day'},
    dtypes={'day': 'date'},
    autotime=True,
    precision=True,
)

day_pipe.sync("val:1")

df = day_pipe.get_data()
print(df)
#    val         day
# 0    1  2025-07-23

The parameter precision may either be a string (the unit) or a dictionary with the following keys:

• unit (required) The precision unit (microsecond, second, etc.)
• interval (default 1) Optionally round to a specific number of units. For example, precision='minute' and interval=15 would round to 15-minute intervals.

• round_to To which direction to round the current timestamp. Supported values are down (default), up, and closest. See meerschaum.utils.dtypes.round_time().

• Add Pipe.get_value(). The convenience function Pipe.get_value() selects single values from result sets of Pipe.get_data():

import meerschaum as mrsm

pipe = mrsm.Pipe(
    'demo', 'value',
    instance='sql:memory',
    temporary=True,
    autoincrement=True,
    columns={'primary': 'id'},
)
pipe.sync([
    {'species': 'parrot', 'name': 'Polly'},
    {'species': 'dog', 'name': 'Spot'},
    {'species': 'rabbit', 'name': 'Peter Cottontail'},
])

spot_id = pipe.get_value('id', params={'name': 'Spot'})
print(spot_id)
# 2

• Add Pipe.get_doc(). Similar to Pipe.get_value(), the method Pipe.get_doc() will return a single row as a dictionary:

import meerschaum as mrsm

pipe = mrsm.Pipe(
    'demo', 'doc',
    instance='sql:memory',
    columns={'primary': 'id'},
)
pipe.sync([
    {'id': 1, 'name': 'Alice', 'fav_color': 'red'},
    {'id': 2, 'name': 'Bob', 'fav_color': 'green'},
    {'id': 3, 'name': 'Charlie', 'fav_color': 'blue'},
])

doc = pipe.get_doc(params={'id': 2})
print(doc)
# {'id': 2, 'name': 'Bob', 'fav_color': 'green'}

• Add the parameter mixed_numerics. Setting mixed_numerics to False will prevent the behavior of coercing integer to float columns as numeric, akin to static=True but just for this behavior.

• Introducing the Meerschaum CLI daemon. Actions are now routed through a long-lived daemon process, simplifying the number of active connections and cutting latency for CLI commands. The shell includes the command daemon which temporarily toggles between the CLI daemon and executing in-process. Adding the flag --no-daemon to any action will disable the CLI daemon routing, and the CLI daemon may be disabled by setting system.experimental.cli_daemon to false. You may selectively allow or restrict actions prefixes under the configuration system.cli.allowed_prefixes (default *) and system.cli.disallowed_prefixes.

• Performance improvements through smarter caching. The metadata caching system has been overhauled, drastically reducing redundant work and increasing performance. Pipes' metadata are cached on-disk, and providing cache_connector_keys to the Pipe constructor will cache to a Valkey instance instead. Set cache=False to disable this behavior.

• Restrict Webterms to API processes. In previous releases, a single webterm process was shared amongst API processes. Now each API process requires its own Webterm server, and tmux sessions are separated by port.

• Fix custom actions with spaces in Web Console.

• Ignore schema from pipes' parameters on SQLite.

• Tweak begin and end input sizes in the pipes card.