Advanced Rust Plugins¶
Start with the single-file Rust plugin guide for quick actions, data sources, and inline dependencies. This page covers packaged Cargo plugins, the host-call API, the C ABI, and the complete single-file reference.
Single-file .rs plugins support every native feature β actions, connectors, instances, sync hooks, setup, CLI flags, HTTP routes, MCP entries, console pages, executors, and streams. Use a Cargo crate when you want multiple source files, typed Arrow frames, bundled assets, or full manifest control.
Trust
Native plugins compile and run inside the Meerschaum process. Only load plugins you trust.
π Layout & Precedence¶
A packaged plugin lives in MRSM_PLUGINS_DIR with a plugin.toml descriptor and either a prebuilt library or a Cargo crate:
plugins/
βββ native_demo/
βββ plugin.toml # name = "native_demo", version = "1.0.0"
βββ libnative_demo.so # .dylib on macOS; libnative_demo.dll on Windows (always `lib` prefix)
βββ Cargo.toml # optional: auto-built by mrsm when cargo is on PATH
Names allow letters, digits, _, and -. Earlier MRSM_PLUGINS_DIR entries win, and invalid plugins are skipped with a stderr warning while others still load.
Rust before Python: If a Rust plugin and a Python plugin share a name, the Rust plugin replaces the Python plugin completely β nothing falls back to Python. To migrate gradually (for example keeping @dash_plugin callbacks or custom FastAPI apps in Python), split them into separate names (e.g. acme_data in Rust and acme_ui in Python) and read the Rust pipe from Python via mrsm.Pipe('plugin:acme_data', ...).
π§ Writing a Cargo Plugin¶
Configure a cdylib crate with panic = "unwind" so the SDK can catch panics across the FFI boundary:
# Cargo.toml
[package]
name = "native_demo"
version = "1.0.0"
edition = "2024"
[lib]
crate-type = ["cdylib"]
[dependencies]
meerschaum-plugin = "5.0"
[profile.release]
panic = "unwind"
Implement Plugin and export it with export_plugin! (name and version come from Cargo.toml unless overridden via const NAME / const VERSION):
use meerschaum_plugin::prelude::*;
#[derive(Default)]
struct Demo;
impl Plugin for Demo {
fn manifest(&self) -> Value {
Manifest::new()
.fetch()
.action("hello", "Return a synthetic greeting.")
.into()
}
fn fetch(&self, _request: &Request) -> Result<Option<Frame>> {
let rows = vec![
json!({"ts": "2024-01-01T00:00:00Z", "id": 1, "value": 1.5}),
json!({"ts": "2024-01-01T01:00:00Z", "id": 2, "value": 2.5}),
];
Frame::from_rows(rows, &[("ts", "datetime"), ("id", "int"), ("value", "float")]).map(Some)
}
fn call(&self, request: &Request, _data: Option<Frame>) -> Result<Reply> {
match request.method.as_str() {
"fetch" => self.fetch(request).map(Reply::data),
"action" if request.context["name"] == "hello" => {
Ok(Reply::json(json!([true, "Hello from native_demo"])))
}
_ => Err(Error::Unsupported(format!("{} is not implemented", request.method))),
}
}
}
meerschaum_plugin::export_plugin!(Demo);
Frame::from_rows(rows, dtypes) builds a typed Arrow Frame using Meerschaum dtypes (datetime, date, int, float, string, bool, json, numeric, uuid, bytes, geometry[...]); frame.to_rows() converts back, and Frame::from_record_batch accepts Arrow batches (arrow, chrono, serde_json are re-exported). Examples: native_demo.rs, native_host.rs.
Request field |
Meaning |
|---|---|
method |
fetch, sync, action, pre_sync_hook, post_sync_hook, api_route, mcp_*, page, register_pipe, get_pipe_data, etc. |
connector_type, label, config |
Target connector keys and configuration. |
context |
Payload (pipe, query, options, name, kwargs). Bounds (begin, end) are ints, ISO strings, or null. |
Handlers are stateless (Default::default() per call; use a Mutex static keyed by label for state), synchronous (host-serialized except concurrent stream and server calls β see Routes and MCP), panic-contained (catch_unwind), and must match the host OS and CPU.
π Manifest & Capabilities¶
manifest() lets --help, completions, and servers discover capabilities without loading the library (cached in <root>/cache/native_plugins_manifest.json, schema 5 / MANIFEST_SCHEMA_VERSION):
fn manifest(&self) -> Value {
Manifest::new()
.fetch()
.action("hello", "Return a greeting.")
.flag("--loud", "Shout.")
.pre_sync_hook("before").post_sync_hook("after")
.connector(ConnectorType::new("memory", ["fetch", "register_pipe", "sync_pipe"]).instance())
.executor_type("procs", ["get_job_names", "create_job"], "Run jobs as local processes.")
.api_route(RouteSpec::new("item", "GET", "/my/items/{id}").scopes(["pipes:read"]))
.mcp_tool(ToolSpec::new("add", "Add two integers."))
.web_page("my/page", "page", true)
.required(["plugin:noaa"])
.into()
}
| Capability | Cargo manifest / method | Notes |
|---|---|---|
| Data & setup | .fetch(), .sync(), .register(), .setup() |
sync overrides built-in sync; register returns default pipe params. |
| Actions & flags | .action(name, doc), .flag(...) / .argument(...) |
Unset flags are absent from context.kwargs. |
| Sync hooks | .pre_sync_hook(name), .post_sync_hook(name) |
Run before Python hooks; failures warn and never fail the sync. |
| Connectors & instances | .connector_type(...) / ConnectorType::new(...).instance() |
Return Error::Unsupported for unimplemented instance methods. |
| HTTP routes & MCP | .api_route(...), .mcp_tool/resource/prompt(...) |
See Routes and MCP. |
| Executors & streams | .executor_type(...), is_streaming + subscribe |
See Executors and Streams. |
| Console pages | .web_page(path, fn_name, login_required) |
Answer page with {html, status?} for meerschaum[dash]. |
ποΈ Building, Distributing & Testing¶
Prebuilt libraries (required for bmeares/meerschaum:slim, which has no Rust toolchain):
cargo build --release -p meerschaum-plugin --example native_demo
mkdir -p "$MRSM_PLUGINS_DIR/native_demo"
cp crates/meerschaum-plugin/examples/plugin.toml target/release/examples/libnative_demo.so \
"$MRSM_PLUGINS_DIR/native_demo/"
Source auto-build: Directories with plugin.toml + Cargo.toml (cdylib, panic = "unwind", matching name) and no newer prebuilt library are auto-built via cargo build --release (CARGO_INCREMENTAL=0) into <root>/cache/native_build/<name>/. Ship Cargo.lock for read-only directories; delete the cache dir to retry a failed build.
Repositories & testing: mrsm register plugins / install plugins / uninstall plugins / edit plugins support both .rs files and packaged crates (skipping Python venvs and rolling back on build errors; prebuilt binaries must match host OS/CPU, and live hot-reload is unsupported). Test on a scratch root:
export MRSM_ROOT_DIR="$(mktemp -d)" MRSM_PLUGINS_DIR="$(mktemp -d)"
mrsm show plugins && mrsm sync pipes -c plugin:my_plugin -m demo -i sql:local --nopretty
π¬ The C ABI¶
Libraries export mrsm_plugin_v1 returning a static #[repr(C)] vtable (status codes: 0 success JSON, 1 error, 2 caught panic, 3 unsupported):
| Field | Purpose |
|---|---|
abi_version, struct_size |
Fixed prefix verified before reading the rest of the table (currently ABI 1). |
describe() |
Returns plugin metadata and manifest JSON. |
invoke(request_json, in_stream, out_stream) |
Single entry point for all methods; tabular data crosses as Arrow C streams (FFI_ArrowArrayStream). |
free_bytes(bytes) |
Releases plugin-allocated buffers (host and plugin always free their own allocations). |
Optional extensions: mrsm_plugin_invoke_host_v1 passes a call-scoped HostV1 callback table for host calls (older hosts degrade gracefully on newer methods), and single-file plugins additionally export mrsm_plugin_invoke_rows_v1 to exchange lightweight JSON rows without compiling Arrow.
Talking to Meerschaum¶
Plugins interact with the host process through host calls on Pipe and host (pre-imported in single-file plugins; meerschaum_plugin::{host, Query} in Cargo crates):
fn sync(pipe: &Pipe) -> Result<SuccessTuple> {
let readings = host::pipe("plugin:readings", "readings", None, None)?;
let daily = host::read_sql(
pipe.instance_keys(),
&format!(
"SELECT date(ts) AS day, station, AVG(temperature) AS temperature FROM {} GROUP BY 1, 2",
readings.target()
),
)?;
pipe.replace(daily, None, None)
}
| Call | Returns | Python equivalent |
|---|---|---|
host::pipe(conn, metric, loc, instance) |
Pipe |
mrsm.Pipe(conn, metric, loc, instance=...) |
pipe.get_data(begin, end, params) |
Rows (Vec<Value> or Frame) |
pipe.get_data(begin=, end=, params=) |
pipe.get_data_with(&Query::new()...) |
Rows | pipe.get_data(select_columns=, omit_columns=, limit=, order=, ...) |
pipe.get_rowcount(begin, end, params) |
i64 |
pipe.get_rowcount(...) (0 if no table) |
pipe.get_sync_time() / _with(params, newest) |
Option<Bound> |
pipe.get_sync_time(params=, newest=) |
pipe.sync(rows) |
SuccessTuple |
pipe.sync(docs) |
pipe.replace(rows, begin, end) |
SuccessTuple |
pipe.sync(docs, begin=, end=, replace=True) |
pipe.clear(begin, end, params) |
SuccessTuple |
pipe.clear(begin=, end=, params=) |
pipe.exists(), id(), is_registered(), get_columns_types(), register() |
Metadata / SuccessTuple |
Matching Pipe methods |
pipe.target(), datetime_column(), parameters(), with_parameters(...) |
Local accessors | pipe.target, pipe.columns['datetime'], pipe.parameters |
host::read_sql(conn, sql) / read_sql_with(conn, sql, ¶ms) |
Rows | conn.read(sql, params=...) |
host::exec_sql(conn, sql) / exec_sql_with(conn, sql, ¶ms) |
SuccessTuple |
conn.exec(sql, params) |
host::transaction(conn, lock_key, \|tx\| ...) |
Result<T> |
Atomic SQL transaction (tx.read_sql_with, tx.exec_sql_with) |
host::root_dir() |
PathBuf |
Resolved MRSM_ROOT_DIR of the current call |
host::plugin_config() / write_plugin_config(&val) |
Value / SuccessTuple |
get_plugin_config() / write_plugin_config(...) |
host::config(&["meerschaum", "instance"]) |
Value |
get_config('meerschaum', 'instance') |
host::mint_token(...), revoke_token(id), draining() |
MintedToken / SuccessTuple / bool |
Route token lifecycle & shutdown (rules) |
Parameterized SQL & transactions: Named :param binds accept JSON scalars, null, arrays (native PostgreSQL = ANY(:ids)), and typed helpers (Bound::to_sql_param(), host::sql_timestamptz, sql_timestamp, sql_date, sql_bytes, sql_json). With lock_key: Some(key), host::transaction acquires pg_advisory_xact_lock on Postgres or BEGIN IMMEDIATE on SQLite, commits on Ok, and rolls back on Err or panic:
let next_id: i64 = host::transaction("sql:main", Some("cmdq:1"), |tx| {
let rows = tx.read_sql_with(
"SELECT COALESCE(MAX(id), 0) + 1 AS next_id FROM cmd_queue WHERE created_at >= :since",
&json!({"since": host::sql_timestamptz("2026-09-25T00:00:00+00:00")}),
)?;
let id = rows[0]["next_id"].as_i64().unwrap_or(1);
tx.exec_sql_with("INSERT INTO cmd_queue (id) VALUES (:id)", &json!({"id": id}))?;
Ok(id)
})?;
Key host-call rules:
- Default instance:
host::pipewithinstance = Noneuses the synced pipe's instance, the action's-i/meerschaum:instance, or the API server's instance. - Writes & safety:
pipe.syncruns normal dtype checks/upserts (never recursively calling pluginfetch/sync),pipe.replaceswaps[begin, end)atomically, and writes toprotectedinstances fail unless--allow-protected/MRSM_ALLOW_PROTECTED=1is set. - Thread-bound: Host calls work synchronously on the calling thread during plugin callbacks (
Pipeis notSend, and re-entering the same native library returns an error). There is no Rust equivalent offrom_plugin_import.
Permissions in routes, MCP, and pages¶
Inside HTTP routes, MCP entries, and console pages, host calls enforce the caller's live scopes, pipe visibility, edit rights, and api:permissions:instances (under --no-auth and MCP stdio, all calls are allowed):
| Host calls | Required scope / rule |
|---|---|
host::pipe, get_data, get_rowcount, get_sync_time, exists, id, is_registered, get_columns_types |
pipes:read |
pipe.sync, pipe.replace |
pipes:write (plus pipes:register if creating) |
pipe.clear / pipe.register |
pipes:delete / pipes:register |
read_sql*, transaction + tx.read_sql* |
sql:read; served sql: connectors only; refuses hidden pipe targets, mrsm_* tables, _mrsm_internal, and dynamic SQL functions (dblink, query_to_xml, etc.) |
exec_sql*, tx.exec_sql* |
actions:execute + caller allowed to run actions (admins, or non_admin: true with jobs:execute; never under --secure) |
host::config |
* (admin or --no-auth) |
plugin_config, write_plugin_config, root_dir, draining |
None (scoped to plugin settings & call root) |
mint_token, revoke_token |
Signed-in caller holding each custom plugin scope (rules) |
Public routes and unauthenticated console pages can grant non-admin scopes for host calls via host_scopes (preserving Everyone visibility and edit rules):
#[api_route(GET, "/status/{metric}", public = true, host_scopes = ["pipes:read"])]
fn status(request: &RouteRequest) -> Result<Value> {
let pipe = host::pipe("plugin:readings", request.path_param("metric").unwrap_or(""), None, None)?;
Ok(json!({"rows": pipe.get_rowcount(None, None, None)?}))
}
Single-file reference¶
Single-file .rs plugins need no Cargo project. Pre-imported types: Pipe, Connector, Bound (Int(i64) / Datetime(String) with as_int(), timestamp(), from_timestamp()), SuccessTuple, Value, json!, Frame (lightweight JSON row container), Result, Error, host, and Query.
Macros & attributes¶
| Attribute / Function | Target | Purpose |
|---|---|---|
fn fetch(...) / fn sync(...) |
Top-level or connector mod |
Data source fetch or sync override for plugin:<name> or <type>:<label>. |
fn setup(...) |
Top-level | Runs on mrsm setup plugins and after install plugins. |
#[make_action] |
Top-level fn |
Registers a CLI action (stackable with hook attributes). |
#[pre_sync_hook] / #[post_sync_hook] |
Top-level fn |
Sync lifecycle hooks (run in file order). |
#[add_plugin_argument(...)] |
File (#![...]) or top-level fn |
Declares CLI flags received in kwargs. |
#[make_connector] / #[make_connector(instance)] |
Inline mod |
Source connector or -i instance connector; use #[method] inside for custom Python-callable methods. |
#[web_page(...)] |
Top-level fn |
HTML fragment served under /dash/ in meerschaum[dash]. |
#[api_route], #[mcp_tool], #[mcp_resource], #[mcp_prompt] |
Top-level fn |
HTTP routes and MCP entries β see Routes and MCP. |
#[make_executor], fn subscribe |
Inline mod / fn |
Job executors and streaming connectors β see Executors and Streams. |
Data functions & setup¶
fetch and sync accept any subset of these named parameters in any order (leading _ allowed; must be synchronous, non-generic):
| Parameter | Type | Meaning |
|---|---|---|
pipe |
&Pipe |
Target pipe metadata and host calls. |
connector |
&Connector |
Connector type, label, and config. |
kwargs |
&Value |
Sync options and plugin flags. |
begin / end |
Option<Bound> |
Inclusive lower / exclusive upper bound (None if unbounded). |
data (sync only) |
Option<Frame> |
Input rows if caller supplied them. |
fetch returns Vec<Value>, Option<Vec<Value>>, Frame, Option<Frame>, or a Result of those (vec![] succeeds with 0 new rows; None fails with "No data were fetched"). sync returns SuccessTuple, (bool, String), (bool, &str), or a Result.
setup optionally takes kwargs: &Value and must return bool, SuccessTuple, (bool, String), or a Result:
fn setup() -> Result<SuccessTuple> {
if host::plugin_config()?.is_null() {
host::write_plugin_config(&json!({"stations": ["north"]}))?;
}
Ok(SuccessTuple::ok("Ready."))
}
Sync hooks¶
Hooks accept any subset of pipe: &Pipe, kwargs: &Value, and (post-sync only) success_tuple: SuccessTuple and sync_duration: f64, returning (), SuccessTuple, (bool, String)/&str, or Result:
#[pre_sync_hook]
fn announce(pipe: &Pipe) -> (bool, String) {
(true, format!("Syncing {}", pipe.metric_key()))
}
#[post_sync_hook]
fn report(pipe: &Pipe, success_tuple: SuccessTuple, sync_duration: f64) -> (bool, String) {
(true, format!("{} took {sync_duration:.1}s: {}", pipe.metric_key(), success_tuple.message))
}
Plugin arguments, VERSION, and REQUIRED¶
Declare plugin version/Python requirements with exact literal constants and CLI flags with #[add_plugin_argument(...)]:
#![add_plugin_argument("--times", "-n", type = int, default = 1, help = "Repeat count.")]
const VERSION: &str = "1.2.0";
const REQUIRED: &[&str] = &["plugin:noaa", "requests>=2"];
#[add_plugin_argument("--loud", action = "store_true", help = "Shout.")]
#[make_action]
fn greet(action: Vec<String>, kwargs: &Value) -> (bool, String) {
let mut text = format!("Hello, {}!", action.join(", "));
if kwargs["loud"] == true { text = text.to_uppercase(); }
(true, text.repeat(kwargs["times"].as_u64().unwrap_or(1) as usize))
}
| Option | Values |
|---|---|
action |
"store" (default), "store_true", "store_false", "store_const", "append", "append_const", "count" |
type |
int, float, or str (for store / append) |
default, const, choices |
Literals (choices = ["north", "south"]) |
help, metavar, dest, required |
Strings / bool (dest defaults to snake_case of first long flag; nargs is unsupported) |
Flags conflicting with built-in or other plugin options emit a warning and are ignored.
Custom connector methods & instance connectors¶
Mark extra connector methods with #[method] to expose them directly to Python (mrsm.get_connector('kuma:main').push(status='up')), and use #[make_connector(instance)] to build a custom -i instance:
#[make_connector]
mod kuma {
#[method]
pub fn push(connector: &Connector, kwargs: &Value) -> Result<SuccessTuple> {
Ok(SuccessTuple::ok(format!("Pushed {} to {}", kwargs["status"], connector.label())))
}
}
#[make_connector(instance)]
mod memory {
fn register_pipe(pipe: &Pipe) -> SuccessTuple { SuccessTuple::success() }
fn pipe_exists(pipe: &Pipe) -> bool { true }
fn sync_pipe(pipe: &Pipe, data: Vec<Value>) -> (bool, String) { (true, format!("Inserted {}.", data.len())) }
fn get_pipe_data(pipe: &Pipe, begin: Option<Bound>, params: &Value) -> Vec<Value> { vec![] }
fn fetch_pipes_keys(query: &Value) -> Value { json!([]) }
}
| Instance methods | Return type (all accept Result<T>) |
|---|---|
register_pipe, delete_pipe, drop_pipe, edit_pipe, sync_pipe, clear_pipe, deduplicate_pipe, drop_pipe_indices, create_pipe_indices, compress_pipe, decompress_pipe, analyze_pipe, vacuum_pipe, partition_pipe |
SuccessTuple, (bool, String), or (bool, &str) |
get_pipe_data |
Vec<Value>, Frame, or Option<Frame> |
test_connection, pipe_exists, acquire_pipe_lease, release_pipe_lease |
bool |
fetch_pipes_keys |
[[connector_keys, metric_key, location_key], ...] |
get_pipe_id, get_pipe_rowcount, get_pipe_size |
Option<i64> or i64 |
get_sync_time |
Option<i64>, ISO timestamp string, or None |
get_pipe_attributes, get_pipe_columns_types, get_pipe_index_names, get_pipe_columns_indices |
JSON object (Value or serializable map) |
Instance method parameters bind by name (pipe, connector, data, context, query, options, params, begin, end, select_columns, omit_columns, limit, order, check_existing, chunksize, extra, newest, patch, no_policy, full, chunk_minutes, holder, ttl_seconds).
Console pages¶
Serve HTML fragments under /dash/ with #[web_page] (following the page contract):
#[web_page("readings/report", login_required = false, page_group = "Readings")]
fn report(page: &PageRequest) -> String {
format!("<h3>Station {}</h3>", page.query_param("station").unwrap_or_default())
}
Options: leading path string (default function name), login_required (default true), skip_navbar (false), page_group (plugin name), dark_theme (true), host_scopes. Accepts optional page: &PageRequest (path, query, query_param, username) and returns String/&str, (u16, T), or Result.
Inline dependencies & build cache¶
Add external Rust crates in a single doc block at the top of the .rs file (only [dependencies] is supported; local path deps resolve relative to the .rs file):
//! ```cargo
//! [dependencies]
//! regex = "1"
//! ```
use regex::Regex;
#[make_action]
fn count_words(action: Vec<String>) -> (bool, String) {
let count = Regex::new(r"\w+").unwrap().find_iter(&action.join(" ")).count();
(true, format!("{count} words"))
}
Generated Cargo projects and stamps live under <root>/cache/native_build/<name>/ (with shared target artifacts in .single_file_target/ and build errors in mrsm-build-error.txt); nothing is written next to your .rs file. External path dependencies are not fingerprinted, relative include_str! resolves from the cache copy, and remember Rust returns the final expression without a trailing semicolon.