Meerschaum 0¶

The 0.6.x series brought a lot of polish to the package, namely through refactoring and changing some legacy features to a meet expected behaviors.

v0.6.3 – v0.6.4: Durable Venvs¶

• Improved durability of the virtual environments. The function meerschaum.utils.packages.manually_import_module() behaves as expected, allowing you to import different versions of modules. More work needs to be done to see if reintroducing import hooks would be beneficial.
• Activate plugin virtual environments for API plugins. If a plugin uses the @api_plugin decorator, its virtual environment will be activated before starting the API server. This could potentially cause problems if you have many API plugins with conflicting dependencies, but this could be mitigated by isolating environments with MRSM_ROOT_DIR.
• Changed name to import_name for determine_version(). The first argument in meerschaum.utils.packages.determine_version() has been renamed from name to the less-ambiguous import_name.
• Shortened the IDs for API environments. Rather than a long UUID, each instance of the API server will have a randomly generated ID of six letters. Keep in mind it is randomly generated, so please excuse any randomly generated words.
• Removed version enforcement for uvicorn and gunicorn. Uvicorn has a lot of hidden imports, and using our home-brewed import system breaks things. Instead, we now use the default attempt_import behavior of check_update=False.
• Reintroduced verify packages to setup.py. Upgrading Meerschaum will check if the virtual environment packages satisfy their required versions.
• Moved pkg_resources patch from the module-level. In v0.6.3, the monkey-patching for flask-compress happened at the module level, but this was quickly moved to a lazy patch in v0.6.4.
• Bugfixes Versions 0.6.3 and 0.6.4 were yanked due to some unforeseen broken features.
• Bumped several dependencies.

v0.6.0 – v0.6.2: Robust Plugins and Beautiful Pipes¶

Potentially Breaking Changes

• Renamed meerschaum.connectors.sql.tools to meerschaum.utils.sql. A dummy module was created at the old import path, but this will be removed in future releases.
• Migrated to meerschaum.core. Important class definitions (e.g. User) have been migrated from meerschaum._internal to meerschaum.core. You can still import meerschaum.core.Pipe as mrsm.Pipe, however.
• Moved meerschaum.actions.shell to meerschaum._internal.shell. Finally marked it off the to-do list!
• Pipe.__str__() and Pipe.__repr__() now return stylized strings. This should make reading logs significantly more pleasant. You can add syntax highlighting back to strings containing Pipe() with meerschaum.utils.formatting.highlight_pipes().

New Features

• Plugins Exposed the meerschaum.Plugin class, which will make cross-pollinating between plugins simpler.
• Uninstall procedure Plugins now ship with a proper uninstall() method.
• Sharing dependencies Plugins may now import dependencies from a required plugin's virtual environment. E.g. if plugin foo requires plugin:bar, and bar requires pandas, then foo will be able to import pandas.
• Allow other repos for required plugins. You can now specify the keys of a required plugin's repo following @, e.g. foo may require plugin:bar@api:main.
• Isolate package cache. Each virtual environment now uses an isolated cache folder.
• Handle multiple versions of packages in determine_version() When verifying packages, if multiple dist-info directories are found in a virtual environment, import the package in a subprocess to determine its __version__.
• Specify a target table. Pipe.target (Pipe.parameters['target']) now governs the name of the underlying SQL table.

Bugfixes

• Circular dependency resolver Multiple plugins may now depend on each other without entering a recursive loop.
• Held back dash_extensions due to breaking API changes. Future releases will migrate to dash_extensions>1.0.0.
• Fixed meerschaum.plugins.add_plugin_argument(). Refactoring broke something awhile back; more plugins-focused tests are needed.
• Fixed an issue with fontawesome and mkdocs-material.
• Fixed pickling issue with mrsm.Pipe.

Documentation

• pdoc changes. Added __pdoc__ and __all__ to public modules to simplify the package docs.
• Lots of cleanup. Almost all of the docstrings have been edited.

The 0.5.x series tied up many loose ends and brought in new features, such as fulling integrating Oracle SQL, rewriting most of the doctrings, and adding tags and key negation. It also added the clear pipes command and introduced the GUI and webterm.

v0.5.14 – v0.5.15¶

• Added tags. Pipes may be grouped together pipes. Check the docs for more information.
• Tags may be negated. Like the key negation added in v0.5.13, you can choose to ignore tags by prefacing them with _.
• Bugfixes for DuckDB
• Updated documentation.
• Fixed issue with flask-compress. When starting the API for the first time, missing flask-compress will not crash the server.

v0.5.13¶

• Key negation when selecting pipes. Prefix connector, metric, or location with _ to select pipes that do NOT have that key.
• Added the setup plugins command. Run the command setup plugins followed by a list of plugins to execute their setup() functions.
• Renamed pipes' keys methods function. The function meerschaum.utils.get_pipes.methods() is renamed to meerschaum.utils.get_pipes.fetch_pipes_keys().
• Improved stability for DuckDB.
• Bumped dependencies. DuckDB, FastAPI, and Uvicorn have been updated to their latest stable versions.

v0.5.11 – v0.5.12¶

• Improved Oracle support. Oracle SQL has been fully integrated into the testing suite, and critical bugs have been addressed.
• Added the install required command. When developing plugins, run the command install required to install the packages in the plugin's required list into its virtual environment.
• Migrated docstrings. To improve legibility, many docstrings have been rewritten from reST- to numpy-style. This will make browsing docs.meerschaum.io easier.

v0.5.10¶

• Added the clear pipes command. Users may now delete specific rows within a pipe using pipe.clear(). This new method includes support for the --begin, --end, and --params flags.
• Changed the default behavior of --begin. The --begin flag is now only included when the user specifies and no longer defaults to the sync time.
• Added --timeout-seconds. The flags --timeout-seconds and --timeout make the syncing engine sync each pipe in a separate subprocess and will kill the process if the sync exceeds the number of provided seconds.
• Fixed shell argparse bug. When editing command line arguments within the shell, edge cases no longer cause the shell to exit.

v0.5.6 – v0.5.9¶

• Added support for gunicorn. Gunicorn may be used to manage API processes with the --production or --gunicorn flags. The --production flag is now default in the Docker image of the API server.
• Updated bootstrap pipes flow. The interactive bootstrapping wizard now makes use of the new register() plugins API as well as asking for the value column.
• Fixed edge cases in Pipe.filter_existing(). Better enforcement of NaT as well as --begin and --end now reduces edge-case bugs and unexpected behavior.
• Re-introduced the full Docker image. Inclusion of the start gui command led to the full version of the Docker image requiring GTK and dependencies. Now you can forward the GUI with docker run -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix bmeares/meerschaum:full start gui
• Added ACKNOWLEDGEMENTS.md to the root directory. Only dynamic dependencies with BSD, LGPL, MIT, and Apache licenses remain.
• Fixed plugin installation bug (again).
• Allow for plugins with hyphens in the name.
• Lots of refactoring and tiny bugfixes.

v0.5.3 – v0.5.5¶

• Refactored the start gui and start webterm commands. The start gui command opens a window which displays the webterm. This terminal will be integrated into the dashboard later.
• Began work on the desktop build. Work on building with PyOxidizer began on these releases.

v0.5.1 – v0.5.2¶

• Added the experimental commands start gui and start webterm. The desktop GUI will be rewritten in the future, but for now it starts a webview for the web console. The webterm is an instance of xterm (not xterm.js, having issues) and will eventually replace the current web console "terminal" output. The desktop GUI will also be replaced and will include the webterm once I can get it working on Windows.
• Isolated API processes. Meerschaum API workers refer to the environment variable MRSM_SERVER_ID to determine the location of the configuration file. This results in the ability to run multiple instances of the API concurrently (at last)!
• Fixed plugin installation bug. When installing plugins, the expected behavior of checking if it's already installed occurs.
• Replaced semver.match() with semver.VersionInfo.match(). This change resolves depreciation warnings when building the package.
• Added the /info endpoint to the API. This allows users to scrape tidbits of information about instances. The current dictionary returns the version and numbers of pipes, plugins, and users. More information will be added to this endpoint in future releases.

v0.5.0¶

• New syncing engine. The sync pipes command reduces concurrency issues while nearly halving syncing times for large batches of pipes.
• Syncing progress bar. The sync pipes command displays a progress bar (only in the shell) to track the number of completed pipes.
• Bumped default TimescaleDB image to PostgreSQL 14. You can continue using PostgreSQL 13 if you already have an existing database.
• Changed API endpoints. An endpoint for deleting pipes was added, and the editing and registration endpoints were changed to match the connector, metric, location path scheme.
• Redesigned test suite. The pytest environment now checks syncing, registration, deletion, etc. for pipes and users with many database flavors.
• Cleanup and small bugfixes. As a result of the updated testing suite, issues with several database flavors as well as the API have been resolved.

The 0.4.x series dramatically updated Meerschaum, such as ensuring compatibility with Python 3.10, migrating to Bootstrap 5, and implementing useful features like the redesigned web console and the shell toolbar.

v0.4.16 – v0.4.18¶

• Rewritten API `register()` methods.
• MySQL / MariaDB and CockroachDB fixes.
• Additional tests.

v0.4.11 – v0.4.15¶

• Change the number of columns when printing items. Depending on the lengths of items and size of the terminal, the number of columns is reduced until most items are not truncated.
• Allow shell jobs with the -f flag. In addition to --allow-shell-job, the --force flag permits non-Meerschaum commands to be run. If these flags are absent, a more informative error message is printed.
• Redesigned the bottom toolbar. The bottom toolbar now uses a black background with white text. Although this technically still prints ANSI when the global ANSI configuration is false, it still does toggle color.
• More bugfixes. A warning when installing plugins has been addressed, and other virtual environment and portable bugs have been fixed.

v0.4.8 – v0.4.10¶

• Added the bottom toolbar to the interactive shell. The includes the current instance, repo, and connection status.
• Fixed parsing issue with the Docker build. There is a strange edge case where multiple levels of JSON-encoding needed to be escaped, and this scenario has been accounted for.
• Enforce MRSM_CONFIG and MRSM_PATCH in the Web Console actions. The Docker version of the API uses environment variables to manage instances, so this information is passed along to children threads.
• Delayed imports when changing instances. This postpones trying to connect to an instance until as late as possible.

v0.4.1 – v0.4.7¶

• Added features to the Web Console. Features such as the Show Pipes button and others were added to give the Web Console better functionality.
• Migrated the Web Console to Bootstrap 5. Many components needed to be modified or rewritten, but ultimately the move to Bootstrap 5 is worth it in the long run.
• Updated to work on Python 3.10. This included creating a standalone internal module for cascadict since the original project is no longer maintained.
• Tighter security. Better enforcement of datetimes in dateadd_str() and denying users access to actions if the permissions setting does not allow non-admins to perform actions.
• Bugfixes for broken dependencies. In addition to migrating to Bootstrap 5, components like PyYAML and fastapi-login changed their function signatures which broke things.

v0.4.0¶

• Allow for other plugins to be specified as dependencies. Other plugins from the same repository may be specified in the required list.
• Added warnings for broken plugins. When plugins fail to be imported, warnings are thrown to help authors identify the problem.
• Added registration to the Web Console. New users may create accounts by clicking the No account? link on the login page.
• Added the verify action. For now, verify packages ensures that the installed dependencies meet the stated requirements for the installed version of Meerschaum.
• Fixed "ghost" background jobs. Ensure that jobs are actually running before marking them as so.

Version 0.3.0 introduced the web interface and added more robust SQL support for various flavors, including MSSQL and DuckDB.

v0.3.12 – v0.3.19¶

• Mostly small bugfixes. Docker-compose fixes, params in get_pipe_rowcount(), unique index names for pipes.
• Added newest flag to pipe.get_sync_time(). Setting newest=False will return the oldest time instead of the newest.
• Migrated filter_existing to a member of Pipe. Although the current implementation for APIConnectors offloads filtering to the SQLConnector, soon filtering will take place locally to save bandwidth.
• Updated Docker base image. Bumped base image from Python 3.7 on Debian Buster Slim to Python 3.9 on Debian Bullseye Slim. Also removed ARM images for the sake of passing builds and reducing build times (e.g. DuckDB fails to compile with QEMU).
• Improved DuckDB support. sql:memory is now the default in-memory DuckDB instance.

v0.3.1 – v0.3.11¶

• Improved Microsoft SQL Server support.
• Added plugins page to the dashboard. Although somewhat hidden away, the path /dash/plugins will show the plugins hosted on the API repository. If the user is logged in, the descriptions of plugins belonging to that user become editable.
• Added locks to resolve race conditions with threading.
• Added --params when searching for data and backtracked data.
• Fixed the --params flag for API pipes.
• Added experimental multiplexed fetching feature To enable this feature, run mrsm edit config system and under the experimental section, set fetch to true.
• Bugfixes and stability improvements

v0.3.0¶

• Introduced the Web Interface. Added the Meerschaum Web Interface, an interactive dashboard for managing Meerschaum instances. Although not a total replacement for the Meerschaum Shell, the Web Interface allows multiple users to share connectors without needing to remote into the same machine.

• Background jobs Actions may be run in the background with the -d or --daemon flags or with the action start job. To assign a name to a job, pass the flag --name.

• Added duckdb as a database flavor The duckdb database flavor is a single file, similar to sqlite. Future releases may use duckdb as the cache store for local pipes' data.

• Added uninstall plugins and uninstall packages. Plugins and virtual environment pip packages may now be removed via the uninstall command.

• Delete plugin from repository The command delete plugins now deletes the archive file and database registration of the plugin on the remote repository. This does not uninstall plugins, so deleted plugins may be re-registered if they are still installed on the client.

• Bound syncing with --begin and --end When performing a sync, you can specify --begin and --end to bound the search for retrieving data.

• Bugfixes and improvements Small bugfixes like including the location None with other locations and improvements like only searching for plugin auto-complete suggestions when the search term is at least 1 character long.

Version 0.2 improved greatly on 0.1, with a greater focus on the user experience, plugins, local performance, and a whole lot more. Read the release notes below for some of the highlights.

v0.2.22¶

• Critical bugfixes. Version 0.2.22 fixes some critical bugs that went unnoticed in v0.2.21 and is another backport from the 0.3.x branch.

v0.2.21¶

• Bugfixes and performance improvements. Improvements that were added to v0.3.0 release candidates were backported to the 0.2.x series prior to the release of v0.3.0. This release is essentially v0.3.0 with the Web Interface disabled.

v0.2.20¶

• Reformatted show columns to tables. The action show columns now displays tables rather than dictionaries.
• SQLConnector bugfixes. The debug flag was breaking functionality of SQLConnector objects, but now connectors are more robust and thread safe.
• Added instance as an alias to mrsm_instance when creating Pipe objects. For convenience, when building Pipes, instance may be used in place of mrsm_instance.

v0.2.19¶

• Added show columns action. The action show columns will now display a pipe's columns and data types.
• docker-compose bugfix. When docker-compose is installed globally, skip using the virtual environment version.
• Refactoring / linting A lot of code was cleaned up to conform with cleaner programming practices.

v0.2.18¶

• Added login action. To verify or correct login credentials for API instance, run the login action. The action will try to log in with your defined usernames and passwords, and if a connector is missing a username or password is incorrect, it will ask if you would like to try different login credentials, and upon success, it will ask if you would like to save the new credentials to the primary configuration file.

• Critical bugfix. Fixed bug where default values were being copied over from the active shell instance. I finally found, deep in the code, the missing .copy().

• Reset api:mrsm to default repository. In my task to move everything to the preconfigured instance, I overstepped and made the default repository into the configured instance, which by default is a SQLConnector, so that broke things! In case you were affected by this change, you can simply reset the value of default_repository to api:mrsm (or your api server) to return to the desired behavior.

• 🧹 Housekeeping (refactoring). I removed nearly all instances of declaring mutable types as optional values, as well as additional typing hints. There may still be some additional cleaning to do, but now the functions are neat and tidy!

v0.2.17¶

• Added CockroachDB as a supported database flavor. CockroachDB may be a data source or a Meerschaum backend. There may be some performance tuning to do, but for now, it is functional. For example, I may implement bulk insert for CockroachDB like what is done for PostgreSQL and TimescaleDB.
• Only attempt to install readline once in Meerschaum portable. The first Meerschaum portable launch will attempt to install readline, but even in case of failure, it won't try to reinstall during subsequent launches or reloads.
• Refactored SQLAlchemy configuration. Under system:connectors:sql, the key create_engine has been added to house all the sqlalchemy configuration settings. WARNING: You might need to run delete config system to refresh this portion of the config file in case any old settings break things.
• Dependency conflict resolution.
• As always, more bugfixes :)

v0.2.16¶

• Hypertable improvements and bugfixes. When syncing a new pipe, if an id column is specified, create partitions for the number of unique id values.
• Only use api:mrsm for plugins, resort to default instance for everything else.
• Fix bug that mirrored changes to main under default.

v0.2.15¶

• MySQL/MariaDB bugfixes.
• Added aiomysql as a driver dependency.

v0.2.14¶

• Implemented bootstrap pipes action. The bootstrap pipes wizard helps guide new users through creating connectors and pipes.
• Added edit pipes definition action. Adding the word definition to the edit pipes command will now open a .sql file for pipes with sql connectors.
• Changed api_instance to symlink to instance by default.
• Registering users applies to instances, not repositories. The action register users now uses the value of instance instead of default_repository. For users to make accounts with api.mrsm.io, they will have to specify -i api:mrsm.

v0.2.13¶

• Fixed symlink handling for nesting dictionaries. For example, the environment variables for the API service now contain clean references to the meerschaum and system keys.
• Added MRSM_PATCH environment variable. The MRSM_PATCH environment variable is treated the same as MRSM_CONFIG and is loaded after MRSM_CONFIG but before patch or permanent patch files. This allows the user to apply a patch on top of a symlinked reference. In the docker-compose configuration, MRSM_PATCH is used to change the sql:main hostname to db, and the entire meerschaum config file is loaded from MRSM_CONFIG.
• Bugfixes, improved robustness. Per usual, many teeny bugs were squashed.

v0.2.12¶

• Improved symlink handling in the configuration dictionary. Symlinks are now stable and persistent but at this time cannot be chained together.
• Improved config file syncing. Generated config files (e.g. Grafana data sources) may only be edited from the main edit config process.
• Upgraded to PostgreSQL 13 TimescaleDB by default. This may break existing installs, but you can revert back to 12 with edit config stack and changing the string latest-pg13-oss under the db image to latest-pg12-oss.
• Bugfixes. Like always, this release includes miscellaneous bugfixes.

v0.2.11 (release notes before this point are back-logged)¶

• API Chaining Set a Meerschaum API as a the parent source connector for a child Meerschaum API, as if it were a SQLConnector.

v0.2.10¶

• MRSM_CONFIG critical bugfix The environment variable MRSM_CONFIG is patched on top of your existing configuration. MRSM_PATH is also a patch that is added after MRSM_CONFIG.

v0.2.9¶

• API and SQL Chunking Syncing data via an APIConnector or SQLConnector uploads the dictionary or DataFrame in chunks (defaults to a chunksize of 900). When calling read() with a SQLConnector, a chunk_hook callable may be passed, and if as_chunks is True, a list of DataFrames will be returned. If as_iterator is True, a dataframe iterator will be returned.

v0.2.8¶

• API Chaining introduction Chaining is first released on v0.2.8, though it is finalized in 0.2.11.

v0.2.7¶

• Shell autocomplete bugfixes

v0.2.6¶

• Miscellaneous bugfixes and dependency updates

v0.2.1 – v0.2.5¶

• Shell improvements Stability, autosuggest, and more.
• Virtual environments Isolate dependencies via virtual environments. The primary entrypoint for virtual environments is meerschaum.utils.packages.attempt_import().

v0.2.0¶

• Plugins Introduced the plugin system, which allows users and developers to easily integrate any data source into Meerschaum. You can read more about plugins here.
• Repositories Repositories are Meerschaum APIs that register and serve plugins. To register a plugin, you need a user login for that API instance.
• Users A user account is required for most functions of the Meerschaum API (for security reasons). By default, user registration is disabled from the API side (but can be enabled with edit config system under permissions). You can register users on a direct SQL connection to a Meerschaum instance.
• Updated shell design Added a new prompt, intro, and more shell design improvements.
• SQLite improvements The connector sql:local may be used as as backend for cases such as when running on a low-powered device like a Raspberry Pi.

Meerschaum's first point release focused on a lot, but mainly stability and improving important functionality, such as syncing.

A lot was accomplished in the first 60 releases of Meerschaum. For the most part, the groundwork for core concepts like pipes, syncing, the config system, SQL and API connectors, bulk inserts, and more was laid.