The catalog is built as data flows through Magpie.

User Interaction & API

The top layer is where humans and other systems interact with Magpie. Four languages span it: MagpieScript, SQL, Python, and Scala.

The Notebook is multi-language interactive computing with real-time collaboration and the Notebook Assistant inline. The Catalog UI is a first-class interface for browsing schemas, projects, sources, repositories, lineage graphs, and field-level history — the Catalog Assistant lives inside it. The Chat / Conversational Query surface handles natural-language data exploration; each response includes the SQL it ran and the catalog objects it touched, reviewable, saveable, shareable.

Two programmatic surfaces sit alongside the three human-facing UIs. The Magpie API is a REST interface to the platform’s capabilities — catalog operations, query generation, conversational queries, notebook execution, saved questions. The JDBC Interface exposes Magpie to existing BI tools (Tableau, Looker, Power BI) and any other JDBC-aware client via the standard protocol. No platform-specific connector required.

MagpieScript

MagpieScript is the platform’s command DSL — every platform action expressed as code. Schema definition, data movement, job orchestration, security policy, quality validation: all of it is scriptable, alongside your SQL, Python, and Scala in the same notebook.

That matters more in an AI-native world than it did before. Because every platform action is scriptable, AI agents work in the same medium your team works in: code. The Catalog Assistant proposes catalog updates as MagpieScript. The Notebook Assistant generates platform operations the same way it generates queries. Every change is reviewable as a diff before anything ships.

The feedback loop runs both ways. Work the platform does gets recorded as metadata; metadata grounds the AI; the AI proposes more code your team reviews and approves. Catalog and execution are the same operation, not two systems you have to reconcile.

MagpieScript is specific to Magpie, and so are the metadata interfaces: the describe statements, the shape of the lineage responses, the profiling and description tooling. That coupling is what lets the assistants work against real schema and lineage, and propose platform operations as reviewable code. What they operate on is standard. Source data stays in your accounts, the analytical work is Spark, SQL, Python, and Scala, and catalog contents export to parquet or delimited files.

Magpie Intelligence Layer

The Intelligence Layer hosts the assistants and the infrastructure that grounds them. There is no second catalog for the assistants: each one reads the same metadata the runtime captured.

The Data Exploration Assistant translates natural-language questions into validated SQL and presents the results — validated against the same schema execution will use.

The Catalog Assistant is an action-planning agent. It traverses lineage, runs validating queries, and proposes catalog updates — every action grounded in the platform’s existing metadata, every change reviewable before it applies.

The Notebook Assistant works alongside the user in the notebook. It reads prior paragraphs and the working catalog for context, generates MagpieScript, SQL, Python, and Scala inline, and proposes every change as a diff the user approves or rejects.

Three services support all three assistants. The Knowledge Service retrieves from a hierarchical documentation corpus by explicit topic ID, not embedding similarity — more predictable behavior, simpler audit path. Memory maintains durable assistant context across conversations through observation logging and background compression. Model Integration treats AI providers as pluggable: OpenAI, Anthropic Claude, AWS Bedrock, and Azure Foundry are all supported, and the assistants don’t change behavior based on which model is wired in.

The Intelligence Layer composes prompts and context; the data itself is processed within the platform.

Magpie Core

Magpie Core is the engine. It owns metadata management, lineage, profiling, security, data quality, job orchestration, and activity tracking.

All of it is backed by the Metadata Repository, the persistent store that records everything the platform knows about the data it operates on. Schema and lineage are captured automatically on every data load — the catalog is a first-class output of execution, not a downstream observation.

Multi-tenant identity, role-based access control, and KMS-encrypted secrets are enforced at the Core. Every action produces an audit record.

The Metadata Repository

Every object the platform manages is a first-class entry with a stable identity, an owner, and an audit trail. The objects fall into four groups.

Tenancy and identity. Organization — top-level tenant; the boundary for users, repositories, clusters, and most other objects. User — individual identity. Role — named permission set, global or organization-scoped.

Data structures. Repository — namespace owned by an organization, containing schemas, data sources, and projects. Data source — connection to an external system. Schema — logical namespace for tables and streams. Table and Stream — persistent or continuous data structures defined within a schema. Field — column on a table or stream. Sink — output target for streaming data.

Orchestration. Project — container for related jobs. Job — executable unit composed of tasks. Task — a single step (SQL mapper, script, nested job). Job schedule — when a job runs; subscriptions notify on schedule events. Cluster — Spark compute target with its own schedule and subscriptions.

Capability and security. Validation suite and Validator — named data-quality rules and their containers. Secret — encrypted credential or configuration value. File — blob storage for scripts and artifacts. Identity provider — SAML/OAuth configuration. Access token — programmatic credential.

Lineage and activity history are not separate objects — they are derived from operations on the objects above. The metadata captured during execution is the metadata the Catalog UI and the assistants use in their processing.

Spark Runtime Environment

The runtime is Apache Spark. Magpie extends it with command processors and lineage capture, but the Spark API itself is unchanged. Existing Spark jobs run on Magpie unmodified.

SQL, DataFrame, structured streaming, and UDFs in Python and Scala all work as expected. Customer Spark, Python, Scala, and SQL skills transfer.

No proprietary connector, driver, file format, or model surface required. JDBC is the access protocol. The knowledge corpus is markdown. Model providers are pluggable.

Source Data

The base layer is the customer’s source data. Magpie reads in place — across object stores, analytics warehouses, conventional relational databases, and any other source the Spark runtime can connect to.

• Object stores — AWS S3, Azure ADLS, Google Cloud Storage
• Analytics warehouses — Snowflake, BigQuery, Amazon Redshift
• Relational databases — PostgreSQL, MySQL, SQL Server, and other JDBC-compatible engines
• Other Spark-native sources — Kafka, Delta Lake, Parquet, Avro, ORC, and the broader Spark connector ecosystem

The customer issues credentials with the access scope they want Magpie to have and revokes them through their cloud’s standard IAM or database authentication. Data sources can live in any cloud account or on-prem environment the runtime can reach. No data copy inside the platform independent of customer-granted access. When upstream access changes, the change applies immediately.