PLATFORM
One platform. One metadata model. Foundation in 4 weeks.
Runtime, catalog, and AI assistants under one metadata model. Software you operate, paired with the experts who get your data AI-ready.
WHY ONE PLATFORM
Three things most data platforms treat as separate products — integrated by design.
The runtime executes the work. The catalog describes it. The AI assistants act on both. They share one metadata model, so there’s no second catalog, no registration step, no shadow store the assistants reason from. No integration tax.
The foundations are open source. Apache Spark is the runtime. SQL and Python are the programming languages. The knowledge corpus is plain markdown. Model providers are pluggable.
AI WORKSPACES
AI everywhere you work.
Catalog, notebook, and data exploration — three places where the metadata layer becomes intelligence. Each surface reads from the same schema, lineage, and history the runtime already knows. Nothing to sync, and nothing to re-index when the data changes.
01 — CATALOG ASSISTANT
Ask your catalog. It answers.
Ask “what tables hold customer data?” or “how is this field computed?” The assistant traverses lineage, runs validating queries, and proposes description updates — grounded in the schema Magpie already knows.
An action-planning agent that knows your catalog. It traverses lineage across sources, runs validating queries to confirm what it found, and proposes catalog updates. Every action is grounded in the schema Magpie already harvested. Every change is reviewable before it applies.
- Natural-language questions about your data — schemas, tables, lineage, job logic
- Auto-generated table and field descriptions, staged as proposals for review
- Validating queries run before answers are shown
- Lineage traversal across data sources, with configurable depth
- Plain-English explanations of jobs and transformations
- Multi-action plans — explore, describe, and generate in a single request
02 — NOTEBOOK ASSISTANT
AI inline across your work.
Generate the query, explain the error, summarize the result. The notebook reads prior paragraphs and the working catalog, so suggestions land in your code, not as generic boilerplate.
Sits inline across MagpieScript, SQL, Python, and Scala. Reads prior cells and the working catalog, so suggestions arrive with context — the right table names, the right column types, the right lineage. Generic AI code assistants don’t know what your data looks like. The Notebook Assistant does.
- Code generation across MagpieScript, SQL, Python, and Scala
- Three interaction modes — Build (write inline), Plan (design multi-turn), Refine (iterate)
- Reads prior paragraphs and the working catalog for grounded context
- Proposes new paragraphs or rewrites existing ones with side-by-side diffs
- Approve / reject queue — nothing writes to the notebook without you
- Generates inline documentation and explanations alongside the code
03 — DATA EXPLORATION
Conversation, with the receipts.
Ask, refine, pivot the answer into a chart. Every response arrives with the SQL it ran and the catalog it touched — so you can audit, save, share, or hand the work to an agent.
Conversation as the primary interface to your data. Ask in natural language; the assistant translates to validated SQL — checked against the same schema execution will use — and returns the answer with the SQL it ran and the catalog objects it touched. Refine, pivot into a chart, save, share, or hand the work off to an agent. Every response carries its own provenance.
- Natural-language questions answered with the exact SQL the assistant ran
- See which catalog objects the answer touched — tables, columns, joins
- Sortable, filterable results table; virtualized for large datasets
- Pivot into bar, line, pie, scatter, or heatmap charts
- Follow-up questions — the assistant maintains context across turns
- Save questions and answers, organized into folders for reuse
THE FOUNDATION
A mature platform beneath the AI.
Beneath the AI workspaces: a mature platform that handles runtime, catalog, connectors, streaming, validation, and security at enterprise scale.
METADATA REPOSITORY
40+ object types, audited.
Schema, lineage, ownership, activity, validation, and access — all tracked as first-class entries in a single catalog. Captured automatically on every load, mutation, and run.
CONNECTORS
Where your data lives.
Native connectors for Snowflake, Redshift, BigQuery, PostgreSQL, MySQL, SQL Server, Kafka, and the broader Spark ecosystem: S3, ADLS, GCS, Delta, Parquet. Sources can sit in any cloud account or on-prem environment the runtime can reach. Read in place; write where you choose.
STREAMING + BATCH
Either. Both. By design.
Continuous streaming ingestion sits alongside scheduled batch loads — same platform, same catalog, same lineage. Real-time and historical data join in the same pipeline.
DATA QUALITY
Validation as a first-class object.
Persistent validator framework, field-level profiling, and quality gates. Every validation run logged to the catalog for trend analysis and anomaly detection.
SECURITY & GOVERNANCE
Multi-tenant by construction.
RBAC at organization, customer, and principal scope. KMS-backed encryption for secrets and field-level data. Every action produces an audit record.
RUNTIME
Built on Apache Spark.
Distributed execution across AWS, Azure, and GCP. Existing Spark jobs run on Magpie unmodified — your team’s Spark, Python, Scala, and SQL skills carry over. MagpieScript, Magpie’s orchestration DSL, drives the platform around those languages — not in place of them. Because every platform action is scriptable, AI agents can write any operation as code — reviewable before anything ships.
THE METADATA LAYER
Built into the runtime, not bolted on.
Most catalog products sit outside the runtime — they observe what the platform produces and infer the rest. Magpie produces the catalog as the runtime works.
Schema and lineage are captured automatically on every data load. Every command in MagpieScript registers itself as a metadata side-effect. Every transformation leaves an audit trail. The catalog isn’t an observation product — it’s the trace of the work.
That distinction is also what grounds the AI. LLMs hallucinate when they don’t know what your tables mean, how fields relate, or where the data came from. The fix isn’t a smarter prompt — it’s a richer substrate. The metadata Magpie’s runtime produces is exactly that substrate.
What gets captured: schema on every load. Lineage on every load. AI-generated descriptions of fields. The working schemas your team actually uses. Validation rules as SQL. Profiling on demand.
The same metadata that runs your queries grounds your AI.
BY DESIGN
Four ways Magpie is different.
-
01
The catalog is a byproduct of execution.
Schema, lineage, ownership, and activity are captured as work happens. A
CREATE TABLEin MagpieScript creates the table and records its metadata in the same operation. The catalog is the trace of the work, not a model of it. -
02
AI agents need the same substrate humans need.
The Catalog Assistant traverses the same lineage graph the runtime captured. The Data Exploration Assistant validates queries against the same schema execution will use. No second catalog. Traceable, not approximate.
-
03
Source data stays in place.
Magpie reads from customer-controlled stores using credentials the customer issues and revokes. No data copy inside the platform independent of customer-granted access. When upstream access changes, the change applies immediately.
-
04
Open at the substrate.
No proprietary connector, driver, file format, or model surface required. Apache Spark is the runtime. JDBC is the access protocol. Model providers are pluggable across OpenAI, Anthropic Claude, AWS Bedrock, and Azure Foundry.
CUSTOMER RESULTS
“
Farm Journal owns its data and its stack — parquet in our own S3, with Silectis Magpie’s Spark-based orchestration layer running on top. The combination of an open platform, a partner who embeds with our team, and AI applied at every stage lets us innovate faster than we could with another vendor that is focused on just pushing subscriptions. In a fast-moving environment, that partnership is the product.
Bob Clinkert
Chief Technology Officer, Farm Journal
PRICING & ENGAGEMENT
The Foundation Package gets your data ready for analytics and AI in four weeks — platform, infrastructure, and the experts who deliver.
Year-one investment typically runs in the low-to-mid six figures; year two and beyond is predictable. Capacity scales with your data footprint, not your seat count.
Talk to us about a tailored package.
Point Magpie at your data.
A working catalog and your first analytics in production. Foundation in 4 weeks.