Data lineage can be a tremendously useful tool for data engineering and analytics, but is often treated as an afterthought both because of the challenges in implementation and the fact that it has not been broadly available within organizations. Many practitioners have never had access to data lineage information and may not know what they are … [Read more...]
Data Engineering Glossary
If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. This data engineering glossary is by no means exhaustive, but should provide some foundational context and … [Read more...]
The Case for the Holistic Data Engineering Platform
We recently published a new report: The Case for the Holistic Data Engineering Platform, which outlines why piecemeal tooling isn't sustainable for most modern companies. In today’s world, it’s well understood that the best businesses make data-driven decisions to push their organizations forward. The business landscape is largely a race - to be … [Read more...]
How CARFAX Uses Magpie to Modernize Its Data Architecture
At CARFAX, a data services company that supplies vehicle history reports to individuals and businesses, accuracy and trustworthiness in their data is paramount, and a modern data architecture is their vehicle for sound decision-making across the company. Karen White, Director of Business Intelligence, recently led her team’s transformation from … [Read more...]
Tutorial: Creating a Multi-Cloud VPN with Terraform between AWS, GCP, and Azure
At Silectis, we deploy Magpie clusters across AWS, Google Cloud, and Azure. But because some of our internal infrastructure resides only on AWS, we need to establish private connections between these environments so that clusters on Google Cloud and Azure can access those private AWS resources. In this post, we’ll walk through the details of how we … [Read more...]
Data Lake Architecture Guide: Choosing the Right Storage Tool
Overview: Build Your Data Architecture to Enable Use Cases One of the things that we often wrestle with in building out data lake architecture is how to best lay out the infrastructure to support different analytical use cases, and more specifically, what storage mechanism might yield the best performance. One of the virtues of data lakes is … [Read more...]