At CARFAX, a data services company that supplies vehicle history reports to individuals and businesses, accuracy and trustworthiness in their data is paramount, and a modern data architecture is their vehicle for sound decision-making across the company. Karen White, Director of Business Intelligence, recently led her team’s transformation from … [Read more...] about How CARFAX Uses Magpie to Modernize Its Data Architecture
Data Lakes
How to Implement a Data Lake with Apache Airflow and Silectis Magpie
Let’s face it, operating in a data-driven environment is hard. Teams, even small ones, can generate a painfully large number of batch processes that need to run on schedules. Drag-and-drop ETL tools become a maze of dependencies as business logic expands. Cron jobs lack transparency, failing silently and sucking away developer time. It’s in … [Read more...] about How to Implement a Data Lake with Apache Airflow and Silectis Magpie
How to Ensure Security In Your Data Lake
THE IMPORTANCE OF SECURITY This post is the third in a series of posts about getting up and running with a Magpie Data Lake. In previous blog posts, we’ve discussed rapidly prototyping a data lake with Magpie, and automating loads into a data lake with Magpie. This post will address a third important piece of data lake infrastructure: security … [Read more...] about How to Ensure Security In Your Data Lake
Tutorial: Using Magpie to Implement a Cloud Data Lake
PILOTING A DATA LAKE WITH MAGPIE More and more companies are turning to data lakes as a way to unify and get value out of their growing collections of data. However, it can be a challenging to navigate the ever-changing technology landscape around these lakes, set one up, and quickly get value from it. At Silectis, we recommend that our … [Read more...] about Tutorial: Using Magpie to Implement a Cloud Data Lake
Data Lake Architecture Guide: Choosing the Right Storage Tool
Overview: Build Your Data Architecture to Enable Use Cases One of the things that we often wrestle with in building out data lake architecture is how to best lay out the infrastructure to support different analytical use cases, and more specifically, what storage mechanism might yield the best performance. One of the virtues of data lakes is … [Read more...] about Data Lake Architecture Guide: Choosing the Right Storage Tool
Magpie in Action: Job Management
A quick explainer on why we built we job management directly into Magpie, our data engineering platform. Plus, a step-by-step tutorial on how to use it. SETTING THE STAGE When first setting up a data lake, it is common for organizations to start with a static export of data. This enables users to immediately take advantage of the advanced … [Read more...] about Magpie in Action: Job Management