At Silectis, we deploy Magpie clusters across AWS, Google Cloud, and Azure. But because some of our internal infrastructure resides only on AWS, we need to establish private connections between these environments so that clusters on Google Cloud and Azure can access those private AWS resources. In this post, we’ll walk through the details of how we … [Read more...] about Tutorial: Creating a Multi-Cloud VPN with Terraform between AWS, GCP, and Azure
How to Ensure Security In Your Data Lake
THE IMPORTANCE OF SECURITYThis post is the third in a series of posts about getting up and running with a Magpie Data Lake. In previous blog posts, we’ve discussed rapidly prototyping a data lake with Magpie, and automating loads into a data lake with Magpie. This post will address a third important piece of data lake infrastructure: security … [Read more...] about How to Ensure Security In Your Data Lake
Tutorial: Using Magpie to Implement a Cloud Data Lake
PILOTING A DATA LAKE WITH MAGPIEMore and more companies are turning to data lakes as a way to unify and get value out of their growing collections of data. However, it can be a challenging to navigate the ever-changing technology landscape around these lakes, set one up, and quickly get value from it.At Silectis, we recommend that our … [Read more...] about Tutorial: Using Magpie to Implement a Cloud Data Lake
Magpie in Action: Job Management
A quick explainer on why we built we job management directly into Magpie, our data engineering platform. Plus, a step-by-step tutorial on how to use it.SETTING THE STAGEWhen first setting up a data lake, it is common for organizations to start with a static export of data. This enables users to immediately take advantage of the advanced … [Read more...] about Magpie in Action: Job Management
Automate Data Exploration with Silectis Magpie
As the first step in the data engineering process, data exploration is crucial. Our data engineering platform, Magpie, makes it simple, fast, and ensures accuracy.GETTING TO KNOW YOUR DATASETWhether as a new employee, a consultant or contractor, or as someone tasked with working on a new project, most data professionals have experienced the … [Read more...] about Automate Data Exploration with Silectis Magpie