The Latest from Silectis
Knowing Your Data Starts with Data Lineage
Data lineage can be a tremendously useful tool for data engineering and analytics, but is often treated as an afterthought both because of the challenges in implementation and the fact that it has not been broadly available within organizations. Many practitioners have never had access to data lineage information and may not know what they…
Technical Tutorial: Machine Learning Model in R Using Spark
In this installment of Silectis Technical Tutorials, we provide a step-by-step introduction to building a machine learning model in R using Apache Spark. This post is intended for R users who understand the basics of machine learning and have an…
Apache Spark: A Beginner’s Guide to Optimizing Spark Scripts
This blog post covers Apache Spark basics and teaches readers why optimizing Spark scripts is important, and how to do it for both memory and runtime efficiency. This blog post is best suited for data analysts and data scientists looking…
Setting Up Your Data Engineering Team for Success: Building Future-Proof Data Architecture
Yesterday’s data is predictable and easy. Our challenge as data engineers is getting ready for tomorrow’s unpredictable new data. With the push toward more flexible solutions, let’s discuss how you can set up your data engineering team for success by…
Tutorial: Creating a Multi-Cloud VPN with Terraform between AWS, GCP, and Azure
At Silectis, we deploy Magpie clusters across AWS, Google Cloud, and Azure. But because some of our internal infrastructure resides only on AWS, we need to establish private connections between these environments so that clusters on Google Cloud and Azure…
Improve Your SQL Skills: Master the Gaps and Islands Problem
The “gaps and islands” problem is a scenario in which you need to identify groups of continuous data (“islands”) and groups where the data is missing (“gaps”) across a particular sequence. The problem may sound arcane at first glance but…
CATEGORIES
- Data Analytics
- Data Engineering
- Data Lakes
- Data Science
- Magpie
- Magpie in Action
- Management Insight
- Silectis News
- Tutorials
- Uncategorized
SUBSCRIBE
Join our mailing list to get updates from Silectis.