The importance of data engineering is on the rise, with organizations increasingly investing in talent and infrastructure. Here at Silectis, we are in the fortunate position of working with a wide range of enterprises across multiple industries. I caught up with a few members of the team to take note of some of the data engineering trends we … [Read more...] about 7 Data Engineering Trends to Watch
Magpie Now Supports Microsoft Azure
Today, we are excited to share that Silectis Magpie, our data engineering platform is supported on Microsoft Azure. Silectis Magpie brings together Azure’s big data capabilities so that organizations can gain business insights from their raw data, faster. Our managed platform provides a centralized hub for analytics, so your team can … [Read more...] about Magpie Now Supports Microsoft Azure
Improve Your SQL Skills: Master the Gaps and Islands Problem
The "gaps and islands" problem is a scenario in which you need to identify groups of continuous data (“islands”) and groups where the data is missing (“gaps”) across a particular sequence. The problem may sound arcane at first glance but knowing how to solve this type of problem is critical for advanced SQL practitioners. These starting and … [Read more...] about Improve Your SQL Skills: Master the Gaps and Islands Problem
How to Implement a Data Lake with Apache Airflow and Silectis Magpie
Let’s face it, operating in a data-driven environment is hard. Teams, even small ones, can generate a painfully large number of batch processes that need to run on schedules. Drag-and-drop ETL tools become a maze of dependencies as business logic expands. Cron jobs lack transparency, failing silently and sucking away developer time. It’s in … [Read more...] about How to Implement a Data Lake with Apache Airflow and Silectis Magpie