Thinking about Analytics Holistically

Here at Silectis, we spend a good deal of time thinking about how the landscape for analytics is changing. As a technology-focused company, the lion’s share of our energy goes to making the technology work effectively and giving data engineering and data science practitioners tools that make their lives easier.

However, we realize that our technology lives in a broader context. It has to dovetail with the processes and organizations that surround it. Ultimately, it needs to serve the consumers of the data, those generating the insight, and those applying it to drive decision-making. Getting value out of data requires a lot of work outside of just the technology.

To that end, we have put together a series of white papers that capture some of the observations that we have made over the course of our client work and expresses our point of view about the broader structure of a successful analytics program in today’s rapidly evolving environment.

The first in the series, EFFECTIVE INSIGHT DELIVERY is now available. We are starting with a focus on how analytics get delivered to stakeholders.

Changing user expectations, more complex analytics, and rapidly changing data require a fresh look at how data insights are packaged and made available. Moving beyond conventional reporting and BI, we examine the new modes of analytics delivery and their implications for upstream processes and technology.

Hopefully, you will find this insight helpful. We welcome your questions and comments.

You can find it here.

Demetrios Kotsikopoulos is the CEO of Silectis. 
You can find him on LinkedIn and Twitter.

Introducing Magpie

edited.jpg

As some of you are aware, we have been working on something pretty big for the last few years. We have been helping clients to build their data and analytics infrastructure, but that isn’t the entire story…

THE MAGPIE PLATFORM

We are finally ready to lift the curtain on our cloud data management platform, Magpie

Magpie helps companies organize their data so they can get to the exciting part, generating insights. Magpie is built to serve as the central hub for analytics data, supporting ETL, Data Lake, and Data Warehousing needs.  

We have been working on it while bootstrapping our consulting business and helping customers solve hard data management and analytics problems every day.

Magpie addresses the real-world challenges of data integration, data organization, and data exploration. As practitioners, building data warehouses or pulling together complex business analytics, we were continuously frustrated by the need to pivot across multiple tools and jump through hoops to get the infrastructure we needed. 

With Magpie we have pulled all of the core tools needed into one, cloud-based platform that readily plugs into existing architectures and lets organizations of all sizes participate in the revolution in data and analytics that is transforming science, business, and government. 

POWERED BY METADATA

The driving principle behind Magpie is the need to understand and organize data before diving into analytics. To that end, much of our focus has been on building a robust, extensible, and powerful data catalog or metadata repository. The catalog makes it easy for us to capture the structure of data and then use that structure to build scalable pipelines for data integration and analysis.

It also allows us to quickly deliver new features that add value to data by capturing more and more knowledge about the meaning, content, and history of each piece of information.

As Magpie evolves, we will build on this foundation to allow users to  collaborate more effectively around analysis, and to further automate learning from data. Our first step in this direction is the profiling available today that helps Magpie users understand the data content in any table with a one line command.

BUILT ON OPEN SOURCE AND OPEN STANDARDS

We have been able to build this platform in a short time and with very limited resources by leveraging the amazing work of the open source community. 

In particular, by leveraging Apache Spark to build our core distributed computing engine, we can achieve world class performance and scalability. It has freed us to focus on adding value to the end-to-end analytics process instead of the core processing engine.

MADE TO SOLVE REAL WORLD PROBLEMS

Most of all, Magpie is grounded in the realities of managing a complex data environment. We are working to eliminate the tedium and frustration that comes from dealing with the alphabet soup of technologies needed to set up a “big” data infrastructure. With Magpie, we are shifting the focus to creating real meaning and value from the data.

We have built Magpie to allow data engineers, analysts, and data scientists ramp up quickly without having to learn a whole new language or new way of working. Most data transformations and jobs can be build using our SQL-like scripting language.  Coupled with our support for Python and Scala, this means that there are skilled resources out there today that can get productive with Magpie in hours, not months.

Our process automation means that you can get from idea to operation quickly without needing to set up separate scheduling, logging, and monitoring. Our robust security layer let’s you keep control of your data while still providing flexible access  to data engineers, data scientists and business analysts.

WHAT’S NEXT

In the coming months, we will continue working with our customers and partners to enhance and expand the Magpie platform and deploy across more organizations.  If you are interested in learning more about Magpie, feel free to email us at info@silect.is.

Demetrios Kotsikopoulos is the CEO of Silectis. 
You can find him on LinkedIn and Twitter.