Datalytyx develops open source webhooks for Snowflake

Over the last 2 years, Datalytyx has been developing Gallium (IoT Smart Data Compression Algorithm) technology, now launched as part of a new era of data services on Snowflake.

To deliver this, the Datalytyx team had to find new ways of triggering external data services from within Snowflake based on table updates without building external polling solutions.

Datalytyx has solved the challenge of triggering actions and processes in external systems from within Snowflake and has made the solution available to all Snowflake customers for free.

The Github repository for Datalytyx’s Snowflake webhooks can be found here.

By implementing webhooks, Snowflake users can now take advantage of powerful bidirectional integration from Snowflake to external systems like Talend and Databricks.

Founders of Snowflake, the fastest growing enterprise data warehouse in the cloud, Benoit Dageville and Thierry Cruanes expressed their enthusiasm for this solution.

“Thierry and I have been working closely with the Datalytyx team for the last year. Datalytyx are one of the most experienced teams around the deep insides of Snowflake and we have been enjoying working with them. Datalytyx work closely with Snowflake as part of their data platform, but are also specialists in IoT timeseries data. They have produced Gallium which is a unique time series compression algorithm with limited loss of fidelity. Whether its 5:1 or 500:1 compression, the business value to companies using Snowflake and Gallium for IoT timeseries data compression will be significant.

This is an example of great solutions coming from the Snowflake ecosystem to help Snowflake users. It’s great to see innovative solutions from leading partners like Datalytyx. This Gallium integration is the first of a new age of data services called natively from within Snowflake, and with the generous opensourcing of this code, we look forward to seeing many more.”

The use cases for webhooks in Snowflake vary from triggering simple Email or SMS alerts based on events in Snowflake to complex integration and orchestration with external systems.

For example, a process in Talend can trigger long running jobs inside Snowflake without having to maintain a session or having to poll to find out if jobs have completed. When the job in Snowflake has finished, a webhook call can be made back into Talend to start the next step in the process.

Databricks users can also benefit from using webhooks in Snowflake. Many Databricks customers hold data inside Snowflake for training models. When a critical mass of new data has been added into Snowflake, a retraining of a model can be triggered within Databricks.

As a partner and service provider of Snowflake, Talend, Databricks, Azure and AWS, Datalytyx saw the need for a universally supported and well accepted method for triggering events in web-based systems from inside Snowflake. So they developed these webhooks and made them available on Github to help support the Snowflake customer community as well as its partner communities.

In the coming weeks, Datalytyx will be publishing more blogs detailing how they used these webhooks with Table Streams, how to use them for Data Services and more.