Airbyte takes the biscuit for open source data integration
Airbyte is the creator of an open source data integration platform.
The company has this month announced its Free Connector Programme to provide free use of any data pipeline on Airbyte Cloud that uses alpha or beta connectors.
The company says that its technology makes moving data easy (and affordable) across almost any source and destination.
With some 600 contributors in its data engineering contributor community, the organisation is seeing open source data code purists work on the tooling to build and maintain connectors with its Connector Development Kit.
Airbyte defines generally available connector reliability as at least 99% synchronisation success rate with more than 40 users, along with other criteria. Beta connectors still on average have a 93% sync success rate and alpha connectors a 90% sync success rate.
Since Airbyte’s start three years ago, there are now more than 300 connectors available, which (so claims the company) means the Airbyte platform has the most [data movement and integration] connectors in the industry.
The ‘long-tail’ of connectors
With its open source model and paid contributor programme, Airbyte hopes it is addressing the industry need for the long-tail of connectors that otherwise never get addressed.
Most of its open source connectors will be available for use with Airbyte Cloud by the end of the quarter, thus making more than 200 connectors part of the free connector program.
“The Free Connector Programme completes the flywheel we put in place to commoditize data integration,” said John Lafleur, the company’s co-founder and chief operating officer. “What we were missing is significant usage across all our long-tail connectors – those less used – in order to help identify the edge cases we don’t fully support yet.”
“By providing free use of alpha and beta connectors, we’re encouraging the industry to use our connectors while helping us improve their reliability,” Lafleur continued. “Reliability is the ultimate goal – we will only charge for use of Airbyte Cloud with data pipelines that have generally available connectors.”
Over the past year-and-a-half, more than 35,000 companies have used Airbyte to sync data from sources such as PostgreSQL, MySQL, Facebook Ads, Salesforce, Stripe and connect to destinations that include Redshift, Snowflake, Databricks and BigQuery.
According to Lafleur and team, “Airbyte’s open-source data integration solves two problems. First, companies always have to build and maintain data connectors on their own because most less popular ‘long-tail’ data connectors are not supported by closed-source ELT (extract, load, transform) technologies. Second, data teams often have to do custom work around pre-built connectors to make them work within their unique data infrastructure.”
Last month, Airbyte launched in Europe with General Data Protection Regulation (GDPR)-compliant data processing that was accomplished by separating Airbyte’s control plane and data plane.
