Confluent Reimagines Data Pipelines for the Streaming Era with Stream Designer

Confluent announced Stream Designer, a visual interface that enables developers to build and deploy streaming data pipelines in minutes. This point-and-click visual builder is a major advancement toward democratising data streams so they are accessible to developers beyond specialised Apache Kafka experts. With more teams able to rapidly build and iterate on streaming pipelines, organisations can quickly connect more data throughout their business for agile development and better, faster, in-the-moment decision making.

“We are in the middle of a major technological shift, where data streaming is making real time the new normal, enabling new business models, better customer experiences, and more efficient operations,” said Jay Kreps, Cofounder and CEO, Confluent. “With Stream Designer we want to democratise this movement towards data streaming and make real time the default for all data flow in an organisation.”

In the streaming era, data streaming is the default mode of data operations for successful modern businesses. The streaming technologies that were once at the edges have become core to critical business functions. This shift is fueled by the growing demand to deliver data instantaneously and scalably across a full range of customer experiences and business operations. Traditional batch processing can no longer keep pace with the growing number of use cases that depend on sub-millisecond updates across an ever-expansive set of data sources.

Organisations are seeking ways to accelerate their data streaming initiatives as more of their business is operating in real time. Kafka is the de facto standard for data streaming, as it enables over 80% of Fortune 100 companies to reliably handle large volumes and varieties of data in real time. However, building streaming data pipelines on open-source Kafka requires large teams of highly specialised engineering talent and time-consuming development spread across multiple tools. This puts pervasive data streaming out of reach for many organisations and leaves existing legacy pipelines clogged with stale and outdated data.

Stream Designer: The First Visual Interface for Rapidly Building Streaming Data Pipelines Natively on Kafka

“Data streaming is quickly becoming the central nervous system of our infrastructure as it powers real-time customer experiences across our 12 countries of operations,” said Enes Hoxha, Enterprise Architect, Raiffeisen Bank International. “Stream Designer’s low-code, visual interface will enable more developers, across our entire organisation, to leverage data in motion. With a unified, end-to-end view of our streaming data pipelines, it will improve our developer productivity by making real-time applications, pipeline development, and troubleshooting much easier.”

Stream Designer provides developers a flexible point-and-click canvas to build pipelines in minutes and describe data flows and business logic easily within the Confluent Cloud UI. It takes a developer-centric approach, where users with different skills and needs can seamlessly switch between the UI, a code editor, and command-line interface to declaratively build data flow logic at top speed. It brings developer-oriented practices to pipelines, making it easier for developers new to Kafka to scale data streaming projects faster.

With Stream Designer, organisations can:

• Boost developer productivity: Instead of spending days or months managing individual components on open-source Kafka, developers can build pipelines with the complete Kafka ecosystem accessible in one visual interface. They can build, iterate and test before deploying into production in a modular fashion, keeping with popular agile development methodologies. There’s no longer a need to work across multiple discrete components, like Kafka Streams and Kafka Connect, that each require their own boilerplate code.
• Unlock a unified end-to-end view: After building a pipeline, the next challenge is maintaining and updating it over its lifecycle as business requirements change and tech stacks evolve. Stream Designer provides a unified, end-to-end view to easily observe, edit, and manage pipelines and keep them up to date.
• Accelerate development of real-time applications: Pipelines built on Stream Designer can be exported as SQL source code for sharing with other teams, deploying to another environment, or fitting into existing CI/CD workflows. Stream Designer allows multiple users to edit and work on the same pipeline live, enabling seamless collaboration and knowledge transfer.