Zero-Code, Infinite Scale: FOSS-Powered Agentic Pipelines with Kafka and Milvus

In this session, discover how you can build fully scalable, and real-time embedding pipelines—without writing a single line of code. I’ll walk you through a working prototype that leverages a 100% FOSS (free open source software) tech stack, showcasing end-to-end automation from data ingestion to vector storage and retrieval.

Using Kafka as the streaming backbone, I’ll demonstrate how a Kafka RSS Source Connector ingests live news feeds into your pipeline. Kafka then acts as the “central nervous system,” orchestrating seamlessly between components with real-time data movement. With a Kafka Sink Connector, we route the processed data directly into Milvus v2.6, a state-of-the-art open-source vector database.

The most powerful part? Milvus’s embedding function capability automatically handles the conversion of incoming text to high-dimensional vector embeddings, eliminating manual coding and complex ETL logic. This fully automated pipeline is zero-maintenance, scalable, and built entirely with open-source connectors and databases.

Whether you’re an engineer, architect, or innovator exploring scalable AI solutions, you’ll leave with practical insights and a new perspective on what’s possible with contemporary open-source technologies.