Cost to register: $799 Apache Pinot is a high-performance database engineered to serve analytical queries with extremely high concurrency, boasting latencies as low as tens of milliseconds. It excels at ingesting streaming data from sources like Apache Kafka and is optimized for real-time, user-facing analytics applications. In this full-day training, we will explore the architectures of Apache Kafka, Apache Flink, and Apache Pinot. We will run local clusters of each system, studying the role each plays in a real-time analytics pipeline. We’ll begin by ingesting static data into Pinot and querying it. Not content to stop there, we’ll add a streaming data source in Kafka, and ingest that into Pinot as well, showing how both data sources can work together to enrich an application. We’ll then examine which analytics operations belong in the analytics data store (Pinot) and which ones should be computed before ingestion. We’ll implement those operations in Flink. Having put all three technologies to use on your own in hands-on exercises, you’ll leave prepared to begin exploring the three of them together for your own real-time, user-facing analytics applications. LEARNING OBJECTIVES At the successful completion of this training, you will be able to: - List the essential components of Pinot, Kafka, and Flink - Explain the architecture of Apache Pinot and its integration with Apache Kafka and Apache Flink - Form an opinion about the proper role of Kafka, Flink, and Pinot in a real-time analytics stack - Implement basic stream processing tasks with Apache Flink - Create a table in Pinot, including schema definition and table configuration - Ingest batch data into an offline table and streaming data from a Kafka topic into a real-time table - Use the Pinot UI to monitor and observe your Pinot cluster PREREQUISITES To participate in this workshop, you will need the following: - Docker Desktop: We will use Docker to run Pinot, Kafka, and Flink locally. If you need to install it, please download Docker Desktop and follow the instructions to install it at https://www.docker.com/get-started/ - Resources: Pinot works well in Docker but is not designed as a desktop solution. Running it locally requires a minimum of 8GB of Memory and 10GB of disk space

Head of Developer Advocacy

Senior Technical Product Manager

Uncorking Analytics with Apache Kafka®, Apache Flink, and Apache Pinot®

In this talk, Kishore Gopalakrishna, the creator of Apache Pinot and CEO of StarTree, reveals critical distinctions between mission-critical analytics workloads and their non-mission critical counterparts. As companies increasingly rely on data-driven insights to steer their strategies, the urgency of real-time analytics solutions becomes evident. Gopalakrishna will explain why certain data operations demand immediate attention and why conventional data warehousing solutions may fall short. Drawing from his expertise in developing Apache Pinot, he highlights the essential features required to meet the demands of mission-critical analytics, and will showcase a framework for easily identifying mission-critical workloads that significantly impact internal &amp; external customer experience. Join us to gain a deeper understanding of why mission-critical data can't afford to wait, and how cutting-edge solutions like Apache Pinot are reshaping the analytics landscape forever.

Co-Founder & CEO

Power of Real-Time Analytics: Stay Ahead or Be Left Behind

Real-time analytics industry leaders will join us on stage for a 30-minute discussion of their recent experiences in the space. We'll discuss what drove them to adopt real-time analytics technologies, the business outcomes they were able to achieve as a result, the experiences of building applications and services in an emerging technology category, and how they see the category unfolding in the immediate future. This panel will provide you with a roadmap to success in the real-time systems you're building now and in the future.

VP, Developer Relations

Director of Engineering

VP, Product Engineering

Chief Product Officer

Vice President & GM

Real-Time Analytics Panel, Moderated by Tim Berglund

In this tech talk we would like to present an overview of how, at Uber, we leverage Pinot as a datasketch store for ML model monitoring use case. We also would like to highlight the power of Apache Pinot's aggregation system and capabilities of recently introduced datasketch functions. Problem and solution: Machine Learning life cycle operates with large-scale data across different storage systems, both online and offline. ML practitioners need to monitor and debug the data quality and distribution for each data set and also cross-compare different data sources. However, it is challenging to have a monitoring application that directly works with raw data from different systems. As a solution, we do data profiling for those datasets, and store the profiling results as data sketches inside Pinot. We leverage Pinot to achieve scalable storage and low-latency queries for data sketches so that we can enable both continuous ML monitoring and adhoc UI based debugging experience. To enable this, we integrated Apache Datasketches into Pinot, particularly focusing on KLL, CPC and Frequent Items sketches, and it clearly shows performance advantages compared to alternative storage solutions like Druid and MySQL. Join us for a deep dive into optimizing real-time analytics in ML applications with Apache Pinot.

Senior Software Engineer

Software Engineer

Real-Time ML Model Monitoring with Datasketches and Apache Pinot at Uber

Stripe has been using Apache Pinot for the past 2 years to store financial data to power both merchant-facing products and critical internal tools. Our use cases require supporting low latency queries with up to 99.99% query availability and large clusters up to petabytes in size. This talk will go over some of the operational lessons learned from running our Pinot clusters in production. We'll go over our cluster topology strategy and key configs we use to maintain our SLAs and keep our clusters stable.

Operating Apache Pinot at Scale

In today's hyper-connected digital landscape, real-time API usage analysis and billing have become paramount. With the proliferation of APIs at the heart of modern applications and services, gaining real-time insights into their use is often mission-critical as it depends on critical factors such as latency, throughput, freshness, and the correctness of generated insights. This talk discusses the architecture of a real-time API usage analytics system composed of Redpanda, Apache Flink, and Apache Pinot. Redpanda, as a scalable streaming data platform, enables high-volume, low-latency API volume data ingestion from API gateways in real time. The ingested data is streamed through Flink for streaming ETL, enabling operations like joins, aggregations, and transformations, feeding the final output of the pipeline to Apache Pinot for serving analytics at scale. In addition to that, we will use Flink for rate-limiting incoming API requests. Having such a system enables businesses to make informed, agile decisions, addressing performance bottlenecks, security threats, and resource allocation issues in the API infrastructure promptly. Also, it ensures seamless user experiences by identifying usage patterns, issues, and opportunities as they occur, proactively enhancing the quality of your product. Whether you're a data engineer, an application developer, or an architect, this talk promises to equip you with the knowledge and tools to build a real-time analytics solution that empowers your organization with the insights it needs to stay ahead in a data-driven world.

Senior Developer Advocate

Transforming Your APIs into Business Gold—Architecting a Real-Time API Usage Analytics Platform

In the dynamic realm of IoT, real-time analytics plays a pivotal role in making data-driven decisions and responding swiftly to changing conditions. I will demonstrate how Apache Pulsar's native MQTT support can be easily combined with Apache Pinot to provide real-time analytics on IoT data. This session goes beyond the theoretical by providing a hands-on code demo, where we will walk through the implementation of an IoT analytics pipeline that will serve as a template for setting up their own real-time IoT analytics solution.

Developer Advocate

Empowering Real-Time IoT Analytics with Apache Pulsar and Apache Pinot

At DoorDash we process 2.5 trillion messages a day across 2500 Real-Time pipelines in total. Our pipelines ingest data from mobile devices and internal services, stream-process the data with Apache Flink and Kafka before writing to Datalake(s). We use Trino, Pinot and other tools to query from Datalake(s). Along the way, we have built a rich set of automation tools to maintain the lifecycle of these pipelines from provisioning to clean-up. The lessons learnt as our business and data-needs scale guide us in both improving our pipelines and the related-automation. We hope to share our learnings with the broader Stream Processing community.

Software Engineer in Data Platform

Automating Real-Time Pipelines at Scale in DoorDash

Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero.

Beyond Tiered Storage: Serverless Kafka with No Local Disks

In this talk, we'll describe our motivation and journey of re-architecting the StarTree Cloud platform, powered by Apache Pinot. The talk will be centered around how we introduced the concept of &quot;workspace&quot; — a logical entity to provide isolation on a single cluster with shared infrastructure. We'll walk through how we architected everything with workspace-first thinking throughout the stack, from the onboarding app, to data management and ingestion, to the Pinot cluster management. We'll also dive into integral security issues like our RBAC implementation.

Founding Engineer

Head of Product

StarTree Serverless

Sometimes you really do need your dashboards to be faster. At Slack, we needed to deliver data at speed, having recently adopted Tableau as our next-generation visualization tool. Not long after, users reported issues with query latency, limitations on row count, and inability to use custom SQL. We solved this by introducing Apache Pinot as a data source for Tableau, replacing the Trino connector for several dashboard use cases. Only two months after this internal transition, we have over a terabyte of data in Pinot powering internal dashboards. And StarTree recently open-sourced the Pinot Tableau connector, removing the need for Slack to mature our home-grown connector. This drastically reduces the time required to integrate Pinot with Tableau while increasing the data quality level of the integration. This talk covers our zero-to-one path to power Tableau dashboards with Pinot data.

Driving Internal Analytics with Pinot and Tableau

In this presentation, we'll discuss our adoption of Apache Pinot to enhance real-time, user-facing analytics for Web3 applications. We'll explore how Pinot was integrated into our analytics stack and the specific use cases it now empowers, from handling massive data volumes to delivering instant query responses. The focus will be on the technical strategies and insights that enabled us to leverage Pinot's capabilities to meet the scalability and performance demands of the Web3 ecosystem.

Co-Founder & CTO

Co-Founder

Getting User-Facing Analytics Right for Web3 at Scale: A Real-Time Story with Apache Pinot

Streaming data is rapidly becoming a key component in modern applications, and Apache Kafka has emerged as a popular and powerful platform for managing and processing these data streams. However, as the volume and complexity of streaming data continue to grow, it becomes increasingly critical to have efficient and effective ways of querying and analyzing this data This is where query engines like Apache Flink, Trino, Timeplus Proton, Materialize, and ksqlDB come in. These powerful tools offer flexible and scalable ways of processing and analyzing streaming data in real-time, enabling users to extract valuable insights from their data streams. In this talk, we will introduce the audience to the world of querying streaming data on Kafka/Pulsar with SQL, compare and contrast the features and capabilities of each of these tools, and provide an in-depth analysis of their respective Pros and Cons. We will also discuss the best practices and scenarios where each tool is most effective. In conclusion, query engines like Apache Flink, Trino, ksqlDB, Timeplus Proton, RisingWave, Materialize and are useful tools in processing and analyzing streaming data. With their ability to extract valuable insights from real-time data streams, these tools are a valuable asset for modern data-driven applications. Coffee elements are added to the talk, to help audience to compare different OSS offerings.

Cofounder and Head of Product

Query Real-Time Data using SQL, a Guide for Coffee Lovers

Data platforms have evolved in amazing ways, tackling increasingly specific and difficult problems. Central to these advancements is data movement - the crucial pathway that gets data to the right place, in the right format. Over the years, numerous methodologies have emerged, including ETL, ELT, event streaming, stream processing, and Change Data Capture (CDC). This talk will take you on a journey through the evolving landscape of data movement technologies and tools, highlighting the trade-offs inherent in each method. We conclude with a pragmatic guide to architecting a unified data movement platform, drawing upon the accumulated knowledge and best practices to date.

Founding Engineer and Product Lead

Old Dogs, New Tricks – A pragmatic guide for modern data movement platforms

In today's impatience economy, milliseconds matter, and the demand for instantaneous insights is paramount. Traditional tools and applications no longer suffice; instead, real-time analytics have become indispensable for driving decisions, entering new markets, and enhancing customer experiences. From IT services to retail, FinTech to AdTech, every industry leverages real-time analytics across a myriad of use cases. This presentation delves into the top six applications of real-time analytics, exploring their impact on various sectors such as Information Technology &amp; Services, Retail &amp; Shopping, FinTech, and more. Through case studies from industry leaders like Cisco Webex, Uber, and LinkedIn, we uncover how organizations are harnessing Apache Pinot, StarTree, and other cutting-edge technologies to achieve subsecond latencies, enhanced observability, and even fortify fraud detection mechanisms. This presentation will introduce the impatience economy and the most common use cases requiring real-time analytics capabilities.

VP of Solutions Engineering and Enablement

Unveiling the Impatience Economy: Real-Time Analytics & The Modern Data Landscape

Sponsored by StarTree and Confluent. On the rooftop pool deck of the Signia by Hilton, we’ll celebrate the triumph of real-time analytics over near-real-time analytics. Bring your badge to the 4th floor for food, drinks, music (our DJ just toured with Pitbull), and a party you''''ll never forget. You''''ll also get the chance to win some prizes, including a new PlayStation®5!

Summit Party, the Pool Deck

Sanjeev Mohan is an established thought leader in the areas of cloud, modern data architectures, analytics, and AI. He researches and advises on changing trends and technologies and is the author of Data Product for Dummies. Until recently, he was a Gartner vice president known for his prolific and detailed research, while directing the research direction for data and analytics. He has been a principal at SanjMo for over two years where he provides technical advisory to elevate category and brand awareness. He has helped several clients in areas like data governance, generative AI, DataOps, data products, and observability. He regularly presents on topics pertaining to end-to-end data pipelines and helps businesses maximize their data assets.

Principal

Understanding the Taxonomy of the Evolving Real-Time Analytics Landscape

Jay Kreps, Founder and CEO of Confluent, will present his vision of the unification of the operational and analytical worlds with data streams.

Streams Forever: Unifying the Operational and Analytical Worlds

Moloco has been leaning heavily on batch pipelines, powered by Airflow, Dataflow, and BigQuery on GCP to run hourly or daily batch pipelines. This session goes through the story of how Moloco introduced streaming, based on the use cases the real-time platform covers, and different phases through which it went through to build its real-time data platform.

How We Used Pinot to Power Adtech Analytics Dashboards

Real-time OLAP databases such as Apache Pinot excel at accommodating on-the-fly analytical queries, while streaming databases like KsqlDB and RisingWave are designed to perform incremental computation over streaming data. Are these two types of systems friends or foes? Can we achieve the best of both worlds? In this presentation, I'll delve into the distinctions between stream processing and OLAP and why blending their strengths isn't a straightforward task. We'll also explore the inherent design differences between streaming databases and real-time OLAP databases, and showcase how Apache Pinot and RisingWave can work together in production settings.

Founder & CEO

Bridging Stream Processing & Real-time OLAP: Can We Have It All?

Query-time JOINs are a new capability leveraged by many user-facing real-time analytics nowadays. With Apache Pinot's new Multi-Stage Query Engine, we are now offering users a wider range of possible ways to gain insights from their data. In this talk we will delve into the world of JOINs, discuss various optimizations we added in Apache Pinot for running fast and efficient JOINs, and highlight the best use cases for query-time JOINs.

Multi-Stage Joins—Industry's First Multi-Stage Approach to Expand Use Cases for Apache Pinot

In this talk, we will share the journey of OLAP at Uber and our consolidation over the years from multiple technologies including Elasticsearch, AresDB, MemSQL, Clickhouse into a single platform built on top of Apache Pinot. We will also share our learnings around open-source technology adoption as well as the development/operation strategies.

Principal Engineer

Evolution of OLAP at Uber

In the digital-first world, the efficiency and performance of APIs play a pivotal role in ensuring seamless interactions between different software applications. This is especially crucial in the on-demand delivery sector, where DoorDash has positioned itself as a market leader. The Developer Insights team at DoorDash has been instrumental in providing invaluable analytics and insights to DoorDash's API partners, ensuring that their interactions with the DoorDash Drive delivery platform are optimized for performance and reliability. In this session, we will delve into an in-depth case study of how the Developer Insights team utilizes Apache Pinot for real-time analytics and StarTree ThirdEye for anomaly detection to serve DoorDash's API partners, including a major player, Mx. With an impressive array of 12 alerts, each capitalizing on the query enumeration feature, the team successfully supports over 550 Mx partners, ensuring they are immediately notified of any anomalies.

Product Lead

Revolutionizing API Partnerships: Real-Time Anomaly Detection at DoorDash

The migration of data-intensive analytics applications to cloud-native environments promises enhanced scalability and flexibility but introduces complex cost models that pose new challenges to traditional optimization strategies. While on-premises setups focused on speed, cloud deployments require a more nuanced approach, factoring in cloud storage operations costs, which can escalate rapidly in real-world scenarios. In a presentation, Hope Wang and Bin Fan will analyze these challenges through a case study on Presto's large deployment. They will show their findings of unexpected cost implications with standard I/O optimizations like table scans, filters, and broadcast joins when implemented in cloud environments. They will also highlight the need for a paradigm shift in optimizing data-intensive applications for the cloud and advocate for developing new I/O strategies, balancing performance and costs while tailored to cloud ecosystems' unique demands. By attending this session, you will: • Understand the complexities and cost challenges of optimizing data-intensive applications in cloud-native environments. • Gain insights from an in-depth case study on the economic implications of traditional I/O optimizations in cloud deployments. • Learn about the need for a paradigm shift towards developing I/O strategies that are both performance-efficient and cost-effective in the cloud.

VP of Technology

A Case Study in API Cost of Running Presto in the Cloud at Scale

Observability data has become some of the most common types of data being generated and stored within the software industry. At the most basic level we have time series and log data, which are used not just for observability but also in business analytics and customer behavior data mining. Shortly after metrics and logs, trace data also became heavily used to provide deeper insights into applications and distributed systems. In this presentation, we will survey the observability ecosystem and common architectures to ingest, store, query and visualize metrics, logs and traces. We will then cover how we expanded Pinot's feature set to allow it to become an observability storage and analytics engine that can power observability platforms such as Grafana.

Building an Observability Platform Using Apache Pinot

Streaming pipelines have become more and more difficult to build and operationalize as complexity increases. In this talk, you will learn how Snowflake is simplifying streaming architecture to enable users to work with streaming data easily through Snowpipe Streaming and Dynamic Tables. Snowpipe streaming is designed for rowsets with variable arrival frequency. Calling the streaming API prompts low-latency loads of streaming data rows using the Snowflake Ingest SDK and your own managed application code. The session includes a live demo of the following: - Record audio from the laptop's default microphone. - Do speech to text transcription - Pipes the words from the transcript to Snowflake using Snowpipe Streaming. - Create a Dynamic Table that aggregates words and their counts. - Use an Ads table that refers to the aforementioned Dynamic Table to determine what ads to show to the user.

Yell at the Laptop and It Might Decide to Show You the Right Ad: Building a Voice-to-Text Streaming Pipeline

In today's fast-paced world of software development, new changes (both in code and infrastructure) are being released at breakneck pace. At Uber, we roll out ~11,000 changes every week and it's important for us to have a way to quickly be able to identify and resolve issues caused by these changes. A delay in detecting issues can create a number of issues including impacts to: user experience, our ability to facilitate transactions on the platform, company revenue, and the overall trust and confidence of our users. At Uber, we have built a system called &quot;Healthline&quot; to help with our Mean Time To Detect (MTTD) and Mean Time To Resolve (MTTR) issues and to avoid potential outages and large-scale user impacts. Due to our ability to detect the issues in real time, this has become the go-to tool for release managers to observe the impact of canary release and decide whether to proceed further or to rollback. In this talk we will be sharing details on how we are leveraging Apache Pinot™ to achieve this in real time at Uber scale.

Engineering Manager

Real-Time Analytics for Mobile App Crashes

Until 2019, the Data Engineering team primarily built analytical data assets on our data lake to serve the needs of our customers. The cadence of this varied from hours to days for the different pipelines. This inconsistency led to missed business opportunities and a less-than-ideal customer experience. For instance, The customer executive couldn't pitch for a product upsell/cross-sell during a call. Stale marketing emails were sent to the consumers. We realized that our Source of Record (SOR) applications were diverse and there was a need to build an operational layer with real-time data aggregation or master data record for key Atlassian entities such as Person, Company, Entitlements, etc to integrate them into core data-entities for Atlassian. To rectify this, we embarked on a journey to build out data micro-services to integrate data from source systems, external 3P applications such as clearbit, etc to set up a real-time, highly available, and consistent operational store using technologies like AWS SQS, Kinesis, DynamoDB, ElasticSearch and event processing via microservice. We constructed a robust, layer-based framework that was capable of handling multiple events concurrently. It was designed to select the most critical event based on priority or weightage, thereby facilitating the construction of a single, comprehensive record. The operational layer then feeds into our analytical data lake. It not only catered to our analytical requirements but also addressed the operational needs of the rest of the company, enabling efficient data application services. The presentation will focus on the practical implementation of the design, and development of real-time systems, including the challenges we have solved.

Agenda

Type of Talks:

Topics Covered:

May 7