Podevcast

Apache Beam with Kenneth Knowles and Pablo Estrada

Google Cloud Platform Podcast

Episode | Podcast

Date: Wed, 06 Apr 2022 16:30:00 +0000

On the podcast this week, your hosts <a href="https://twitter.com/stephr_wong">Stephanie Wong</a> and <a href="https://twitter.com/markmirch">Mark Mirchandani</a> talk about the data processing tool Apache Beam with guests <a href="https://twitter.com/polecito">Pablo Estrada</a> and <a href="https://twitter.com/KennKnowles">Kenneth Knowles</a>. Kenn starts us off with an overview of how Apache Beam began and how Cloud Dataflow was involved. The unique batch and stream method and emphasis on correctness garnered support from developers early on and continues to attract users. Pablo helps us understand why Beam is a better option for certain projects looking to process large amounts of data. Our guests describe how Beam may be a better fit than microservices that could become obsolete as company needs change. Next, we step back and take a look at why batch and stream is the gold standard of data processing because of its balance between low latency and ease of “being done” with data collection. Beam’s focus on the correctness of data and correctness in processing that data is a core component. With good data, processing becomes easier, more reliable, and cheaper. Kenn gives examples of how things can go wrong with bad data processing. Beam strives for the perfect combination of low latency, correct data, and affordability. Users can choose where to run Beam pipelines, from other Apache software offerings to Dataflow, which means excellent flexibility. Our guests talk about the pros and cons of some of these options and we hear examples of how companies are using Beam along with supporting software to solve data processing challenges. To get started with Beam, check out Beam College or attend Beam Summit 2022. <h5>Kenneth Knowles</h5> <a href="https://twitter.com/KennKnowles">Kenn Knowles</a> is chair of the Apache Beam Project Management Committee. Kenn has been working on Google Cloud Dataflow—Google’s Beam backend—since 2014. Kenn holds a PhD in programming languages from the University of California, Santa Cruz. <h5>Pablo Estrada</h5> <a href="https://twitter.com/polecito">Pablo</a> is a Software Engineer at Google, and a management committee member for Apache Beam. Pablo is big into working on an open source project, and has worked all across the Apache Beam stack. <h5>Cool things of the week</h5> <ul> <li style="font-weight: 400;">Under the sea: Building the world’s fiber optic internet <a href="https://www.youtube.com/watch?v=N0ng8R0_Tis">video</a></li> <li style="display: inline;"> <ul> <li style="font-weight: 400;">Discovering Data Centers <a href="https://www.youtube.com/watch?v=2R-UVdw6thI&list=PLIivdWyY5sqI7lzvVHfp4zbwp3Xaub2jm"> videos</a></li> </ul> </li> <li style="font-weight: 400;">Google Data Cloud Summit <a href="https://cloudonair.withgoogle.com/events/summit-data-cloud-2022"> site</a></li> <li style="font-weight: 400;">It’s official—Google Distributed Cloud Edge is generally available <a href="https://cloud.google.com/blog/products/infrastructure-modernization/google-distributed-cloud-edge-is-ga"> blog</a></li> <li style="display: inline;"> <ul> <li style="font-weight: 400;">GCP Podcast Episode 228: Fastly with Tyler McMullen <a href="https://www.gcppodcast.com/post/episode-228-fastly-with-tyler-mcmullen/"> podcast</a></li> </ul> </li> <li style="font-weight: 400;">Save big by temporarily suspending unneeded Compute Engine VMs—now GA <a href="https://cloud.google.com/blog/products/compute/save-by-suspending-vms-on-google-compute-engine"> blog</a></li> </ul> <h5>Interview</h5> <ul> <li style="font-weight: 400;">Apache Beam <a href="https://beam.apache.org/">site</a></li> <li style="font-weight: 400;">Apache Beam Documentation <a href="https://beam.apache.org/documentation/">site</a></li> <li style="font-weight: 400;">Dataflow <a href="https://cloud.google.com/dataflow">site</a></li> <li style="font-weight: 400;">Apache Flink <a href="https://flink.apache.org/">site</a></li> <li style="font-weight: 400;">Apache Spark <a href="https://spark.apache.org/">site</a></li> <li style="font-weight: 400;">Apache Samza <a href="https://samza.apache.org/">site</a></li> <li style="font-weight: 400;">Apache Nemo <a href="https://nemo.apache.org/">site</a></li> <li style="font-weight: 400;">Spanner <a href="https://cloud.google.com/spanner">site</a></li> <li style="font-weight: 400;">BigQuery <a href="https://cloud.google.com/bigquery">site</a></li> <li style="font-weight: 400;">Beam College <a href="https://beamcollege.dev/">site</a></li> <li style="font-weight: 400;">Beam College on Github <a href="https://github.com/griscz/beam-college/blob/main/day2/B1_Beam_College_Advanced_Windows_and_Triggers_a_practical_guide_v0_9_0.ipynb"> site</a></li> <li style="font-weight: 400;">Beam Developer Mailing List email</li> <li style="font-weight: 400;">Beam User Mailing List email</li> <li style="font-weight: 400;">Beam Summit <a href="https://2022.beamsummit.org/">site</a></li> </ul> <h5>What’s something cool you’re working on?</h5> Mark is working on a new Apache Beam video series <a href="https://www.youtube.com/watch?v=65lmwL7rSy4&list=PLIivdWyY5sqIEiHGunZXg_yoS7unlHNJt"> Getting Started Wtih Apache Beam</a> <h5>Hosts</h5> Stephanie Wong and Mark Mirchandani