Spark with Scala Course - #13 Partitions, AST, Logical Plan & Optimizations

Apache Spark with Scala Course: From Beginner to Pro
Spark Rockstar Academy: https://academy.philipp-brunenberg.de
Spark Rockstar Newsletter: https://newsletter.philipp-brunenberg.de
Entire playlist: https://www.youtube.com/playlist?listPLeEh_6coH9EpIwsiNiDCW0V-UjNZElF0d
Next video: https://youtu.be/7YZpiaH9eCE
In this video, we will explore the concepts of Spark SQL which allow the parallel execution of workload on a cluster. The level of parallelism stems from a concept called partitioning, while the query planning and optimization is enabled by a translation from the user program in an abstract logical plan.
00:00 Intro & Outline
00:12 Partitioning
03:32 AST, Logical Plan & Optimizations
06:25 Exploring the logical plan in our IDE
10:05 Lazy evaluation
11:48 Summary & Conclusion