Apache Spark is one of the most used distributed engines to deal with large amounts of data. Multiple tools can be used to run Spark: Spark Standalone, Apache Hadoop Yarn, Apache Mesos, and Kubernetes. In this hands-on article, we’ll show how to use spark-k8s-operator to run Spark on Kubernetes. Prerequisites Before you start, make sure…
In this article, we will explore how leveraging Predicate Pushdown can enhance the performance of your Spark SQL queries, providing insights into a powerful optimization technique for efficient data processing. We’ll address the questions: What is Predicate Pushdown ? Why should you use it ? And how to implement it ? What is Predicate Pushdown?…
In Apache Spark, it’s well-known that using User-Defined Functions (UDFs), especially with PySpark, can aggressively compromise your application’s performance. In this article, we’ll explore why and how UDFs can impact performance. Let’s dive into the intricacies of Apache Spark UDF impacts on performance. What are User-Defined Functions (UDFs)? Before we get started, let’s briefly review what…
Docker is a powerful containerization technology that allows you to package and distribute applications along with their dependencies in a consistent and portable way. One of the key components of creating a Docker image is the Dockerfile, which is a script that contains instructions for building the image. In this hands-on article, we will explore…
When working with Apache Spark, it’s crucial to understand the concepts of logical and physical plans, as they play a pivotal role in the execution of your data processing tasks. In this blog post, we will break down these two fundamental aspects of Spark’s query optimization process. You can refer to this blog post Spark…