ERR_SPARK_FAILED_YARN_KILLED_MEMORY: Spark failure: killed by YARN (excessive memory usage)
Description
This error can happen when running any Spark-enabled recipe, when Spark is running in “YARN” deployment mode.
This error indicates that YARN (the resource manager) has forcefully killed the Spark components, because they ran above their allocated memory allocation.
When a Spark application starts on YARN, it tells YARN how much memory it will use at maximum. YARN accordingly reserves this amount of memory. If, during runtime, the memory usage (per container) goes above this limit, YARN kills the process for breaching its promise.
Remediation
When a Spark application runs on YARN, it requests YARN containers with an amount of memory computed as:
spark.executor.memory
+
spark.yarn.executor.memoryOverhead
spark.executor.memory
is the amount of Java memory (Xmx) that Spark executors will get. However, Java processes always consume a bit more memory, which is accounted for by
spark.yarn.executor.memoryOverhead
By default, the memory overhead is 10% of executor memory (with a minimum of 384 MB). This value is often not enough.
The remediation is to increase the value of
spark.yarn.executor.memoryOverhead
Spark setting. For more information about how to set Spark settings, please see
Spark configurations
. Note that your administrator may need to perform this change.
Beware: unlike
spark.executor.memory
where values like
3g
are permitted, the value for
spark.yarn.executor.memoryOverhead
must always be an integer, in megabytes.
We generally recommend setting a value between
500
and
1000
.