ERR_SPARK_SQL_LEGACY_UNION_SUPPORT: Your current Spark version doesn’t support UNION clause but only supports UNION ALL, which does not remove duplicates
The Spark SQL cannot be executed because it includes a UNION clause that is not supported by this version of Spark
Remediation
If you’re running a stack recipe with Distinct rows post-filter there are following possibilities:
-
Use stack recipe without distinct postfilter + create a distinct recipe for a result of a stack recipe
-
Apply distinct recipe on each of the stacked datasets and then stack them
-
Create a Spark SQL recipe with a query that uses UNION ALL and then wrap the whole query with SELECT DISTINCT {your column names} from {select with UNION ALL}