org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063.

github.com | 7 months ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    [SPARK-13758][Streaming][core]enhance exception message to avoid misleading by mwws · Pull Request #11595 · apache/spark · GitHub

    github.com | 9 months ago
    org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063.
  2. 0

    Streaming df kinesis updated by zsxwing · Pull Request #33 · tdas/spark · GitHub

    github.com | 7 months ago
    org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063.
  3. 0

    How to join dstream and JDBCRDD with checkpointing enabled?

    Stack Overflow | 4 months ago | saurabh dubey
    org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.spark.SparkException

      RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063.

      at org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$sc()
    2. Spark
      RDD.union
      1. org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$sc(RDD.scala:87)
      2. org.apache.spark.rdd.RDD.withScope(RDD.scala:352)
      3. org.apache.spark.rdd.RDD.union(RDD.scala:565)
      3 frames
    3. Spark Project Streaming
      DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply
      1. org.apache.spark.streaming.Repo$$anonfun$createContext$1.apply(Repo.scala:23)
      2. org.apache.spark.streaming.Repo$$anonfun$createContext$1.apply(Repo.scala:19)
      3. org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627)
      3 frames