Ошибка искробезопасности - ClassDefNotFound: SparkConf

Question

После установки и создания Apache Spark (хотя и с несколькими предупреждениями) компиляция нашего приложения Spark (с использованием пакета «sbt») успешно завершена. Однако при попытке запустить наше приложение с использованием сценария spark-submit возникает ошибка времени выполнения, указывающая, что определение класса SparkConf не найдено. Файл SparkConf.scala присутствует в нашей системе, но кажется, что он не построен правильно. Любые идеи о том, как это решить?

user@compname:~/Documents/TestApp$ /opt/Spark/spark-1.4.0/bin/spark-submit --master local[4] --jars /opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar target/scala-2.11/sparkesingest_2.11-1.0.0.jar ~/Desktop/CSV/data.csv es-index localhost 
Warning: Local jar /opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar does not exist, skipping. 
log4j:WARN No appenders could be found for logger (App). 
log4j:WARN Please initialize the log4j system properly. 
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
15/07/01 13:56:58 INFO SparkContext: Running Spark version 1.4.0 
15/07/01 13:56:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
15/07/01 13:56:59 WARN Utils: Your hostname, compname resolves to a loopback address: 127.0.1.1; using [IP ADDRESS] instead (on interface eth0) 
15/07/01 13:56:59 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 
15/07/01 13:56:59 INFO SecurityManager: Changing view acls to: user 
15/07/01 13:56:59 INFO SecurityManager: Changing modify acls to: user 
15/07/01 13:56:59 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); users with modify permissions: Set(user) 
15/07/01 13:56:59 INFO Slf4jLogger: Slf4jLogger started 
15/07/01 13:56:59 INFO Remoting: Starting remoting 
15/07/01 13:56:59 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@[IP ADDRESS]] 
15/07/01 13:56:59 INFO Utils: Successfully started service 'sparkDriver' on port 34276. 
15/07/01 13:56:59 INFO SparkEnv: Registering MapOutputTracker 
15/07/01 13:56:59 INFO SparkEnv: Registering BlockManagerMaster 
15/07/01 13:56:59 INFO DiskBlockManager: Created local directory at /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/blockmgr-8d273f32-589e-4f55-98a2-cf0322a05d45 
15/07/01 13:56:59 INFO MemoryStore: MemoryStore started with capacity 265.4 MB 
15/07/01 13:56:59 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/httpd-f4c3c67a-d058-4aba-bd65-5352feb5f12e 
15/07/01 13:56:59 INFO HttpServer: Starting HTTP Server 
15/07/01 13:56:59 INFO Utils: Successfully started service 'HTTP file server' on port 33599. 
15/07/01 13:56:59 INFO SparkEnv: Registering OutputCommitCoordinator 
15/07/01 13:56:59 INFO Utils: Successfully started service 'SparkUI' on port 4040. 
15/07/01 13:56:59 INFO SparkUI: Started SparkUI at http://[IP ADDRESS]:4040 
15/07/01 13:57:00 ERROR SparkContext: Jar not found at file:/opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar 
15/07/01 13:57:00 INFO SparkContext: Added JAR file:/home/user/Documents/TestApp/target/scala-2.11/sparkesingest_2.11-1.0.0.jar at http://[IP ADDRESS]:33599/jars/sparkesingest_2.11-1.0.0.jar with timestamp 1435784220028 
15/07/01 13:57:00 INFO Executor: Starting executor ID driver on host localhost 
15/07/01 13:57:00 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44746. 
15/07/01 13:57:00 INFO NettyBlockTransferService: Server created on 44746 
15/07/01 13:57:00 INFO BlockManagerMaster: Trying to register BlockManager 
15/07/01 13:57:00 INFO BlockManagerMasterEndpoint: Registering block manager localhost:44746 with 265.4 MB RAM, BlockManagerId(driver, localhost, 44746) 
15/07/01 13:57:00 INFO BlockManagerMaster: Registered BlockManager 
15/07/01 13:57:00 INFO MemoryStore: ensureFreeSpace(143840) called with curMem=0, maxMem=278302556 
15/07/01 13:57:00 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 140.5 KB, free 265.3 MB) 
15/07/01 13:57:00 INFO MemoryStore: ensureFreeSpace(12635) called with curMem=143840, maxMem=278302556 
15/07/01 13:57:00 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 12.3 KB, free 265.3 MB) 
15/07/01 13:57:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:44746 (size: 12.3 KB, free: 265.4 MB) 
15/07/01 13:57:00 INFO SparkContext: Created broadcast 0 from textFile at Ingest.scala:159 
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf 
    at org.elasticsearch.spark.rdd.CompatUtils.<clinit>(CompatUtils.java:20) 
    at java.lang.Class.forName0(Native Method) 
    at java.lang.Class.forName(Class.java:274) 
    at org.elasticsearch.hadoop.util.ObjectUtils.loadClass(ObjectUtils.java:71) 
    at org.elasticsearch.spark.package$.<init>(package.scala:14) 
    at org.elasticsearch.spark.package$.<clinit>(package.scala) 
    at build.Ingest$.main(Ingest.scala:176) 
    at build.Ingest.main(Ingest.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) 
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) 
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) 
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) 
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358) 
    ... 17 more 
15/07/01 13:57:00 INFO SparkContext: Invoking stop() from shutdown hook 
15/07/01 13:57:00 INFO SparkUI: Stopped Spark web UI at http://[IP ADDRESS]:4040 
15/07/01 13:57:00 INFO DAGScheduler: Stopping DAGScheduler 
15/07/01 13:57:00 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 
15/07/01 13:57:00 INFO Utils: path = /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/blockmgr-8d273f32-589e-4f55-98a2-cf0322a05d45, already present as root for deletion. 
15/07/01 13:57:00 INFO MemoryStore: MemoryStore cleared 
15/07/01 13:57:00 INFO BlockManager: BlockManager stopped 
15/07/01 13:57:01 INFO BlockManagerMaster: BlockManagerMaster stopped 
15/07/01 13:57:01 INFO SparkContext: Successfully stopped SparkContext 
15/07/01 13:57:01 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 
15/07/01 13:57:01 INFO Utils: Shutdown hook called 
15/07/01 13:57:01 INFO Utils: Deleting directory /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd 

Вот файл build.sbt:

scalaVersion := "2.11.6" 

name := "SparkEsIngest" 

version := "1.0.0" 

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "1.4.0" % "provided", 
    "org.apache.spark" %% "spark-streaming" % "1.4.0" % "provided", 
    "org.apache.spark" %% "spark-sql" % "1.4.0" % "provided", 
    "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2" exclude("org.spark-project.akka", "akka-remote_2.10") exclude("org.spark-project.akka", "akka-slf4j_2.10") exclude("org.json4s", "json4s-ast_2.10") exclude("org.apache.spark", "spark-catalyst_2.10") exclude("com.twitter", "chill_2.10") exclude("org.apache.spark", "spark-sql_2.10") exclude("org.json4s", "json4s-jackson_2.10") exclude("org.json4s", "json4s-core_2.10") exclude("org.apache.spark", "spark-core_2.10") 
) 

if (System.getenv("QUERY_ES_RESOURCE") != null) { 
    println("[info] Using lib/es-hadoop-build-snapshot/ unmanagedBase dir") 
    unmanagedBase <<= baseDirectory { base => base/"lib/es-hadoop-build-snapshot" } 
} else { 
    println("[info] Using lib/ unmanagedBase dir") 
    unmanagedBase <<= baseDirectory { base => base/"lib" } 
} 

resolvers += "conjars.org" at "http://conjars.org/repo" 

resolvers += "clojars" at "https://clojars.org/repo" 

Answer 1

Является ли искра JAR внутри JAR вы представившего? кажется, что вы говорите sbt, что вы предоставляете банку, но я не вижу «неуправляемыхJars в компиляции + = файл (...)» в вас sbt, если вы считаете, что банка находится в машине i предположил бы, что вы этого не сделали бы, так как этот коврик ведет к таким проблемам.

Попробуйте unrar, вы JAR и посмотрите, есть ли искра JAR, если не используется sbt-assembly или другой инструмент, который вы выбрали.

Answer 2

Не используйте область provided, поскольку она позволяет использовать зависимости только во время компиляции, а не во время запуска кода. Измененный код выглядит следующим образом.

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "1.4.0", 
    "org.apache.spark" %% "spark-streaming" % "1.4.0", 
    "org.apache.spark" %% "spark-sql" % "1.4.0", 
    "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2" exclude("org.spark-project.akka", "akka-remote_2.10") exclude("org.spark-project.akka", "akka-slf4j_2.10") exclude("org.json4s", "json4s-ast_2.10") exclude("org.apache.spark", "spark-catalyst_2.10") exclude("com.twitter", "chill_2.10") exclude("org.apache.spark", "spark-sql_2.10") exclude("org.json4s", "json4s-jackson_2.10") exclude("org.json4s", "json4s-core_2.10") exclude("org.apache.spark", "spark-core_2.10") 
)