2023年12月15日发(作者：)

Spark示例——SparkPi

在运行中出现ClassNotFound异常的问题解决

前言：按照《Spark实战高手之路-第1章》的前四节，搭建完Spark集群及IDEA集成环境后，最后一步是用IDEA集成环境运行SparkPi例子。可就在这最后一步，让我花了三天时间才最终完成。所以，这里详细介绍解决方法，让接下来以《 Spark实战高手之路》入门的后来者少走些弯路。 文中将大量引用网上的资源，在可能的情况下我会尽量标明出处，但不到之处还请原著作者见谅。

1.在《Spark实战高手之路-第1章（4）》的最后，说要以本地模式过行，则在 Edit

Configurations->Program arguments中输入“local”就不会报错了，但事实是：加之前与加之后都报错，且错误相同。此次报错内容总共只有6行，主要内容是“A master URL must be „...”,大致意思是找不到master 的URL 路径。

解决方法：

找不到URL路径，要在代码中添加ter("spark://192.168.18.211:7077").自行设置URL路径，其中，

192.168.18.211 是我的Master电脑的IP地址，不同的集群根据自己的情况而定。这个方法在《Spark实战高手之路-第1章（5）》的最后也提到过，可能原文的本意是仅本地运行时填“local”,当要在集群上运行时要添加master的URL路径，但好像在本地运行不是很happy.好在重点不在本地运行，而在集群，所以就不管“local”了。下面看一下这一行代码集体插入的位置和方法：

原代码：

package es

import

import ._

/** Computes an approximation to pi */

object SparkPi {

def main(args: Array[String]) {

val conf = new SparkConf().setAppName("SparkPi")

val spark = new SparkContext(conf)

val slices = if ( > 0) args(0).toInt else 2

val n = 100000 * slices

val count = elize(1 to n, slices).map { i =>

val x = random * 2 - 1

val y = random * 2 - 1

if (x*x + y*y < 1) 1 else 0

}.reduce(_ + _)

println("Pi is roughly " + 4.0 * count / n)

()

}

}

添加后：

package es

import

import ._

/** Computes an approximation to pi */

object SparkPi {

def main(args: Array[String]) {

val conf = new SparkConf().setAppName("SparkPi")

ter("spark://192.168.18.211:7077")

val spark = new SparkContext(conf)

val slices = if ( > 0) args(0).toInt else 2

val n = 100000 * slices

val count = elize(1 to n, slices).map { i =>

val x = random * 2 - 1

val y = random * 2 - 1

if (x*x + y*y < 1) 1 else 0

}.reduce(_ + _)

println("Pi is roughly " + 4.0 * count / n)

()

}

}

到此，第一种异常（6行）就解决了。

2.在解决完第一种异常后运行，发现会出现新的异常，而且这次异常所用的行数远多于6行。主要内容为”otFoundException:”,意思应该是没有将jar包提交到spark的worker上面 ，导致运行的worker找不到被调用的类。

解决方法：

将要运行的程序达成jar包，然后调用JavaSparkContext的addJar方法将该jar包提交到spark集群中，然后spark的master会将该jar包分发到各个worker上面。

（本段内容参考了

/?mod=viewthread&tid=322668

网站的内容。）

接下来我就偿试打jar包，再提交。先说一下提交的方法：

原代码：

package es

import

import ._

/** Computes an approximation to pi */

object SparkPi {

def main(args: Array[String]) {

val conf = new SparkConf().setAppName("SparkPi")

ter("spark://192.168.18.211:7077")

val spark = new SparkContext(conf)

val slices = if ( > 0) args(0).toInt else 2

val n = 100000 * slices

val count = elize(1 to n, slices).map { i =>

val x = random * 2 - 1

val y = random * 2 - 1

if (x*x + y*y < 1) 1 else 0

}.reduce(_ + _)

println("Pi is roughly " + 4.0 * count / n)

()

}

}

添加后：

package es

import

import ._

/** Computes an approximation to pi */

object SparkPi {

def main(args: Array[String]) {

val conf = new SparkConf().setAppName("SparkPi")

ter("spark://192.168.18.211:7077")

val spark = new SparkContext(conf)

("/root/")

val slices = if ( > 0) args(0).toInt else 2

val n = 100000 * slices

val count = elize(1 to n, slices).map { i =>

val x = random * 2 - 1

val y = random * 2 - 1

if (x*x + y*y < 1) 1 else 0

}.reduce(_ + _)

println("Pi is roughly " + 4.0 * count / n)

()

}

}

其中 就是我们打的包。

下面聚体说下怎么打包：

起初我是在终端用命令行：jar cvf 自行打包，但是不对。正确的打包方法是这样的：

依次选择“File”–> “Project Structure” –> “Artifact”，选择“+”–>

“Jar” –> “From Modules with dependencies”，选择main函数，并在弹出框中选择输出jar位置，并选择“OK”。

最后依次选择“Build”–> “Build Artifact”编译生成jar包。具体如下图所示。

转载自董的博客 ：/framework-on-yarn/apache-spark-intellij-idea/

这里打成的包还比较大，我打了三次，都是一百二三十兆。包打成后，可以将.jar文件移到目录较浅的地方，方便代码的编写。

3.总结

要使Spark自带的示例能够在集群上运行，需要增加两行代码，一行用来指明master路径：ter("spark://192.168.18.211:7077")，另一行用来把打好的jar 包发到集群上：("/root/")。然后注意一下打包的方法，和生成包的大小。

最后，祝你成功！！

附：成功后的运行结果：

/usr/lib/java/jdk1.7.0_67/bin/java -=7535

-=/usr/local/idea/idea-IC-135.1289/bin

-ng=UTF-8 -classpath

/usr/lib/java/jdk1.7.0_67/jre/lib/:/usr/lib/java/jdk1.7.0_67/jre/lib/:/usr/lib/java/jdk1.7.0_67/jre/lib/:/usr/lib/java/jdk1.7.0_67/jre/lib/:/usr/lib/java/jdk1.7.0_67/jr

e/lib/:/usr/lib/java/jdk1.7.0_67/jre/lib/:/usr/lib/java/jdk1.7.0_67/jre/lib/:/usr/lib/java/jdk1.7.0_67/jre/lib/:/usr/lib/java/jdk1.7.0_67/jre/lib/:/usr/lib/java/jdk1.7.0_67/jre/lib/:/usr/lib/java/jdk1.7.0_67/jre/lib/:/usr/lib/java/jdk1.7.0_67/jre/lib/ext/:/usr/lib/java/jdk1.7.0_67/jre/lib/ext/:/usr/lib/java/jdk1.7.0_67/jre/lib/ext/:/usr/lib/java/jdk1.7.0_67/jre/lib/ext/:/usr/lib/java/jdk1.7.0_67/jre/lib/ext/:/usr/lib/java/jdk1.7.0_67/jre/lib/ext/sunjce_:/root/IdeaProjects/HelloScala/target/scala-2.10/classes:/root/.sbt/boot/scala-2.10.4/lib/:/usr/local/spark/spark-1.1.0/lib/:/usr/local/idea/idea-IC-135.1289/lib/idea_

n SparkPi

Using Spark's default log4j profile:

org/apache/spark/ties

14/11/05 15:19:47 INFO SecurityManager: Changing view acls to: root,

14/11/05 15:19:47 INFO SecurityManager: Changing modify acls to: root,

14/11/05 15:19:47 INFO SecurityManager: SecurityManager: authentication

disabled; ui acls disabled; users with view permissions: Set(root, );

users with modify permissions: Set(root, )

14/11/05 15:19:47 INFO Slf4jLogger: Slf4jLogger started

14/11/05 15:19:47 INFO Remoting: Starting remoting

14/11/05 15:19:47 INFO Remoting: Remoting started; listening on

addresses :[://sparkDriver@Master:60777]

14/11/05 15:19:47 INFO Remoting: Remoting now listens on addresses:

[://sparkDriver@Master:60777]

14/11/05 15:19:47 INFO Utils: Successfully started service 'sparkDriver'

on port 60777.

14/11/05 15:19:47 INFO SparkEnv: Registering MapOutputTracker

14/11/05 15:19:47 INFO SparkEnv: Registering BlockManagerMaster

14/11/05 15:19:47 INFO DiskBlockManager: Created local directory at

/tmp/spark-local-27-2b9c

14/11/05 15:19:47 INFO Utils: Successfully started service 'Connection

manager for block manager' on port 50031.

14/11/05 15:19:47 INFO ConnectionManager: Bound socket to port 50031 with

id = ConnectionManagerId(Master,50031)

14/11/05 15:19:47 INFO MemoryStore: MemoryStore started with capacity

491.5 MB

14/11/05 15:19:47 INFO BlockManagerMaster: Trying to register

BlockManager

14/11/05 15:19:47 INFO BlockManagerMasterActor: Registering block

manager Master:50031 with 491.5 MB RAM

14/11/05 15:19:47 INFO BlockManagerMaster: Registered BlockManager

14/11/05 15:19:47 INFO HttpFileServer: HTTP File server directory is

/tmp/spark-b5af6f99-7900-4de7-8379-89650b305b36

14/11/05 15:19:47 INFO HttpServer: Starting HTTP Server

14/11/05 15:19:48 INFO Utils: Successfully started service 'HTTP file

server' on port 52570.

14/11/05 15:19:53 INFO Utils: Successfully started service 'SparkUI' on

port 4040.

14/11/05 15:19:53 INFO SparkUI: Started SparkUI at Master:4040

14/11/05 15:19:53 INFO AppClient$ClientActor: Connecting to master

spark://192.168.18.

14/11/05 15:19:53 INFO SparkDeploySchedulerBackend: SchedulerBackend is

ready for scheduling beginning after reached minRegisteredResourcesRatio:

0.0

14/11/05 15:19:54 INFO SparkDeploySchedulerBackend: Connected to Spark

cluster with app ID app-24-0008

14/11/05 15:19:54 INFO AppClient$ClientActor: Executor added:

app-24-0008/0 on worker-22-Master-53897

(Master:53897) with 8 cores

14/11/05 15:19:54 INFO SparkDeploySchedulerBackend: Granted executor ID

app-24-0008/0 on hostPort Master:53897 with 8 cores, 512.0

MB RAM

14/11/05 15:19:54 INFO AppClient$ClientActor: Executor added:

app-24-0008/1 on worker-23-Slave1-49080

(Slave1:49080) with 2 cores

14/11/05 15:19:54 INFO SparkDeploySchedulerBackend: Granted executor ID

app-24-0008/1 on hostPort Slave1:49080 with 2 cores, 512.0

MB RAM

14/11/05 15:19:54 INFO AppClient$ClientActor: Executor added:

app-24-0008/2 on worker-23-Slave2-46320

(Slave2:46320) with 2 cores

14/11/05 15:19:54 INFO SparkDeploySchedulerBackend: Granted executor ID

app-24-0008/2 on hostPort Slave2:46320 with 2 cores, 512.0

MB RAM

14/11/05 15:19:54 INFO AppClient$ClientActor: Executor updated:

app-24-0008/1 is now RUNNING

14/11/05 15:19:54 INFO AppClient$ClientActor: Executor updated:

app-24-0008/0 is now RUNNING

14/11/05 15:19:54 INFO AppClient$ClientActor: Executor updated:

app-24-0008/2 is now RUNNING

14/11/05 15:19:56 INFO SparkDeploySchedulerBackend: Registered executor:

Actor[://sparkExecutor@Slave1:38597/user/Executor#-463599812]

with ID 1

14/11/05 15:19:56 INFO SparkDeploySchedulerBackend: Registered executor:

Actor[://sparkExecutor@Slave2:34634/user/Executor#-1528704844]

with ID 2

14/11/05 15:19:56 INFO BlockManagerMasterActor: Registering block

manager Slave1:33325 with 265.1 MB RAM

14/11/05 15:19:56 INFO BlockManagerMasterActor: Registering block

manager Slave2:57790 with 265.1 MB RAM

14/11/05 15:19:57 INFO SparkDeploySchedulerBackend: Registered executor:

Actor[://sparkExecutor@Master:51501/user/Executor#454736183]

with ID 0

14/11/05 15:19:57 INFO BlockManagerMasterActor: Registering block

manager Master:51783 with 265.1 MB RAM

14/11/05 15:20:39 INFO SparkContext: Added JAR

/root/IdeaProjects/HelloScala/out/artifacts/helloscala_jar/ at 192.168.18.211:52570/jars/ with timestamp

23

14/11/05 15:20:40 INFO SparkContext: Starting job: reduce at

:27

14/11/05 15:20:40 INFO DAGScheduler: Got job 0 (reduce at

:27) with 2 output partitions (allowLocal=false)

14/11/05 15:20:40 INFO DAGScheduler: Final stage: Stage 0(reduce at

:27)

14/11/05 15:20:40 INFO DAGScheduler: Parents of final stage: List()

14/11/05 15:20:40 INFO DAGScheduler: Missing parents: List()

14/11/05 15:20:40 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[1] at

map at :23), which has no missing parents

14/11/05 15:20:40 WARN SizeEstimator: Failed to check whether

UseCompressedOops is set; assuming yes

14/11/05 15:20:40 INFO MemoryStore: ensureFreeSpace(1680) called with

curMem=0, maxMem=515411804

14/11/05 15:20:40 INFO MemoryStore: Block broadcast_0 stored as values

in memory (estimated size 1680.0 B, free 491.5 MB)

14/11/05 15:20:40 INFO MemoryStore: ensureFreeSpace(1114) called with

curMem=1680, maxMem=515411804

14/11/05 15:20:40 INFO MemoryStore: Block broadcast_0_piece0 stored as

bytes in memory (estimated size 1114.0 B, free 491.5 MB)

14/11/05 15:20:40 INFO BlockManagerInfo: Added broadcast_0_piece0 in

memory on Master:50031 (size: 1114.0 B, free: 491.5 MB)

14/11/05 15:20:40 INFO BlockManagerMaster: Updated info of block

broadcast_0_piece0

14/11/05 15:20:40 INFO DAGScheduler: Submitting 2 missing tasks from

Stage 0 (MappedRDD[1] at map at :23)

14/11/05 15:20:40 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks

14/11/05 15:20:40 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID

0, Slave2, PROCESS_LOCAL, 1209 bytes)

14/11/05 15:20:40 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID

1, Slave1, PROCESS_LOCAL, 1209 bytes)

14/11/05 15:21:01 INFO ConnectionManager: Accepted connection from

[Slave1/192.168.30.194:49874]

14/11/05 15:21:01 INFO SendingConnection: Initiating connection to

[Slave1/192.168.30.194:33325]

14/11/05 15:21:01 INFO SendingConnection: Connected to

[Slave1/192.168.30.194:33325], 1 messages pending

14/11/05 15:21:01 INFO BlockManagerInfo: Added broadcast_0_piece0 in

memory on Slave1:33325 (size: 1114.0 B, free: 265.1 MB)

14/11/05 15:21:01 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID

1) in 21559 ms on Slave1 (1/2)

14/11/05 15:21:03 INFO BlockManagerInfo: Added broadcast_0_piece0 in

memory on Slave2:57790 (size: 1114.0 B, free: 265.1 MB)

14/11/05 15:21:03 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID

0) in 22924 ms on Slave2 (2/2)

14/11/05 15:21:03 INFO DAGScheduler: Stage 0 (reduce at :27)

finished in 22.926 s

14/11/05 15:21:03 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose

tasks have all completed, from pool

14/11/05 15:21:03 INFO SparkContext: Job finished: reduce at

:27, took 23.224325592 s

Pi is roughly 3.14668

14/11/05 15:21:03 INFO SparkUI: Stopped Spark web UI at

Master:4040

14/11/05 15:21:03 INFO DAGScheduler: Stopping DAGScheduler

14/11/05 15:21:03 INFO SparkDeploySchedulerBackend: Shutting down all

executors

14/11/05 15:21:03 INFO SparkDeploySchedulerBackend: Asking each executor

to shut down

14/11/05 15:21:03 INFO ConnectionManager: Removing SendingConnection to

ConnectionManagerId(Slave1,33325)

14/11/05 15:21:03 INFO ConnectionManager: Removing ReceivingConnection

to ConnectionManagerId(Slave1,33325)

14/11/05 15:21:03 ERROR ConnectionManager: Corresponding

SendingConnection to ConnectionManagerId(Slave1,33325) not found

14/11/05 15:21:04 INFO MapOutputTrackerMasterActor:

MapOutputTrackerActor stopped!

14/11/05 15:21:04 INFO ConnectionManager: Selector thread was

interrupted!

14/11/05 15:21:04 INFO ConnectionManager: ConnectionManager stopped

14/11/05 15:21:04 INFO MemoryStore: MemoryStore cleared

14/11/05 15:21:04 INFO BlockManager: BlockManager stopped

14/11/05 15:21:04 INFO BlockManagerMaster: BlockManagerMaster stopped

14/11/05 15:21:04 INFO SparkContext: Successfully stopped SparkContext

14/11/05 15:21:04 INFO RemoteActorRefProvider$RemotingTerminator:

Shutting down remote daemon.

14/11/05 15:21:04 INFO RemoteActorRefProvider$RemotingTerminator:

Remote daemon shut down; proceeding with flushing remote transports.

Process finished with exit code 0