Mac怎么部署spark2.4.4
这篇文章主要讲解了“Mac怎么部署spark2.4.4”,文中的讲解内容简单清晰,易于学习与理解,下面请大家跟着小编的思路慢慢深入,一起来研究和学习“Mac怎么部署spark2.4.4”吧!
创新互联坚持“要么做到,要么别承诺”的工作理念,服务领域包括:成都网站设计、成都网站制作、企业官网、英文网站、手机端网站、网站推广等服务,满足客户于互联网时代的清江浦网站设计、移动媒体设计的需求,帮助企业找到有效的互联网解决方案。努力成为您成熟可靠的网络建设合作伙伴!
环境信息
操作系统:macOS Mojave 10.14.6
JDK:1.8.0_211 (安装位置:/Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home)
前提条件
请参考《Mac部署hadoop3(伪分布式)》一文,在Mac上事先部署好hadoop3
部署步骤
安装scala:
brew install scala
配置scala的环境变量,打开文件~/.bash_profile,增加以下配置内容:
export SCALA_HOME=/usr/local/Cellar/scala/2.13.0 export PATH=$PATH:$SCALA_HOME/bin
执行命令source ~/.bash_profile,再验证scala:
base) zhaoqindeMBP:~ zhaoqin$ scala -version Scala code runner version 2.13.0 -- Copyright 2002-2019, LAMP/EPFL and Lightbend, Inc.
下载spark,地址是:http://spark.apache.org/downloads.html ,如下图红框:
将下载的文件在/usr/local/目录下解压,并将文件夹名字从spark-2.4.4-bin-hadoop2.7改为spark
配置spark的环境变量,打开文件~/.bash_profile,增加以下配置内容:
export SPARK_HOME=/usr/local/spark export PATH=$PATH:$SPARK_HOME/bin
执行命令source ~/.bash_profile使配置生效;
打开文件spark/conf/spark-env.sh,在尾部增加以下三行:
export SCALA_HOME=/usr/local/Cellar/scala/2.13.0 export SPARK_MASTER_IP=localhost export SPARK_WORKER_MEMORY=2G
确保hdfs和yarn已经启动,然后执行命令spark-shell,即可启动spark服务:
To update your account to use zsh, please run `chsh -s /bin/zsh`. For more details, please visit https://support.apple.com/kb/HT208050. (base) zhaoqindeMBP:~ zhaoqin$ spark-shell 19/10/27 13:33:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://zhaoqindembp:4040 Spark context available as 'sc' (master = local[*], app id = local-1572154437623). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.4 /_/ Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_211) Type in expressions to have them evaluated. Type :help for more information. scala>
感谢各位的阅读,以上就是“Mac怎么部署spark2.4.4”的内容了,经过本文的学习后,相信大家对Mac怎么部署spark2.4.4这一问题有了更深刻的体会,具体使用情况还需要大家实践验证。这里是创新互联,小编将为大家推送更多相关知识点的文章,欢迎关注!
网站栏目:Mac怎么部署spark2.4.4
标题来源:http://scyanting.com/article/jieecd.html