)
容器化搭建可以看前面的搭建hadoop,我的3个节点是在3个容器内搭建的分别为master,slave1,slave2此后我直接说节点一.spark的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp spark-3.1.1-bin-hadoop3.2.tgz master:/root[rootmaster ~]# tar -zxvf spark-3.1.1-bin-hadoop3.2.tgz -C /opt/module/2.配置环境变量(1)添加如下几行[rootmaster module]# vim /etc/profileexportSPARK_HOME/opt/module/spark-3.1.1-bin-hadoop3.2exportPATH$PATH:$SPARK_HOME/bin(2)使环境变量生效[rootmaster module]# source /etc/profile(3)验证是否成功[rootmaster module]# spark-submit --versionWelcome to ____ __ / __/__ ___ _____/ /__ _\\/ _\/ _ / __/ _/ /___/ .__/\_,_/_/ /_/\_\version3.1.1 /_/ Using Scala version2.12.10, Java HotSpot(TM)64-Bit Server VM,1.8.0_371 Branch HEAD Compiled by user ubuntu on2021-02-22T01:33:19Z Revision 1d550c4e90275ab418b9161925049239227f3dc9 Url https://github.com/apache/spark Type--helpformoreinformation.3.配置相关环境[rootmaster hadoop]# cd /opt/module/spark-3.1.1-bin-hadoop3.2/conf/[rootmaster conf]# cp spark-env.sh.template spark-env.sh[rootmaster conf]# cp workers.template workers[rootmaster conf]# vim spark-env.sh #添加如下几行exportJAVA_HOME/opt/module/jdk1.8.0_371exportHADOOP_HOME/opt/module/hadoop-3.2.1exportYARN_CONF_DIR/opt/module/hadoop-3.2.1/etc/hadoopexportHADOOP_CONF_DIR/opt/module/hadoop-3.2.1/etc/hadoop[rootmaster conf]# vim workers #添加3行,删除原来的localhostmaster slave1 slave24.把master节点的传给其他两个节点[rootmaster conf]# scp -r /etc/profile slave1:/etc/profile[rootmaster conf]# scp -r /etc/profile slave2:/etc/profile[rootmaster conf]# scp -r /opt/module/spark-3.1.1-bin-hadoop3.2/ slave1:/opt/module/[rootmaster conf]# scp -r /opt/module/spark-3.1.1-bin-hadoop3.2/ slave2:/opt/module/5.测试[rootmaster spark-3.1.1-bin-hadoop3.2]# ./bin/run-example SparkPi 10Assignments Map()2025-10-2017:58:46,196 INFO scheduler.TaskSetManager: Finished task5.0instage0.0(TID5)in47ms on master(executor driver)(6/10)2025-10-2017:58:46,198 INFO executor.Executor: Running task7.0instage0.0(TID7)2025-10-2017:58:46,211 INFO executor.Executor: Finished task6.0instage0.0(TID6).957bytes result sent to driver2025-10-2017:58:46,213 INFO scheduler.TaskSetManager: Starting task8.0instage0.0(TID8)(master, executor driver, partition8, PROCESS_LOCAL,4578bytes)taskResourceAssignments Map()2025-10-2017:58:46,214 INFO scheduler.TaskSetManager: Finished task6.0instage0.0(TID6)in36ms on master(executor driver)(7/10)2025-10-2017:58:46,217 INFO executor.Executor: Running task8.0instage0.0(TID8)2025-10-2017:58:46,233 INFO executor.Executor: Finished task7.0instage0.0(TID7).957bytes result sent to driver2025-10-2017:58:46,238 INFO scheduler.TaskSetManager: Starting task9.0instage0.0(TID9)(master, executor driver, partition9, PROCESS_LOCAL,4578bytes)taskResourceAssignments Map()2025-10-2017:58:46,239 INFO scheduler.TaskSetManager: Finished task7.0instage0.0(TID7)in44ms on master(executor driver)(8/10)2025-10-2017:58:46,239 INFO executor.Executor: Running task9.0instage0.0(TID9)2025-10-2017:58:46,255 INFO executor.Executor: Finished task8.0instage0.0(TID8).957bytes result sent to driver2025-10-2017:58:46,263 INFO scheduler.TaskSetManager: Finished task8.0instage0.0(TID8)in51ms on master(executor driver)(9/10)2025-10-2017:58:46,273 INFO executor.Executor: Finished task9.0instage0.0(TID9).957bytes result sent to driver2025-10-2017:58:46,274 INFO scheduler.TaskSetManager: Finished task9.0instage0.0(TID9)in37ms on master(executor driver)(10/10)2025-10-2017:58:46,276 INFO scheduler.TaskSchedulerImpl: Removed TaskSet0.0, whose tasks have all completed, from pool2025-10-2017:58:46,285 INFO scheduler.DAGScheduler: ResultStage0(reduce at SparkPi.scala:38)finishedin1.230s2025-10-2017:58:46,292 INFO scheduler.DAGScheduler: Job0is finished. Cancelling potential speculative or zombie tasksforthis job2025-10-2017:58:46,292 INFO scheduler.TaskSchedulerImpl: Killing all running tasksinstage0: Stage finished2025-10-2017:58:46,296 INFO scheduler.DAGScheduler: Job0finished: reduce at SparkPi.scala:38, took1.313123s Pi is roughly3.1379711379711382025-10-2017:58:46,315 INFO server.AbstractConnector: Stopped Spark53cdecf6{HTTP/1.1,(http/1.1)}{0.0.0.0:4040}2025-10-2017:58:46,321 INFO ui.SparkUI: Stopped Spark web UI at http://master:40402025-10-2017:58:46,345 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!2025-10-2017:58:46,390 INFO memory.MemoryStore: MemoryStore cleared2025-10-2017:58:46,391 INFO storage.BlockManager: BlockManager stopped2025-10-2017:58:46,407 INFO storage.BlockManagerMaster: BlockManagerMaster stopped2025-10-2017:58:46,411 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!2025-10-2017:58:46,444 INFO spark.SparkContext: Successfully stopped SparkContext2025-10-2017:58:46,449 INFO util.ShutdownHookManager: Shutdown hook called2025-10-2017:58:46,449 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-46db54ca-ca2c-4114-b9fb-58f0fc3a62b42025-10-2017:58:46,452 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-97095aab-a35e-4bb4-9edb-9177e4b23cb7输出如下说明没问题二.Hbase的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp hbase-2.2.3-bin.tar.gz master:/root[rootmaster ~]# tar -zxvf hbase-2.2.3-bin.tar.gz -C /opt/module/[rootmaster ~]# cd /opt/module/[rootmaster module]# mv hbase-2.2.3 hbase2.配置环境变量[rootmaster ~]# vim /etc/profileexportHBASE_HOME/opt/module/hbaseexportPATH$PATH:$HBASE_HOME/bin[rootmaster module]# source /etc/profile[rootmaster module]# hbase versionSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]HBase2.2.3 Source code repository git://hao-OptiPlex-7050/home/hao/open_source/hbaserevision6a830d87542b766bd3dc4cfdee28655f62de3974 Compiled by hao on2020? 01?10? ???18:27:51 CST Fromsourcewith checksum 097925184b85f6995e20da5462b10f3f3.配置相关环境[rootmaster conf]# vim hbase-env.shexportJAVA_HOME/opt/module/jdk1.8.0_371exportHBASE_MANAGES_ZKfalse[rootmaster conf]# vim hbase-site.xmlconfigurationpropertynamehbase.rootdir/namevaluehdfs://master:9000/hbase/value/propertypropertynamehbase.cluster.distributed/namevaluetrue/value/propertypropertynamehbase.zookeeper.quorum/namevaluemaster,slave1,slave2/value/propertypropertynamehbase.unsafe.stream.capability.enforce/namevaluefalse/value/property/configuration[rootmaster conf]# vim regionserversmaster slave1 slave2[rootmaster conf]# cp /opt/module/hadoop-3.2.1/etc/hadoop/core-site.xml /opt/module/hbase/conf/[rootmaster conf]# cp /opt/module/hadoop-3.2.1/etc/hadoop/hdfs-site.xml /opt/module/hbase/conf/4把master节点的传给其他两个节点跟前面一样这里就不写了之后都是5.启动[rootmaster conf]# start-hbase.shSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]running master, logging to /opt/module/hbase/logs/hbase-root-master-master.out slave2: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-slave2.out slave1: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-slave1.out master: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-master.out[rootmaster conf]# hbase shell # 启动 HBase 交互式命令行工具SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]HBase Shell Usehelpto get list of supported commands. Useexitto quit this interactive shell. For Reference, please visit: http://hbase.apache.org/2.0/book.html#shellVersion2.2.3, r6a830d87542b766bd3dc4cfdee28655f62de3974,2020年 01月10日 星期五18:27:51 CST Took0.0034seconds hbase(main):001:0三.hudi,maven的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp apache-maven-3.6.3-bin.tar.gz master:/root[rootlocalhost ~]# docker cp hudi-0.11.0.src.tgz master:/root[rootmaster ~]# tar -zxvf apache-maven-3.6.3-bin.tar.gz -C /opt/module/[rootmaster ~]# tar -zxvf hudi-0.11.0.src.tgz -C /opt/module/[rootmaster modulel# mv apache-maven-3.6.3/ maven2.配置环境变量[rootmaster ~]# vim /etc/profileexportMAVEN_HOME/opt/module/mavenexportPATH$PATH:$MAVEN_HOME/bin[rootmaster module]# source /etc/profile[rootmaster ~]# mvn -vApache Maven3.6.3(cecedd343002696d0abb50b32b541b8a6ba2883f)Maven home: /opt/module/maven Java version:1.8.0_371, vendor: Oracle Corporation, runtime: /opt/module/jdk1.8.0_371/jre Default locale: en_US, platform encoding: ANSI_X3.4-1968 OS name:linux, version:3.10.0-1160.el7.x86_64, arch:amd64, family:unix3.配置相关环境[rootmaster ~]# cd /opt/module/maven/conf/[rootmaster conf]# vim settings.xml/locallocalRepository/path/to/local/repo/localRepository有需要在这里改仓库位置 /mirrorsmirroridnexus-aliyun/idmirrorOfcentral/mirrorOfnameNexus aliyun/nameurlhttp://maven.aliyun.com/nexus/content/groups/public/url/mirror[rootmaster conf]# cd /opt/module/hudi-0.11.0/[rootmaster hudi-0.11.0]# vim pom.xml/repoidnexus-aliyun/idnamenexus-aliyun/nameurlhttp://maven.aliyun.com/nexus/content/groups/public/url/hadoophadoop.version3.2.1/hadoop.versionhive.version3.1.2/hive.version[rootmaster hudi-0.11.0]# cd hudi-common/src/main/java/org/apache/hudi/common/table/log/block/[rootmaster block]# vim HoodieParquetDataBlock.java/try try(FSDataOutputStream outputStreamnew FSDataOutputStream(baos,satas:null)){4.启动[rootmaster block]# spark-shell \--jars/opt/software/hudi-spark3.1-bundle_2.12-0.12.0.jar\--confspark.serializerorg.apache.spark.serializer.KryoSerializer\--confspark.sql.extensionsorg.apache.spark.sql.hudi.HoodieSparkSessionExtension2025-10-2020:14:29,222 INFO repl.Main: Created Spark session with Hive support Spark context Web UI available at http://master:4040 Spark context available assc(masterlocal[*], appidlocal-1760991268524). Spark session available asspark.Welcome to ____ __ / __/__ ___ _____/ /__ _\\/ _\/ _ / __/ _/ /___/ .__/\_,_/_/ /_/\_\version3.1.1 /_/ Using Scala version2.12.10(Java HotSpot(TM)64-Bit Server VM, Java1.8.0_371)Typeinexpressions to have them evaluated. Type :helpformoreinformation. scala四.flink的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp flink-1.14.0-bin-scala_2.12.tgz master:/root[rootmaster ~]# tar -zxvf flink-1.14.0-bin-scala_2.12.tgz -C /opt/module/[rootmaster ~]# cd /opt/modulerootmaster module]#:mv flink-1.14.0 flink2.配置环境变量[rootmaster ~]# vim /etc/profileexportFLINK_HOME/opt/module/flinkexportPATH$PATH:$FLINK_HOME/binexportHADOOP_CLASSPATH$(hadoop classpath)[rootmaster module]# source /etc/profile[rootmaster ~]# flink --versionSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/flink/lib/log4j-slf4j-impl-2.14.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.apache.logging.slf4j.Log4jLoggerFactory]Version:1.14.0, Commit ID: 460b3863.配置相关环境[rootmaster ~]# cd /opt/module/flink/conf/[rootmaster conf]# vim flink-conf.yamlclassloader.check-leaked-classloader:falsejobmanager.rpc.address: master[rootmaster conf]# vim mastersmaster:8081[rootmaster conf]# vim workersmaster[rootmaster conf]# vim zoo.cfgserver.1master:2888:38884把master节点的传给其他两个节点5.测试[rootmaster conf]# flink run -m yarn-cluster -p 2 -yjm 2G -ytm 2G $FLINK_HOME/examples/batch/WordCount.jar(rather,1)(regard,1)(remember,1)(resolution,1)(respect,1)(rub,1)(sins,1)(sleep,5)(slings,1)(so,1)(soft,1)(spurns,1)(suffer,1)(sweat,1)(take,1)(that,7)(their,1)(there,2)(thought,1)(time,1)(to,15)(under,1)(undiscover,1)(we,4)(weary,1)(when,2)(whips,1)(who,2)(whose,1)(will,1)(wish,1)(would,2)(wrong,1)(you,1)
容器化搭建可以看前面的搭建hadoop,我的3个节点是在3个容器内搭建的分别为master,slave1,slave2此后我直接说节点一.spark的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp spark-3.1.1-bin-hadoop3.2.tgz master:/root[rootmaster ~]# tar -zxvf spark-3.1.1-bin-hadoop3.2.tgz -C /opt/module/2.配置环境变量(1)添加如下几行[rootmaster module]# vim /etc/profileexportSPARK_HOME/opt/module/spark-3.1.1-bin-hadoop3.2exportPATH$PATH:$SPARK_HOME/bin(2)使环境变量生效[rootmaster module]# source /etc/profile(3)验证是否成功[rootmaster module]# spark-submit --versionWelcome to ____ __ / __/__ ___ _____/ /__ _\\/ _\/ _ / __/ _/ /___/ .__/\_,_/_/ /_/\_\version3.1.1 /_/ Using Scala version2.12.10, Java HotSpot(TM)64-Bit Server VM,1.8.0_371 Branch HEAD Compiled by user ubuntu on2021-02-22T01:33:19Z Revision 1d550c4e90275ab418b9161925049239227f3dc9 Url https://github.com/apache/spark Type--helpformoreinformation.3.配置相关环境[rootmaster hadoop]# cd /opt/module/spark-3.1.1-bin-hadoop3.2/conf/[rootmaster conf]# cp spark-env.sh.template spark-env.sh[rootmaster conf]# cp workers.template workers[rootmaster conf]# vim spark-env.sh #添加如下几行exportJAVA_HOME/opt/module/jdk1.8.0_371exportHADOOP_HOME/opt/module/hadoop-3.2.1exportYARN_CONF_DIR/opt/module/hadoop-3.2.1/etc/hadoopexportHADOOP_CONF_DIR/opt/module/hadoop-3.2.1/etc/hadoop[rootmaster conf]# vim workers #添加3行,删除原来的localhostmaster slave1 slave24.把master节点的传给其他两个节点[rootmaster conf]# scp -r /etc/profile slave1:/etc/profile[rootmaster conf]# scp -r /etc/profile slave2:/etc/profile[rootmaster conf]# scp -r /opt/module/spark-3.1.1-bin-hadoop3.2/ slave1:/opt/module/[rootmaster conf]# scp -r /opt/module/spark-3.1.1-bin-hadoop3.2/ slave2:/opt/module/5.测试[rootmaster spark-3.1.1-bin-hadoop3.2]# ./bin/run-example SparkPi 10Assignments Map()2025-10-2017:58:46,196 INFO scheduler.TaskSetManager: Finished task5.0instage0.0(TID5)in47ms on master(executor driver)(6/10)2025-10-2017:58:46,198 INFO executor.Executor: Running task7.0instage0.0(TID7)2025-10-2017:58:46,211 INFO executor.Executor: Finished task6.0instage0.0(TID6).957bytes result sent to driver2025-10-2017:58:46,213 INFO scheduler.TaskSetManager: Starting task8.0instage0.0(TID8)(master, executor driver, partition8, PROCESS_LOCAL,4578bytes)taskResourceAssignments Map()2025-10-2017:58:46,214 INFO scheduler.TaskSetManager: Finished task6.0instage0.0(TID6)in36ms on master(executor driver)(7/10)2025-10-2017:58:46,217 INFO executor.Executor: Running task8.0instage0.0(TID8)2025-10-2017:58:46,233 INFO executor.Executor: Finished task7.0instage0.0(TID7).957bytes result sent to driver2025-10-2017:58:46,238 INFO scheduler.TaskSetManager: Starting task9.0instage0.0(TID9)(master, executor driver, partition9, PROCESS_LOCAL,4578bytes)taskResourceAssignments Map()2025-10-2017:58:46,239 INFO scheduler.TaskSetManager: Finished task7.0instage0.0(TID7)in44ms on master(executor driver)(8/10)2025-10-2017:58:46,239 INFO executor.Executor: Running task9.0instage0.0(TID9)2025-10-2017:58:46,255 INFO executor.Executor: Finished task8.0instage0.0(TID8).957bytes result sent to driver2025-10-2017:58:46,263 INFO scheduler.TaskSetManager: Finished task8.0instage0.0(TID8)in51ms on master(executor driver)(9/10)2025-10-2017:58:46,273 INFO executor.Executor: Finished task9.0instage0.0(TID9).957bytes result sent to driver2025-10-2017:58:46,274 INFO scheduler.TaskSetManager: Finished task9.0instage0.0(TID9)in37ms on master(executor driver)(10/10)2025-10-2017:58:46,276 INFO scheduler.TaskSchedulerImpl: Removed TaskSet0.0, whose tasks have all completed, from pool2025-10-2017:58:46,285 INFO scheduler.DAGScheduler: ResultStage0(reduce at SparkPi.scala:38)finishedin1.230s2025-10-2017:58:46,292 INFO scheduler.DAGScheduler: Job0is finished. Cancelling potential speculative or zombie tasksforthis job2025-10-2017:58:46,292 INFO scheduler.TaskSchedulerImpl: Killing all running tasksinstage0: Stage finished2025-10-2017:58:46,296 INFO scheduler.DAGScheduler: Job0finished: reduce at SparkPi.scala:38, took1.313123s Pi is roughly3.1379711379711382025-10-2017:58:46,315 INFO server.AbstractConnector: Stopped Spark53cdecf6{HTTP/1.1,(http/1.1)}{0.0.0.0:4040}2025-10-2017:58:46,321 INFO ui.SparkUI: Stopped Spark web UI at http://master:40402025-10-2017:58:46,345 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!2025-10-2017:58:46,390 INFO memory.MemoryStore: MemoryStore cleared2025-10-2017:58:46,391 INFO storage.BlockManager: BlockManager stopped2025-10-2017:58:46,407 INFO storage.BlockManagerMaster: BlockManagerMaster stopped2025-10-2017:58:46,411 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!2025-10-2017:58:46,444 INFO spark.SparkContext: Successfully stopped SparkContext2025-10-2017:58:46,449 INFO util.ShutdownHookManager: Shutdown hook called2025-10-2017:58:46,449 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-46db54ca-ca2c-4114-b9fb-58f0fc3a62b42025-10-2017:58:46,452 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-97095aab-a35e-4bb4-9edb-9177e4b23cb7输出如下说明没问题二.Hbase的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp hbase-2.2.3-bin.tar.gz master:/root[rootmaster ~]# tar -zxvf hbase-2.2.3-bin.tar.gz -C /opt/module/[rootmaster ~]# cd /opt/module/[rootmaster module]# mv hbase-2.2.3 hbase2.配置环境变量[rootmaster ~]# vim /etc/profileexportHBASE_HOME/opt/module/hbaseexportPATH$PATH:$HBASE_HOME/bin[rootmaster module]# source /etc/profile[rootmaster module]# hbase versionSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]HBase2.2.3 Source code repository git://hao-OptiPlex-7050/home/hao/open_source/hbaserevision6a830d87542b766bd3dc4cfdee28655f62de3974 Compiled by hao on2020? 01?10? ???18:27:51 CST Fromsourcewith checksum 097925184b85f6995e20da5462b10f3f3.配置相关环境[rootmaster conf]# vim hbase-env.shexportJAVA_HOME/opt/module/jdk1.8.0_371exportHBASE_MANAGES_ZKfalse[rootmaster conf]# vim hbase-site.xmlconfigurationpropertynamehbase.rootdir/namevaluehdfs://master:9000/hbase/value/propertypropertynamehbase.cluster.distributed/namevaluetrue/value/propertypropertynamehbase.zookeeper.quorum/namevaluemaster,slave1,slave2/value/propertypropertynamehbase.unsafe.stream.capability.enforce/namevaluefalse/value/property/configuration[rootmaster conf]# vim regionserversmaster slave1 slave2[rootmaster conf]# cp /opt/module/hadoop-3.2.1/etc/hadoop/core-site.xml /opt/module/hbase/conf/[rootmaster conf]# cp /opt/module/hadoop-3.2.1/etc/hadoop/hdfs-site.xml /opt/module/hbase/conf/4把master节点的传给其他两个节点跟前面一样这里就不写了之后都是5.启动[rootmaster conf]# start-hbase.shSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]running master, logging to /opt/module/hbase/logs/hbase-root-master-master.out slave2: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-slave2.out slave1: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-slave1.out master: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-master.out[rootmaster conf]# hbase shell # 启动 HBase 交互式命令行工具SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]HBase Shell Usehelpto get list of supported commands. Useexitto quit this interactive shell. For Reference, please visit: http://hbase.apache.org/2.0/book.html#shellVersion2.2.3, r6a830d87542b766bd3dc4cfdee28655f62de3974,2020年 01月10日 星期五18:27:51 CST Took0.0034seconds hbase(main):001:0三.hudi,maven的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp apache-maven-3.6.3-bin.tar.gz master:/root[rootlocalhost ~]# docker cp hudi-0.11.0.src.tgz master:/root[rootmaster ~]# tar -zxvf apache-maven-3.6.3-bin.tar.gz -C /opt/module/[rootmaster ~]# tar -zxvf hudi-0.11.0.src.tgz -C /opt/module/[rootmaster modulel# mv apache-maven-3.6.3/ maven2.配置环境变量[rootmaster ~]# vim /etc/profileexportMAVEN_HOME/opt/module/mavenexportPATH$PATH:$MAVEN_HOME/bin[rootmaster module]# source /etc/profile[rootmaster ~]# mvn -vApache Maven3.6.3(cecedd343002696d0abb50b32b541b8a6ba2883f)Maven home: /opt/module/maven Java version:1.8.0_371, vendor: Oracle Corporation, runtime: /opt/module/jdk1.8.0_371/jre Default locale: en_US, platform encoding: ANSI_X3.4-1968 OS name:linux, version:3.10.0-1160.el7.x86_64, arch:amd64, family:unix3.配置相关环境[rootmaster ~]# cd /opt/module/maven/conf/[rootmaster conf]# vim settings.xml/locallocalRepository/path/to/local/repo/localRepository有需要在这里改仓库位置 /mirrorsmirroridnexus-aliyun/idmirrorOfcentral/mirrorOfnameNexus aliyun/nameurlhttp://maven.aliyun.com/nexus/content/groups/public/url/mirror[rootmaster conf]# cd /opt/module/hudi-0.11.0/[rootmaster hudi-0.11.0]# vim pom.xml/repoidnexus-aliyun/idnamenexus-aliyun/nameurlhttp://maven.aliyun.com/nexus/content/groups/public/url/hadoophadoop.version3.2.1/hadoop.versionhive.version3.1.2/hive.version[rootmaster hudi-0.11.0]# cd hudi-common/src/main/java/org/apache/hudi/common/table/log/block/[rootmaster block]# vim HoodieParquetDataBlock.java/try try(FSDataOutputStream outputStreamnew FSDataOutputStream(baos,satas:null)){4.启动[rootmaster block]# spark-shell \--jars/opt/software/hudi-spark3.1-bundle_2.12-0.12.0.jar\--confspark.serializerorg.apache.spark.serializer.KryoSerializer\--confspark.sql.extensionsorg.apache.spark.sql.hudi.HoodieSparkSessionExtension2025-10-2020:14:29,222 INFO repl.Main: Created Spark session with Hive support Spark context Web UI available at http://master:4040 Spark context available assc(masterlocal[*], appidlocal-1760991268524). Spark session available asspark.Welcome to ____ __ / __/__ ___ _____/ /__ _\\/ _\/ _ / __/ _/ /___/ .__/\_,_/_/ /_/\_\version3.1.1 /_/ Using Scala version2.12.10(Java HotSpot(TM)64-Bit Server VM, Java1.8.0_371)Typeinexpressions to have them evaluated. Type :helpformoreinformation. scala四.flink的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp flink-1.14.0-bin-scala_2.12.tgz master:/root[rootmaster ~]# tar -zxvf flink-1.14.0-bin-scala_2.12.tgz -C /opt/module/[rootmaster ~]# cd /opt/modulerootmaster module]#:mv flink-1.14.0 flink2.配置环境变量[rootmaster ~]# vim /etc/profileexportFLINK_HOME/opt/module/flinkexportPATH$PATH:$FLINK_HOME/binexportHADOOP_CLASSPATH$(hadoop classpath)[rootmaster module]# source /etc/profile[rootmaster ~]# flink --versionSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/flink/lib/log4j-slf4j-impl-2.14.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.apache.logging.slf4j.Log4jLoggerFactory]Version:1.14.0, Commit ID: 460b3863.配置相关环境[rootmaster ~]# cd /opt/module/flink/conf/[rootmaster conf]# vim flink-conf.yamlclassloader.check-leaked-classloader:falsejobmanager.rpc.address: master[rootmaster conf]# vim mastersmaster:8081[rootmaster conf]# vim workersmaster[rootmaster conf]# vim zoo.cfgserver.1master:2888:38884把master节点的传给其他两个节点5.测试[rootmaster conf]# flink run -m yarn-cluster -p 2 -yjm 2G -ytm 2G $FLINK_HOME/examples/batch/WordCount.jar(rather,1)(regard,1)(remember,1)(resolution,1)(respect,1)(rub,1)(sins,1)(sleep,5)(slings,1)(so,1)(soft,1)(spurns,1)(suffer,1)(sweat,1)(take,1)(that,7)(their,1)(there,2)(thought,1)(time,1)(to,15)(under,1)(undiscover,1)(we,4)(weary,1)(when,2)(whips,1)(who,2)(whose,1)(will,1)(wish,1)(would,2)(wrong,1)(you,1)
