1、在CDH中已经添加了hive
2、配置hive中的hive-site.xml参数 /opt/cloudera/parcels/CDH/lib/hive/conf/hive-site.xml
3、配置spark中的hive-site.xml 目录/opt/cloudera/parcels/CDH/lib/spark3/conf/hive-site.xml
<property> <name>hive.metastore.warehouse.dir</name> <value>/user/hive/warehouse</value> </property> <property> <name>hive.metastore.local</name> <value>false</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://hadoop103:9083</value> </property> 4、修改spark-env.sh,目录/opt/cloudera/parcels/CDH/lib/spark3/conf/
export PYSPARK_PYTHON=/opt/cloudera/anaconda3/bin/python export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark3 5、如果自己在代码中实现这
spark = SparkSession.builder.appName('feat-eng')\ .config("hive.metastore.warehouse.dir",'hdfs://hadoop103:8020/user/hive/warehouse')\ .config("hive.metastore.uris","thrift://hadoop103:9083") \ .enableHiveSupport().getOrCreate() 6、下面就是spark on hive 对接数据
import findspark findspark.init(spark_home='/opt/cloudera/parcels/CDH/lib/spark3',python_path='/opt/cloudera/anaconda3/bin/python') import pyspark from pyspark.context import SparkContext from pyspark.sql import SparkSession from pyspark.sql.functions import * spark = SparkSession.builder.appName('feat-eng')\ .config("hive.metastore.warehouse.dir",'hdfs://hadoop103:8020/user/hive/warehouse')\ .config("hive.metastore.uris","thrift://hadoop103:9083") \ .enableHiveSupport().getOrCreate() '''读取csv的原始数据''' anime_df = spark.read.csv('hdfs://hadoop101:8020/recommandvideo/anime.csv',header=True,inferSchema=True) anime_df.createOrReplaceTempView("temp_anime") '''将数据入库hive''' spark.sql('insert into sparktest.anime select "1",anime_id,name,genre,type,episodes,rating,members from temp_anime') '''读取hive中表的数据''' spark.sql("select * from sparktest.anime").show()