Mac上安装hive问题解决
安装
一开始按照书上选择源码安装,结果配置环境变量太麻烦了,后选择通过brew安装。简单命令如下
brew install hadoop
brew install hive
踩坑
guava版本不一致问题
问题
直接敲 hive
命令,遇到以下报错
LF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/3.1.2/libexec/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/3.2.1/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)
at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)
at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:97)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
原因
谷歌之得到以下答案
关键在: com.google.common.base.Preconditions.checkArgument 这是因为hive内依赖的guava.jar和hadoop内的版本不一致造成的。
解决
# 查看版本
$ ls /usr/local/Cellar/hadoop/3.2.1/libexec/share/hadoop/common/lib/guava*
/usr/local/Cellar/hadoop/3.2.1/libexec/share/hadoop/common/lib/guava-27.0-jre.jar
$ ls /usr/local/Cellar/hive/3.1.2/libexec/lib/guava*
/usr/local/Cellar/hive/3.1.2/libexec/lib/guava-19.0-jre.jar
# hadoop 版本比 hive 高,替换之
$ mv /usr/local/Cellar/hive/3.1.2/libexec/lib/guava-19.0.jar /tmp # 个人习惯保留下副本
$ cp /usr/local/Cellar/hadoop/3.2.1/libexec/share/hadoop/common/lib/guava-27.0-jre.jar /usr/local/Cellar/hive/3.1.2/libexec/lib/ # 替换
jdk 版本不一致问题
问题
解决完上面 jar 包问题后再次敲入 hive
命令遇到以下报错
Hive Session ID = c2bd3ba5-9902-48d3-8d7b-e63bf64444cf
Exception in thread "main" java.lang.ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and java.net.URLClassLoader are in module java.base of loader 'bootstrap')
at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:413)
at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:389)
at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:60)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
原因
谷歌之找到问题在于 JDK 版本不一致。个人系统是 MacOS 10.15 ,里面的 JDK 版本 11
解决
安装低版本 jdk ,可直接在oracle官网下载的 1.8 版本 jdk 安装。
安装后的系统默认 jdk
还是 11,但是在 /Library/Java/JavaVirtualMachines
目录下可看到有多个 jdk
版本共存。
$ ll /Library/Java/JavaVirtualMachines
total 0
drwxr-xr-x 4 root wheel 128B Nov 3 17:01 ./
drwxr-xr-x 4 root wheel 128B Oct 24 08:35 ../
drwxr-xr-x 3 root wheel 96B Nov 15 2018 jdk-11.0.1.jdk/
drwxr-xr-x 3 root wheel 96B Nov 3 16:58 jdk1.8.0_212.jdk/
因此修改 hive
启动环境下的 JAVA_HOME
变量:
添加写入权限,默认
hive
可执行文件是只读的chmod +w `which hive`
修改文件内容
原内容如下
#!/bin/bash JAVA_HOME="$(/usr/libexec/java_home --version 1.7+)" HIVE_HOME="/usr/local/Cellar/hive/3.1.2/libexec" exec "/usr/local/Cellar/hive/3.1.2/libexec/bin/hive" "$@"
修改后如下
#!/bin/bash #JAVA_HOME="$(/usr/libexec/java_home --version 1.7+)" JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_212.jdk/Contents/Home" HIVE_HOME="/usr/local/Cellar/hive/3.1.2/libexec" exec "/usr/local/Cellar/hive/3.1.2/libexec/bin/hive" "$@"