Mac上安装hive问题解决

Author Avatar
呃哦 11月 08, 2019

安装

一开始按照书上选择源码安装,结果配置环境变量太麻烦了,后选择通过brew安装。简单命令如下

brew install hadoop
brew install hive

踩坑

guava版本不一致问题

问题

直接敲 hive 命令,遇到以下报错

LF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/3.1.2/libexec/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/3.2.1/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
    at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)
    at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)
    at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)
    at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)
    at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:97)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

原因

谷歌之得到以下答案

关键在: com.google.common.base.Preconditions.checkArgument 这是因为hive内依赖的guava.jar和hadoop内的版本不一致造成的。

解决

# 查看版本
$ ls /usr/local/Cellar/hadoop/3.2.1/libexec/share/hadoop/common/lib/guava*
/usr/local/Cellar/hadoop/3.2.1/libexec/share/hadoop/common/lib/guava-27.0-jre.jar
$ ls /usr/local/Cellar/hive/3.1.2/libexec/lib/guava* 
/usr/local/Cellar/hive/3.1.2/libexec/lib/guava-19.0-jre.jar
# hadoop 版本比 hive 高,替换之
$ mv /usr/local/Cellar/hive/3.1.2/libexec/lib/guava-19.0.jar /tmp # 个人习惯保留下副本
$ cp /usr/local/Cellar/hadoop/3.2.1/libexec/share/hadoop/common/lib/guava-27.0-jre.jar /usr/local/Cellar/hive/3.1.2/libexec/lib/ # 替换

jdk 版本不一致问题

问题

解决完上面 jar 包问题后再次敲入 hive 命令遇到以下报错

Hive Session ID = c2bd3ba5-9902-48d3-8d7b-e63bf64444cf
Exception in thread "main" java.lang.ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and java.net.URLClassLoader are in module java.base of loader 'bootstrap')
    at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:413)
    at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:389)
    at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:60)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

原因

谷歌之找到问题在于 JDK 版本不一致。个人系统是 MacOS 10.15 ,里面的 JDK 版本 11

解决

安装低版本 jdk ,可直接在oracle官网下载的 1.8 版本 jdk 安装。

安装后的系统默认 jdk 还是 11,但是在 /Library/Java/JavaVirtualMachines 目录下可看到有多个 jdk 版本共存。

$ ll /Library/Java/JavaVirtualMachines
total 0
drwxr-xr-x  4 root  wheel   128B Nov  3 17:01 ./
drwxr-xr-x  4 root  wheel   128B Oct 24 08:35 ../
drwxr-xr-x  3 root  wheel    96B Nov 15  2018 jdk-11.0.1.jdk/
drwxr-xr-x  3 root  wheel    96B Nov  3 16:58 jdk1.8.0_212.jdk/

因此修改 hive 启动环境下的 JAVA_HOME 变量:

  1. 添加写入权限,默认 hive 可执行文件是只读的

    chmod +w `which hive`
    
  2. 修改文件内容

    原内容如下

    #!/bin/bash
    JAVA_HOME="$(/usr/libexec/java_home --version 1.7+)" HIVE_HOME="/usr/local/Cellar/hive/3.1.2/libexec" exec "/usr/local/Cellar/hive/3.1.2/libexec/bin/hive" "$@"
    

    修改后如下

    #!/bin/bash
    #JAVA_HOME="$(/usr/libexec/java_home --version 1.7+)"
    JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_212.jdk/Contents/Home"
    HIVE_HOME="/usr/local/Cellar/hive/3.1.2/libexec"
    exec "/usr/local/Cellar/hive/3.1.2/libexec/bin/hive" "$@"