CrazyAirhead

疯狂的傻瓜,傻瓜也疯狂——傻方能执著,疯狂才专注!

0%

Windows编译QuickSQL问题记录

环境

Windows 10

IDEA Comunity 2021.1

JAVA 1.8.0_281

问题及处理

  1. 编译时无法下载依赖

配置阿里云仓库,需要在Maven的settgins.xml中指定

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<mirrors>

<mirror>

<id>aliyunmaven</id>

<mirrorOf>*</mirrorOf>

<name>阿里云公共仓库</name>

<url>https://maven.aliyun.com/repository/public</url>

</mirror>

</mirrors>

打开QuickSQL项目,直接编译即可。

1
mvn -DskipTests clean package
  1. 配置Spark和Hadoop环境

QuickSQL对要求的Spark版本是2.3+,我这里使用了2.4.7版本,

https://www.apache.org/dyn/closer.lua/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz

同时依赖的hadoop版本2.7,我选择的是为hadoop2.7.5

http://archive.apache.org/dist/hadoop/core/hadoop-2.7.5/hadoop-2.7.5.tar.gz

整体的配置过程参考《解决报错:Failed to locate the winutils binary in the hadoop binary path

  1. 运行CsvScanExample报错,主要提示
1
Error while running command to get file permissions : ExitCodeException exitCode=-1073741515

winutils.exe需要VS2010的dll依赖,需要安装VCRedist_x64.exe

参考链接

hadoop - java.lang.RuntimeException: Error while running command to get file permissions : ExitCodeException exitCode=-1073741515 - Stack Overflow

但文章给出的链接以及失效,从如下链接找到可以的地址,

Visual C++ 2010 Runtime Redistributable Package (x86, x64, ia64) Free Download

点击下载,VCRedist_x64.exe

  1. 运行CsvScanExample报错,主要提示
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
java.lang.RuntimeException: java.lang.UnsupportedOperationException 

at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:163)

at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:100)

at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:94)

at com.qihoo.qsql.codegen.ClassBodyWrapper$WithClassPathInMemoryCompiler.compile(ClassBodyWrapper.java:156)

at com.qihoo.qsql.codegen.ClassBodyWrapper$WithClassPathInMemoryCompiler.singleCompile(ClassBodyWrapper.java:123)

at com.qihoo.qsql.codegen.ClassBodyWrapper.compileSourceAndLoadClass(ClassBodyWrapper.java:64)

at com.qihoo.qsql.codegen.ClassBodyWrapper.compile(ClassBodyWrapper.java:101)

at com.qihoo.qsql.exec.AbstractPipeline.compileRequirement(AbstractPipeline.java:51)

at com.qihoo.qsql.exec.spark.SparkPipeline.show(SparkPipeline.java:105)

at com.qihoo.qsql.CsvJoinWithEsExample.main(CsvJoinWithEsExample.java:29)

检查IDEA的Project Struct,找到Project SDK,可能默认选中了JDK11,需要选择JDK8,包括Project language level,也需要选择8。

img

  1. 运行CsvScanExample报错,主要提示
1
The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: ---------;

进入%HADOOP_HOME%/bin目录,执行如下命令:

1
winutils.exe chmod 777 D:\tmp\hive

/tmp/hive目录是否在D盘需要自行确认。

参考链接

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- (on Windows) - Stack Overflow

  1. 运行CsvScanExample报错,主要提示
1
java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V

此时为spark引用的net.jpountz.lz4:lz4的引起的,排除qsql-core的net.jpountz.lz4依赖

参考链接

java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
<dependency>

<groupId>com.qihoo.qsql</groupId>

<artifactId>qsql-core</artifactId>

<version>${project.parent.version}</version>

<exclusions>

<exclusion>

<groupId>net.jpountz.lz4</groupId>

<artifactId>lz4</artifactId>

</exclusion>

</exclusions>

</dependency>
  1. 运行CsvScanExample报错,主要提示
1
ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.5.3ANTLR Runtime version 4.7 used for parser compilation does not match the current runtime version 4.5.3Exception in thread "main" java.lang.ExceptionInInitializerError

与spark引用的org.antlr:antlr4-runtime引起的,排除相关依赖。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
<dependency>

<groupId>org.codelibs.elasticsearch.module</groupId>

<artifactId>lang-painless</artifactId>

<version>${elasticsearch.version}</version>

<exclusions>

<exclusion>

<artifactId>antlr4-runtime</artifactId>

<groupId>org.antlr</groupId>

</exclusion>

</exclusions>

</dependency>

欢迎联系我

微信号 :Crazy_Airhead

Mixin ID : 1091586

定投课堂邀请码:6DYMBFP061

李笑来写作课邀请码:38MDGFYZK8

水龙头邀请码:FDJQHJ

欢迎关注我的其它发布渠道