Solution
A similar question was asked at Running unit tests with Spark 3.3.0 on Java 17 fails with IllegalAccessError: class StorageUtils cannot access class sun.nio.ch.DirectBuffer, but that question (and solution) was only about unit tests. For me Spark is breaking actually running the program.
Please, consider adding the appropriate Java Virtual Machine command-line options.
The exact way to add them depends on how you run the program: by using a command line, an IDE, etc.
Examples
The command-line options have been taken from the JavaModuleOptions
class: spark/JavaModuleOptions.java at v3.3.0 · apache/spark.
Command line
For example, to run the program (the .jar
file) by using the command line:
java \
--add-opens=java.base/java.lang=ALL-UNNAMED \
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED \
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
--add-opens=java.base/java.io=ALL-UNNAMED \
--add-opens=java.base/java.net=ALL-UNNAMED \
--add-opens=java.base/java.nio=ALL-UNNAMED \
--add-opens=java.base/java.util=ALL-UNNAMED \
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED \
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED \
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED \
--add-opens=java.base/sun.security.action=ALL-UNNAMED \
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED \
--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED \
-jar <JAR_FILE_PATH>
IDE: IntelliJ IDEA
References:
- Run/debug configurations | IntelliJ IDEA: Search for the «Add VM options» string.
- java – How to set JVM arguments in IntelliJ IDEA? – Stack Overflow.
References
-
Running unit tests with Spark 3.3.0 on Java 17 fails with IllegalAccessError: class StorageUtils cannot access class sun.nio.ch.DirectBuffer – Stack Overflow.
-
The answer.
- Re: [Java 17] –add-exports required?-Apache Mail Archives.
-
-
[SPARK-33772] Build and Run Spark on Java 17 – ASF JIRA.
-
spark/JavaModuleOptions.java at v3.3.0 · apache/spark.