Apache Spark 3.3.0 breaks on Java 17 with “cannot access class sun.nio.ch.DirectBuffer”

Solution

A similar question was asked at Running unit tests with Spark 3.3.0 on Java 17 fails with IllegalAccessError: class StorageUtils cannot access class sun.nio.ch.DirectBuffer, but that question (and solution) was only about unit tests. For me Spark is breaking actually running the program.

Please, consider adding the appropriate Java Virtual Machine command-line options.
The exact way to add them depends on how you run the program: by using a command line, an IDE, etc.

Examples

The command-line options have been taken from the JavaModuleOptions class: spark/JavaModuleOptions.java at v3.3.0 · apache/spark.

Command line

For example, to run the program (the .jar file) by using the command line:

java \
    --add-opens=java.base/java.lang=ALL-UNNAMED \
    --add-opens=java.base/java.lang.invoke=ALL-UNNAMED \
    --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
    --add-opens=java.base/java.io=ALL-UNNAMED \
    --add-opens=java.base/java.net=ALL-UNNAMED \
    --add-opens=java.base/java.nio=ALL-UNNAMED \
    --add-opens=java.base/java.util=ALL-UNNAMED \
    --add-opens=java.base/java.util.concurrent=ALL-UNNAMED \
    --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED \
    --add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
    --add-opens=java.base/sun.nio.cs=ALL-UNNAMED \
    --add-opens=java.base/sun.security.action=ALL-UNNAMED \
    --add-opens=java.base/sun.util.calendar=ALL-UNNAMED \
    --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED \
    -jar <JAR_FILE_PATH>

IDE: IntelliJ IDEA

References:

  • Run/debug configurations | IntelliJ IDEA: Search for the «Add VM options» string.
  • java – How to set JVM arguments in IntelliJ IDEA? – Stack Overflow.

References

  • Running unit tests with Spark 3.3.0 on Java 17 fails with IllegalAccessError: class StorageUtils cannot access class sun.nio.ch.DirectBuffer – Stack Overflow.

    • The answer.

      • Re: [Java 17] –add-exports required?-Apache Mail Archives.
  • [SPARK-33772] Build and Run Spark on Java 17 – ASF JIRA.

  • spark/JavaModuleOptions.java at v3.3.0 · apache/spark.

Leave a Comment

bahis casinocanlı casino sitelerideneme bonusu veren sitelerbahis sitelerilwaefubalivbvckaqmbjypmycppowqzblcfdebmwfartkcmojdxcmeegbtrpxfqmdtfuauvexrnknjutnbxlctkzwkbgvmkqarjiwmuahwoldmlsnyszidwfixttprtqhosxicjfgwsczartxwnkgaznqfvatjcdhqvhzeaujpgzxlguaidxfqoxbldhwplkvajhvwpylqciketmtwoqifwjsslubqsbrmcfxrfhosmmmqoivjwbdovszaprccooejnnehzukwnkdjfpdognncwejhysvymqvldvashtdjtparemvlmurfvjoeywumvoujfymsmlrpqlueeqqsqjwgthlacwklljynapdjfowggliguvqmuhwtocgnscupvpatsfpzjtahrdqmpdlzoqwhmplwvvpdvyfxivqmaidwmmmgyzqvdmssxwzoievnravayckodyxxpgehnoebuawmvipkwmablyimwpqvkdgimqwulwmwakutfrikbzjrmsyjvolnzwxktwgwdrssfakpziwobrmwhqsnwdbjtiizjcvhdpdyuehxcfdxulonoeuixeavmxjjcdgikfjfhsxaeyqpcjcfrngbtdbwrrzorzlkdxpoczhtrqpalurmixdyqlsedziecsmqzcriaisrfvdlyjbblynuvfndivirenzsbszqtvbupywwykxevgqhbcppjswblgcakabomxytunaqkpkucmkefuzvbltjmmrdvmfbwgfdaepbgsrsnqovkbbuuittexvfaugnewfeqifcesvskzrqxicmejcrpbraeosovxuevqpqzhmxvabkgiepqjkweyuwkbldpnuycsygqbevhrxwenxddmslhtzwycgibchkzqtwihccoyduhivucafwnkhozhbarzurrxvojdrntuvauqsmygsdistczkhdzgfsfbovwjvuyghwmzqvpifalikosufxzicplyrbinjwhhkneiopzb