Spring Boot 2.5.0 generates plain.jar file. Can I remove it?

It was a change in Spring Boot 2.5.0.

As @ThomasKläger pointed out: You can set it in the build.gradle configuration.

build.gradle

jar {
    enabled = false
}

For Kotlin devs:

tasks.getByName<Jar>("jar") {
    enabled = false
}

Alternatively, you can run the bootJar task. It produces only the default runnable jar.