I agree with @JB Nizet about respecting standard conventions. If you still insist on being an Anarchist though:
You already have src declared in your sourceset, why not add src1 and src2 as well? You can add them to the same sourceset, or define a sourceset per module if you want.
sourceSets {
main {
java {
srcDirs 'src'
srcDirs 'src1'
srcDirs 'src2'
}
}
}
To reference files outside the project, see this answer.