How do you organise a python project that contains multiple packages so that each file in a package can still be run individually?

Once you move to your desired configuration, the absolute imports you are using to load the modules that are specific to my_tool no longer work.

You need three modifications after you create the my_tool subdirectory and move the files into it:

1. Create my_tool/__init__.py. (You seem to already do this but I wanted to mention it for completeness.)

2. In the files directly under in my_tool: change the import statements to load the modules from the current package. So in my_tool.py change:

   import c
   import d
   import k
   import s
   

   to:

   from . import c
   from . import d
   from . import k
   from . import s
   

   You need to make a similar change to all your other files. (You mention having tried setting __package__ and then doing a relative import but setting __package__ is not needed.)

3. In the files located in my_tool/tests: change the import statements that import the code you want to test to relative imports that load from one package up in the hierarchy. So in test_my_tool.py change:

   import my_tool
   

   to:

   from .. import my_tool
   

   Similarly for all the other test files.

With the modifications above, I can run modules directly:

$ python -m my_tool.my_tool
C!
D!
F!
V!
K!
T!
S!
my_tool!
my_tool main!
|main tool!||detected||tar edit!||installed||keys||LOL||ssl connect||parse ASN.1||config|

$ python -m my_tool.k
F!
V!
K!
K main!
|keys||LOL||ssl connect||parse ASN.1|

and I can run tests:

$ nosetests 
........
----------------------------------------------------------------------
Ran 8 tests in 0.006s

OK

Note that I can run the above both with Python 2.7 and Python 3.

Rather than make the various modules under my_tool be directly executable, I suggest using a proper setup.py file to declare entry points and let setup.py create these entry points when the package is installed. Since you intend to distribute this code, you should use a setup.py to formally package it anyway.

1. Modify the modules that can be invoked from the command line so that, taking my_tool/my_tool.py as example, instead of this:

   if __name__ == "__main__":
       print("my_tool main!")
       print(do_something())
   

   You have:

   def main():
       print("my_tool main!")
       print(do_something())
   
   if __name__ == "__main__":
       main()
   

2. Create a setup.py file that contains the proper entry_points. For instance:

   from setuptools import setup, find_packages
   
   setup(
       name="my_tool",
       version="0.1.0",
       packages=find_packages(),
       entry_points={
           'console_scripts': [
               'my_tool = my_tool.my_tool:main'
           ],
       },
       author="",
       author_email="",
       description="Does stuff.",
       license="MIT",
       keywords=[],
       url="",
       classifiers=[
       ],
   )
   

   The file above instructs setup.py to create a script named my_tool that will invoke the main method in the module my_tool.my_tool. On my system, once the package is installed, there is a script located at /usr/local/bin/my_tool that invokes the main method in my_tool.my_tool. It produces the same output as running python -m my_tool.my_tool, which I’ve shown above.