Python Module Import Errors

Created At: 12.07.2019, Last Update: 12.07.2019

Disclaimer: Please help me improve the article. I am eager to learn more about the Python internals. Please contact me, if you can provide more insights and get Kudos in my article :)

Sample Project

Suppose, we have the following project structure:

project_root/
  DirA/
    a.py
  Test/
    test_A.py
    test_B.py
    b.py
    d.py
main.py
c.py
test_C.py
test_D.py

Each file contains a class with a hello world method like this:

class C:
    @staticmethod
    def hello_world():
        return 42

And a test like this:

import unittest
from unittest import TestCase

from c import C


class TestC(TestCase):
    def test_hello_world(self):
        self.assertEqual(C.hello_world(), 42)


if __name__ == "__main__":
    unittest.main()

Calling the test from the main directory yields the following output:

.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

While this is nice, it is usually not the way real-world projects are structured. You want to organize your code into modules that are located in directories (or packages, we will come back to this later).

Let us further assume that you want to run the test_A.py file as a preceding step to execute the main project in main.py.

Depending on your current location, you can issue one of the two following commands. It does not matter as they yield the same output.

python test_A.py
python Test/test_A.py

And you will most likely see the following error message:

Traceback (most recent call last):
  File "test_A.py", line 4, in 
    from DirA.a import A
ModuleNotFoundError: No module named 'DirA'

Tl:dr Elegant Solution: Do not Run Scripts in Subdirectories without -m

The root cause of this behavior is the desired way to use Python as stated by its creators. They basically say:

Do not run scripts within sub-directories. (See here: E-Mail archive)

Now, I could basically end the blog post as you are supposed to run a test like this:

python -m Test.test_B
# or
python -m unittest Test/test_B.py

The -m flag indicates to run the library module as a script and adds everything to the sys.path in the correct way. Both imports now work:

from Test.b import B
from .b import B

You can also use test discovery from unittest, which will search for files starting with test_:

python -m unittest

Be aware that every reference to a file needs to be relative to the root directory!

Let's not give up yet with this answer and dig a little deeper...

Python3 ModuleNotFoundError

The Python interpreter is not able to find the module as it is in another directory. Finding the root cause of this behavior is not that easy as it looks like. You probably find thousands of posts via Google on this topic. In my opinion, only a selected few of these cover all the details related to this issue.

Let me extend the problem a little bit by focusing on test_B.py:

import unittest
from unittest import TestCase

from Test.b import B


class TestB(TestCase):
    def test_hello_world(self):
        self.assertEquals(B.hello_world(), 42)


if __name__ == "__main__":
    unittest.main()

In the current setting, you will even get an error in test_B.py with the file under test in the same directory:

python test_B.py
Traceback (most recent call last):
  File "test_B.py", line 4, in <module>
    from Test.b import B
ModuleNotFoundError: No module named 'Test'

Modifying the import to a relative import does only change the details regarding the error. The trailing dot means that the import should be resolved relative to the current directory.

from .b import B

This yields the following output:

Traceback (most recent call last):
  File "test_B.py", line 4, in 
    from .b import B
ModuleNotFoundError: No module named '__main__.b'; '__main__' is not a package

Let me add one more extension. Let's have a look at test_D.py:

import unittest
from unittest import TestCase

from Test.d import D


class TestC(TestCase):
    def test_hello_world(self):
        self.assertEqual(D.hello_world(), 42)


if __name__ == "__main__":
    unittest.main()

This test actually works, even though the module under test is in another directory and not marked as package yet. This leads to the first conclusion:

Python implicit package imports are a new feature since Python 3.3 (see here). However, they only seem to work correctly from directories at the root level. All other directories (even directories in directories) do not work. So there is a problem with the so called intra-package references.

Marking Directories as Packages

The Python3 documentation reveals that modules should be annotated with a special file called "__init__.py" ( link to documentation, packages). The interpreter only considers directories as modules that are stored in the variable sys.path. It is similar to Java's classpath.

sys.path

So you might find some hacks in the internet that manipulate this variable to add certain folders to sys.path so the interpreter is tricked into thinking that this is a module.

However, after adding such a file to the directory "Test", you will still see the error. Even after adding an init file to the root directory as suggested in the documentation, you will see the same error message. Referring to the "Implicit Namespace Packages" feature, this seems logical as Python 3.3 should recognize the directories as packages automatically.

project_root/
  __init__.py
  DirA/
    a.py
  Test/
    __init__.py
    test_A.py
    test_B.py
    b.py
main.py
c.py
test_C.py

So it seems that even though the modules are in sys.path, they are not considered. That means, the hierarchy of the sys.path entries play a crucial role.

Solving Intra-Package Referencing Issues

We can further push this idea by the following relative import:

import unittest
from unittest import TestCase

from ..Test.b import B


class TestB(TestCase):
    def test_hello_world(self):
        self.assertEquals(B.hello_world(), 42)


if __name__ == "__main__":
    unittest.main()

The import yields the following error that exactly fits to this idea. So clearly, the Python developers do not want you to run scripts below top level.

Traceback (most recent call last):
  File "Test/test_B.py", line 4, in 
    from ..Test.b import B
ValueError: attempted relative import beyond top-level package

Import Workaround

The only way to make this work is to use a little workaround that makes the file in the subdirectory think it is actually a valid root level script (besides the main directory).

import sys
import os

import unittest
from unittest import TestCase

PACKAGE_PARENT = '..'
SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))

from Test.b import B


class TestB(TestCase):
    def test_hello_world(self):
        self.assertEqual(B.hello_world(), 42)
        pass


if __name__ == "__main__":
    # print(str(Path(__file__).resolve().parents[1]))
    unittest.main()

Now both calls (from root and within Test directory) work like intended. However, what about test_A.py?

import sys
import os

import unittest
from unittest import TestCase

PACKAGE_PARENT = '..'
SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))

from DirA.a import A


class TestA(TestCase):
    def test_hello_world(self):
        self.assertEqual(A.hello_world(), 42)


if __name__ == "__main__":
    unittest.main()

The file test_A.py does also work without modification of a.py in the directory "DirA". So only the script that imports something needs to be adjusted, not the thing that is actually imported.