{"id":27,"date":"2022-02-10T17:54:44","date_gmt":"2022-02-10T17:54:44","guid":{"rendered":"https:\/\/blogs.oregonstate.edu\/techgooder\/?p=27"},"modified":"2022-02-10T17:59:10","modified_gmt":"2022-02-10T17:59:10","slug":"managing-imports-in-a-python-project","status":"publish","type":"post","link":"https:\/\/blogs.oregonstate.edu\/techgooder\/2022\/02\/10\/managing-imports-in-a-python-project\/","title":{"rendered":"Managing imports in a Python project"},"content":{"rendered":"\n

I’ve been working on a group project this term, and we’ve been having some trouble with imports. The project structure is shown (with some details removed for simplicity).<\/p>\n\n\n\n

project\n\u2502   .gitignore\n\u2502   README.md\n\u2502   requirements.txt\n\u2502   setup.py\n\u2502\n\u2514\u2500\u2500\u2500project_name\n    \u2502   __init__.py\n    \u2502\n    \u251c\u2500\u2500\u2500data\n    \u2502       data_loader.py\n    \u2502       feature_extractor.py\n    \u2502       feature_recorder.py\n    \u2502       gtzan_utils.py\n    \u2502       __init__.py\n    \u2502\n    \u251c\u2500\u2500\u2500ml\n    \u2502   \u2502   __init__.py\n    \u2502   \u2502\n    \u2502   \u2514\u2500\u2500\u2500c\n    \u2502           ml_algo_c.py\n    \u2502           training_script.py\n    \u2502           write_features_script.py\n    \u2502           __init__.py\n    \u2502\n    \u251c\u2500\u2500\u2500resources\n    \u2502       .gitignore\n    \u2502\n    \u2514\u2500\u2500\u2500tests\n            testing.py\n            __init__.py<\/code><\/pre>\n\n\n\n
Please forgive the placeholder project_name<\/code>. I swear we’ll replace it with something brilliant any day now.<\/p>\n\n\n\n
The problem is, I need to access the data<\/code> classes in training_script.py<\/code>. I spent some time trying to get relative imports working the way I wanted. Python did not like this at all, and I wound up with this terrible code.<\/p>\n\n\n\n
import sys\nimport os\nimport time\n\nimport tensorflow as tf\n\n# This import works fine since it's in the same folder\n# as training_script.py\nfrom ml_c import MlAlgoC\n\n# Doing work to crawl up the top-level directory of the project\n# and append that directory to sys.path so that Python can\n# find our project files\nscript_dir = os.path.dirname(__file__)\nmymodule_dir = os.path.join(script_dir, '..', '..')\nsys.path.append(mymodule_dir)\n\n# The preceding code was necessary to make these imports\n# of local project files work.\n# This is terrible!\nimport data.data_loader as dl           # noqa: E402\nimport data.feature_recorder as fr      # noqa: E402\n\n# Note the inline comments needed to stop the linter\n# from complaining about imports not being at the\n# top of the file. Ugh!<\/code><\/pre>\n\n\n\n
In fact, the whole group was having this issue. In most languages, you would just do something like this.<\/p>\n\n\n\n
# Not in Python, but many languages would allow an import\n# like this\nimport '..\/..\/data\/data_loader.py'\nimport '..\/..\/data\/feature_recorder.py'<\/code><\/pre>\n\n\n\n
But no dice, as Python does not allow any such import syntax. What’s the problem?<\/p>\n\n\n\n
When looking for imports, Python crawls sys.path<\/code>, which is a list of paths where Python expects to find modules and packages. This, by default, holds the paths Python needs to find Python library files and installed packages. It also contains the folder in which your Python execution started.<\/p>\n\n\n\n
When I’m trying to run my script files in the project_name\/ml\/c<\/code> folder, either directly through the command line, or using my IDE (VS Code), the local folder Python supplies to sys.path<\/code> is the c<\/code> folder. Python has no idea how to find the data<\/code> folder from here — it can only find modules\/packages at the same level or lower in the tree.<\/p>\n\n\n\n
One note here is that PyCharm might automatically add your project folder to sys.path<\/code>, so those using PyCharm might not have this issue.<\/p>\n\n\n\n
Another note: you could argue that the script files should be located together in a bin<\/code> folder somewhere, but that doesn’t really help with this particular problem.<\/p>\n\n\n\n
I looked into various solutions for this, and the cleanest one I could find is to simply install the project like you would any other Python package. Then, you can access it using the same absolute import syntax you would use for any built-in Python library or installed package.<\/p>\n\n\n\n
This sounds problematic, though, because do we really want a duplicate of our project files installed somewhere else on our system? Worse, are we going to have to update\/reinstall the package every time we change a single line of code? Finally, what if we just want a small project and don’t want to do extra work to make it installable as a robust, reusable package?<\/p>\n\n\n\n
Luckily, pip has a method to handle this. Using an editable install<\/a>, you can do a very lightweight install that essentially sets up a symbolic link to your project folder, wherever it is located. The command is simple.<\/p>\n\n\n\n
pip install -e path\/to\/SomeProject\n\n# Or, just use this while navigated to the directory\n# where your setup.py file is located\npip install -e .<\/code><\/pre>\n\n\n\n
Then, Python can access your project folder just like it would any installed package. Since this links to your actual project folder, any changes you make will be reflected and available in your imports immediately. You are just linking from the Python installed package folder directly to your real project directory.<\/p>\n\n\n\n
Of course, you will need a setup.py<\/code> file to make this work. Fortunately, you don’t need all the fields filled in that a true installable package would require. If you just want to ease your intra-project imports, a setup.py<\/code> as simple as this is sufficient.<\/p>\n\n\n\n
from setuptools import setup\n\n\nsetup(name='project_name')\n<\/code><\/pre>\n\n\n\n
The following pip commands may be helpful as well.<\/p>\n\n\n\n
# Display info about your installed project\npip show project_name\n\n# Uninstall your project if you want to clean up later\n# Don't worry, this just removes the symbolic link and\n# doesn't delete any of your project files in its\n# original home.\npip uninstall project_name<\/code><\/pre>\n\n\n\n
With this setup, that abysmal code from earlier becomes much cleaner and more Pythonic.<\/p>\n\n\n\n
import sys\nimport time\n\nimport tensorflow as tf\n\nfrom project_name.ml.c.ml_algo_c import MlAlgoC\nimport project_name.data.data_loader as dl\nimport project_name.data.feature_recorder as fr<\/code><\/pre>\n\n\n\n
This will work wherever you’re doing these imports, as Python finds your project files in the same place as other installed packages!<\/p>\n\n\n\n
We’re no longer trying to awkwardly force Python to do imports in a way it doesn’t want to. If you can’t beat them…<\/p>\n\n\n\n
(join them… learn their secrets… then plot your revenge)<\/p>\n\n\n\n
Song of the week: Cat Clyde – All the Black (acoustic)<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
I’ve been working on a group project this term, and we’ve been having some trouble with imports. The project structure is shown (with some details removed for simplicity). Please forgive the placeholder project_name. I swear we’ll replace it with something brilliant any day now. The problem is, I need to access the data classes in … Continue reading “Managing imports in a Python project”<\/span><\/a><\/p>\n","protected":false},"author":12022,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-27","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.oregonstate.edu\/techgooder\/wp-json\/wp\/v2\/posts\/27","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.oregonstate.edu\/techgooder\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.oregonstate.edu\/techgooder\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.oregonstate.edu\/techgooder\/wp-json\/wp\/v2\/users\/12022"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.oregonstate.edu\/techgooder\/wp-json\/wp\/v2\/comments?post=27"}],"version-history":[{"count":4,"href":"https:\/\/blogs.oregonstate.edu\/techgooder\/wp-json\/wp\/v2\/posts\/27\/revisions"}],"predecessor-version":[{"id":31,"href":"https:\/\/blogs.oregonstate.edu\/techgooder\/wp-json\/wp\/v2\/posts\/27\/revisions\/31"}],"wp:attachment":[{"href":"https:\/\/blogs.oregonstate.edu\/techgooder\/wp-json\/wp\/v2\/media?parent=27"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.oregonstate.edu\/techgooder\/wp-json\/wp\/v2\/categories?post=27"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.oregonstate.edu\/techgooder\/wp-json\/wp\/v2\/tags?post=27"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}