Project Setup with Python

If possible use a project tool to set up and develop your Python projects. It will implement these concepts for you.

Some Python projects have specialized requirements that mean that you will decide to create a customised project, rather than using one of the popular project tools. If you decide not to use a project tool, set up your projects to follow the best practices in this article.

For more details about modern Python tools and good practices, read my article.

Use a pyproject.toml File #

Create a pyproject.toml file in the root directory of each Python project. Use this file as the central place to store configuration information about the project and the tools that it uses. For example, you list the dependencies of your project in the pyproject.toml file.

Python project tools like PDM, uv and Hatch automatically create and use a pyproject.toml file.

The pyOpenSci project documentation on pyproject.toml provides an introduction to the file format. The various features of pyproject.toml files are defined these PEPs: PEP 517, PEP 518, PEP 621 and PEP 660.

Use the Correct Directory Structure #

Python itself does not require a specific directory structure for your projects. The Python packaging documentation describes two popular directory structures: the src layout and the flat layout. The pyOpenSci project documentation on directory structures explains the practical differences between the two.

Use the src layout for a project that creates Python wheel packages. This requires you to use editable installs of the packages in your project. PDM, uv and Hatch support editable installs.

Use the flat layout for applications, such as Web services and command-line tools.

By default, uv will create a project with the flat layout. Use the –lib flag to create a project with the src layout.

Use Virtual Environments for Development #

The virtual environments feature enables you to define one or more separate sets of packages for each Python project, and switch between them. This ensures that a set of packages that you use for a specific purpose do not conflict with any other Python packages on the system. Always use Python virtual environments for your projects.

Several tools automate virtual environments. The mise version manager includes support for virtual environments. The pyenv version manager supports virtual environments with the virtualenv plugin. If you use a tool like uv, PDM or Hatch to develop your projects, these also manage Python virtual environments for you.

You can set up and use virtual environments with venv, which is part of the Python standard library. This is a manual process.

Use Requirements Files to Install Packages Into Environments #

Avoid using pip commands to install individual packages into virtual environments. If you use uv, PDM or Hatch to develop your project, they can manage the contents of virtual environments for development and testing.

For other cases, use requirements files. A requirements file can specify the exact version and hash for each required package.

You run a tool to read the dependencies in the pyproject.toml file and generate a requirements file that lists the specific packages that are needed to provide those dependencies for the Python version and operating system. PDM, pip-tools and uv include features to create requirements files.

You can then use pip-sync or the sync feature of uv to make the packages in a target virtual environment match the list in the requirements file. This process ensures that any extra packages are removed from the virtual environment.

You can also run pip install with a requirements file. This only attempts to install the specified packages. For example, these commands install the packages that are specified by the file requirements-macos-dev.txt into the virtual environment .venv-dev:

source ./.venv-dev/bin/activate
python3 -m pip install --require-virtualenv -r requirements-macos-dev.txt

Requirements files: If you use requirements files to build or deploy projects then configure your tools to use hashes.

pip-compile: Use the Correct Virtual Environment #

If you do not already have a tool that can create requirements files, you can use the pip-compile utility that is provided by pip-tools.

To ensure that it calculates the correct requirements for your application, the pip-compile tool must be run in a virtual environment that includes your application package. This means that you cannot use pipx to install pip-compile.