Modern Good Practices for Python Development

Python has a long history, and it has evolved over time. This article describes some agreed modern best practices.

Tooling #

Use The Most Recent Version of Python That You Can #

Use the most recent stable version of Python 3. This ensures that you have the latest security fixes, as well as the fastest performance. If your operating system includes an older version of Python, you can use tools like asdf, pyenv or Homebrew to install a separate copy of Python.

If you need to use an older version of Python for compatibility, check that you are using the most recent version that you can. The Python development team usually support each version for five years, but some Python libraries may only support each version of Python for a shorter period of time.

Avoid using Python 2. It is not supported by the Python development team. The current versions of many libraries are not compatible with Python 2.

Use pipx To Install Python Applications #

Always use pipx to install Python applications, rather than pip. This ensures that each application has the correct libraries. Unlike pip, pipx automatically installs the libraries for each application into a separate Python virtual environment.

The Python Packaging Authority maintain pipx, but it is not included with Python. You can install pipx with Homebrew on macOS, or with your system package manager on Linux.

PEP 668 - Marking Python base environments as “externally managed” recommends that users install Python applications with pipx.

Use Virtual Environments for Development #

The virtual environments feature enables you to define separate sets of packages for each Python project, so that the packages for a project do not conflict with any other Python packages on the system. Always use Python virtual environments for your projects.

If you use a tool like Hatch or Poetry to develop your projects it will manage Python virtual environments for you.

If you prefer, you can also manually set up and manage virtual environments with venv, which is part of the Python standard library.

Use a Code Linter #

Use a code linting tool with a plugin to your editor, so that your code is automatically checked for issues.

flake8 is currently the most popular linter for Python, but it is being superseded by Ruff.

You can also run the linter with your CI system, to reject code that does not meet the standards for your project.

Format Your Code #

Use a formatting tool with a plugin to your editor, so that your code is automatically formatted to a consistent style.

If possible, use Black to format your code. Black is now the leading code formatter for Python. Black has been adopted by the Python Software Foundation and other major Python projects. It formats Python code to a style that follows the PEP 8 standard, but allows longer line lengths.

You can also run the formatter with your CI system, to reject code that does not match the format for your project.

Test with pytest #

Use pytest for testing. It has superseded nose as the most popular testing system for Python. Use the unittest module in the standard library for situations where you cannot add pytest to the project.

Language Syntax #

Use Type Hinting #

Current versions of Python support type hinting. Consider using type hints in any critical application. If you develop a shared library, use type hints.

Once you add type hints, Mypy tool can check your code as you develop it. Code editors can also read type hints to display information about the code that you are working with.

If you add Pydantic to your software, it uses type hints in your software to validate data as an application runs.

PEP 484 - Type Hints and PEP 526 – Syntax for Variable Annotations define the notation for type hinting.

Format Strings with f-strings #

The new f-string syntax is both more readable and has better performance than older methods. Use f-strings instead of % formatting, str.format() or str.Template().

The older features for formatting strings will not be removed, to avoid breaking backward compatibility.

The f-strings feature was added in version 3.6 of Python. Alternate implementations of Python may include this specific feature, even when they do not support version 3.6 syntax.

PEP 498 explains f-strings in detail.

Use Datetime Objects with Time Zones #

Always use datetime objects that are aware of time zones. By default, Python creates datetime objects that do not include a time zone. The documentation refers to datetime objects without a time zone as naive.

Avoid using date objects, except where the time of day is completely irrelevant. The date objects are always naive, and do not include a time zone.

Use aware datetime objects with the UTC time zone for timestamps, logs and other internal features.

To get the current time and date in UTC as an aware datetime object, specify the UTC time zone with now(). For example:

from datetime import datetime, timezone

dt = datetime.now(timezone.utc)

Python 3.9 and above include the zoneinfo module. This provides access to the standard IANA database of time zones. Previous versions of Python require a third-party library for time zones.

PEP 615 describes support for the IANA time zone database with zoneinfo.

Use enum or Named Tuples for Immutable Sets of Key-Value Pairs #

Use the enum type in Python 3.4 or above for immutable collections of key-value pairs. Enums can use class inheritance.

Python 3 also has collections.namedtuple() for immutable key-value pairs. Named tuples do not use classes.

Create Data Classes for Custom Data Objects #

The data classes feature enables you to reduce the amount of code that you need to define classes for objects that exist to store values. The new syntax for data classes does not affect the behavior of the classes that you define with it. Each data class is a standard Python class.

You can set a frozen option to make frozen instances of a data class.

Data classes were introduced in version 3.7 of Python.

PEP 557 describes data classes.

Use collections.abc for Custom Collection Types #

The abstract base classes in collections.abc provide the components for building your own custom collection types.

Use these classes, because they are fast and well-tested. The implementations in Python 3.7 and above are written in C, to provide better performance than Python code.

Use breakpoint() for Debugging #

This function drops you into the debugger at the point where it is called. Both the built-in debugger and external debuggers can use these breakpoints.

The breakpoint() feature was added in version 3.7 of Python.

PEP 553 describes the breakpoint() function.

Application Design #

Use Logging for Diagnostic Messages, Rather Than print() #

The built-in print() statement is convenient for adding debugging information, but you should include logging in your scripts and applications. Use the logging module in the standard library, or a third-party logging module.

Use The TOML Format for Configuration #

Use TOML for data files that must be written or edited by human beings. Use the JSON format for data that is transferred between computer programs. Avoid using the INI or YAML formats.

Python 3.11 and above include tomllib to read the TOML format. Use tomli to add support for reading TOML to applications that run on older versions of Python.

If your Python software needs to generate TOML, add Tomli-W.

PEP 680 - tomllib: Support for Parsing TOML in the Standard Library explains why TOML is now included with Python.

Only Use async Where It Makes Sense #

The asynchronous features of Python enable a single process to avoid blocking on I/O operations. To achieve concurrency with Python, you must run multiple Python processes. Each of these processes may or may not use asynchronous I/O.

To run multiple application processes, either use an application server like Gunicorn or use a container system, with one container per process. If you need to build a custom application that manages muliple processes, use the multiprocessing package in the Python standard library.

Code that uses asynchronous I/O must not call any function that uses synchronous I/O, such as open(), or the logging module in the standard library. Instead, you need to use either the equivalent functions from asyncio in the standard library or a third-party library that is designed to support asynchronous code.

The FastAPI Web framework supports using both synchronous and asynchronous functions in the same application. You must still ensure that asynchronous functions never call any synchronous function.

If you would like to work with asyncio, use Python 3.7 or above. Version 3.7 of Python introduced context variables, which enable you to have data that is local to a specific task, as well as the asyncio.run() function.

PEP 0567 describes context variables.

Libraries #

Handle Command-line Input with argparse #

The argparse module is now the recommended way to process command-line input. Use argparse, rather than the older optparse and getopt.

The optparse module is officially deprecated, so update code that uses optparse to use argparse instead.

Refer to the argparse tutorial in the official documentation for more details.

Use pathlib for File and Directory Paths #

Use pathlib objects instead of strings whenever you need to work with file and directory pathnames.

Consider using the the pathlib equivalents for os functions.

The existing methods in the standard library have been updated to support Path objects.

To list all of the the files in a directory, use either the .iterdir() function of a Path object, or the os.scandir() function.

This RealPython article provides a full explanation of the different Python functions for working with files and directories.

The pathlib module was added to the standard library in Python 3.4, and other standard library functions were updated to support Path objects in version 3.5 of Python.

Use os.scandir() Instead of os.listdir() #

The os.scandir() function is significantly faster and more efficient than os.listdir(). Use os.scandir() wherever you previously used the os.listdir() function.

This function provides an iterator, and works with a context manager:

import os

with os.scandir('some_directory/') as entries:
    for entry in entries:
        print(entry.name)

The context manager frees resources as soon as the function completes. Use this option if you are concerned about performance or concurrency.

The os.walk() function now calls os.scandir(), so it automatically has the same improved performance as this function.

The os.scandir() function was added in version 3.5 of Python.

PEP 471 explains os.scandir().

Run External Commands with subprocess #

The subprocess module provides a safe way to run external commands. Use subprocess rather than shell backquoting or the functions in os, such as spawn, popen2 and popen3. The subprocess.run() function in current versions of Python is sufficient for most cases.

PEP 324 explains the technical details of subprocess in detail.

Use httpx for Web Clients #

Use httpx for Web client applications. It supports HTTP/2, and async. The httpx package supersedes requests, which only supports HTTP 1.1.

Avoid using urllib.request from the Python standard library. It was designed as a low-level library, and lacks the features of httpx.