Modern Python Practices

Page content

Python has a long history, and has evolved over time. This article describes some agreed modern best practices.


Use Python 3

Use Python 3 for all new work. Python 2 will no longer be supported, as of 2020. All of the major libraries and frameworks now support Python 3. Some projects have removed support for Python 2 in their latest releases.

Use Virtual Environments for Development

The virtual environments feature enables you to define separate sets of packages for each Python project, so that they do not conflict with each other.

There are several tools that help you to manage your Python projects, and use virtual environments. The most popular are pipenv and poetry. Poetry is considered to be better designed, but pipenv is more widely supported. If you prefer, you can also manually set up and manage virtual environments.

Format Your Code

Use a formatting tool with a plugin to your editor, so that your code is automatically formatted to a consistent style.

Black is being adopted by the Python Software Foundation and other projects. It formats Python code to a style that follows the PEP 8 standard, but allows longer line lengths. Use Black for new projects.

Language Syntax

Format Strings with f-strings

The new f-string syntax is both more readable and has better performance than older methods. Use f-strings instead of % formatting, str.format() or str.Template().

The older features for formatting strings will not be removed, to avoid breaking backward compatibility.

The f-strings feature was added in version 3.6 of Python. Alternate implementations of Python may include this specific feature, even when they do not support version 3.6 syntax.

PEP 498 explains f-strings in detail.

Use enum or Named Tuples for Immutable Sets of Key-Value Pairs

Use the enum type in Python 3.4 or above for immutable collections of key-value pairs. Python 3 also has collections.namedtuple() for immutable key-value pairs.

Create Data Classes for Custom Data Objects

The data classes feature enables you to reduce the amount of code that you need to define classes for objects that exist to store values. The new syntax for data classes does not affect the behavior of the classes that you define with it. Each data class is a standard Python class.

Data classes were introduced in version 3.7 of Python.

PEP 557 describes data classes.

Use for Custom Collection Types

The abstract base classes in provide the components for building your own custom collection types.

Use these classes, because they are fast and well-tested. The implementations in Python 3.7 and above are written in C, to provide better performance than Python code.

Use breakpoint() for Debugging

This function drops you into the debugger at the point where it is called. Both the built-in debugger and external debuggers can use these breakpoints.

The breakpoint() feature was added in version 3.7 of Python.

PEP 553 describes the breakpoint() function.

Application Design

Use Logging for Diagnostic Messages, Rather Than print()

The built-in print() statement is convenient for adding debugging information, but you should include logging in your scripts and applications. Use the logging module in the standard library, or a third-party logging module.

Only Use asyncio Where It Makes Sense

The asynchronous features of Python enable a single process to avoid blocking on I/O operations. You can achieve concurrency by running multiple Python processes, with or without asynchronous I/O.

To run multiple Web application processes, use Gunicorn or another WSGI server. Use the multiprocessing package in the Python standard library to build custom applications that run as multiple processes.

Code that needs asynchronous I/O must not call any function in the standard library that synchronous I/O, such as open(), or the logging module.

If you would like to work with asyncio, always use the most recent version of Python. Each new version of Python has improved the performance and features of async. For example, version 3.7 of Python introduced context variables, which enable you to have data that is local to a specific task.

The initial version of asyncio was included in version 3.4 of Python. Keywords for async and await were added in Python 3.5. Context variables and the function were introduced in version 3.7 of Python.

PEP 0567 describes context variables.


Handle Command-line Input with argparse

The argparse module is now the recommended way to process command-line input. Use argparse, rather than the older optparse and getopt.

The optparse module is officially deprecated, so update code that uses optparse to use argparse instead.

Refer to the argparse tutorial in the official documentation for more details.

Use pathlib for File and Directory Paths

Use pathlib objects instead of strings whenever you need to work with file and directory pathnames.

Consider using the the pathlib equivalents for os functions.

The existing methods in the standard library have been updated to support Path objects.

To list all of the the files in a directory, use either the .iterdir() function of a Path object, or the os.scandir() function.

This RealPython article provides a full explanation of the different Python functions for working with files and directories.

The pathlib module was added to the standard library in Python 3.4, and other standard library functions were updated to support Path objects in version 3.5 of Python.

Use os.scandir() Instead of os.listdir()

The os.scandir() function is significantly faster and more efficient than os.listdir(). Use os.scandir() wherever you previously used the os.listdir() function.

This function provides an iterator, and works with a context manager:

import os

with os.scandir('some_directory/') as entries:
    for entry in entries:

The context manager frees resources as soon as the function completes. Use this option if you are concerned about performance or concurrency.

The os.walk() function now calls os.scandir(), so it automatically has the same improved performance as this function.

The os.scandir() function was added in version 3.5 of Python.

PEP 471 explains os.scandir().

Run External Commands with subprocess

The subprocess module provides a safe way to run external commands. Use subprocess rather than shell backquoting or the functions in os, such as spawn, popen2 and popen3. The function in current versions of Python is sufficient for most cases.

PEP 324 explains the technical details of subprocess in detail.

Use Requests for HTTP Clients

Use the requests package for HTTP, rather than the urllib.request in the standard library.

Test with pytest

The pytest package has superceded nose as the most popular testing system for Python. Use the unittest module in the standard library for situations where you cannot add pytest to the project.