Virtual environment, pip and pyproject.toml in Python

1. Overview

When working on multiple Python projects on the same machine, you’ll soon encounter version conflicts. Project A needs requests 2.28, Project B needs requests 2.32. This article guides you through three tools to solve this problem. venv isolates the environment for each project. pip installs and manages packages. pyproject.toml declares the project according to modern standards. After reading, you’ll be able to set up a Python project with proper structure, install it in editable mode, and run tests with pytest.

2. Virtual environment

A virtual environment (venv) is a directory containing three things: a separate Python interpreter (usually a symlink), a separate site-packages directory, and activation scripts. Packages you install in a venv only exist in that directory and don’t affect the system’s Python.

2.1. Creating and activating a venv

The venv module is built into the standard library since Python 3.3, so you don’t need to install anything:

mkdir shopcart && cd shopcart
python3 -m venv .venv

# Linux / macOS
source .venv/bin/activate

# Windows (PowerShell)
.venv\Scripts\Activate.ps1

The command python3 -m venv .venv creates a .venv directory in your project. Naming it .venv is a common convention, and most IDEs will recognize it automatically. After activation, your shell prompt will have a (.venv) prefix. The python and pip commands now point to the venv.

To exit the venv, run:

deactivate

2.2. Check which interpreter you’re using

This is a step many people skip, and it’s the source of many errors like “I installed it but still get ModuleNotFoundError”:

which python
python -c "import sys; print(sys.prefix); print(sys.prefix != sys.base_prefix)"

/home/lan/shopcart/.venv/bin/python
/home/lan/shopcart/.venv
True

sys.prefix is the root directory of the running environment. sys.base_prefix is the directory of the base Python installation. If they’re different, you’re in a venv.

Note: Don’t commit the .venv directory to Git. Add .venv/ to .gitignore. Venvs depend on absolute paths and the operating system, so copying to another machine usually breaks it. The correct way is to commit the dependency declaration file and recreate the venv on the new machine.

3. Managing packages with pip

pip is the default package installer and retrieves packages from PyPI.

3.1. Basic commands

python -m pip install --upgrade pip
python -m pip install "requests>=2.31,<3"
python -m pip list
python -m pip show requests
python -m pip uninstall -y requests

What each command does:

• install --upgrade pip upgrades pip itself. Older pip versions sometimes don’t read pyproject.toml correctly.
• "requests>=2.31,<3" is a version specifier. You need to quote it so the shell doesn’t interpret < as redirection.
• list shows installed packages. show prints detailed information, including version, installation location, and dependencies.

3.2. Why use python -m pip instead of pip

The bare pip command might point to a different Python than the python you’re using. This often happens when a machine has multiple Python versions or when you forget to activate the venv. The python -m pip command always runs pip from the correct interpreter, so packages are guaranteed to go into your intended environment.

3.3. requirements.txt and pip freeze

The traditional way to share a package list is using requirements.txt:

python -m pip freeze > requirements.txt
python -m pip install -r requirements.txt

certifi==2024.8.30
charset-normalizer==3.4.0
idna==3.10
requests==2.32.3
urllib3==2.2.3

pip freeze records all packages in the venv with exact versions, including indirect dependencies like urllib3. This approach is good for reproducing an exact runtime environment. However, it doesn’t tell you which packages you actually need versus which are just pulled in as dependencies. It also doesn’t describe your project: name, version, Python requirements. That’s what pyproject.toml is for.

4. Declaring a project with pyproject.toml

pyproject.toml is the standard configuration file for a Python project. It’s defined in PEP 518, PEP 621, and related PEPs. This file replaces setup.py, setup.cfg, and centralizes configuration for many tools like pytest and ruff into one place.

4.1. Project structure

We use src layout, which means putting code in src/. This structure ensures tests always run against the installed package, not accidentally importing directly from the working directory:

shopcart/
├── pyproject.toml
├── src/
│   └── shopcart/
│       ├── __init__.py
│       ├── order.py
│       └── cli.py
└── tests/
    └── test_order.py

4.2. Complete pyproject.toml file

[build-system]
requires = ["setuptools>=69"]
build-backend = "setuptools.build_meta"

[project]
name = "shopcart"
version = "0.1.0"
description = "Simple order calculation"
requires-python = ">=3.11"
dependencies = [
    "requests>=2.31,<3",
]

[project.optional-dependencies]
dev = [
    "pytest>=8",
    "ruff>=0.5",
]

[project.scripts]
shopcart = "shopcart.cli:main"

[tool.pytest.ini_options]
testpaths = ["tests"]

What each section means:

• [build-system] tells pip which tools to use for building the project. Here it’s setuptools. You could switch to hatchling or flit_core without changing anything else.
• [project] contains standard metadata. requires-python prevents installing on unsupported Python versions.
• dependencies are required packages to run the project. You should specify version ranges (>=2.31,<3) rather than pinning (==2.32.3).
• [project.optional-dependencies] groups optional packages. The dev group contains development tools that end users don’t need.
• [project.scripts] creates a shopcart command on the terminal. This command calls the main function in the shopcart.cli module.
• [tool.pytest.ini_options] is pytest-specific configuration. Every tool has its own [tool.<name>] namespace.

With src layout, setuptools automatically finds packages in src/, so you don’t need to declare anything extra.

4.3. Package code

# src/shopcart/order.py
from dataclasses import dataclass, field
from decimal import Decimal


@dataclass(frozen=True)
class OrderLine:
    product: str
    unit_price: Decimal
    quantity: int


@dataclass
class Order:
    customer: str
    lines: list[OrderLine] = field(default_factory=list)

    def add(self, product: str, unit_price: str, quantity: int = 1) -> None:
        if quantity <= 0:
            raise ValueError("quantity must be greater than 0")
        self.lines.append(OrderLine(product, Decimal(unit_price), quantity))

    def total(self) -> Decimal:
        return sum((l.unit_price * l.quantity for l in self.lines), Decimal("0"))

Order holds a list of OrderLines. Prices use Decimal to avoid floating-point errors, so unit_price is passed as a string. The total() method sums the line totals, with an initial value of Decimal("0").

# src/shopcart/cli.py
from shopcart.order import Order


def main() -> None:
    order = Order(customer="Alice Smith")
    order.add("Coffee with milk", "29000", 2)
    order.add("Bread", "25000")
    print(f"Customer: {order.customer}")
    print(f"Total: {order.total():,} VND")

The file src/shopcart/__init__.py can be empty.

4.4. Installing in editable mode

python -m pip install -e ".[dev]"
shopcart

Customer: Alice Smith
Total: 83,000 VND

The -e flag (editable) links the package to the source code instead of copying it to site-packages. This way, when you edit order.py, changes take effect immediately without reinstalling. The ".[dev]" part means install the current project plus the dev group, so pytest and ruff are also installed. The shopcart command is available in the terminal because of the [project.scripts] declaration.

Note: Some shells like zsh treat [...] as glob patterns. Always quote ".[dev]" to be safe.

5. Writing and running tests

# tests/test_order.py
from decimal import Decimal

import pytest

from shopcart.order import Order


def test_total_sums_all_lines():
    order = Order(customer="Bob Johnson")
    order.add("Peach tea", "35000", 2)
    order.add("Flan", "15000", 3)

    assert order.total() == Decimal("115000")


def test_empty_order_has_zero_total():
    assert Order(customer="Charlie Brown").total() == Decimal("0")


def test_add_rejects_non_positive_quantity():
    order = Order(customer="Diana Prince")
    with pytest.raises(ValueError, match="quantity"):
        order.add("Water", "10000", 0)

python -m pytest -q

...                                                     [100%]
3 passed in 0.02s

Tests import shopcart.order as an installed package. This is why you need the pip install -e step above. Without it, src layout will raise ModuleNotFoundError, and this is intentional: it forces you to test the same thing users will install. The third test uses pytest.raises to check that invalid quantity is rejected with a message containing “quantity”.

6. Version pinning: pyproject.toml or requirements.txt?

These two files aren’t mutually exclusive. They serve different purposes:

	pyproject.toml	Lock file (pinned requirements.txt)
Purpose	Describe what the project needs	Reproduce an exact environment
Version format	Range (>=2.31,<3)	Fixed (==2.32.3)
Used for	Libraries and applications	Applications deployed (servers, Docker)

For a library you’ll publish to PyPI, you only need pyproject.toml. If you pin versions, library users will run into conflicts.

For an application running on a server, add a lock file to ensure every deployment installs the exact same set of versions:

python -m pip install -e .
python -m pip freeze --exclude-editable > requirements.lock

When deploying, install from the lock file first, then the project, to prevent pip from automatically upgrading dependencies:

python -m pip install -r requirements.lock
python -m pip install --no-deps .

For a cleaner lock process, you can use pip-tools (command pip-compile pyproject.toml) or uv. Both read pyproject.toml directly and generate lock files with hashes.

7. Common errors

• Installing packages with sudo pip install. This overwrites system packages and can break system tools. Many modern Linux distributions block this with an externally-managed-environment error. The fix is to always use a venv.
• Forgetting to activate the venv before installing or running. Check using which python as shown in section 2.2, or call .venv/bin/python -m pip ... directly.
• Creating a venv with one Python, running with another. Venvs are tied to the interpreter that created them. When you upgrade Python (e.g., from 3.11 to 3.12), delete .venv and recreate it.
• Moving or renaming the project directory. Scripts in .venv/bin contain absolute paths, so they break after moving. You need to recreate the venv.
• Mixing old setup.py with pyproject.toml and declaring dependencies in both places. Choose a single source of truth. For new projects, that’s the [project] table.
• Putting dev tools in dependencies. pytest and ruff should go in optional-dependencies (or [dependency-groups] if your tool supports it), so end users don’t have to install them.

Note: Don’t use venv if you just want to run a standalone CLI tool like black or httpie system-wide. In that case, pipx is better: it creates a separate venv for each tool and puts the command in PATH.

8. Conclusion

This article covered the core trio of tools for a Python project:

• python -m venv .venv isolates dependencies per project.
• python -m pip installs packages into the correct interpreter.
• pyproject.toml declares metadata, dependencies, optional groups, CLI commands, and tool configuration in one file.

Combined with src layout, pip install -e ".[dev]", and pytest, you have a clean, testable, and shareable project. For applications that need stable deployments, add a lock file alongside pyproject.toml to ensure every environment installs the same set of versions.