Appendix M — Poetry - Modern Python Packaging and Dependency Management

M.1 Introduction to Poetry

Poetry is a modern Python package management tool designed to simplify dependency management and packaging in Python projects. Developed by Sébastien Eustace and released in 2018, Poetry aims to solve common problems in the Python ecosystem by providing a single tool to handle dependency installation, package building, and publishing.

Poetry’s core philosophy is to make Python packaging more deterministic and user-friendly through declarative dependency specification, lock files for reproducible environments, and simplified commands for common workflows. By combining capabilities that traditionally required multiple tools (pip, setuptools, twine, etc.), Poetry offers a more cohesive development experience.

M.2 Key Features of Poetry

M.2.1 Dependency Management

Poetry’s dependency resolution is one of its strongest features:

Deterministic builds: Poetry resolves dependencies considering the entire dependency graph, preventing many common conflicts
Lock file: The poetry.lock file ensures consistent installations across different environments
Easy version specification: Simple syntax for version constraints
Dependency groups: Organize dependencies into development, testing, and other logical groups

M.2.2 Project Setup and Configuration

Poetry uses a single configuration file for project metadata and dependencies:

pyproject.toml: All project configuration lives in one standard-compliant file
Project scaffolding: poetry new command creates a standardized project structure
Environment management: Automatic handling of virtual environments

M.2.3 Build and Publish Workflow

Poetry streamlines the package distribution process:

Unified build command: poetry build creates both source and wheel distributions
Simplified publishing: poetry publish handles uploading to PyPI
Version management: Tools to bump version numbers according to semantic versioning

M.3 Getting Started with Poetry

M.3.1 Installation

Poetry can be installed in several ways:

# Using the official installer (recommended)
curl -sSL https://install.python-poetry.org | python3 -

# Using pipx
pipx install poetry

# Using pip (not recommended for most cases)
pip install poetry

After installation, verify that Poetry is working:

poetry --version

M.3.2 Creating a New Project

To create a new project with Poetry:

# Create a new project
poetry new my-project

# Project structure created:
# my-project/
# ├── my_project/
# │   └── __init__.py
# ├── tests/
# │   └── __init__.py
# ├── pyproject.toml
# └── README.md

Alternatively, initialize Poetry in an existing project:

# Navigate to existing project
cd existing-project

# Initialize Poetry
poetry init

This interactive command helps you create a pyproject.toml file with your project’s metadata and dependencies.

M.3.3 Basic Configuration

The pyproject.toml file is the heart of a Poetry project. Here’s a sample:

[tool.poetry]
name = "my-project"
version = "0.1.0"
description = "A sample Python project"
authors = ["Your Name <your.email@example.com>"]
readme = "README.md"
packages = [{include = "my_project"}]

[tool.poetry.dependencies]
python = "^3.8"
requests = "^2.28.0"
pandas = "^2.0.0"

[tool.poetry.group.dev.dependencies]
pytest = "^7.0.0"
black = "^23.0.0"
mypy = "^1.0.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

M.4 Essential Poetry Commands

M.4.1 Managing Dependencies

# Install all dependencies
poetry install

# Install only main dependencies (no dev dependencies)
poetry install --without dev

# Add a new dependency
poetry add requests

# Add a development dependency
poetry add pytest --group dev

# Update all dependencies
poetry update

# Update specific packages
poetry update requests pandas

# Show installed packages
poetry show

# Show dependency tree
poetry show --tree

M.4.2 Environment Management

# Create/use virtual environment
poetry env use python3.10

# List available environments
poetry env list

# Get information about the current environment
poetry env info

# Remove an environment
poetry env remove python3.9

M.4.3 Building and Publishing

# Build source and wheel distributions
poetry build

# Publish to PyPI
poetry publish

# Build and publish in one step
poetry publish --build

# Publish to a custom repository
poetry publish -r my-repository

M.4.4 Running Scripts

# Run a Python script in the Poetry environment
poetry run python script.py

# Run a command defined in pyproject.toml
poetry run my-command

# Activate the shell in the Poetry environment
poetry shell

M.5 Advanced Poetry Features

M.5.1 Dependency Groups

Poetry allows organizing dependencies into logical groups:

[tool.poetry.dependencies]
python = "^3.8"
requests = "^2.28.0"

[tool.poetry.group.dev.dependencies]
pytest = "^7.0.0"
black = "^23.0.0"

[tool.poetry.group.docs.dependencies]
sphinx = "^5.0.0"
sphinx-rtd-theme = "^1.0.0"

Install specific groups:

# Install only production and docs dependencies
poetry install --without dev

# Install with specific groups
poetry install --only main,dev

M.5.2 Version Constraints

Poetry supports various version constraint syntaxes:

^1.2.3: Compatible with 1.2.3 <= version < 2.0.0
~1.2.3: Compatible with 1.2.3 <= version < 1.3.0
>=1.2.3,<1.5.0: Version between 1.2.3 (inclusive) and 1.5.0 (exclusive)
1.2.3: Exactly version 1.2.3
*: Any version

M.5.3 Private Repositories

Configure private package repositories:

# Add a repository
poetry config repositories.my-repo https://my-repository.example.com/simple/

# Add credentials
poetry config http-basic.my-repo username password

# Install from the repository
poetry add package-name --source my-repo

M.5.4 Script Commands

Define custom commands in your pyproject.toml:

[tool.poetry.scripts]
my-command = "my_package.cli:main"
start-server = "my_package.server:start"

These commands become available through poetry run or when the package is installed.

M.6 Best Practices with Poetry

M.6.1 Project Structure

A recommended project structure for Poetry projects:

my_project/
├── src/
│   └── my_package/         # Main package code
│       ├── __init__.py
│       └── module.py
├── tests/                  # Test files
│   ├── __init__.py
│   └── test_module.py
├── docs/                   # Documentation
├── pyproject.toml          # Poetry configuration
├── poetry.lock             # Lock file (auto-generated)
└── README.md               # Project documentation

To use the src layout with Poetry:

[tool.poetry]
# ...
packages = [{include = "my_package", from = "src"}]

M.6.2 Dependency Management Strategies

Minimal Version Specification: Use ^ (caret) constraint to allow compatible updates

[tool.poetry.dependencies]
requests = "^2.28.0"  # Allows any 2.x.y version >= 2.28.0

Development vs. Production Dependencies: Use groups to separate dependencies

[tool.poetry.dependencies]
# Production dependencies

[tool.poetry.group.dev.dependencies]
# Development-only dependencies

Update Strategy: Regularly update the lock file

# Update dependencies and lock file
poetry update

# Regenerate lock file based on pyproject.toml
poetry lock --no-update

M.6.3 Version Control Practices

Always commit the lock file: The poetry.lock file ensures reproducible builds

Consider a CI step to verify lock file consistency:

# In GitHub Actions
- name: Verify poetry.lock is up to date
  run: poetry lock --check

M.6.4 Integration with Development Tools

M.6.4.1 Code Formatting and Linting

Configure tools like Black and Ruff in pyproject.toml:

[tool.black]
line-length = 88
target-version = ["py39"]

[tool.ruff]
select = ["E", "F", "I"]
line-length = 88

M.6.4.2 Type Checking

Configure mypy in pyproject.toml:

[tool.mypy]
python_version = "3.9"
warn_return_any = true
disallow_untyped_defs = true

M.7 Integration with Development Workflows

M.7.1 IDE Integration

Poetry integrates well with most Python IDEs:

M.7.1.1 VS Code

Install the Python extension
Configure VS Code to use Poetry’s environment:
- It should detect the Poetry environment automatically
- Or set python.poetryPath in settings

M.7.1.2 PyCharm

Go to Settings → Project → Python Interpreter
Add the Poetry-created interpreter (typically in ~/.cache/pypoetry/virtualenvs/)
Or use PyCharm’s Poetry plugin

M.7.2 CI/CD Integration

M.7.2.1 GitHub Actions Example

name: Python CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.10"

    - name: Install Poetry
      uses: snok/install-poetry@v1
      with:
        version: "1.5.1"

    - name: Install dependencies
      run: poetry install

    - name: Run tests
      run: poetry run pytest

M.8 Troubleshooting Common Issues

M.8.1 Dependency Resolution Errors

If Poetry can’t resolve dependencies:

# Show more detailed error information
poetry install -v

# Try updating Poetry itself
poetry self update

# Try with specific versions to identify the conflict
poetry add package-name==specific.version

M.8.2 Virtual Environment Problems

For environment-related issues:

# Get environment information
poetry env info

# Create a fresh environment
poetry env remove --all
poetry install

# Use a specific Python version
poetry env use /path/to/python

M.8.3 Package Publishing Issues

When facing publishing problems:

# Verify your PyPI credentials
poetry config pypi-token.pypi your-token

# Check build before publishing
poetry build
# Examine the resulting files in dist/

# Publish with more information
poetry publish -v

M.9 Comparison with Other Tools

M.9.1 Poetry vs. pip + venv

Poetry: Single tool for environment, dependencies, and packaging
pip + venv: Separate tools for different aspects of the workflow
Key difference: Poetry adds dependency resolution and lock file

M.9.2 Poetry vs. Pipenv

Poetry: Stronger focus on packaging and publishing
Pipenv: Primarily focused on application development
Key difference: Poetry’s packaging capabilities make it more suitable for libraries

M.9.3 Poetry vs. PDM

Poetry: More opinionated, integrated experience
PDM: More standards-compliant, supports PEP 582
Key difference: Poetry’s custom installer vs. PDM’s closer adherence to PEP standards

M.9.4 Poetry vs. Hatch

Poetry: Focus on dependency management and packaging
Hatch: Focus on project management and multi-environment workflows
Key difference: Poetry’s stronger dependency resolution vs. Hatch’s project lifecycle features

M.10 When to Use Poetry

Poetry is particularly well-suited for:

Library Development: Its packaging and publishing tools shine for creating distributable packages
Team Projects: The lock file ensures consistent environments across team members
Projects with Complex Dependencies: The resolver helps manage intricate dependency requirements
Developers Wanting an All-in-One Solution: The unified interface simplifies the development workflow

Poetry might not be ideal for:

Simple Scripts: May be overkill for very small projects
Projects with Unusual Build Requirements: Complex custom build processes might need more specialized tools
Integration with Existing pip-Based Workflows: Requires adapting established processes

M.11 Conclusion

Poetry represents a significant evolution in Python package management, offering a more integrated and user-friendly approach to dependencies, environments, and packaging. Its focus on deterministic builds through the lock file mechanism and simplified workflow commands addresses many pain points in traditional Python development.

While Poetry introduces its own conventions and may require adaptation for teams used to traditional tools, the benefits in terms of reproducibility and developer experience make it worth considering for both new and existing Python projects. As the tool continues to mature and the ecosystem around it grows, Poetry is establishing itself as a standard part of the modern Python development toolkit.