SHIP PYTHON, ORCHESTRATE AI Professional Python in the AI Era By Michael Borck ============================================================ ============================================================ SOURCE: index.qmd ============================================================ # Preface ## Why This Book Exists You can write Python. Now what? The gap between writing scripts that work on your machine and shipping software that other people can use is enormous. The Python ecosystem offers hundreds of tools to bridge that gap — and that is exactly the problem. Which virtual environment tool? Which formatter? Which testing framework? How do you manage dependencies? How do you structure a project? The questions feel endless, and every answer leads to three more. This book grew out of my own search for a sane, repeatable workflow for Python projects. I needed a pipeline that could target multiple platforms — web apps, PWAs, Docker containers, Windows, Linux, Mac — from a single codebase. I wanted something that was simple by default but could scale when needed, without bolting on complexity for edge cases that did not yet exist. The result is what you will find in these pages: a deliberate 80/20 approach. The 20% of practices that yield 80% of the benefits. We settle on specific tools — uv, ruff, mypy, pytest — not because they are the only options, but because making a decision and committing to it is more productive than endlessly evaluating alternatives. If you are working in a large team with established conventions, you may choose differently. This book is for everyone else: solo developers, small teams, prototypers, and anyone who wants a professional workflow without the overhead. ## Who This Book Is For - Python developers ready to move beyond scripts to professional projects - Solo developers and small teams who need a complete workflow without enterprise overhead - Prototypers who want to ship to multiple platforms from one codebase - Anyone overwhelmed by Python tooling choices who wants someone to just make the decisions You should be comfortable writing Python. If you are still learning the language itself, start with Code Python, Consult AI. This book assumes you can write functions, use data structures, and debug basic errors. It teaches you what to do with that skill. ## What This Book Is Not This is not a Python tutorial. It does not teach the language. It teaches the professional practices around the language: project structure, version control, dependency management, testing, documentation, and deployment. It is not an enterprise DevOps guide. It does not cover large-team workflows, complex CI/CD orchestration, or Kubernetes. It covers what a solo developer or small team needs to ship reliable software. The practices scale, but the book does not add complexity for situations that may never apply to you. It is not a tool comparison. It does not present five options for every choice and leave you to decide. It picks one, explains why, and moves on. Simple but not simplistic — that is the guiding principle. And it is not a book that ignores AI. But it uses AI differently from the other books in this series. Code Python, Consult AI teaches you to think with AI, to explore concepts and build understanding through conversation. This book teaches you to orchestrate AI, to set up the project structure, testing, and automation so that AI-generated code has somewhere safe to land. Without a proper pipeline, AI produces code you cannot test, cannot deploy, and cannot maintain. With one, AI becomes a force multiplier. The pipeline is what makes the difference. ## If You Are Feeling Overwhelmed That is the normal response to the Python ecosystem. The number of tools, frameworks, and opinions about "the right way" to do things is genuinely staggering. This book exists to cut through that noise. You do not need to evaluate every option. You need one good workflow that works, and the understanding to adapt it when your needs change. ## How This Book Is Structured The book builds a complete development pipeline in stages: 1. Setting the Foundation — project structure, version control, virtual environments 2. Advancing Your Workflow — dependency management, code quality (ruff), testing (pytest), type checking (mypy) 3. Documentation and Deployment — documentation, CI/CD automation 4. Case Study — applying everything to a real project (SimpleBot) 5. Multi-Platform Distribution — shipping to web, desktop, PWA, and containers from one codebase Companion templates let you start new projects with the recommended structure immediately: - Cookiecutter template: `cookiecutter gh:michael-borck/ship-python-cookiecutter` - GitHub template: ship-python-template - Example project: ship-python-example ## Conventions Used in This Book Code examples appear with syntax highlighting: Terminal output and configuration content appear as plain text blocks: Throughout the chapters you will find: - AI Tips — practical advice for using AI with the practice being taught - Callout boxes — important notes, tips, and warnings highlighted for reference ## The Series This book is part of a series designed to help you master modern software development in the AI era. Conversation, Not Delegation — the general methodology for working with AI across any discipline. Converse Python, Partner AI — intentional prompting methodology applied to software development. Think Python, Direct AI — computational thinking for absolute beginners. Code Python, Consult AI — focused Python fundamentals with AI integration. Ship Python, Orchestrate AI (this book) — professional Python development practices and tooling. Build Web, Guide AI — web development with AI as your development partner. All titles are available at books.borck.education. ## Ways to Engage with This Book This book is available in several formats. Pick whichever fits how you work and learn. - Read it online. The full book is freely available at the companion website, with dark mode, search, and navigation. - Read it on paper or e-reader. Available as a paperback and ebook through Amazon KDP. - Converse with it. The online edition includes a chatbot grounded in the book's content. - Feed it to your own AI. The `llm.txt` file provides a clean text version of the entire book, ready to paste into ChatGPT, Claude, or any AI tool. - Run the code. All code examples and templates are available on GitHub. DeepWiki provides an AI-navigable view of the repository. - Browse all books. This book is part of a series. See all titles at books.borck.education. The online version is always the most current. ## Source Code & Feedback All code examples, templates, and supplementary materials are available at: - This book's repository: https://github.com/michael-borck/ship-python-orchestrate-ai - Cookiecutter template: `cookiecutter gh:michael-borck/ship-python-cookiecutter` - GitHub template: https://github.com/michael-borck/ship-python-template - Example project: https://github.com/michael-borck/ship-python-example Found an error? Have a suggestion? - Open an issue: https://github.com/michael-borck/ship-python-orchestrate-ai/issues - Email: michael@borck.me ============================================================ SOURCE: copyright.qmd ============================================================ # Copyright \thispagestyle{empty} Ship Python, Orchestrate AI: Professional Python in the AI Era Copyright 2026 Michael Borck. All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the author, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law. First Edition, 2026 While every precaution has been taken in the preparation of this book, the author assumes no responsibility for errors or omissions, or for damages resulting from the use of information contained herein. --- Cover Art Created using AI image generation, with prompts crafted by the author depicting the themes of professional Python development and human-AI collaboration. Production Notes This book was written in Markdown using Quarto. The text is set in system fonts optimised for screen and print reading. Code examples use a monospace font for clarity. Editorial assistance provided by Claude (Anthropic). The author reviewed and approved all content. --- Online Edition A free online version of this book is available at: https://michael-borck.github.io/ship-python-orchestrate-ai --- michael@borck.me \newpage ============================================================ SOURCE: chapters/foundation.qmd ============================================================ # Setting the Foundation ## Python Project Structure Best Practices A well-organised project structure is the cornerstone of maintainable Python code. Even before writing a single line of code, decisions about how to organise your files will impact how easily you can test, document, and expand your project. The structure we recommend follows modern Python conventions, prioritizing clarity and separation of concerns: ### Why Use the `src` Layout? The `src` layout (placing your package inside a `src` directory rather than at the project root) provides several advantages: 1. Enforces proper installation: When developing, you must install your package to use it, ensuring you're testing the same way users will experience it. 2. Prevents accidental imports: You can't accidentally import from your project without installing it, avoiding confusing behaviours. 3. Clarifies package boundaries: Makes it explicit which code is part of your distributable package. While simpler projects might skip this layout, adopting it early builds good habits and makes future growth easier. Ask your AI assistant to generate a project structure tailored to your needs: "I'm building a CLI tool that processes CSV files. Generate the directory structure and initial files for a Python project using the src layout, including tests and a pyproject.toml." AI is excellent at generating boilerplate — the value you add is knowing whether the structure it suggests follows the conventions in this chapter. ### Key Components Explained - src/my_package/: Contains your actual Python code. The package name should be unique and descriptive. - tests/: Keeps tests separate from implementation but adjacent in the repository. - docs/: Houses documentation, starting simple and growing as needed. - .gitignore: Tells Git which files to ignore (like virtual environments, cache files, etc.). - README.md: The first document anyone will see—provide clear instructions on installation and basic usage. - requirements.txt: Lists your project's dependencies. We'll explore more advanced dependency management techniques in Part 2. - pyproject.toml: Configuration for development tools like Ruff and mypy, following modern standards. ### Getting Started Creating this structure is straightforward. Here's how to initialize a basic project: ### Applications vs. Packages: Knowing Your Project Type Understanding whether you're building an application or a package influences structure decisions and development priorities: Python Applications are end-user focused programs: - Web applications (Django/Flask projects) - Command-line tools and utilities - Desktop applications - Data processing scripts - Have clear entry points and user interfaces - Often include configuration files and deployment considerations Python Packages are developer-focused libraries: - Reusable code modules (like `requests` or `pandas`) - APIs and frameworks - Plugin systems - Focus on import interfaces and documentation - Published to PyPI for others to use Key Differences in Practice: | Aspect | Applications | Packages | |--------|-------------|----------| | Entry point | `main.py`, CLI commands | Import interfaces | | Dependencies | Can pin exact versions | Should use flexible ranges | | Documentation | User guides, deployment | API docs, examples | | Testing focus | End-to-end workflows | Unit tests, edge cases | | Configuration | Settings files, env vars | Initialization parameters | Most projects start as applications and may later extract reusable components into packages. Our recommended structure accommodates both paths—you can begin with application-focused development and naturally evolve toward package-like modularity as your codebase matures. This chapter focuses on script-based projects, but notebooks (Jupyter, Colab) are also a legitimate way to ship Python work. For data scientists, educators, and researchers, a well-structured notebook with a sharing link is shipping. We cover notebooks, dashboards, and interactive tools in . Practical example: A data analysis script (application) might extract its core algorithms into a separate analytics package, while keeping the command-line interface and configuration handling in the main application. This structure promotes maintainability and follows Python's conventions. It might seem like overkill for tiny scripts, but as your project grows, you'll appreciate having this organisation from the start. In the next section, we'll build on this foundation by implementing version control best practices. ## Version Control Fundamentals Version control is an essential part of modern software development, and Git has become the de facto standard. Even for small solo projects, proper version control offers invaluable benefits for tracking changes, experimenting safely, and maintaining a clear history of your work. ### Setting Up Git If you haven't set up Git yet, here's how to get started: A good `.gitignore` file is essential for Python projects. Here's a simplified version to start with: ### Basic Git Workflow For beginners, a simple Git workflow is sufficient: 1. Make changes to your code 2. Stage changes you want to commit 3. Commit with a meaningful message 4. Push to a remote repository (like GitHub) Here's what this looks like in practice: ### Effective Commit Messages Good commit messages are vital for understanding project history. Follow these simple guidelines: 1. Use the imperative mood ("Add feature" not "Added feature") 2. Keep the first line under 50 characters as a summary 3. When needed, add more details after a blank line 4. Explain why a change was made, not just what changed Example of a good commit message: ### Branching for Features and Fixes As your project grows, a branching workflow helps manage different streams of work: For team projects, consider using pull/merge requests on platforms like GitHub or GitLab rather than direct merges to the main branch. This enables code review and discussion before changes are incorporated. ### Integrating with GitHub or GitLab Hosting your repository on GitHub, GitLab, or similar services provides: 1. A backup of your code 2. Collaboration tools (issues, pull requests) 3. Integration with CI/CD services 4. Visibility for your project To connect your local repository to GitHub: ### Git Best Practices for Beginners 1. Commit frequently: Small, focused commits are easier to understand and review 2. Never commit sensitive data: Passwords, API keys, etc. should never enter your repository 3. Pull before pushing: Always integrate others' changes before pushing your own 4. Use meaningful branch names: Names like `feature/user-login` or `fix/validation-bug` explain the purpose AI assistants can help you write commit messages, generate `.gitignore` files for specific frameworks, and explain Git commands you are unsure about. Try: "I just added user authentication with Flask-Login. Write a commit message following conventional commit format." But always review what AI generates — it does not know your project's commit history or conventions unless you provide them. Version control may seem like an overhead for very small projects, but establishing these habits early will pay dividends as your projects grow in size and complexity. It's much easier to start with good practices than to retrofit them later. In the next section, we'll set up a virtual environment and explore basic dependency management to isolate your project and manage its requirements. ## Virtual Environments and Basic Dependencies Python's flexibility with packages and imports is powerful, but can quickly lead to conflicts between projects. Virtual environments solve this problem by creating isolated spaces for each project's dependencies. ### Understanding Virtual Environments A virtual environment is an isolated copy of Python with its own packages, separate from your system Python installation. This isolation ensures: * Different projects can use different versions of the same package * Installing a package for one project won't affect others * Your development environment closely matches production ### Setting Up a Virtual Environment with `venv` Python comes with `venv` built in, making it the simplest way to create virtual environments: Once activated, any packages you install will be confined to this environment. When you're done working on the project, you can deactivate the environment: Using `.venv` as the environment name (with the leading dot) makes it hidden in many file browsers, reducing clutter. Make sure `.venv/` is in your `.gitignore` file — you never want to commit this directory. ### Basic Dependency Management With your virtual environment active, you can install packages using pip: When working on a team project or deploying to production, you'll need to track and share these dependencies. For basic projects, you can manually maintain a `requirements.txt` file with the packages you need: This approach works well for simple projects, especially when you're just getting started. However, as we'll see in Part 2, there are limitations to this basic method: * It doesn't handle indirect dependencies (dependencies of your dependencies) automatically * It doesn't distinguish between your project's requirements and development tools * It doesn't provide version locking for reproducible environments In Part 2, we'll explore more robust dependency management with tools like pip-tools and uv, which solve these limitations by creating proper "lock files" while maintaining a clean list of direct dependencies. We'll also see how these tools help ensure deterministic builds — a crucial feature as your projects grow in complexity. ### Practical Example: Setting Up a New Project Let's combine what we've learned so far with a practical example. Here's how to set up a new project with good practices: ## Jumpstarting Your Projects with Templates Now that we've covered the essential foundation for Python development, you might be wondering how to apply these practices efficiently when starting new projects. Rather than recreating this structure manually each time, we offer two approaches to jumpstart your projects: ### Simple Scaffolding Script For those who prefer a transparent, straightforward approach, we've created a simple bash script that creates the basic project structure we've discussed: This script creates a minimal but well-structured Python project with: - The recommended `src` layout - Basic test setup - Simple `pyproject.toml` configuration - Version control initialization - Placeholder documentation The script is intentionally simple and readable, allowing you to understand exactly what's happening and modify it for your specific needs. This approach is ideal for learning or for smaller projects where you want maximum visibility into the setup process. ### Cookiecutter Template (For More Comprehensive Setup) For more complex projects or when you want a more feature-rich starting point, we also provide a cookiecutter template that implements the full development pipeline described throughout this book: The cookiecutter template offers more customisation options and includes: - All the foundational structure from the simple script - Comprehensive tool configurations - Optional documentation setup with MkDocs - CI/CD workflow configurations - Advanced dependency management - Security scanning integration This approach is covered in detail in Appendix C and is recommended when you're ready to adopt more advanced practices or when working with larger teams. ### GitHub Repository Templates (For No-Installation Simplicity) For the ultimate in simplicity, we also provide a GitHub repository template that requires no local tool installation. GitHub templates offer a frictionless way to create new projects with the same structure and files: 1. Visit the template repository at https://github.com/michael-borck/ship-python-template 2. Click the "Use this template" button 3. Name your new repository and create it 4. Clone your new repository locally While GitHub templates don't offer the same parameterization as cookiecutter (file contents remain exactly as they were in the template), they provide the lowest barrier to entry for getting started with a well-structured project. After creating your repository from the template, you can manually customise file contents like project name, author information, and other details. The GitHub template includes: - The recommended `src` layout - Basic test structure - `.gitignore` and `pyproject.toml` configuration - Documentation structure - Example code and tests This approach is ideal for quickly starting new projects when you don't want to install additional tools or when you're introducing others to Python best practices with minimal setup overhead. All these options—the simple script, the cookiecutter template—embody, and GitHub repository templates embody our philosophy of "Simple but not Simplistic." Choose the option that best fits your current needs and comfort level. As your projects grow in complexity, you can gradually adopt more sophisticated practices while maintaining the solid foundation established here. In Part 2, we'll build on this foundation by exploring robust dependency management, code quality tools, testing strategies, and type checking — the next layers in our Python development pipeline. ============================================================ SOURCE: chapters/workflow.qmd ============================================================ # Advancing Your Workflow ## Robust Dependency Management with pip-tools and uv As your projects grow in complexity or involve more developers, the basic `pip freeze > requirements.txt` approach starts to show limitations. You need a dependency management system that gives you more control and ensures truly reproducible environments. ### The Problem with `pip freeze` While `pip freeze` is convenient, it has several drawbacks: 1. No distinction between direct and indirect dependencies: You can't easily tell which packages you explicitly need versus those that were installed as dependencies of other packages. 2. Maintenance challenges: When you want to update a package, you may need to regenerate the entire requirements file, potentially changing packages you didn't intend to update. 3. No environment synchronization: Installing from a requirements.txt file adds packages but doesn't remove packages that are no longer needed. 4. No explicit dependency specification: You can't easily specify version ranges (e.g., "I need any Django 4.x version") or extras. Let's explore two powerful solutions: `pip-tools` and `uv`. ### Solution 1: pip-tools pip-tools introduces a two-file approach to dependency management: 1. `requirements.in`: A manually maintained list of your direct dependencies, potentially with version constraints. 2. `requirements.txt`: A generated lock file containing exact versions of all dependencies (direct and indirect). #### Getting Started with pip-tools The generated `requirements.txt` will contain exact versions of your specified packages plus all their dependencies, including hashes for security. #### Managing Development Dependencies For a cleaner setup, you can separate production and development dependencies: #### Updating Dependencies When you need to update packages: ### Solution 2: uv `uv` is a newer, Rust-based tool that provides significant speed improvements while maintaining compatibility with existing Python packaging standards. It combines environment management, package installation, and dependency resolution in one tool. #### Getting Started with uv #### Key Advantages of uv 1. Speed: uv is significantly faster than standard pip and pip-tools, especially for large dependency trees. 2. Global caching: uv implements efficient caching, reducing redundant downloads across projects. 3. Consolidated tooling: Acts as a replacement for multiple tools (pip, pip-tools, virtualenv) with a consistent interface. 4. Enhanced dependency resolution: Often provides clearer error messages for dependency conflicts. #### Managing Dependencies with uv uv supports the same workflow as pip-tools but with different commands: ### Choosing Between pip-tools and uv Both tools solve the core problem of creating reproducible environments, but with different tradeoffs: | Factor | pip-tools | uv | |--------|-----------|---| | Speed | Good | Excellent (often 10x+ faster) | | Installation | Simple Python package | External tool (but simple to install) | | Maturity | Well-established | Newer but rapidly maturing | | Functionality | Focused on dependency locking | Broader tool combining multiple functions | | Learning curve | Minimal | Minimal (designed for compatibility) | For beginners or smaller projects, pip-tools offers a gentle introduction to proper dependency management with minimal new concepts. For larger projects or when speed becomes important, uv provides significant benefits with a similar workflow. ### Best Practices for Either Approach Regardless of which tool you choose: 1. Commit both `.in` and `.txt` files to version control. The `.in` files represent your intent, while the `.txt` files ensure reproducibility. 2. Use constraints carefully. Start with loose constraints (just package names) and add version constraints only when needed. 3. Regularly update dependencies to get security fixes, using `--upgrade` or `--upgrade-package`. 4. Always use `pip-sync` or `uv pip sync` instead of `pip install -r requirements.txt` to ensure your environment exactly matches the lock file. When choosing between packages that serve similar purposes, ask your AI assistant for a comparison: "Compare requests vs httpx for making HTTP calls in a Python 3.11 project. I need async support and type hints." AI can surface trade-offs quickly, but verify its claims about package popularity and maintenance status — check PyPI download stats and GitHub activity yourself. In the next section, we'll explore how to maintain code quality through automated formatting and linting with Ruff, taking your workflow to the next professional level. ## Code Quality Tools with Ruff Writing code that works is only part of the development process. Code should also be readable, maintainable, and free from common errors. This is where code quality tools come in, helping you enforce consistent style and catch potential issues early. ### The Evolution of Python Code Quality Tools Traditionally, Python developers used multiple specialized tools: - Black for code formatting - isort for import sorting - Flake8 for linting (style checks) - Pylint for deeper static analysis While effective, maintaining configuration for all these tools was cumbersome. Enter Ruff – a modern, Rust-based tool that combines formatting and linting in one incredibly fast package. ### Why Ruff? Ruff offers several compelling advantages: 1. Speed: Often 10-100x faster than traditional Python linters 2. Consolidation: Replaces multiple tools with one consistent interface 3. Compatibility: Implements rules from established tools (Flake8, Black, isort, etc.) 4. Configuration: Single configuration in your pyproject.toml file 5. Automatic fixing: Can automatically fix many issues it identifies ### Getting Started with Ruff First, install Ruff in your virtual environment: ### Basic Configuration Configure Ruff in your `pyproject.toml` file: This configuration enables: - `E` rules from pycodestyle (PEP 8 style guide) - `F` rules from Pyflakes (logical and syntax error detection) - `I` rules for import sorting (like isort) ### Using Ruff in Your Workflow Ruff provides two main commands: To automatically fix issues that Ruff can solve: ### Hands-on: Setting Up Ruff Step-by-Step Let's walk through a practical example that demonstrates Ruff's impact on code quality. Starting with some intentionally messy Python code: This code has several quality issues: - Multiple imports on one line - Inconsistent spacing around operators - Missing spaces in type hints - Unused imports and variables - Inconsistent string quote styles First, add Ruff to your project: Now configure Ruff in your `pyproject.toml`: Run Ruff to identify issues: This will show output like: ```text example.py:2:1: E401 Multiple imports on one line example.py:2:8: F401 `sys` imported but unused example.py:4:1: F401 `json` imported but unused example.py:15:5: F841 Local variable `unused_var` is assigned to but never used bash uv run ruff check --fix example.py uv run ruff format example.py python # example.py - After Ruff import os from pathlib import Path def calculate_average(numbers: list) -> float: return sum(numbers) / len(numbers) if __name__ == "__main__": data = [1, 2, 3, 4, 5] result = calculate_average(data) print(f"Average: {result}") ``` Notice the improvements: - Unused imports automatically removed - Imports properly sorted and formatted - Consistent spacing around operators and type hints - Proper string quote style - Clean, readable formatting ### Integrating Ruff with Pre-commit Hooks To automatically apply these fixes before each commit, add this to your `.pre-commit-config.yaml`: Install and activate the hooks: Now Ruff will automatically clean up your code before each commit, ensuring consistent quality across your entire project. ### Real-world Configuration Example Here's a more comprehensive configuration that balances strictness with practicality: ### Integrating Ruff into Your Editor Ruff provides editor integrations for: - VS Code (via the Ruff extension) - PyCharm (via third-party plugin) - Vim/Neovim - Emacs For example, in VS Code, install the Ruff extension and add to your settings.json: This configuration automatically formats code and fixes issues whenever you save a file. ### Gradually Adopting Ruff If you're working with an existing codebase, you can adopt Ruff gradually: 1. Start with formatting only: Begin with `ruff format` to establish consistent formatting 2. Add basic linting: Enable a few rule sets like `E`, `F`, and `I` 3. Gradually increase strictness: Add more rule sets as your team adjusts 4. Use per-file ignores: For specific issues in specific files ### Enforcing Code Quality in CI Add Ruff to your CI pipeline to ensure code quality standards are maintained: The `--check` flag on `ruff format` makes it exit with an error if files would be reformatted, instead of actually changing them. ### Beyond Ruff: When to Consider Other Tools While Ruff covers a wide range of code quality checks, some specific needs might require additional tools: - mypy for static type checking (covered in a later section) - bandit for security-focused checks - vulture for finding dead code However, Ruff's rule set continues to expand, potentially reducing the need for these additional tools over time. When Ruff flags an error you do not understand, paste the rule code into your AI assistant: "What does Ruff rule B006 mean, and how do I fix it in this code?" AI is particularly good at explaining why a pattern is problematic and suggesting the idiomatic alternative. This turns linting errors into learning opportunities. By incorporating Ruff into your workflow, you'll catch many common errors before they reach production and maintain a consistent, readable codebase. In the next section, we'll explore how to ensure your code works as expected through automated testing with pytest. ## Automated Testing with pytest Testing is a crucial aspect of software development that ensures your code works as intended and continues to work as you make changes. Python's testing ecosystem offers numerous frameworks, but pytest has emerged as the most popular and powerful choice for most projects. ### Why Testing Matters Automated tests provide several key benefits: 1. Verification: Confirm that your code works as expected 2. Regression prevention: Catch when changes break existing functionality 3. Documentation: Tests demonstrate how code is meant to be used 4. Refactoring confidence: Change code structure while ensuring behaviour remains correct 5. Design feedback: Difficult-to-test code often indicates design problems ### Getting Started with pytest Add pytest as a development dependency to your project: ### Setting Up a Testing Project Structure Create a proper test directory structure in your project: Your project structure should look like: ### Writing Your First Test Let's assume you have a simple function in `src/my_package/calculations.py`: Create a test file in `tests/test_calculations.py`: ### Running Tests Run all tests from your project root: ### pytest Features That Make Testing Easier pytest has several features that make it superior to Python's built-in unittest framework: #### 1. Simple Assertions Instead of methods like `assertEqual` or `assertTrue`, pytest lets you use Python's built-in `assert` statement, making tests more readable. #### 2. Fixtures Fixtures are a powerful way to set up preconditions for your tests: #### 3. Parameterized Tests Test multiple inputs without repetitive code: #### 4. Marks for Test organisation organise tests with marks: ### Test Coverage Track which parts of your code are tested using pytest-cov: A coverage report helps identify untested code: ### Configuring pytest for Your Project Set up pytest configuration in your `pyproject.toml` to customise default behaviour: This configuration provides several benefits: 1. Automatic coverage: Every test run includes coverage reporting 2. Clean output: Suppresses unnecessary warnings while still showing errors 3. Test organisation: Markers help categorize and selectively run tests 4. Consistent behaviour: Same settings for all developers With this configuration, running `uv run pytest` automatically: - Discovers tests in the `tests/` directory - Calculates code coverage for your `src/` directory - Generates both terminal and HTML coverage reports - Applies your chosen settings consistently ### Testing Best Practices 1. Write tests as you develop: Don't wait until the end 2. Name tests clearly: Include the function name and scenario being tested 3. One assertion per test: Focus each test on a single behaviour 4. Test edge cases: Empty input, boundary values, error conditions 5. Avoid test interdependence: Tests should work independently 6. Mock external dependencies: APIs, databases, file systems 7. Keep tests fast: Slow tests get run less often ### Common Testing Patterns #### Testing Exceptions Verify that your code raises the right exceptions: #### Testing with Temporary Files Test file operations safely: #### Mocking Isolate your code from external dependencies using the pytest-mock plugin: ### Testing Strategy As your project grows, organise tests into different categories: 1. Unit tests: Test individual functions/classes in isolation 2. Integration tests: Test interactions between components 3. Functional tests: Test entire features from a user perspective 4. End-to-end (E2E) tests: Test the full application as a user would interact with it Most projects should have a pyramid shape: many unit tests, fewer integration tests, and even fewer functional/E2E tests. AI excels at generating test cases, especially edge cases you might miss. Try: "Write pytest tests for this function, including edge cases for empty input, None values, and boundary conditions." Then paste your function. The key is reviewing what AI generates — it often writes tests that pass trivially or miss the actual business logic. Use AI-generated tests as a starting point, not a finished product. ## GUI and End-to-End Testing with Playwright When your project has a web frontend or a desktop interface (such as an Electron app), consider adding end-to-end tests using Playwright. Playwright lets you automate real browser interactions—clicking buttons, filling forms, verifying that pages render correctly—giving you confidence that your application works as users will actually experience it. We cover Playwright in detail in the Multi-Platform Distribution chapter, where we show how to test web deployments, PWAs, and Electron desktop applications. ### Continuous Testing Make testing a habitual part of your workflow: 1. Run relevant tests as you code: Many editors integrate with pytest 2. Run full test suite before committing: Use pre-commit hooks 3. Run tests in CI: Catch issues that might only appear in different environments By incorporating comprehensive testing into your development process, you'll catch bugs earlier, ship with more confidence, and build a more maintainable codebase. In the next section, we'll explore static type checking with mypy, which can help catch a whole new category of errors before your code even runs. ## Type Checking with mypy Python is dynamically typed, which provides flexibility but can also lead to type-related errors that only appear at runtime. Static type checking with mypy adds an extra layer of verification, catching many potential issues before your code executes. ### Understanding Type Hints Python 3.5+ supports type hints, which are annotations indicating what types of values functions expect and return: These annotations don't change how Python runs—they're ignored by the interpreter at runtime. However, tools like mypy can analyse them statically to catch potential type errors. ### Getting Started with mypy First, install mypy in your development environment: Let's check a simple example: Run mypy to check: Output: mypy caught the type mismatch without running the code! ### Configuring mypy Configure mypy in your `pyproject.toml` file for a consistent experience: Start with a lenient configuration and gradually increase strictness: ### Gradual Typing One major advantage of Python's type system is gradual typing—you can add types incrementally: 1. Start with critical or error-prone modules 2. Add types to public interfaces first 3. Increase type coverage over time ### Essential Type Annotations #### Basic Types #### Function Annotations #### Class Annotations ### Advanced Type Hints #### Union Types Use Union to indicate multiple possible types (use the `|` operator in Python 3.10+): #### Optional and None `Optional[T]` is equivalent to `Union[T, None]` or `T | None`: #### Type Aliases Create aliases for complex types: #### Callable Type hint for functions: ### Common Challenges and Solutions #### Working with Third-Party Libraries Not all libraries provide type hints. For popular packages, you can often find stub files: For others, you can silence mypy warnings selectively: #### Dealing with Dynamic Features Python's dynamic features can be challenging to type. Use `Any` when necessary: ### Integration with Your Workflow #### Running mypy #### Integrating with CI/CD Add mypy to your continuous integration workflow: #### Editor Integration Most Python-friendly editors support mypy: - VS Code: Use the Pylance extension - PyCharm: Has built-in type checking - vim/neovim: Use ALE or similar plugins ### The Broader Type Checking Landscape While mypy remains the most widely adopted and beginner-friendly type checker, Python's type checking ecosystem is rapidly evolving. Other notable options include: - pyright/pylance: Microsoft's fast, strict type checker that powers VS Code's Python extension - basedmypy: A mypy fork with stricter defaults and additional features - basedpyright: An even more aggressive fork of pyright - ty: Astral's upcoming type checker (from the makers of ruff and uv), with an alpha preview expected by PyCon 2025 For learning and establishing good type annotation habits, mypy provides an excellent foundation with extensive documentation and community support. As your expertise grows, you can explore these alternatives to find the right balance of speed, strictness, and features for your projects. ### Benefits of Type Checking 1. Catch errors early: Find type-related bugs before running code 2. Improved IDE experience: Better code completion and refactoring 3. Self-documenting code: Types serve as documentation 4. Safer refactoring: Change code with more confidence 5. Gradual adoption: Add types where they provide the most value ### When to Use Type Hints Type hints are particularly valuable for: - Functions with complex parameters or return values - Public APIs used by others - Areas with frequent bugs - Critical code paths - Large codebases with multiple contributors Type checking isn't an all-or-nothing proposition. Even partial type coverage can significantly improve code quality and catch common errors. Start small, focus on interfaces, and expand your type coverage as your team becomes comfortable with the system. AI is excellent at adding type hints to untyped code. Paste a function and ask: "Add type hints to this function, including the return type." AI can also help you understand complex type annotations like `Optional[dict[str, list[int]]]`. For larger codebases, use AI to type-annotate one module at a time rather than trying to do everything at once. ## Security Analysis with Bandit Software security is a critical concern in modern development, yet it's often overlooked until problems arise. Bandit is a tool designed to find common security issues in Python code through static analysis. ### Understanding Security Static Analysis Unlike functional testing or linting, security-focused static analysis looks specifically for patterns and practices that could lead to security vulnerabilities: - Injection vulnerabilities - Use of insecure functions - Hardcoded credentials - Insecure cryptography - And many other security issues ### Getting Started with Bandit First, install Bandit in your virtual environment: Run a basic scan: ### Security Issues Bandit Can Detect Bandit identifies a wide range of security concerns, including: #### 1. Hardcoded Secrets #### 2. SQL Injection #### 3. Shell Injection #### 4. Insecure Cryptography #### 5. Unsafe Deserialization ### Configuring Bandit You can configure Bandit using a `.bandit` file or your `pyproject.toml`: The most critical findings are categorized with high severity and confidence levels: ### Integrating Bandit in Your Workflow #### Add Bandit to CI/CD Add security scanning to your continuous integration pipeline: #### Pre-commit Hook Configure a pre-commit hook to run Bandit before commits: ### Responding to Security Findings When Bandit identifies security issues: 1. Understand the risk: Read the detailed explanation to understand the potential vulnerability 2. Fix high-severity issues immediately: These represent significant security risks 3. Document deliberate exceptions: If a finding is a false positive, document why and use an inline ignore comment 4. Review regularly: Security standards evolve, so regular scanning is essential ### False Positives Like any static analysis tool, Bandit can produce false positives. You can exclude specific findings: By incorporating security scanning with Bandit, you add an essential layer of protection against common security vulnerabilities, helping to ensure that your code is not just functional but also secure. ## Finding Dead Code with Vulture As projects evolve, code can become obsolete but remain in the codebase, creating maintenance burdens and confusion. Vulture is a static analysis tool that identifies unused code – functions, classes, and variables that are defined but never used. ### The Problem of Dead Code Dead code creates several issues: 1. Maintenance overhead: Every line of code needs maintenance 2. Cognitive load: Developers need to understand code that serves no purpose 3. False security: Tests might pass while dead code goes unchecked 4. Misleading documentation: Dead code can appear in documentation generators ### Getting Started with Vulture Install Vulture in your virtual environment: Run a basic scan: ### What Vulture Detects Vulture identifies: #### 1. Unused Variables #### 2. Unused Functions #### 3. Unused Classes #### 4. Unused Imports ### Handling False Positives Vulture can sometimes flag code that's actually used but in ways it can't detect. Common cases include: - Classes used through reflection - Functions called in templates - Code used in an importable public API You can create a whitelist file to suppress these reports: Run Vulture with the whitelist: ### Configuration and Integration Add Vulture to your workflow: #### Command Line Options #### CI Integration ### Best Practices for Dead Code Removal 1. Verify before removing: Confirm the code is truly unused 2. Use version control: Remove code through proper commits with explanations 3. Update documentation: Ensure documentation reflects the changes 4. Run tests: Confirm nothing breaks when the code is removed 5. Look for patterns: Clusters of dead code often indicate larger architectural issues ### When to Run Vulture - Before major refactoring - During codebase cleanup - As part of regular maintenance - When preparing for a significant release - When onboarding new team members (helps them focus on what matters) Regularly checking for and removing dead code keeps your codebase lean and maintainable. It also provides insights into how your application has evolved and may highlight areas where design improvements could be made. With these additional security and code quality tools in place, your Python development workflow is now even more robust. Let's move on to Part 3, where we'll explore documentation and deployment options. ============================================================ SOURCE: chapters/documentation.qmd ============================================================ # Documentation and Deployment ## Documentation Options: From pydoc to MkDocs Documentation is often neglected in software development, yet it's crucial for ensuring others (including your future self) can understand and use your code effectively. Python offers a spectrum of documentation options, from simple built-in tools to sophisticated documentation generators. ### Starting Simple with Docstrings The foundation of Python documentation is the humble docstring - a string literal that appears as the first statement in a module, function, class, or method: Docstrings become particularly useful when following a consistent format. Common conventions include: - Google style (shown above) - NumPy style (similar to Google style but with different section headers) - reStructuredText (used by Sphinx) ### Viewing Documentation with pydoc Python's built-in `pydoc` module provides a simple way to access documentation: You can also generate basic HTML documentation: While simple, this approach has limitations: - Minimal styling - No cross-linking between documents - Limited navigation options For beginner projects, however, it provides a fast way to make documentation available with zero dependencies. ### Simple Script for Basic Documentation Site For slightly more organised documentation than plain pydoc, you can create a simple script that: 1. Generates pydoc HTML for all modules 2. Creates a basic index.html linking to them Here's a minimal example script (`build_docs.py`): This script generates slightly more organised documentation than raw pydoc but still leverages built-in tools. ### Moving to MkDocs for Comprehensive Documentation When your project grows and needs more sophisticated documentation, MkDocs provides an excellent balance of simplicity and features. MkDocs generates a static site from Markdown files, making it easy to write and maintain documentation. #### Getting Started with MkDocs First, install MkDocs and a theme: Initialize a new documentation project: This creates a `mkdocs.yml` configuration file and a `docs/` directory with an `index.md` file. #### Basic Configuration Edit `mkdocs.yml`: #### Creating Documentation Content MkDocs uses Markdown files for content. Create `docs/user-guide/installation.md`: #### Testing Documentation Locally Preview your documentation while writing: This starts a development server at http://127.0.0.1:8000 that automatically refreshes when you update files. #### Building and Deploying Documentation Generate static HTML files: This creates a `site/` directory with the HTML documentation site. For GitHub projects, you can publish to GitHub Pages: ### Hosting Documentation with GitHub Pages GitHub Pages provides a simple, free hosting solution for your project documentation that integrates seamlessly with your GitHub repository. #### Setting Up GitHub Pages There are two main approaches to hosting documentation on GitHub Pages: 1. Repository site: Serves content from a dedicated branch (typically `gh-pages`) 2. User/organisation site: Serves content from a special repository named `username.github.io` For most Python projects, the repository site approach works best: 1. Go to your repository on GitHub 2. Navigate to Settings → Pages 3. Under "Source", select your branch (either `main` or `gh-pages`) 4. Choose the folder that contains your documentation (`/` or `/docs`) 5. Click Save Your documentation will be published at `https://username.github.io/repository-name/`. #### Automating Documentation Deployment MkDocs has built-in support for GitHub Pages deployment: This command: 1. Builds your documentation into the `site/` directory 2. Creates or updates the `gh-pages` branch 3. Pushes the built site to that branch 4. GitHub automatically serves the content For a fully automated workflow, integrate this into your GitHub Actions CI pipeline: This workflow automatically deploys your documentation whenever you push changes to documentation files on the main branch. #### GitHub Pages with pydoc Even if you're using the simpler pydoc approach, you can still host the generated HTML on GitHub Pages: 1. Create a `docs/` folder in your repository 2. Generate HTML documentation with pydoc: 3. Add a simple `docs/index.html` that links to your module documentation 4. Configure GitHub Pages to serve from the `docs/` folder of your main branch #### Custom Domains For more established projects, you can use your own domain: 1. Purchase a domain from a registrar 2. Add a `CNAME` file to your documentation with your domain name 3. Configure your DNS settings according to GitHub's instructions 4. Enable HTTPS in GitHub Pages settings By hosting your documentation on GitHub Pages, you make it easily accessible to users and maintainable alongside your codebase. It's a natural extension of the Git-based workflow we've established. #### Enhancing MkDocs MkDocs supports numerous plugins and extensions: - Code highlighting: Built-in support for syntax highlighting - Admonitions: Create warning, note, and info boxes - Search: Built-in search functionality - Table of contents: Automatic generation of section navigation Example of enhanced configuration: ### Integrating API Documentation MkDocs alone is great for manual documentation, but you can also integrate auto-generated API documentation: #### Using mkdocstrings Install mkdocstrings to include docstrings from your code: Update `mkdocs.yml`: Then in your `docs/api-reference.md`: This automatically generates documentation from docstrings in your `my_package.core` module. ### Documentation Best Practices Regardless of which documentation tool you choose, follow these best practices: 1. Start with a clear README: Include installation, quick start, and basic examples 2. Document as you code: Write documentation alongside code, not as an afterthought 3. Include examples: Show how to use functions and classes with realistic examples 4. Document edge cases and errors: Explain what happens in exceptional situations 5. Keep documentation close to code: Use docstrings for API details 6. Maintain a changelog: Track major changes between versions 7. Consider different audiences: Write for both new users and experienced developers ### Choosing the Right Documentation Approach | Approach | When to Use | |----------|-------------| | Docstrings only | For very small, personal projects | | pydoc | For simple projects with minimal documentation needs | | Custom pydoc script | Small to medium projects needing basic organisation | | MkDocs | Medium to large projects requiring structured, attractive documentation | | Sphinx | Large, complex projects, especially with scientific or mathematical content | For most applications, the journey often progresses from simple docstrings to MkDocs as the project grows. By starting with good docstrings from the beginning, you make each subsequent step easier. AI is excellent at writing docstrings and README content. Try: "Write Google-style docstrings for all the public functions in this module" and paste your code. For project-level documentation, ask AI to draft a README based on your pyproject.toml and source code. The key is providing AI with your actual code — generic documentation prompts produce generic results. Always review for accuracy, especially around installation steps and API descriptions. In the next section, we'll explore how to automate your workflow with CI/CD using GitHub Actions. ## CI/CD Workflows with GitHub Actions Continuous Integration (CI) and Continuous Deployment (CD) automate the process of testing, building, and deploying your code, ensuring quality and consistency throughout the development lifecycle. GitHub Actions provides a powerful and flexible way to implement CI/CD workflows directly within your GitHub repository. ### Understanding CI/CD Basics Before diving into implementation, let's understand what each component achieves: - Continuous Integration: Automatically testing code changes when pushed to the repository - Continuous Deployment: Automatically deploying code to testing, staging, or production environments A robust CI/CD pipeline typically includes: 1. Running tests 2. Verifying code quality (formatting, linting) 3. Static analysis (type checking, security scanning) 4. Building documentation 5. Building and publishing packages or applications 6. Deploying to environments ### Setting Up GitHub Actions GitHub Actions workflows are defined using YAML files stored in the `.github/workflows/` directory of your repository. Each workflow file defines a set of jobs and steps that execute in response to specified events. Start by creating the directory structure: ### Basic Python CI Workflow Let's create a file named `.github/workflows/ci.yml`: This workflow: 1. Triggers on pushes to `main` and on pull requests 2. Runs on the latest Ubuntu environment 3. Tests against multiple Python versions 4. Sets up caching to speed up dependency installation 5. Runs our full suite of quality checks and tests 6. Uploads coverage reports to Codecov (if you've set up this integration) ### Using Dependency Caching To speed up your workflow, GitHub Actions provides caching capabilities: For more specific control over caching: ### Adapting for Different Dependency Tools If you're using uv instead of pip, adjust your workflow: ### Building and Publishing Documentation Add a job to build documentation with MkDocs: This job builds your documentation with MkDocs and deploys it to GitHub Pages when changes are pushed to the main branch. ### Building and Publishing Python Packages For projects that produce packages, add a job for publication to PyPI: This job: 1. Only runs after tests and documentation have passed 2. Only triggers on tagged commits (releases) 3. Builds the package using the `build` package 4. Validates the package with `twine` 5. Publishes to PyPI using a secure token You would need to add the `PYPI_API_TOKEN` to your repository secrets. ### Running Tests in Multiple Environments For applications that need to support multiple operating systems or Python versions: This configuration runs your tests on three operating systems with three Python versions each, for a total of nine environments. ### Branch Protection and Required Checks To ensure code quality, set up branch protection rules on GitHub: 1. Go to your repository → Settings → Branches 2. Add a rule for your main branch 3. Enable "Require status checks to pass before merging" 4. Select the checks from your CI workflow This prevents merging pull requests until all tests pass, maintaining your code quality standards. ### Scheduled Workflows Run your tests on a schedule to catch issues with external dependencies: ### Notifications and Feedback Configure notifications for workflow results: This example sends notifications to Slack, but similar actions exist for other platforms. ### A Complete CI/CD Workflow Example Here's a comprehensive workflow example bringing together many of the concepts we've covered: This comprehensive workflow: 1. Checks code quality (formatting, linting, type checking, security, dead code) 2. Runs tests on multiple Python versions and operating systems 3. Builds and deploys documentation 4. Publishes packages to PyPI on tagged releases 5. Creates GitHub releases with release notes ### CI/CD Best Practices 1. Keep workflows modular: Split complex workflows into logical jobs 2. Fail fast: Run quick checks (like formatting) before longer ones (like testing) 3. Cache dependencies: Speed up workflows by caching pip packages 4. Be selective: Only run necessary jobs based on changed files 5. Test thoroughly: Include all environments your code supports 6. Secure secrets: Use GitHub's secret storage for tokens and keys 7. Monitor performance: Watch workflow execution times and optimise slow steps With these CI/CD practices in place, your development workflow becomes more reliable and automatic. Quality checks run on every change, documentation stays up to date, and releases happen smoothly and consistently. GitHub Actions YAML can be fiddly to get right. Ask AI to generate workflow files: "Write a GitHub Actions workflow that runs pytest, ruff, and mypy on push to main and on pull requests, for a Python project using uv." AI handles the YAML syntax and action versions well, but always test the workflow — AI may reference outdated action versions or miss environment-specific details. In the final section, we'll explore how to publish and distribute Python packages to make your work available to others. ## Package Publishing and Distribution When your Python project matures, you may want to share it with others through the Python Package Index (PyPI). Publishing your package makes it installable via `pip`, allowing others to easily use your work. ### Preparing Your Package for Distribution Before publishing, your project needs the right structure. Let's ensure everything is ready: #### 1. Package Structure Review A distributable package should have this basic structure: #### 2. Package Configuration with pyproject.toml Modern Python packaging uses `pyproject.toml` for configuration: This configuration: - Defines basic metadata (name, version, description) - Lists dependencies (both required and optional) - Sets up entry points for command-line scripts - Specifies the package location (src layout) #### 3. Include Essential Files Ensure you have these files: ### Building Your Package With configuration in place, you're ready to build distribution packages: This creates two files in the `dist/` directory: - A source distribution (`.tar.gz`) - A wheel file (`.whl`) Always check your distributions before publishing: ### Publishing to Test PyPI Before publishing to the real PyPI, test your package on TestPyPI: 1. Create a TestPyPI account at https://test.pypi.org/account/register/ 2. Upload your package: 3. Test installation from TestPyPI: ### Publishing to PyPI When everything works correctly on TestPyPI: 1. Create a PyPI account at https://pypi.org/account/register/ 2. Upload your package: Your package is now available to the world via `pip install my-package`! ### Automating Package Publishing To automate publishing with GitHub Actions, add a workflow that: 1. Builds the package 2. Uploads to PyPI when you create a release tag For better security, use API tokens instead of your PyPI password: 1. Generate a token from your PyPI account settings 2. Add it as a GitHub repository secret 3. Use the token in your workflow: ### Versioning Best Practices Follow Semantic Versioning (MAJOR.MINOR.PATCH): - MAJOR: Incompatible API changes - MINOR: New functionality (backward-compatible) - PATCH: Bug fixes (backward-compatible) Track versions in one place, usually in `__init__.py`: Or with a dynamic version from your git tags using setuptools-scm: ### Creating Releases A good release process includes: 1. Update documentation: - Ensure README is current - Update changelog with notable changes 2. Create a new version: - Update version number - Create a git tag: 3. Monitor the CI/CD pipeline: - Ensure tests pass - Verify package build succeeds - Confirm successful publication 4. Announce the release: - Create GitHub release notes - Post in relevant community forums - Update documentation site ### Package Maintenance Once published, maintain your package responsibly: 1. Monitor issues on GitHub or GitLab 2. Respond to bug reports promptly 3. Review and accept contributions from the community 4. Regularly update dependencies to address security issues 5. Create new releases when significant improvements are ready ### Advanced Distribution Topics As your package ecosystem grows, consider these advanced techniques: #### 1. Binary Extensions For performance-critical components, you might include compiled C extensions: - Use Cython to compile Python to C - Configure with the `build-system` section in pyproject.toml - Build platform-specific wheels #### 2. Namespace Packages For large projects split across multiple packages: #### 3. Conditional Dependencies For platform-specific dependencies: #### 4. Data Files Include non-Python files (data, templates, etc.): Create a `MANIFEST.in` file: By following these practices, you'll create a professional, well-maintained package that others can easily discover, install, and use. Publishing your work to PyPI is not just about sharing code—it's about participating in the Python ecosystem and contributing back to the community. ### Modern vs. Traditional Python Packaging Python packaging has evolved significantly over the years: #### Traditional `setup.py` Approach Historically, Python packages required a `setup.py` file: This approach is still common and has advantages for: - Compatibility with older tooling - Dynamic build processes that need Python code - Complex build requirements (e.g., C extensions, custom steps) #### Modern `pyproject.toml` Approach Since PEP 517/518, packages can use `pyproject.toml` exclusively: This declarative approach is recommended for new projects because it: - Provides a standardized configuration format - Supports multiple build systems (not just setuptools) - Simplifies dependency specification - Avoids executing Python code during installation #### Which Approach Should You Use? - For new, straightforward packages: Use `pyproject.toml` only - For packages with complex build requirements: You may need both `pyproject.toml` and `setup.py` - For maintaining existing packages: Consider gradually migrating to `pyproject.toml` Many projects use a hybrid approach, with basic metadata in `pyproject.toml` and complex build logic in `setup.py`. ============================================================ SOURCE: chapters/case-study.qmd ============================================================ # Case Study: Building SimpleBot - A Python Development Workflow Example This case study demonstrates how to apply the Python development pipeline practices to a real project. We'll walk through the development of SimpleBot, a lightweight wrapper for Large Language Models (LLMs) designed for educational settings. ## Project Overview SimpleBot is an educational tool that makes it easy for students to interact with Large Language Models through simple Python functions. Key features include: - Simple API for sending prompts to LLMs - Pre-defined personality bots (pirate, Shakespeare, emoji, etc.) - Error handling and user-friendly messages - Support for local LLM servers like Ollama This project is ideal for our case study because: - It solves a real problem (making LLMs accessible in educational settings) - It's small enough to understand quickly but complex enough to demonstrate real workflow practices - It includes both pure Python and compiled Cython components Let's see how we can develop this project using our Python development pipeline. Along the way, we will see how AI can accelerate each phase — from scaffolding the project structure to generating tests and documentation. ## Prefer Notebooks? If you develop primarily in Jupyter notebooks, see for a parallel case study that builds TextKit using nbdev—same destination (published package), different workflow. ## 1. Setting the Foundation ### Project Structure We'll set up the project using the recommended `src` layout: ### Setting Up Version Control First, we initialize a Git repository and create a `.gitignore` file: ### Creating Essential Files Let's create the basic files: ```bash # Create the project structure mkdir -p src/simplebot tests docs # Create a file name # SimpleBot > LLMs made simple for students and educators SimpleBot is a lightweight Python wrapper that simplifies interactions with Large Language Models (LLMs) for educational settings. ## Installation \`\`\`bash pip install simplebot \`\`\` ## Quick Start \`\`\`python from simplebot import get_response, pirate_bot # Basic usage response = get_response("Tell me about planets") print(response) # Use a personality bot pirate_response = pirate_bot("Tell me about sailing ships") print(pirate_response) \`\`\` ## License This project is licensed under the MIT License - see the LICENSE file for details. EOF # Create a file named LICENSE with the following contents: MIT License Copyright (c) 2025 SimpleBot Authors Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. EOF git add README.md LICENSE git commit -m "Add README and LICENSE" bash # Create virtual environment python -m venv .venv # Activate the environment (Linux/macOS) source .venv/bin/activate # On Windows: .venv\Scripts\activate # Initial package installation for development pip install pytest ruff mypy build bash # Create the package structure mkdir -p src/simplebot bash # Create the package __init__.py # Create a file named src/simplebot/__init__.py with the following contents: """SimpleBot - LLMs made simple for students and educators.""" from .core import get_response from .personalities import ( pirate_bot, shakespeare_bot, emoji_bot, teacher_bot, coder_bot, ) __version__ = "0.1.0" __all__ = [ "get_response", "pirate_bot", "shakespeare_bot", "teacher_bot", "emoji_bot", "coder_bot", ] # Create the core module # Create a file named src/simplebot/core.py with the following contents: """Core functionality for SimpleBot.""" import requests import random import time from typing import Optional, Dict, Any # Cache for the last used model to avoid redundant loading messages _last_model: Optional[str] = None def get_response( prompt: str, model: str = "llama3", system: str = "You are a helpful assistant.", stream: bool = False, api_url: Optional[str] = None, ) -> str: """ Send a prompt to the LLM API and retrieve the model's response. Args: prompt: The text prompt to send to the language model model: The name of the model to use system: System instructions that control the model's behaviour stream: Whether to stream the response api_url: Custom API URL (defaults to local Ollama server) Returns: The model's response text, or an error message if the request fails """ global _last_model # Default to local Ollama if no API URL is provided if api_url is None: api_url = "http://localhost:11434/api/generate" # Handle model switching with friendly messages if model != _last_model: warmup_messages = [ f"🧠 Loading model '{model}' into RAM... give me a sec...", f"💾 Spinning up the AI core for '{model}'...", f"⏳ Summoning the knowledge spirits... '{model}' booting...", f"🤖 Thinking really hard with '{model}'...", f"⚙️ Switching to model: {model} ... (may take a few seconds)", ] print(random.choice(warmup_messages)) # Short pause to simulate/allow for model loading time.sleep(1.5) _last_model = model # Validate input if not prompt.strip(): return "⚠️ Empty prompt." # Prepare the request payload payload: Dict[str, Any] = { "model": model, "prompt": prompt, "system": system, "stream": stream } try: # Send request to the LLM API response = requests.post( api_url, json=payload, timeout=10 ) response.raise_for_status() data = response.json() return data.get("response", "⚠️ No response from model.") except requests.RequestException as e: return f"❌ Connection Error: {str(e)}" except Exception as e: return f"❌ Error: {str(e)}" EOF # Create the personalities module # Create a file named src/simplebot/personalities.py with the following contents: """Pre-defined personality bots for SimpleBot.""" from .core import get_response from typing import Optional def pirate_bot(prompt: str, model: Optional[str] = None) -> str: """ Generate a response in the style of a 1700s pirate with nautical slang. Args: prompt: The user's input text/question model: Optional model override Returns: A response written in pirate vernacular """ return get_response( prompt, system="You are a witty pirate from the 1700s. " "Use nautical slang, say 'arr' occasionally, " "and reference sailing, treasure, and the sea.", model=model or "llama3" ) def shakespeare_bot(prompt: str, model: Optional[str] = None) -> str: """ Generate a response in the style of William Shakespeare. Args: prompt: The user's input text/question model: Optional model override Returns: A response written in Shakespearean style """ return get_response( prompt, system="You respond in the style of William Shakespeare, " "using Early Modern English vocabulary and phrasing.", model=model or "llama3" ) def emoji_bot(prompt: str, model: Optional[str] = None) -> str: """ Generate a response primarily using emojis with minimal text. Args: prompt: The user's input text/question model: Optional model override Returns: A response composed primarily of emojis """ return get_response( prompt, system="You respond using mostly emojis, mixing minimal words " "and symbols to convey meaning. You love using expressive " "emoji strings.", model=model or "llama3" ) def teacher_bot(prompt: str, model: Optional[str] = None) -> str: """ Generate a response in the style of a patient, helpful educator. Args: prompt: The user's input text/question model: Optional model override Returns: A response with an educational approach """ return get_response( prompt, system="You are a patient, encouraging teacher who explains " "concepts clearly at an appropriate level. Break down " "complex ideas into simpler components and use analogies " "when helpful.", model=model or "llama3" ) def coder_bot(prompt: str, model: Optional[str] = None) -> str: """ Generate a response from a coding assistant optimised for programming help. Args: prompt: The user's input programming question or request model: Optional model override (defaults to a coding-specific model) Returns: A technical response focused on code-related assistance """ return get_response( prompt, system="You are a skilled coding assistant who explains and writes " "code clearly and concisely. Prioritize best practices, " "readability, and proper error handling.", model=model or "codellama" ) EOF git add src/ git commit -m "Add core SimpleBot functionality" ``` ## 3. Package Configuration Let's set up the package configuration in `pyproject.toml`: This case study uses the newer `pyproject.toml`-only approach for simplicity and to follow current best practices. Many existing Python projects still use `setup.py`, either alongside `pyproject.toml` or as their primary configuration. The `setup.py` approach remains valuable for packages with complex build requirements, custom build steps, or when supporting older tools and Python versions. For SimpleBot, our straightforward package requirements allow us to use the cleaner, declarative `pyproject.toml` approach. ## Create a file named pyproject.toml with the following contents: Let's set up the package configuration in `pyproject.toml`: ## 4. Writing Tests Let's create some tests for our SimpleBot functionality: ## 5. Applying Code Quality Tools Let's run our code quality tools and fix any issues: ## 6. Documentation Let's create basic documentation: ```bash # Create docs directory mkdir -p docs # Create main documentation file # Create a file named docs/index.md with the following contents: # SimpleBot Documentation > LLMs made simple for students and educators SimpleBot is a lightweight Python wrapper that simplifies interactions with Large Language Models (LLMs) for educational settings. It abstracts away the complexity of API calls, model management, and error handling, allowing students to focus on learning programming concepts through engaging AI interactions. ## Installation \`\`\`bash pip install simplebot \`\`\` ## Basic Usage \`\`\`python from simplebot import get_response # Basic usage with default model response = get_response("Tell me about planets") print(response) \`\`\` ## Personality Bots SimpleBot comes with several pre-defined personality bots: \`\`\`python from simplebot import pirate_bot, shakespeare_bot, emoji_bot, teacher_bot, coder_bot # Get a response in pirate speak pirate_response = pirate_bot("Tell me about sailing ships") print(pirate_response) # Get a response in Shakespearean style shakespeare_response = shakespeare_bot("What is love?") print(shakespeare_response) # Get a response with emojis emoji_response = emoji_bot("Explain happiness") print(emoji_response) # Get an educational response teacher_response = teacher_bot("How do photosynthesis work?") print(teacher_response) # Get coding help code_response = coder_bot("Write a Python function to check if a string is a palindrome") print(code_response) \`\`\` ## API Reference ### get_response() \`\`\`python def get_response( prompt: str, model: str = "llama3", system: str = "You are a helpful assistant.", stream: bool = False, api_url: Optional[str] = None, ) -> str: \`\`\` The core function for sending prompts to an LLM and getting responses. #### Parameters: - `prompt`: The text prompt to send to the language model - `model`: The name of the model to use (default: "llama3") - `system`: System instructions that control the model's behaviour - `stream`: Whether to stream the response (default: False) - `api_url`: Custom API URL (defaults to local Ollama server) #### Returns: - A string containing the model's response or an error message EOF # Create examples file # Create a file named docs/examples.md with the following contents: # SimpleBot Examples Here are some examples of using SimpleBot in educational settings. ## Creating Custom Bot Personalities You can create custom bot personalities: \`\`\`python from simplebot import get_response def scientist_bot(prompt): """A bot that responds like a scientific researcher.""" return get_response( prompt, system="You are a scientific researcher. Provide evidence-based " "responses with references to studies when possible. " "Be precise and methodical in your explanations." ) result = scientist_bot("What happens during photosynthesis?") print(result) \`\`\` ## Building a Simple Quiz System \`\`\`python from simplebot import teacher_bot quiz_questions = [ "What is the capital of France?", "Who wrote Romeo and Juliet?", "What is the chemical symbol for water?" ] def generate_quiz(): print("=== Quiz Time! ===") for i, question in enumerate(quiz_questions, 1): print(f"Question {i}: {question}") user_answer = input("Your answer: ") # Generate feedback on the answer feedback = teacher_bot( f"Question: {question}\nStudent answer: {user_answer}\n" "Provide brief, encouraging feedback on whether this answer is " "correct. If incorrect, provide the correct answer." ) print(f"Feedback: {feedback}\n") # Run the quiz generate_quiz() \`\`\` ## Simulating a Conversation Between Bots \`\`\`python from simplebot import pirate_bot, shakespeare_bot def bot_conversation(topic, turns=3): """Simulate a conversation between two bots on a given topic.""" print(f"=== A conversation about {topic} ===") # Start with the pirate current_message = f"Tell me about {topic}" current_bot = "pirate" for i in range(turns): if current_bot == "pirate": response = pirate_bot(current_message) print(f"🏴‍☠️ Pirate: {response}") current_message = f"Respond to this: {response}" current_bot = "shakespeare" else: response = shakespeare_bot(current_message) print(f"🎭 Shakespeare: {response}") current_message = f"Respond to this: {response}" current_bot = "pirate" print() # Run a conversation about the ocean bot_conversation("the ocean", turns=4) \`\`\` EOF git add docs/ git commit -m "Add documentation" bash # Create GitHub Actions workflow directory mkdir -p .github/workflows # Create CI workflow file # Create a file named .github/workflows/ci.yml with the following contents: name: Python CI on: push: branches: [ main ] pull_request: branches: [ main ] jobs: test: runs-on: ubuntu-latest strategy: matrix: python-version: ["3.7", "3.8", "3.9", "3.10"] steps: - uses: actions/checkout@v3 - name: Set up Python \${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: \${{ matrix.python-version }} cache: pip - name: Install dependencies run: | python -m pip install --upgrade pip python -m pip install -e ".[dev]" - name: Check formatting with Ruff run: ruff format --check . - name: Lint with Ruff run: ruff check . - name: Type check with mypy run: mypy src/ - name: Test with pytest run: pytest --cov=src/ tests/ - name: Build package run: python -m build EOF # Create release workflow # Create a file named .github/workflows/release.yml with the following contents: name: Publish to PyPI on: release: types: [created] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: "3.10" - name: Install dependencies run: | python -m pip install --upgrade pip pip install build twine - name: Build and publish env: TWINE_USERNAME: \${{ secrets.PYPI_USERNAME }} TWINE_PASSWORD: \${{ secrets.PYPI_PASSWORD }} run: | python -m build twine check dist/* twine upload dist/* EOF git add .github/ git commit -m "Add CI/CD workflows" bash # Install the package in development mode pip install -e . # Run the tests pytest # Build the package python -m build # Verify the package twine check dist/* ``` ## 9. Project Summary By following the Python Development Workflow, we've transformed the SimpleBot concept into a well-structured, tested, and documented Python package that's ready for distribution. Let's review what we've accomplished: 1. Project Foundation: - Created a clear, organised directory structure - Set up version control with Git - Added essential files (README, LICENSE) 2. Development Environment: - Created a virtual environment - Managed dependencies cleanly 3. Code Quality: - Applied type hints throughout the codebase - Used Ruff for formatting and linting - Used mypy for static type checking 4. Testing: - Created comprehensive unit tests with pytest - Used mocking to test external API interactions 5. Documentation: - Added clear docstrings - Created usage documentation with examples 6. Packaging & Distribution: - Configured the package with pyproject.toml - Set up CI/CD with GitHub Actions ## 10. Next Steps If we were to continue developing SimpleBot, potential next steps might include: 1. Enhanced Features: - Add more personality bots - Support for conversation memory/context - Configuration file support 2. Advanced Documentation: - Set up MkDocs for a full documentation site - Add tutorials for classroom usage 3. Performance Improvements: - Add caching for responses - Implement Cython optimisation for performance-critical sections 4. Security Enhancements: - Add API key management - Implement content filtering for educational settings Notice how AI could have accelerated every phase of this case study. You could ask AI to generate the initial project structure, draft the core module, write pytest tests, create the pyproject.toml configuration, generate GitHub Actions workflows, and draft the README. The pipeline you learned in earlier chapters gives you the knowledge to evaluate everything AI produces. Without that knowledge, you would accept AI output uncritically. With it, you orchestrate AI as a force multiplier. This case study demonstrates how following a structured Python development workflow leads to a high-quality, maintainable, and distributable package — even for relatively small projects. ============================================================ SOURCE: chapters/advanced-techniques.qmd ============================================================ # Advanced Development Techniques As your Python projects grow in complexity and requirements, you'll encounter challenges that require more sophisticated approaches than the foundational practices we've established. This chapter explores advanced techniques that build upon our core development pipeline, focusing on principles and patterns that scale with your project's needs. Rather than diving into the specifics of every advanced tool, we'll focus on understanding when and why to adopt more complex solutions, maintaining our philosophy of "simple but not simplistic." ## Performance optimisation: Measure First, optimise Second Performance optimisation often feels compelling, but premature optimisation is a common trap. The key principle: measure before you optimise. Our development pipeline already includes the foundation for performance work through comprehensive testing and quality gates. ### Establishing Performance Baselines Before optimising, establish measurable baselines using tools that integrate naturally with your existing workflow: Add performance dependencies to your development requirements: This approach integrates performance measurement into your existing development workflow rather than introducing entirely new tools. ### Performance optimisation Strategy When benchmarks indicate performance issues, follow a systematic approach: 1. Profile to identify bottlenecks - Don't guess where the slowness is 2. optimise the algorithms first - Better algorithms beat micro-optimisations 3. Consider caching strategically - Cache expensive computations, not everything 4. Measure the impact - Ensure optimisations actually improve performance The key insight: optimise within your existing architecture before considering more complex solutions like Cython or asyncio. ## Containerization: Development Environment Consistency Containers address the challenge of environment reproducibility across different development machines and deployment environments. However, containerization should enhance, not replace, your existing development workflow. ### Development Containers vs. Production Containers Development containers prioritize developer experience: - Fast rebuild times - Volume mounts for live code editing - Development tools and debugging capabilities - Integration with your existing toolchain Production containers prioritize runtime efficiency: - Minimal attack surface - optimised for size and startup time - No development dependencies - Security-focused configurations ### Integrating Containers with Your Workflow Create a `Dockerfile` that builds upon your existing dependency management: Add container management to your task automation: This approach uses containers to enhance reproducibility without disrupting your core development workflow. ### When to Containerize Consider containerization when you encounter: - Environment inconsistencies between team members - Complex system dependencies that are difficult to install - Deployment environment differences from development - Service integration challenges (databases, message queues, etc.) Don't containerize simply because it's trendy — use it to solve specific reproducibility problems. Dockerfiles are perfect candidates for AI generation. Try: "Write a multi-stage Dockerfile for a Python FastAPI application using uv for dependency management. The final image should be minimal." AI handles the syntax and layer ordering well. But always review the base images it suggests (are they up to date? are they the slim variants?) and test the build — AI often gets volume mounts and port mappings wrong in docker-compose files. ## Scaling Your Development Process As projects grow, you'll need techniques for managing complexity while maintaining development velocity. ### Modular Architecture Patterns Design your codebase for growth by establishing clear module boundaries: This interface-based design allows you to: 1. Test implementations independently with mocks and stubs 2. Swap implementations without changing dependent code 3. Add new implementations without modifying existing code 4. Maintain clear boundaries between different parts of your system ### Configuration Management As projects grow, configuration becomes more complex. Establish patterns early: This approach provides: - Type safety through dataclasses and type hints - Environment-based configuration for different deployment contexts - Testable configuration through dependency injection - Clear documentation of required configuration values ### Database Integration Patterns When your application needs persistent storage, integrate database operations cleanly with your existing testing and development workflow: Test database operations with fixtures: This pattern maintains clean separation between business logic and data persistence while integrating smoothly with your testing infrastructure. ## API Development and Integration When building applications that expose or consume APIs, maintain the same development quality principles. ### API Design Principles Design APIs that are: 1. Consistent - Similar operations work similarly 2. Documented - Clear, up-to-date documentation 3. Versioned - Handle changes without breaking existing clients 4. Testable - Easy to test both as provider and consumer ### API Testing Strategy Test APIs at multiple levels: This approach integrates API testing with your existing pytest infrastructure and maintains the same quality standards. ## Cross-Platform Development Considerations When your Python application needs to run across different operating systems, handle platform differences gracefully within your existing development workflow. ### Path and Environment Handling Use `pathlib` and environment-aware patterns: ### Testing Across Platforms Use your existing CI/CD pipeline to test across platforms: This extends your existing quality gates to ensure cross-platform compatibility. ## When to Adopt Advanced Techniques The key to advanced techniques is selective adoption based on actual needs: ### Adopt Containerization When: - Team members struggle with environment setup - You need to integrate with external services during development - Deployment environments differ significantly from development ### Adopt Performance optimisation When: - Benchmarks show actual performance problems - Performance requirements are clearly defined - You have established baseline measurements ### Adopt Advanced Architecture When: - Code complexity makes maintenance difficult - You need to support multiple implementations of core functionality - Team size makes modular development beneficial ### Don't Adopt Advanced Techniques When: - Your current approach works well - The complexity cost exceeds the benefits - You haven't mastered the foundational practices ## Maintaining Development Velocity The most important principle for advanced techniques: they should enhance, not replace, your core development practices. Your testing, code quality, documentation, and automation should continue to work as you adopt more sophisticated approaches. Advanced techniques are tools for solving specific problems, not goals in themselves. Focus on delivering value through your software while maintaining the solid development foundation you've established. When deciding whether to adopt an advanced technique, describe your situation to your AI assistant: "My Flask app handles 50 requests per second and response times are around 200ms. Users are complaining about slow page loads. Should I add caching, switch to async, or look at the database queries first?" AI can help you reason through trade-offs, but the decision is yours — AI does not know your team's capacity or your deployment constraints. ============================================================ SOURCE: chapters/project-management.qmd ============================================================ # Project Management and Automation Moving beyond individual development practices, this chapter focuses on project-level management, automation, and collaboration workflows. We'll explore tools and techniques that help you maintain consistency, automate repetitive tasks, and establish sustainable development practices. ## Task Automation with Poe the Poet One of the first challenges in any Python project is managing the growing number of commands you need to run: testing, linting, formatting, building documentation, and more. While traditional Unix environments might use Makefiles, Python projects benefit from a more integrated approach. Poe the Poet provides a powerful task runner that integrates seamlessly with your `pyproject.toml` file, offering a cross-platform alternative to Makefiles that works naturally with your existing Python toolchain. ### Setting Up Poe the Poet Add Poe the Poet as a development dependency to your project: This aligns with our philosophy of keeping project tooling within the project itself, ensuring every developer has access to the same automation tools. ### Defining Project Tasks Define your common development tasks in your `pyproject.toml` file: This configuration demonstrates several key principles: 1. Single source of truth: All project automation is defined in one place 2. Composable tasks: Complex workflows are built from simpler tasks 3. Cross-platform compatibility: Tasks work on Windows, macOS, and Linux 4. Integration with existing tools: Works seamlessly with uv, ruff, pytest, and other tools in our stack ### Advanced Task Configuration For more complex scenarios, Poe supports parameterized tasks and conditional execution: ### Running Tasks Execute your defined tasks using the `poe` command through uv: ### Integration with Development Workflow The power of Poe the Poet becomes apparent when integrated into your daily development routine: Pre-commit hooks can reference your Poe tasks: IDE integration allows running tasks directly from your editor, while CI/CD pipelines can use the same task definitions: This approach eliminates the disconnect between local development and automated systems — everyone uses the same commands. AI can generate Poe the Poet task definitions, Makefiles, and shell scripts from natural language descriptions. Try: "Write Poe the Poet tasks for: formatting with ruff, linting, running tests with coverage, and a 'check' task that runs all three in sequence." Since these are configuration files with well-defined syntax, AI output is usually accurate. But test each task before committing it — path assumptions and tool flags can vary between projects. ## Project Setup and Structure Consistent project structure is fundamental to maintainable Python development. While Python is famously flexible, establishing conventions early saves significant time and confusion as projects grow. ### Modern Python Project Layout Our recommended project structure balances simplicity with scalability: This structure follows several important principles: Src Layout: Placing source code in a `src/` directory prevents accidental imports of uninstalled code during development and testing. This is particularly important for ensuring your tests run against the installed package, not just local files. Clear Separation: Tests, documentation, and source code are clearly separated, making the project structure immediately understandable to new contributors. Configuration Co-location: All project configuration lives in `pyproject.toml`, providing a single source of truth for project metadata, dependencies, and tool configuration. ### Initializing New Projects Create new projects following this structure using uv: ### Application vs. Package Considerations The structure varies slightly depending on whether you're building an application (end-user focused) or a package (library for other developers): Applications typically include: - Configuration files and settings management - Entry point scripts or CLI interfaces - Deployment configurations - User documentation focused on usage Packages emphasize: - Clean, documented APIs - Comprehensive test coverage - Developer documentation - Distribution metadata for PyPI Most projects start as applications and may later extract reusable components into packages. Our recommended structure accommodates both paths naturally. ## Team Collaboration Workflows ### Code Review Standards Establish clear expectations for code reviews that align with your automated tooling: 1. Automated checks must pass: All pre-commit hooks and CI checks should be green before review 2. Test coverage requirements: New code should include appropriate tests 3. Documentation updates: Public API changes require documentation updates 4. Consistent style: Rely on automated formatting (Ruff) rather than manual style discussions ### Release Management Define clear release processes that leverage your automation: ### Managing Technical Debt Use your automation to continuously monitor and address technical debt: Regular execution of these tasks helps maintain code quality and security over time. ## Development Environment Standards ### Editor-Agnostic Configuration While developers may prefer different editors, project configuration should work consistently across environments. Our approach centers on `pyproject.toml` configuration that most modern Python editors understand: This configuration works automatically with VS Code, PyCharm, Vim, Emacs, and other editors with Python support. ### Development Environment Reproducibility Ensure consistent development environments across team members: New team members can quickly verify their setup with `uv run poe doctor`. AI can help you write contributing guides, development setup instructions, and team conventions documents. Provide your pyproject.toml and project structure, then ask: "Write a CONTRIBUTING.md that explains how to set up the development environment, run tests, and submit a pull request for this project." This is documentation that benefits everyone on the team and that AI can draft well from your project's configuration. This chapter has established the foundation for scalable project management through automation, consistent structure, and collaborative workflows. These practices become increasingly valuable as projects grow in size and complexity, ensuring that good habits established early continue to serve the project throughout its lifecycle. ============================================================ SOURCE: chapters/distribution.qmd ============================================================ # Multi-Platform Distribution Moving beyond traditional Python package publishing, this chapter explores how to distribute complete applications to users across web, desktop, and mobile platforms—all from a single codebase. We'll focus on an architecture pattern that works exceptionally well with AI-assisted development: FastAPI backend + React frontend + multiple distribution targets. ## A Note on Technologies Beyond Python This chapter introduces React and Electron—technologies outside Python's ecosystem. Don't worry: the goal isn't to master these tools, but to understand the architecture that lets you ship to multiple platforms. Your Python skills remain central (the backend is still FastAPI), and the companion templates handle the frontend implementation details. With AI assistance, you can work effectively with React and Electron by understanding their role in the architecture rather than memorizing their APIs. ## The Modern Distribution Challenge Today's users expect applications everywhere: - Web: Accessible from any browser, no installation required - Desktop: Native experience on Windows, macOS, and Linux - Mobile: Installable apps or Progressive Web Apps (PWAs) For indie developers and small teams, maintaining separate codebases for each platform is impractical. The solution is an API-first architecture where a single backend serves multiple frontends, and those frontends can be packaged for different platforms. ### What We're Building By the end of this chapter, you'll understand how to structure a project that can ship to: 1. Web: Docker container deployable to any cloud platform 2. PWA: Installable web app with offline capabilities 3. Desktop: Native applications for Windows, macOS, and Linux via Electron All from a single codebase, with automated builds through GitHub Actions. ## Why FastAPI + React? This isn't about which technologies are theoretically "best"—it's about which combination is most practical for AI-assisted development in 2025 and beyond. ### Training Data Density Python and JavaScript/React represent the largest pools of training data for AI models. This means: - Better code generation: AI has seen more examples of these patterns - More accurate suggestions: Edge cases are better handled - Richer ecosystem knowledge: AI understands popular libraries deeply When you ask AI to help with a FastAPI endpoint or a React component, you're working with the technologies AI knows best. ### Ecosystem Maturity Both ecosystems have: - Solved most common problems: Authentication, forms, state management, API design - Extensive documentation: AI can reference official docs and community resources - Active communities: Stack Overflow answers, GitHub issues, blog posts This maturity translates directly to better AI assistance. ### Clean Separation The API-first approach creates natural boundaries: This separation means: - AI can work on backend or frontend independently - Changes to one side don't break the other (if the API contract holds) - Different team members (or AI sessions) can work in parallel - Testing is straightforward at each boundary ## The Architecture Pattern ### Project Structure for AI Collaboration Here's the recommended structure for a multi-platform application: This structure provides clear boundaries that both humans and AI can navigate effectively. ### The CLAUDE.md Pattern A key practice for AI-assisted development is maintaining a `CLAUDE.md` file (or similar context file) at the project root. This file provides AI assistants with project-specific context: ```markdown # Project: My Application ## Architecture - Backend: FastAPI (Python 3.11+) - Frontend: React 18 + TypeScript + Vite - Desktop: Electron - Database: SQLite (development), PostgreSQL (production) ## Conventions - Backend API routes follow RESTful conventions - All API responses use Pydantic models - Frontend components use functional style with hooks - State management via React Query for server state ## Directory Structure - `/backend/src/my_app/api/` - API route definitions - `/backend/src/my_app/models/` - Pydantic schemas - `/frontend/src/components/` - React components - `/frontend/src/services/` - API client functions ## Development Commands - Backend: `cd backend && uv run uvicorn my_app.main:app --reload` - Frontend: `cd frontend && npm run dev` - Both: `docker-compose up` ## Key Decisions - Using SQLite for simplicity; PostgreSQL in production - Electron for desktop instead of Tauri (more mature, better AI support) - Single Docker container for deployment (backend serves frontend static files) python # backend/src/my_app/main.py """Main FastAPI application.""" from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from fastapi.staticfiles import StaticFiles from pathlib import Path from .api.routes import router app = FastAPI( title="My Application", description="API for my multi-platform application", version="1.0.0", ) # CORS configuration for development app.add_middleware( CORSMiddleware, allow_origins=["http://localhost:5173"], # Vite dev server allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # API routes app.include_router(router, prefix="/api") # Serve frontend static files in production frontend_path = Path(__file__).parent.parent.parent.parent / "frontend" / "dist" if frontend_path.exists(): app.mount("/", StaticFiles(directory=frontend_path, html=True), name="frontend") python # backend/src/my_app/api/routes.py """API route definitions.""" from fastapi import APIRouter, HTTPException from typing import List from ..models.schemas import Item, ItemCreate, ItemUpdate router = APIRouter() # In-memory storage for demonstration items_db: dict[int, Item] = {} next_id = 1 @router.get("/items", response_model=List[Item]) async def list_items(): """List all items.""" return list(items_db.values()) @router.post("/items", response_model=Item, status_code=201) async def create_item(item: ItemCreate): """Create a new item.""" global next_id new_item = Item(id=next_id, item.model_dump()) items_db[next_id] = new_item next_id += 1 return new_item @router.get("/items/{item_id}", response_model=Item) async def get_item(item_id: int): """Get a specific item by ID.""" if item_id not in items_db: raise HTTPException(status_code=404, detail="Item not found") return items_db[item_id] @router.put("/items/{item_id}", response_model=Item) async def update_item(item_id: int, item: ItemUpdate): """Update an existing item.""" if item_id not in items_db: raise HTTPException(status_code=404, detail="Item not found") stored_item = items_db[item_id] update_data = item.model_dump(exclude_unset=True) updated_item = stored_item.model_copy(update=update_data) items_db[item_id] = updated_item return updated_item @router.delete("/items/{item_id}", status_code=204) async def delete_item(item_id: int): """Delete an item.""" if item_id not in items_db: raise HTTPException(status_code=404, detail="Item not found") del items_db[item_id] python # backend/src/my_app/models/schemas.py """Pydantic models for API request/response validation.""" from pydantic import BaseModel, Field from datetime import datetime from typing import Optional class ItemBase(BaseModel): """Base item schema with common fields.""" title: str = Field(..., min_length=1, max_length=100) description: Optional[str] = Field(None, max_length=500) completed: bool = False class ItemCreate(ItemBase): """Schema for creating a new item.""" pass class ItemUpdate(BaseModel): """Schema for updating an item (all fields optional).""" title: Optional[str] = Field(None, min_length=1, max_length=100) description: Optional[str] = Field(None, max_length=500) completed: Optional[bool] = None class Item(ItemBase): """Schema for item responses.""" id: int created_at: datetime = Field(default_factory=datetime.utcnow) class Config: from_attributes = True bash # Create React project with TypeScript npm create vite@latest frontend -- --template react-ts cd frontend npm install npm install @tanstack/react-query axios typescript // frontend/src/services/api.ts import axios from 'axios'; const API_BASE_URL = import.meta.env.VITE_API_URL || 'http://localhost:8000/api'; const api = axios.create({ baseURL: API_BASE_URL, headers: { 'Content-Type': 'application/json', }, }); // Types matching backend Pydantic models export interface Item { id: number; title: string; description: string | null; completed: boolean; created_at: string; } export interface ItemCreate { title: string; description?: string; completed?: boolean; } export interface ItemUpdate { title?: string; description?: string; completed?: boolean; } // API functions export const itemsApi = { list: async (): Promise => { const response = await api.get('/items'); return response.data; }, get: async (id: number): Promise => { const response = await api.get(`/items/${id}`); return response.data; }, create: async (item: ItemCreate): Promise => { const response = await api.post('/items', item); return response.data; }, update: async (id: number, item: ItemUpdate): Promise => { const response = await api.put(`/items/${id}`, item); return response.data; }, delete: async (id: number): Promise => { await api.delete(`/items/${id}`); }, }; typescript // frontend/src/components/ItemList.tsx import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query'; import { itemsApi, Item, ItemCreate } from '../services/api'; import { useState } from 'react'; export function ItemList() { const queryClient = useQueryClient(); const [newTitle, setNewTitle] = useState(''); // Fetch items const { data: items, isLoading, error } = useQuery({ queryKey: ['items'], queryFn: itemsApi.list, }); // Create mutation const createMutation = useMutation({ mutationFn: (item: ItemCreate) => itemsApi.create(item), onSuccess: () => { queryClient.invalidateQueries({ queryKey: ['items'] }); setNewTitle(''); }, }); // Toggle completion mutation const toggleMutation = useMutation({ mutationFn: ({ id, completed }: { id: number; completed: boolean }) => itemsApi.update(id, { completed }), onSuccess: () => { queryClient.invalidateQueries({ queryKey: ['items'] }); }, }); // Delete mutation const deleteMutation = useMutation({ mutationFn: (id: number) => itemsApi.delete(id), onSuccess: () => { queryClient.invalidateQueries({ queryKey: ['items'] }); }, }); if (isLoading) return Loading...; if (error) return Error loading items; return ( { e.preventDefault(); if (newTitle.trim()) { createMutation.mutate({ title: newTitle }); } }} > setNewTitle(e.target.value)} placeholder="Add new item..." /> Add {items?.map((item: Item) => ( toggleMutation.mutate({ id: item.id, completed: !item.completed, }) } /> {item.title} deleteMutation.mutate(item.id)}> Delete ))} ); } dockerfile # docker/Dockerfile.combined # Multi-stage build for combined backend + frontend # Stage 1: Build frontend FROM node:20-alpine AS frontend-build WORKDIR /app/frontend COPY frontend/package*.json ./ RUN npm ci COPY frontend/ ./ RUN npm run build # Stage 2: Production image FROM python:3.11-slim WORKDIR /app # Install uv RUN pip install uv # Copy backend COPY backend/pyproject.toml backend/uv.lock ./ RUN uv sync --frozen --no-dev COPY backend/src ./src # Copy built frontend COPY --from=frontend-build /app/frontend/dist ./frontend/dist # Expose port EXPOSE 8000 # Run the application CMD ["uv", "run", "uvicorn", "my_app.main:app", "--host", "0.0.0.0", "--port", "8000"] yaml # docker/docker-compose.yml version: '3.8' services: backend: build: context: ../backend dockerfile: Dockerfile ports: - "8000:8000" volumes: - ../backend/src:/app/src environment: - DEBUG=true command: uv run uvicorn my_app.main:app --reload --host 0.0.0.0 frontend: build: context: ../frontend dockerfile: Dockerfile.dev ports: - "5173:5173" volumes: - ../frontend/src:/app/src environment: - VITE_API_URL=http://localhost:8000/api command: npm run dev -- --host bash # Build the production image docker build -f docker/Dockerfile.combined -t my-app:latest . # Push to registry (example: GitHub Container Registry) docker tag my-app:latest ghcr.io/username/my-app:latest docker push ghcr.io/username/my-app:latest # Deploy to fly.io (example) flyctl deploy bash npm install vite-plugin-pwa -D typescript // frontend/vite.config.ts import { defineConfig } from 'vite'; import react from '@vitejs/plugin-react'; import { VitePWA } from 'vite-plugin-pwa'; export default defineConfig({ plugins: [ react(), VitePWA({ registerType: 'autoUpdate', includeAssets: ['favicon.ico', 'robots.txt', 'apple-touch-icon.png'], manifest: { name: 'My Application', short_name: 'MyApp', description: 'A multi-platform application', theme_color: '#ffffff', icons: [ { src: 'pwa-192x192.png', sizes: '192x192', type: 'image/png', }, { src: 'pwa-512x512.png', sizes: '512x512', type: 'image/png', }, ], }, workbox: { globPatterns: ['/.{js,css,html,ico,png,svg}'], runtimeCaching: [ { urlPattern: /^https:\/\/api\.example\.com\/.*/i, handler: 'NetworkFirst', options: { cacheName: 'api-cache', expiration: { maxEntries: 100, maxAgeSeconds: 60 60 24, // 24 hours }, }, }, ], }, }), ], }); javascript // electron/main.js const { app, BrowserWindow, Menu } = require('electron'); const path = require('path'); const { autoUpdater } = require('electron-updater'); // Handle creating/removing shortcuts on Windows when installing/uninstalling if (require('electron-squirrel-startup')) { app.quit(); } let mainWindow; function createWindow() { mainWindow = new BrowserWindow({ width: 1200, height: 800, webPreferences: { preload: path.join(__dirname, 'preload.js'), contextIsolation: true, nodeIntegration: false, }, }); // In development, load from Vite dev server if (process.env.NODE_ENV === 'development') { mainWindow.loadURL('http://localhost:5173'); mainWindow.webContents.openDevTools(); } else { // In production, load the built frontend mainWindow.loadFile(path.join(__dirname, '../frontend/dist/index.html')); } } app.whenReady().then(() => { createWindow(); // Check for updates in production if (process.env.NODE_ENV !== 'development') { autoUpdater.checkForUpdatesAndNotify(); } }); app.on('window-all-closed', () => { if (process.platform !== 'darwin') { app.quit(); } }); app.on('activate', () => { if (BrowserWindow.getAllWindows().length === 0) { createWindow(); } }); // Auto-updater events autoUpdater.on('update-available', () => { console.log('Update available'); }); autoUpdater.on('update-downloaded', () => { console.log('Update downloaded'); // Optionally prompt user to restart }); javascript // electron/preload.js const { contextBridge, ipcRenderer } = require('electron'); // Expose protected methods that allow the renderer process to use // the ipcRenderer without exposing the entire object contextBridge.exposeInMainWorld('electronAPI', { getVersion: () => ipcRenderer.invoke('get-version'), platform: process.platform, }); json { "name": "my-app-desktop", "version": "1.0.0", "description": "My Application - Desktop", "main": "main.js", "scripts": { "start": "electron .", "build": "electron-builder", "build:win": "electron-builder --win", "build:mac": "electron-builder --mac", "build:linux": "electron-builder --linux" }, "build": { "appId": "com.example.myapp", "productName": "My Application", "directories": { "output": "dist" }, "files": [ "main.js", "preload.js", "../frontend/dist//" ], "win": { "target": ["nsis", "portable"], "icon": "icons/icon.ico" }, "mac": { "target": ["dmg", "zip"], "icon": "icons/icon.icns", "category": "public.app-category.productivity" }, "linux": { "target": ["AppImage", "deb"], "icon": "icons", "category": "Utility" }, "publish": { "provider": "github", "owner": "username", "repo": "my-app" } }, "dependencies": { "electron-updater": "^6.1.0" }, "devDependencies": { "electron": "^28.0.0", "electron-builder": "^24.0.0" } } javascript // electron/main.js - Embedded backend example const { spawn } = require('child_process'); const path = require('path'); let backendProcess; function startBackend() { const backendPath = path.join(__dirname, '../backend'); backendProcess = spawn('python', ['-m', 'uvicorn', 'my_app.main:app', '--port', '8000'], { cwd: backendPath, env: { ...process.env, PYTHONUNBUFFERED: '1' }, }); backendProcess.stdout.on('data', (data) => { console.log(`Backend: ${data}`); }); backendProcess.stderr.on('data', (data) => { console.error(`Backend error: ${data}`); }); } app.whenReady().then(() => { startBackend(); // Wait for backend to start, then create window setTimeout(createWindow, 2000); }); app.on('before-quit', () => { if (backendProcess) { backendProcess.kill(); } }); yaml # .github/workflows/ci.yml name: CI on: push: branches: [main] pull_request: branches: [main] jobs: backend-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: '3.11' - name: Install uv run: pip install uv - name: Install dependencies working-directory: ./backend run: uv sync --all-extras - name: Run tests working-directory: ./backend run: uv run pytest --cov frontend-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Node uses: actions/setup-node@v4 with: node-version: '20' cache: 'npm' cache-dependency-path: frontend/package-lock.json - name: Install dependencies working-directory: ./frontend run: npm ci - name: Run tests working-directory: ./frontend run: npm test - name: Build working-directory: ./frontend run: npm run build yaml # .github/workflows/build.yml name: Build on: push: tags: - 'v*' jobs: build-web: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - name: Login to GitHub Container Registry uses: docker/login-action@v3 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Build and push uses: docker/build-push-action@v5 with: context: . file: ./docker/Dockerfile.combined push: true tags: ghcr.io/${{ github.repository }}:${{ github.ref_name }} build-desktop: runs-on: ${{ matrix.os }} strategy: matrix: os: [ubuntu-latest, windows-latest, macos-latest] steps: - uses: actions/checkout@v4 - name: Set up Node uses: actions/setup-node@v4 with: node-version: '20' - name: Install frontend dependencies working-directory: ./frontend run: npm ci - name: Build frontend working-directory: ./frontend run: npm run build - name: Install Electron dependencies working-directory: ./electron run: npm ci - name: Build Electron app working-directory: ./electron run: npm run build env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Upload artifacts uses: actions/upload-artifact@v4 with: name: desktop-${{ matrix.os }} path: electron/dist/* release: needs: [build-web, build-desktop] runs-on: ubuntu-latest steps: - name: Download all artifacts uses: actions/download-artifact@v4 - name: Create Release uses: softprops/action-gh-release@v1 with: files: | desktop-ubuntu-latest/* desktop-windows-latest/* desktop-macos-latest/* generate_release_notes: true ``` ## End-to-End Testing with Playwright Once you're distributing across web, PWA, and desktop, you need confidence that the full application works as users experience it. Unit tests verify your Python backend logic; Playwright verifies that buttons click, pages load, and workflows complete. ### Why Playwright Playwright is a browser automation framework with first-class Python support. Compared to alternatives like Selenium or Cypress: - Multi-browser: Tests run against Chromium, Firefox, and WebKit from a single API - Python-native: Integrates directly with pytest via `pytest-playwright` - Electron support: Can test desktop apps, not just web pages - Auto-waiting: No manual sleep statements—Playwright waits for elements to be ready - Trace viewer: Built-in debugging tool that records screenshots, DOM snapshots, and network activity ### Installation ### Testing Your Web Application A basic test that verifies your FastAPI + React app works end-to-end: ### Testing Electron Desktop Apps Playwright can connect to Electron applications directly, testing your desktop builds with the same API: ### Integrating E2E Tests into CI/CD Add Playwright to your existing GitHub Actions workflow: ### E2E Testing Best Practices 1. Test user workflows, not implementation details: Click buttons by their text, not by CSS selectors that change with refactors 2. Keep E2E tests focused: A handful of critical path tests is worth more than exhaustive UI coverage 3. Use the trace viewer for debugging: Run `playwright show-trace trace.zip` to step through failures visually 4. Run against a real backend: E2E tests should exercise the full stack—mocking defeats the purpose 5. Mark E2E tests separately: Use `@pytest.mark.e2e` so you can run them independently of unit tests ## The Human's Role in This Architecture With this architecture in place, the human developer's role shifts to: ### Architecture Decisions - Defining the API contract (endpoints, data models) - Choosing the right distribution targets for the use case - Making tradeoffs between complexity and capability ### Verification - Reviewing AI-generated code for correctness - Ensuring tests cover critical paths - Validating that the API contract is maintained ### Orchestration - Managing the build and release pipeline - Coordinating between backend and frontend work - Handling platform-specific edge cases ### User Experience - Making design decisions AI can't make - Understanding user needs and requirements - Testing across platforms and environments The AI handles implementation details—writing the FastAPI routes, React components, Docker configurations—while you focus on the architecture, verification, and delivery. ## Summary Multi-platform distribution is achievable for indie developers through: 1. API-first architecture: FastAPI backend + React frontend with clear boundaries 2. Container deployment: Docker for web distribution 3. PWA capabilities: Installable web apps with offline support 4. Electron packaging: Native desktop apps for Windows, macOS, and Linux 5. Automated CI/CD: GitHub Actions for multi-platform builds This architecture is deliberately simple—"simple but not simplistic"—because simplicity enables effective AI collaboration. Clear structures, typed interfaces, and well-defined boundaries let you leverage AI for implementation while maintaining control over the architecture and delivery. In the next chapter, we'll explore how to structure these templates and configurations for maximum reusability across projects. ============================================================ SOURCE: chapters/notebooks.qmd ============================================================ # Beyond Scripts: Notebooks, Dashboards, and Interactive Python ## Chapter Overview Scripts aren't the only way to ship Python. This chapter explores notebooks, dashboards, and interactive tools—legitimate ways to share, deploy, and deliver Python work. ## Why This Chapter Appears Late (But Isn't an Afterthought) This chapter comes near the end of the book, but that's intentional—not because notebooks are less important. We wanted you to first understand the complete script-based workflow: project structure, testing, documentation, packaging, and distribution. These fundamentals apply whether you're writing scripts OR notebooks. Now that you understand the "full" workflow, you can appreciate both when notebooks simplify things and when they create limitations. You'll recognise that a `requirements.txt` matters for Binder just as it does for pip installs, that documentation practices transfer to notebook markdown cells, and that nbdev's testing approach builds on pytest concepts you already know. Notebooks are a first-class citizen in the Python ecosystem—not an alternative for people who can't handle "real" development. Many professional data scientists, researchers, and educators ship exclusively through notebooks. This chapter gives you the complete picture. ## The Trade-Off Up Front Notebooks are often easier to share than packaged scripts—a Colab link gives anyone instant access with zero installation. But they expose your code by default, which creates friction for some audiences. Not everyone wants to scroll past Python cells to see results. Tools like Voilà and Mercury address this, but it's worth knowing: the simplicity of notebooks comes with visibility trade-offs. ## The Three Ways to Write Python There are only three ways to write Python code: 1. REPL - Interactive exploration (not for shipping) 2. Scripts - Traditional apps, packages, CLI tools (this book's main focus) 3. Notebooks - Data analysis, reports, teaching, prototypes This book has focused primarily on scripts. But for many Python practitioners—especially in data science, education, and research—notebooks ARE the deliverable. A well-structured notebook with a sharing link is shipping. ## When Notebooks Make Sense Notebooks excel when: - The narrative matters - Analysis with explanation, teaching materials - Exploration is the product - Data investigation, research findings - Visuals are central - Charts, plots, interactive widgets - Reproducibility is key - Share exact environment and execution order - Zero-install is required - Viewers shouldn't need to set up Python Notebooks are less suited for: - Production APIs or services - CLI tools - Reusable libraries (though nbdev challenges this) - Long-running applications ## Sharing and Viewing ### GitHub Native Rendering GitHub renders `.ipynb` files automatically. Simply push your notebook: Viewers see rendered output without running code. Limitations: large notebooks may not render, and formatting can be inconsistent. ### nbviewer nbviewer.org provides cleaner rendering: - Better formatting than GitHub - Supports Gists - Cacheable links for sharing ### Gists for Quick Sharing For standalone notebooks: 1. Create a Gist at gist.github.com 2. Upload your `.ipynb` file 3. Share via nbviewer: `https://nbviewer.org/gist/username/gist_id` ## Zero-Install Execution The power of notebooks: viewers can RUN your code without installing anything. ### Google Colab The most accessible option. Add a badge to your README: Colab advantages: - Zero setup for viewers - Free GPU/TPU access - Google Drive integration - GitHub integration (open directly from repos) Colab workflow with GitHub: 1. Develop locally or in Colab 2. Save to GitHub (File → Save a copy to GitHub) 3. Share Colab link that opens from GitHub 4. Viewers get latest version automatically This gives you version control (GitHub) with zero-install execution (Colab)—a "clunky Dropbox" that's actually better because it's versioned. ### Binder mybinder.org turns any GitHub repo into interactive notebooks: Binder advantages: - Works with `requirements.txt` or `environment.yml` - Full JupyterLab environment - No Google account required Binder limitations: - Slower startup (builds environment) - Sessions timeout - Limited resources ### Kaggle Kernels For data science work, Kaggle provides: - Free GPU access - Built-in datasets - Community sharing - Competition integration ## Notebooks as Applications Transform notebooks into interactive applications that hide the code. ### Voilà Voilà converts notebooks into standalone dashboards: - Renders only output cells (code hidden) - Supports ipywidgets for interactivity - Deploy on Heroku, Binder, or your own server ### Mercury Mercury turns notebooks into web apps: - Automatic widget generation from parameters - PDF/HTML export - Authentication support - Self-hostable ### Streamlit (Notebook-Adjacent) Not notebooks, but similar rapid-development workflow: - Python scripts, not notebooks - But similar "write and see" iteration - Easy deployment via Streamlit Cloud ### Panel and HoloViz For more complex dashboards: - Panel - Flexible dashboarding from notebooks - HoloViews - High-level plotting - hvPlot - Interactive pandas plots ## Notebooks as Libraries: nbdev nbdev flips the script: develop libraries FROM notebooks. ## See It In Action We dedicate to building a complete package with nbdev—TextKit, a text analysis library. That case study parallels the SimpleBot chapter () but follows the notebook-first workflow. Think of nbdev as an alternative path to the same destination: instead of writing `.py` files and separate docs, you write notebooks that generate both. The end result—a published package—is the same. The nbdev philosophy: - Write code, tests, and documentation together - Export specific cells to modules - Generate API docs automatically - Literate programming for Python Basic workflow: When to use nbdev: - You naturally develop in notebooks - Documentation and code should live together - Teaching libraries where explanation matters ## Low-Code Alternatives ### Anvil Anvil provides a different model: - Drag-and-drop UI builder - Write Python for event handlers - Hosted deployment included - Database and user management built-in Anvil is good for: - Internal business tools - Forms and data entry - Quick prototypes with real UIs - Teaching event-driven programming Trade-offs: - Vendor lock-in - Less "Pythonic" project structure - Limited customisation ## Notebook Best Practices for Shipping ### Structure Your Notebooks ### Environment Management Include a requirements cell: Or ship with `requirements.txt` / `environment.yml` for Binder. ### Clear Outputs vs. Keep Outputs | Approach | When | |----------|------| | Clear outputs | Version control (smaller diffs) | | Keep outputs | Sharing (viewers see results immediately) | Consider: clear for development, render for sharing. ### Use nbstripout Automatically strip outputs on commit: ### Cell Tags and Metadata Use tags for tools like Voilà and nbdev: - `#| hide` - Hide cell in output - `#| export` - Export to module (nbdev) - `#| test` - Mark as test (nbdev) ## Comparison Table | Tool | Type | Hosting | Best For | |------|------|---------|----------| | GitHub + nbviewer | View only | Free | Simple sharing | | Colab | Interactive | Free (Google) | Zero-install, GPU | | Binder | Interactive | Free | Reproducible environments | | Voilà | Dashboard | Self-host/Binder | Hide code, show results | | Mercury | Web app | Self-host | Parameterized reports | | Streamlit | Web app | Streamlit Cloud | Rapid app development | | nbdev | Library dev | PyPI | Literate programming | | Anvil | Full app | Anvil servers | Low-code business apps | ## When to Graduate from Notebooks Notebooks are great, but sometimes you need to move to scripts: - Tests are growing - pytest is better than notebook tests - Reuse across projects - Package your code - Production deployment - APIs, services, CLI tools - Team collaboration - Notebooks have merge conflicts The path: Notebook → extract functions to `.py` → package → tests → CI/CD. Or use nbdev to keep working in notebooks while generating proper packages. ## Summary - Notebooks are a legitimate shipping format - "Shipping" can mean a Colab link, not just a packaged app - Multiple tools exist to share, execute, and transform notebooks - Choose based on your audience: viewers, runners, or users - Know when to graduate to scripts (or use nbdev to avoid the choice) Many notebook environments now include built-in AI assistants. Google Colab has Gemini, and JupyterLab supports extensions for Copilot and other AI tools. These are particularly useful for exploratory data analysis — ask AI to generate plotting code, explain error messages, or suggest the next analysis step. The interactive, cell-by-cell nature of notebooks makes AI collaboration especially natural: generate code in one cell, review and modify it, then run it immediately. Next: In , we'll build TextKit — a complete Python package developed entirely in notebooks using nbdev. ## Exercises 1. Share a notebook: Push a notebook to GitHub and create both an nbviewer link and a Colab badge. 2. Try Binder: Add a `requirements.txt` to a repo and create a Binder link. 3. Build a dashboard: Take an analysis notebook and convert it to a Voilà dashboard. 4. Explore nbdev: Create a simple function in a notebook and export it to a Python module using nbdev. ============================================================ SOURCE: chapters/notebook-case-study.ipynb ============================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Case Study: From Notebook to Package with nbdev \n", "\n", "::: \n", "## Chapter Overview\n", "This case study parallels (SimpleBot), but follows a notebook-first workflow. We'll build TextKit\u2014a text analysis library\u2014entirely in Jupyter notebooks, then ship it as a published Python package using nbdev.\n", ":::\n", "\n", "## Project Overview\n", "\n", "TextKit is a lightweight text analysis library that provides simple utilities for analyzing text. Key features include:\n", "\n", "- Word and character statistics\n", "- Readability scoring (Flesch-Kincaid, etc.)\n", "- Basic sentiment indicators\n", "- Text cleaning utilities\n", "\n", "This project is ideal for our notebook case study because:\n", "\n", "- Natural notebook fit: Text analysis involves exploration and visualization\n", "- Keeps the theme: Complements SimpleBot's chatbot focus (analyzing what bots produce)\n", "- Real utility: Functions you'd actually use in data analysis\n", "- Right size: Small enough to complete, complex enough to demonstrate the workflow\n", "\n", "By the end of this chapter, you'll have a package published to PyPI\u2014built entirely from notebooks.\n", "\n", "## Why nbdev for This Project?\n", "\n", "In , we introduced nbdev as a way to develop libraries from notebooks. Here's why it fits TextKit:\n", "\n", "| Traditional Workflow | nbdev Workflow |\n", "|---------------------|----------------|\n", "| Write code in `.py` files | Write code in notebooks |\n", "| Write separate test files | Tests live next to code |\n", "| Write docs separately | Docs generated from notebooks |\n", "| Context switching | Single environment |\n", "\n", "For exploratory, iterative work like text analysis, nbdev keeps everything together.\n", "\n", "## 1. Setting Up the nbdev Project\n", "\n", "### Installing nbdev\n", "\n", "\n", "\n", "### Creating the Project\n", "\n", "\n", "\n", "This creates:\n", "\n", "\n", "\n", "### Key Insight: You Edit Notebooks, Not .py Files\n", "\n", "The `textkit/` directory contains generated code. Your source of truth is `nbs/*.ipynb`.\n", "\n", "## 2. Building the Core Module\n", "\n", "### The First Notebook: `00_core.ipynb`\n", "\n", "Open `nbs/00_core.ipynb` in Jupyter. The structure:\n", "\n", "\n", "\n", "This directive tells nbdev: \"export cells from this notebook to `textkit/core.py`\".\n", "\n", "### Exporting Functions\n", "\n", "\n", "\n", "The `#| export` directive marks this cell for inclusion in the generated module.\n", "\n", "### Exploring as You Build\n", "\n", "This is where notebooks shine. Between exported cells, add exploration:\n", "\n", "\n", "\n", "Your notebook becomes both implementation AND documentation of your thinking.\n", "\n", "## 3. Adding Tests with nbdev\n", "\n", "### Inline Doctests\n", "\n", "The docstring examples above ARE tests. nbdev runs them automatically:\n", "\n", "\n", "\n", "### Dedicated Test Cells\n", "\n", "For more complex tests:\n", "\n", "\n", "\n", "### Running Tests\n", "\n", "\n", "\n", "## 4. Building More Functionality\n", "\n", "### Readability Scores\n", "\n", "\n", "\n", "### Helper Functions\n", "\n", "\n", "\n", "\n", "\n", "## 5. Visualizations in Your Notebook\n", "\n", "Notebooks excel at visual exploration. Add analysis cells (not exported):\n", "\n", "\n", "\n", "This visualization appears in your generated documentation\u2014showing users what the library can do.\n", "\n", "## 6. Building the Text Analyzer Class\n", "\n", "For a more complete API, add a class that combines functionality:\n", "\n", "\n", "\n", "## 7. Adding an Interactive Widget\n", "\n", "End with something users can interact with\u2014demonstrating the notebook as an application:\n", "\n", "\n", "\n", "When viewed in Colab or Binder, users can interact with your library without installing anything.\n", "\n", "## 8. Generating the Package\n", "\n", "### Export to Python Modules\n", "\n", "\n", "\n", "This generates `textkit/core.py` from your notebook's `#| export` cells.\n", "\n", "### Verify Everything Works\n", "\n", "\n", "\n", "### The Generated Code\n", "\n", "Look at `textkit/core.py`\u2014it contains clean Python code generated from your notebooks, with proper imports and structure.\n", "\n", "## 9. Documentation\n", "\n", "### The Index Notebook\n", "\n", "`nbs/index.ipynb` becomes both your README.md and documentation homepage. Include:\n", "\n", "1. Installation instructions\n", "2. Quick start example\n", "3. Feature overview\n", "\n", "bash\n", "pip install textkit\n", "python\n", "from textkit.core import TextAnalyzer\n", "\n", "text = \"Your text here. Analyze it easily.\"\n", "analyzer = TextAnalyzer(text)\n", "print(analyzer.summary())\n", "\n", "\n", "### Build Documentation\n", "\n", "\n", "\n", "This generates a Quarto-based documentation site in `_docs/`.\n", "\n", "## 10. Publishing to PyPI\n", "\n", "### Prepare for Release\n", "\n", "\n", "\n", "### Publish\n", "\n", "\n", "\n", "### The Result\n", "\n", "\n", "\n", "You've shipped a Python package\u2014developed entirely in notebooks.\n", "\n", "## 11. Sharing the Notebook Itself\n", "\n", "Beyond the package, share the development notebook:\n", "\n", "### Colab Badge\n", "\n", "\n", "\n", "### Binder Badge\n", "\n", "\n", "\n", "Users can:\n", "\n", "1. Install the package via pip (traditional)\n", "2. Explore the notebook to understand the code (educational)\n", "3. Run interactively in Colab/Binder (zero-install)\n", "\n", "## Comparing Workflows\n", "\n", "Here's how this case study compares to the SimpleBot approach ():\n", "\n", "| Aspect | SimpleBot (Scripts) | TextKit (nbdev) |\n", "|--------|---------------------|-----------------|\n", "| Source files | `.py` in `src/` | `.ipynb` in `nbs/` |\n", "| Tests | Separate `tests/` directory | Inline with code |\n", "| Documentation | Separate `docs/` | Generated from notebooks |\n", "| Exploration | Separate REPL/scratch files | Integrated in notebooks |\n", "| Output | Package on PyPI | Package on PyPI |\n", "| Best for | Traditional dev, teams | Exploratory, teaching |\n", "\n", "Both workflows produce the same result: a published package. Choose based on how you like to work.\n", "\n", "## When to Use This Workflow\n", "\n", "The nbdev approach works best when:\n", "\n", "- Exploration is central: You're figuring things out as you build\n", "- Teaching matters: Others will learn from your notebooks\n", "- Docs should show execution: You want live examples in documentation\n", "- Solo or small team: Git conflicts in notebooks are real\n", "\n", "Consider traditional scripts when:\n", "\n", "- Large teams: Notebook diffs are harder to review\n", "- Complex architecture: Many interconnected modules\n", "- Heavy IDE reliance: Refactoring tools work better with `.py` files\n", "- Existing codebase: Converting to nbdev is non-trivial\n", "\n", "## Summary\n", "\n", "- nbdev inverts the workflow: Notebooks are source, `.py` files are generated\n", "- Tests live with code: Doctests and `#| test` cells eliminate context switching\n", "- Exploration becomes documentation: Your investigative work helps users\n", "- Same destination: Published package, installable via pip\n", "- Different journey: Iterative, visual, integrated\n", "\n", "## Exercises\n", "\n", "1. Extend TextKit: Add a `sentiment_words()` function that counts positive/negative words from a simple word list. Include doctests.\n", "\n", "2. Add a notebook: Create `01_advanced.ipynb` with functions for text comparison (e.g., similarity between two texts).\n", "\n", "3. Publish to TestPyPI: Go through the full publication workflow to TestPyPI.\n", "\n", "4. Create a Voil\u00e0 dashboard: Convert the interactive widget section into a standalone Voil\u00e0 dashboard.\n", "\n", "5. Compare workflows: Take one function from TextKit and rewrite it in the traditional script workflow. Reflect on the differences." ] } ], "metadata": { "kernelspec": { "name": "python3", "language": "python", "display_name": "Python 3 (ipykernel)", "path": "/Users/michael/Library/Application Support/uv/envs/general/share/jupyter/kernels/python3" } }, "nbformat": 4, "nbformat_minor": 4 } ============================================================ SOURCE: chapters/conclusion.qmd ============================================================ # Conclusion: Embracing Efficient Python Development Throughout this guide, we've built a comprehensive Python development pipeline that balances simplicity with professional practices. From project structure to deployment, we've covered tools and techniques that help create maintainable, reliable, and efficient Python code. ## The Power of a Complete Pipeline Each component of our development workflow serves a specific purpose: - Project structure provides organisation and clarity - Version control enables collaboration and change tracking - Virtual environments isolate dependencies - Dependency management ensures reproducible environments - Code formatting and linting maintain consistent, error-free code - Testing verifies functionality - Type checking catches type errors early - Security scanning prevents vulnerabilities - Dead code detection keeps projects lean - Documentation makes code accessible to others - CI/CD automates quality checks and deployment - Package publishing shares your work with the world Together, these practices create a development experience that is both efficient and enjoyable. You spend less time on repetitive tasks and more time solving the real problems your code addresses. ## Your Path Forward: A Practical Adoption Strategy The concepts in this book are most valuable when applied systematically. Here's a concrete roadmap for implementing these practices, tailored to different project stages and team sizes: ### For Your Next New Project (Week 1) Immediate implementation - Use these from day one: 1. Project structure: Start with the `src` layout and proper directory organisation 2. Version control: Initialize Git immediately with a proper `.gitignore` 3. Virtual environment: Use `uv` or `pip-tools` for dependency management 4. Basic automation: Set up Poe the Poet with essential tasks (`lint`, `test`, `format`) ### For Existing Projects (Month 1-2) Gradual integration - Add one practice per week: - Week 1: Add code formatting with Ruff (`uv run ruff format .`) - Week 2: Introduce basic testing with pytest - Week 3: Add pre-commit hooks for automated quality checks - Week 4: Set up task automation with Poe the Poet - Week 5: Add type checking with mypy - Week 6: Implement basic CI/CD with GitHub Actions This pace prevents workflow disruption while building better practices. ### For Team Environments (Month 2-3) Collaborative workflows - Focus on consistency and shared practices: - Documentation standards: Establish README templates and docstring conventions - Code review processes: Define what automated checks must pass before review - Shared configurations: centralise tool configuration in `pyproject.toml` - Development environment parity: Use containers or detailed setup documentation ### Advanced Techniques (Month 3+) Only after mastering the fundamentals: - Performance optimisation: When benchmarks indicate actual problems - Advanced architecture: When code complexity impedes development - Containerization: When environment consistency becomes problematic ## AI as Your Development Partner Throughout this book, we have built a professional Python development pipeline. AI amplifies every part of it: - Project scaffolding: AI generates directory structures, configuration files, and boilerplate - Code quality: AI explains linting errors, suggests type annotations, and writes docstrings - Testing: AI generates test cases, including edge cases you might miss - Documentation: AI drafts READMEs, API docs, and contributing guides - CI/CD: AI generates GitHub Actions workflows and Dockerfile configurations - Debugging: AI analyses error messages and suggests fixes The critical insight is that AI works best when you understand what it produces. The pipeline practices in this book give you that understanding. Without knowing what a good `pyproject.toml` looks like, you cannot evaluate AI-generated configuration. Without understanding pytest fixtures, you cannot judge AI-generated tests. The fundamentals make AI useful rather than dangerous. The title of this book is deliberate. Orchestrate AI means you direct the work, review the output, and make the decisions. AI generates code faster than you can type it — but you still need to know whether that code is correct, secure, and maintainable. The pipeline you have built is your quality gate. Everything AI produces passes through it: linted by Ruff, checked by mypy, tested by pytest, reviewed by you. ## Beyond Tools: Engineering Culture The most important outcome isn't just using specific tools—it's developing habits and values that lead to better software: - Think defensively: Use tools that catch mistakes early - Value maintainability: Write code for humans, not just computers - Embrace automation: Let computers handle repetitive tasks - Practice continuous improvement: Regularly refine your workflow - Share knowledge: Document not just what code does, but why ## When to Consider More Advanced Tools As your projects grow more complex, you might explore more sophisticated tools: - Containerization with Docker for consistent environments - Orchestration with Kubernetes for complex deployments - Monorepo tools like Pants or Bazel for large codebases - Feature flagging for controlled feature rollouts - Advanced monitoring for production insights However, the core practices we've covered will remain valuable regardless of the scale you reach. ## Common Implementation Challenges and Solutions As you implement these practices, you'll likely encounter some common obstacles. Here's how to address them: ### "This Seems Like Too Much Overhead" Symptom: Tools feel burdensome and slow down development Solution: Start smaller and focus on automation - Begin with just `ruff format` and `pytest` - Use pre-commit hooks to make quality checks automatic - Remember: 5 minutes of setup saves hours of debugging later ### "My Team Resists New Processes" Symptom: Team members bypass or ignore new practices Solution: Lead by example and demonstrate value - Start with your own projects and show improved outcomes - Introduce practices that solve existing pain points - Make adherence easy with good tooling and clear documentation ### "Tool Configuration is Confusing" Symptom: Conflicting configurations or unclear settings Solution: Use our recommended starting templates - Copy configuration from successful projects - Use the companion templates to bootstrap correctly - Focus on standard configurations before customising ### "I Don't Know When to Add Advanced Practices" Symptom: Uncertainty about when complexity is justified Solution: Let pain points guide your decisions - Add testing when manual verification becomes tedious - Add CI/CD when manual releases cause errors - Add advanced architecture when code becomes hard to maintain - Never add complexity that doesn't solve an actual problem ## Staying Updated and Growing Python's ecosystem continues to evolve. Maintain relevance by: ### Following Core Development Principles - Python Enhancement Proposals (PEPs): Understand the direction of the language - Community discussions: Participate in forums like Python Discourse or Reddit r/Python - Release notes: Read updates for your core dependencies (pytest, ruff, uv, etc.) ### Practical Learning Approach - Test new tools in small projects before adopting them in production - Attend conferences or meetups (virtual or in-person) for broader perspective - Read other people's code to see different implementation approaches - Contribute to open source to deepen understanding of development practices ### Continuous Improvement Mindset - Regular retrospectives: What's working well? What's causing friction? - Experiment with alternatives: Try new tools when they solve specific problems - Share knowledge: Write about your experiences and learn from feedback ## Final Thoughts This book represents more than a collection of Python tools—it's a philosophy of development that prioritizes sustainability, maintainability, and developer happiness. The practices we've explored create a foundation that serves projects from first prototype to production scale. ### The Universal Principles Behind the Tools While we've used Python tooling as our examples, the core concepts transfer across languages and domains: - Clear project structure reduces cognitive load in any language - Automated quality checks catch errors early regardless of the technology stack - Comprehensive testing provides confidence when making changes - Thoughtful automation eliminates repetitive work and reduces human error - Progressive complexity allows practices to evolve with project needs These principles remain constant even as specific tools evolve. ### Your Development Journey Continues The practices in this book form a foundation, not a destination. As you apply these concepts: - Trust the process: Initially, some practices may feel like overhead, but their value becomes clear as projects grow - Adapt to your context: Not every practice fits every project, but understanding the principles helps you make informed decisions - Share your knowledge: Teaching others these practices deepens your own understanding and improves the broader development community ### Your Pipeline Is Agent-Ready Infrastructure AI coding agents, tools like Cursor, Claude Code, and OpenHands, are getting remarkably capable. They can generate code, run tests, fix errors, and iterate. But they work best when they have structure to work within. Everything in this book builds that structure. A test suite gives an agent a way to verify its own output. A CI pipeline catches mistakes before they reach users. A clear project layout gives an agent context about where code belongs. Pre-commit hooks enforce quality standards automatically. Without this infrastructure, an agent is generating code into a void. With it, an agent has guardrails, feedback loops, and a deployment path. The pipeline you have built is what turns AI from a code generator into a reliable development partner. That is what orchestration means. ### Starting Your Next Project You now have everything needed to begin any Python project with professional practices from day one. Whether you use our bash script for transparency, GitHub templates for convenience, or cookiecutter templates for customisation, you can establish solid foundations in minutes rather than hours. More importantly, you understand why these practices matter and when to apply them. This knowledge will serve you well as you encounter new challenges and evaluate new tools. ### A Personal Note Remember that perfect is the enemy of good. Start with the basics, improve incrementally, and focus on delivering value through your code. The best development pipeline is one that you'll actually use consistently. The Python ecosystem will continue evolving—new tools will emerge, and current tools will improve—but the underlying principles of clear structure, automated quality, comprehensive testing, and thoughtful automation will remain valuable throughout your development career. We hope this guide helps you build software that not only works but is also maintainable, reliable, and enjoyable to develop. The investment you make in better development practices pays dividends for years to come. Happy coding, and may your development pipeline serve you well! ============================================================ SOURCE: acknowledgments.qmd ============================================================ # Acknowledgments This book started as a personal quest: find a sane Python development workflow that could target multiple platforms without drowning in complexity. The practices in these pages were tested on real projects before they were written up, and the ones that survived are the ones that actually worked under pressure. Students who used earlier versions of these workflows — in project templates, lab exercises, and assignment scaffolding — provided the practical feedback that separated good ideas from impractical ones. When a workflow looked elegant in documentation but caused confusion in practice, they said so. The Python community deserves particular credit. The developers behind uv, ruff, mypy, pytest, and the broader ecosystem of tools covered in this book have made professional Python development accessible in ways that were not possible even a few years ago. Their work is the foundation this book builds on. The open source community behind Quarto, GitHub, and GitHub Pages made it possible to write, build, and publish across multiple formats. The cookiecutter and template ecosystems made it possible to turn the book's recommendations into something readers can use immediately. AI tools were used throughout the writing process. Claude (Anthropic) served as a conversation partner for drafting, iterating, and refining. The process was the same one the book advocates: orchestrate the AI, but own the decisions. Every recommendation, every tool choice, every architectural opinion reflects the author's judgement. The AI made the work faster. It did not make the decisions. ============================================================ SOURCE: about-author.qmd ============================================================ # About the Author Michael Borck is a software developer and educator passionate about the intersection of human expertise and artificial intelligence. He developed the Intentional Prompting methodology to help programmers maintain agency and deepen their understanding while leveraging AI tools effectively. Michael believes that the future of programming lies not in delegating to AI, but in conversing with it---treating AI as a collaborative partner that enhances human capability rather than replacing human understanding. When not writing about AI collaboration, Michael works on practical applications of these principles across software development, education, and creative projects. He creates educational software and resources, and explores the 80/20 principle in learning and productivity. --- Connect - michaelborck.dev --- Professional work and projects - michaelborck.education --- Educational software and resources - 8020workshop.com --- Passion projects and workshops - LinkedIn --- Other Books in This Series Foundational Methodology: - Conversation, Not Delegation: Your Expertise + AI's Breadth = Amplified Thinking - Converse Python, Partner AI: The Python Edition Python Track: - Think Python, Direct AI: Computational Thinking for Beginners - Code Python, Consult AI: Python Fundamentals for the AI Era Web Track: - Build Web, Guide AI: Business Web Development with AI For Educators: - Partner, Don't Police: AI in the Business Classroom ============================================================ SOURCE: appendices/glossary.qmd ============================================================ # Glossary of Python Development Terms ## A - API (Application Programming Interface): A set of definitions and protocols for building and integrating application software. - Artifact: Any file or package produced during the software development process, such as documentation or distribution packages. ## C - CI/CD (Continuous Integration/Continuous Deployment): Practices where code changes are automatically tested (CI) and deployed to production (CD) when they pass quality checks. - CLI (Command Line Interface): A text-based interface for interacting with software using commands. - Code Coverage: A measure of how much of your code is executed during testing. - Code Linting: The process of analysing code for potential errors, style issues, and suspicious constructs. ## D - Dependency: An external package or module that your project requires to function properly. - Docstring: A string literal specified in source code that is used to document a specific segment of code. - Dynamic Typing: A programming language feature where variable types are checked during runtime rather than compile time. - - Cookiecutter: A project templating tool that helps developers create new projects with a predefined structure, configuration files, and boilerplate code. Cookiecutter uses Jinja2 templating to customise files based on user inputs during project generation. ## E - Entry Point: A function or method that serves as an access point to an application, module, or library. ## F - Fixture: In testing, a piece of code that sets up a system for testing and provides test data. ## G - Git: A distributed version control system for tracking changes in source code. - GitHub Repository Template: A repository that can be used as a starting point for new projects on GitHub. - GitHub/GitLab: Web-based platforms for hosting Git repositories with collaboration features. ## I - Integration Testing: Testing how different parts of the system work together. ## L - Lock File: A file that records the exact versions of dependencies needed by a project to ensure reproducible installations. ## M - Mocking: Simulating the behaviour of real objects in controlled ways during testing. - Module: A file containing Python code that can be imported and used by other Python files. - Monorepo: A software development strategy where many projects are stored in the same repository. ## N - Namespace Package: A package split across multiple directories or distribution packages. ## P - Package: A directory of Python modules containing an additional `__init__.py` file. - PEP (Python Enhancement Proposal): A design document providing information to the Python community, often proposing new features. - PEP 8: The style guide for Python code. - PyPI (Python Package Index): The official repository for third-party Python software. ## R - Refactoring: Restructuring existing code without changing its external behaviour. - Repository: A storage location for software packages and version control. - Requirements File: A file listing the dependencies required for a Python project. - Reproducible Build: A build that can be recreated exactly regardless of when or where it's built. ## S - Semantic Versioning: A versioning scheme in the format MAJOR.MINOR.PATCH, where each number increment indicates the type of change. - Static Analysis: analysing code without executing it to find potential issues. - Static Typing: Specifying variable types at compile time instead of runtime. - Stub Files: Files that contain type annotations for modules that don't have native typing support. ## T - Test-Driven Development (TDD): A development process where tests are written before the code. - Type Annotation: Syntax for indicating the expected type of variables, function parameters, and return values. - Type Hinting: Adding type annotations to Python code to help with static analysis and IDE assistance. ## U - Unit Testing: Testing individual components in isolation from the rest of the system. ## V - Virtual Environment: An isolated Python environment that allows packages to be installed for use by a particular project, without affecting other projects. ## W - Wheel: A built-package format for Python that can be installed more quickly than source distributions. ============================================================ SOURCE: appendices/distribution-templates.qmd ============================================================ # Distribution Templates This appendix provides ready-to-use templates and configurations for multi-platform distribution. These templates complement the architecture patterns discussed in Chapter 8: Multi-Platform Distribution. ## Companion Repositories To quickly start a new project with this architecture, use one of our companion repositories: ### Cookiecutter Template For full customisation with interactive prompts: The cookiecutter template prompts for: - Project name and description - Author information - Backend options (database choice, authentication) - Frontend options (styling framework, state management) - Distribution targets (web, PWA, desktop) - CI/CD configuration ### GitHub Template For quick starts without installing tools: 1. Visit https://github.com/michael-borck/ship-python-template 2. Click "Use this template" 3. Clone your new repository and customise ### Example Repository A complete working example with all features: - https://github.com/michael-borck/ship-python-example ## FastAPI Backend Template ### pyproject.toml ### Application Configuration ### Database Setup with SQLAlchemy ### Backend Dockerfile ## React Frontend Template ### package.json ### Vite Configuration with PWA ### TypeScript Configuration ### API Client Template ```typescript // frontend/src/services/api.ts import axios, { AxiosInstance, AxiosError } from 'axios'; const API_BASE_URL = import.meta.env.VITE_API_URL || '/api'; class ApiClient { private client: AxiosInstance; constructor() { this.client = axios.create({ baseURL: API_BASE_URL, headers: { 'Content-Type': 'application/json', }, }); // Response interceptor for error handling this.client.interceptors.response.use( (response) => response, (error: AxiosError) => { if (error.response?.status === 401) { // Handle unauthorized window.location.href = '/login'; } return Promise.reject(error); } ); } async get(url: string): Promise { const response = await this.client.get(url); return response.data; } async post(url: string, data: unknown): Promise { const response = await this.client.post(url, data); return response.data; } async put(url: string, data: unknown): Promise { const response = await this.client.put(url, data); return response.data; } async delete(url: string): Promise { await this.client.delete(url); } setAuthToken(token: string): void { this.client.defaults.headers.common['Authorization'] = `Bearer ${token}`; } clearAuthToken(): void { delete this.client.defaults.headers.common['Authorization']; } } export const apiClient = new ApiClient(); javascript // electron/main.js const { app, BrowserWindow, Menu, shell, ipcMain } = require('electron'); const path = require('path'); const { autoUpdater } = require('electron-updater'); const log = require('electron-log'); // Configure logging log.transports.file.level = 'info'; autoUpdater.logger = log; // Handle Squirrel events for Windows if (require('electron-squirrel-startup')) { app.quit(); } const isDev = process.env.NODE_ENV === 'development'; let mainWindow; function createWindow() { mainWindow = new BrowserWindow({ width: 1200, height: 800, minWidth: 800, minHeight: 600, webPreferences: { preload: path.join(__dirname, 'preload.js'), contextIsolation: true, nodeIntegration: false, sandbox: true, }, show: false, // Show when ready titleBarStyle: 'hiddenInset', // macOS }); // Show window when ready mainWindow.once('ready-to-show', () => { mainWindow.show(); }); // Load content if (isDev) { mainWindow.loadURL('http://localhost:5173'); mainWindow.webContents.openDevTools(); } else { mainWindow.loadFile(path.join(__dirname, '../frontend/dist/index.html')); } // Open external links in browser mainWindow.webContents.setWindowOpenHandler(({ url }) => { shell.openExternal(url); return { action: 'deny' }; }); // Build menu const template = [ { label: 'File', submenu: [ { role: 'quit' } ] }, { label: 'Edit', submenu: [ { role: 'undo' }, { role: 'redo' }, { type: 'separator' }, { role: 'cut' }, { role: 'copy' }, { role: 'paste' }, { role: 'selectAll' } ] }, { label: 'View', submenu: [ { role: 'reload' }, { role: 'forceReload' }, { role: 'toggleDevTools' }, { type: 'separator' }, { role: 'resetZoom' }, { role: 'zoomIn' }, { role: 'zoomOut' }, { type: 'separator' }, { role: 'togglefullscreen' } ] }, { label: 'Help', submenu: [ { label: 'Documentation', click: async () => { await shell.openExternal('https://example.com/docs'); } }, { label: 'Check for Updates', click: () => { autoUpdater.checkForUpdatesAndNotify(); } } ] } ]; const menu = Menu.buildFromTemplate(template); Menu.setApplicationMenu(menu); } // App lifecycle app.whenReady().then(() => { createWindow(); // Check for updates (production only) if (!isDev) { autoUpdater.checkForUpdatesAndNotify(); } app.on('activate', () => { if (BrowserWindow.getAllWindows().length === 0) { createWindow(); } }); }); app.on('window-all-closed', () => { if (process.platform !== 'darwin') { app.quit(); } }); // IPC handlers ipcMain.handle('get-version', () => { return app.getVersion(); }); ipcMain.handle('check-for-updates', () => { return autoUpdater.checkForUpdatesAndNotify(); }); // Auto-updater events autoUpdater.on('checking-for-update', () => { log.info('Checking for update...'); }); autoUpdater.on('update-available', (info) => { log.info('Update available:', info); }); autoUpdater.on('update-not-available', (info) => { log.info('Update not available:', info); }); autoUpdater.on('error', (err) => { log.error('Error in auto-updater:', err); }); autoUpdater.on('download-progress', (progressObj) => { log.info(`Download speed: ${progressObj.bytesPerSecond} - Downloaded ${progressObj.percent}%`); }); autoUpdater.on('update-downloaded', (info) => { log.info('Update downloaded:', info); // Optionally prompt user and restart }); javascript // electron/preload.js const { contextBridge, ipcRenderer } = require('electron'); contextBridge.exposeInMainWorld('electronAPI', { // App info getVersion: () => ipcRenderer.invoke('get-version'), platform: process.platform, // Updates checkForUpdates: () => ipcRenderer.invoke('check-for-updates'), // Window controls (if using frameless window) minimise: () => ipcRenderer.send('window-minimise'), maximise: () => ipcRenderer.send('window-maximise'), close: () => ipcRenderer.send('window-close'), }); json { "name": "my-app", "version": "1.0.0", "description": "My Application", "main": "main.js", "author": "Your Name ", "license": "MIT", "scripts": { "start": "electron .", "build": "electron-builder", "build:all": "electron-builder -mwl", "build:mac": "electron-builder --mac", "build:win": "electron-builder --win", "build:linux": "electron-builder --linux", "release": "electron-builder --publish always" }, "build": { "appId": "com.yourname.myapp", "productName": "My Application", "copyright": "Copyright © 2025 Your Name", "directories": { "output": "dist", "buildResources": "build" }, "files": [ "main.js", "preload.js", "../frontend/dist//" ], "extraMetadata": { "main": "main.js" }, "mac": { "category": "public.app-category.productivity", "icon": "build/icon.icns", "hardenedRuntime": true, "gatekeeperAssess": false, "entitlements": "build/entitlements.mac.plist", "entitlementsInherit": "build/entitlements.mac.plist", "target": [ { "target": "dmg", "arch": ["x64", "arm64"] }, { "target": "zip", "arch": ["x64", "arm64"] } ] }, "win": { "icon": "build/icon.ico", "target": [ { "target": "nsis", "arch": ["x64"] }, { "target": "portable", "arch": ["x64"] } ] }, "nsis": { "oneClick": false, "perMachine": false, "allowToChangeInstallationDirectory": true, "deleteAppDataOnUninstall": true }, "linux": { "icon": "build/icons", "category": "Utility", "target": [ { "target": "AppImage", "arch": ["x64"] }, { "target": "deb", "arch": ["x64"] } ] }, "publish": { "provider": "github", "owner": "your-username", "repo": "my-app", "releaseType": "release" } }, "dependencies": { "electron-log": "^5.0.0", "electron-updater": "^6.1.0" }, "devDependencies": { "electron": "^28.0.0", "electron-builder": "^24.9.0" } } dockerfile # docker/Dockerfile.combined # Build stage for frontend FROM node:20-alpine AS frontend-builder WORKDIR /app/frontend COPY frontend/package*.json ./ RUN npm ci --production=false COPY frontend/ ./ RUN npm run build # Build stage for backend dependencies FROM python:3.11-slim AS backend-builder WORKDIR /app RUN pip install uv COPY backend/pyproject.toml backend/uv.lock ./ RUN uv sync --frozen --no-dev # Production stage FROM python:3.11-slim AS production WORKDIR /app # Install uv for running RUN pip install uv # Copy backend dependencies COPY --from=backend-builder /app/.venv /app/.venv # Copy backend source COPY backend/src ./src # Copy frontend build COPY --from=frontend-builder /app/frontend/dist ./static # Create non-root user RUN useradd -m -u 1000 appuser && \ chown -R appuser:appuser /app USER appuser # Environment ENV PATH="/app/.venv/bin:$PATH" ENV PYTHONPATH="/app/src" ENV STATIC_FILES_DIR="/app/static" EXPOSE 8000 HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" CMD ["uvicorn", "my_app.main:app", "--host", "0.0.0.0", "--port", "8000"] yaml # docker/docker-compose.yml version: "3.9" services: backend: build: context: ../backend dockerfile: Dockerfile.dev ports: - "8000:8000" volumes: - ../backend/src:/app/src:ro environment: - DEBUG=true - DATABASE_URL=sqlite:///./dev.db command: uv run uvicorn my_app.main:app --reload --host 0.0.0.0 --port 8000 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 30s timeout: 10s retries: 3 frontend: build: context: ../frontend dockerfile: Dockerfile.dev ports: - "5173:5173" volumes: - ../frontend/src:/app/src:ro - ../frontend/public:/app/public:ro environment: - VITE_API_URL=http://localhost:8000/api command: npm run dev -- --host 0.0.0.0 depends_on: - backend # Optional: PostgreSQL for production-like development db: image: postgres:16-alpine ports: - "5432:5432" environment: - POSTGRES_USER=myapp - POSTGRES_PASSWORD=devpassword - POSTGRES_DB=myapp volumes: - postgres_data:/var/lib/postgresql/data volumes: postgres_data: yaml # .github/workflows/ci-cd.yml name: CI/CD on: push: branches: [main] tags: ['v*'] pull_request: branches: [main] env: REGISTRY: ghcr.io IMAGE_NAME: ${{ github.repository }} jobs: # ============================================ # Testing # ============================================ test-backend: name: Test Backend runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: '3.11' - name: Install uv run: pip install uv - name: Install dependencies working-directory: ./backend run: uv sync --all-extras - name: Run linting working-directory: ./backend run: | uv run ruff check . uv run ruff format --check . - name: Run type checking working-directory: ./backend run: uv run mypy src/ - name: Run tests working-directory: ./backend run: uv run pytest --cov --cov-report=xml - name: Upload coverage uses: codecov/codecov-action@v3 with: files: ./backend/coverage.xml flags: backend test-frontend: name: Test Frontend runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Node.js uses: actions/setup-node@v4 with: node-version: '20' cache: 'npm' cache-dependency-path: frontend/package-lock.json - name: Install dependencies working-directory: ./frontend run: npm ci - name: Run linting working-directory: ./frontend run: npm run lint - name: Run type checking working-directory: ./frontend run: npm run typecheck - name: Run tests working-directory: ./frontend run: npm run test:coverage - name: Build working-directory: ./frontend run: npm run build # ============================================ # Docker Build # ============================================ build-docker: name: Build Docker Image needs: [test-backend, test-frontend] runs-on: ubuntu-latest if: github.event_name == 'push' permissions: contents: read packages: write steps: - uses: actions/checkout@v4 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - name: Log in to Container Registry uses: docker/login-action@v3 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Extract metadata id: meta uses: docker/metadata-action@v5 with: images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} tags: | type=ref,event=branch type=semver,pattern={{version}} type=semver,pattern={{major}}.{{minor}} type=sha - name: Build and push uses: docker/build-push-action@v5 with: context: . file: ./docker/Dockerfile.combined push: true tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} cache-from: type=gha cache-to: type=gha,mode=max # ============================================ # Desktop Builds # ============================================ build-desktop: name: Build Desktop (${{ matrix.os }}) needs: [test-backend, test-frontend] if: startsWith(github.ref, 'refs/tags/v') runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: include: - os: ubuntu-latest platform: linux - os: windows-latest platform: win - os: macos-latest platform: mac steps: - uses: actions/checkout@v4 - name: Set up Node.js uses: actions/setup-node@v4 with: node-version: '20' - name: Install frontend dependencies working-directory: ./frontend run: npm ci - name: Build frontend working-directory: ./frontend run: npm run build - name: Install Electron dependencies working-directory: ./electron run: npm ci - name: Build Electron app working-directory: ./electron run: npm run build:${{ matrix.platform }} env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Upload artifacts uses: actions/upload-artifact@v4 with: name: desktop-${{ matrix.platform }} path: | electron/dist/*.dmg electron/dist/*.zip electron/dist/*.exe electron/dist/*.AppImage electron/dist/*.deb if-no-files-found: ignore # ============================================ # Release # ============================================ release: name: Create Release needs: [build-docker, build-desktop] if: startsWith(github.ref, 'refs/tags/v') runs-on: ubuntu-latest permissions: contents: write steps: - uses: actions/checkout@v4 - name: Download all artifacts uses: actions/download-artifact@v4 with: path: artifacts - name: Create Release uses: softprops/action-gh-release@v1 with: files: artifacts// generate_release_notes: true draft: false prerelease: ${{ contains(github.ref, '-') }} text # Project: [Your Project Name] ## Overview [Brief description of what this project does] ## Architecture ### Stack - Backend: FastAPI (Python 3.11+) - Frontend: React 18 + TypeScript + Vite - Desktop: Electron 28 - Database: SQLite (dev) / PostgreSQL (prod) - Deployment: Docker, GitHub Actions ### Directory Structure project/ ├── backend/ # FastAPI application │ ├── src/my_app/ # Source code │ └── tests/ # Backend tests ├── frontend/ # React application │ └── src/ # Source code ├── electron/ # Desktop wrapper ├── docker/ # Container configs └── .github/ # CI/CD workflows ## Conventions ### Backend - All API routes under /api/ prefix - Pydantic models for all request/response schemas - SQLAlchemy for database operations - Async handlers where beneficial ### Frontend - Functional components with hooks - React Query for server state - Axios for API calls - TypeScript strict mode ### General - Semantic versioning (v1.2.3) - Conventional commits - All code formatted and linted before commit ## Development Commands ### Backend cd backend uv run uvicorn my_app.main:app --reload uv run pytest uv run ruff check . ### Frontend cd frontend npm run dev npm test npm run lint ### Docker docker-compose -f docker/docker-compose.yml up ## API Endpoints - GET /api/health - Health check - GET /api/items - List items - POST /api/items - Create item - GET /api/items/{id} - Get item - PUT /api/items/{id} - Update item - DELETE /api/items/{id} - Delete item ## Environment Variables - DEBUG - Enable debug mode (default: false) - DATABASE_URL - Database connection string - SECRET_KEY - Application secret key - CORS_ORIGINS - Allowed CORS origins ## Key Decisions 1. Using uv for Python dependency management (fast, reliable) 2. Electron over Tauri (more mature, better AI support) 3. SQLite for development (simple, no setup required) 4. Single Docker container for deployment (simpler ops) 5. GitHub Actions for CI/CD (integrated with repo) ## Common Tasks ### Adding a new API endpoint 1. Define Pydantic models in backend/src/my_app/models/ 2. Create route handler in backend/src/my_app/api/ 3. Add tests in backend/tests/ 4. Update TypeScript types in frontend/src/services/api.ts ### Adding a new React component 1. Create component in frontend/src/components/ 2. Use React Query hooks for data fetching 3. Add to relevant pages/routes ### Creating a release 1. Update version in relevant package files 2. Create git tag: git tag v1.2.3 3. Push tag: git push origin v1.2.3 4. GitHub Actions builds and publishes automatically ``` This appendix provides the essential templates for implementing the multi-platform distribution architecture. For complete, up-to-date templates with all configurations, see the companion repositories listed at the beginning of this appendix. ============================================================ SOURCE: appendices/ai-tools.qmd ============================================================ # AI Tools for Python Development The integration of AI into software development represents one of the most significant shifts in how developers work. This appendix provides an overview of AI tools available for Python development, guidance on how to use them effectively, and important considerations for their ethical use. ## Overview of Current AI Tools and Their Strengths ### Code Assistants and Completion Tools - GitHub Copilot: - Strengths: Real-time code suggestions directly in your IDE; trained on public GitHub repositories; understands context from open files - Best for: Rapid code generation, boilerplate reduction, exploring implementation alternatives - Integration: Available for VS Code, Visual Studio, JetBrains IDEs, and Neovim - JetBrains AI Assistant: - Strengths: Deeply integrated with JetBrains IDEs; code explanation and generation; documentation creation - Best for: PyCharm users; explaining complex code; generating docstrings - Integration: Built into PyCharm and other JetBrains products - Tabnine: - Strengths: Code completion with local models option; privacy-focused; adapts to your coding style - Best for: Teams with strict data privacy requirements; personalized code suggestions - Integration: Works with most popular IDEs including VS Code and PyCharm ### Conversational AI Assistants - Claude (Anthropic): - Strengths: Excellent reasoning capabilities; strong Python knowledge; handles lengthy context - Best for: Complex problem-solving; explaining algorithms; reviewing code; documentation creation - Access: Web interface, API, Claude Code (terminal) - ChatGPT/GPT-4 (OpenAI): - Strengths: Wide knowledge base; good at generating code and explaining concepts - Best for: Troubleshooting; learning concepts; brainstorming ideas; code generation - Access: Web interface, API, plugins for various platforms - Gemini (Google): - Strengths: Strong code analysis and generation; multimodal capabilities useful for analysing diagrams - Best for: Code support; learning resources; teaching concepts - Access: Web interface, API, Duet AI integrations ### AI-Enhanced Code Review Tools - DeepSource: - Strengths: Continuous analysis; focuses on security issues, anti-patterns, and performance - Best for: Automated code reviews; maintaining code quality standards - Integration: GitHub, GitLab, Bitbucket - Codiga: - Strengths: Real-time code analysis; custom rule creation; automated PR comments - Best for: Enforcing team-specific best practices; providing quick feedback - Integration: GitHub, GitLab, Bitbucket, and various IDEs - Sourcery: - Strengths: Python-specific refactoring suggestions; explains why changes are recommended - Best for: Learning better Python patterns; gradual code quality improvement - Integration: VS Code, JetBrains IDEs, GitHub ### AI Documentation Tools - Mintlify Writer: - Strengths: Auto-generates documentation from code; supports various docstring formats - Best for: Quickly documenting existing codebases; maintaining consistent documentation - Integration: VS Code, JetBrains IDEs - Docstring Generator AI: - Strengths: Creates detailed docstrings following specified formats (Google, NumPy, etc.) - Best for: Consistently formatting documentation across a project - Integration: VS Code extension ## Guidelines for Effective Prompting The quality of AI output depends significantly on how you formulate your requests. Here are strategies to get the most from AI tools when working with Python: ### General Prompting Principles 1. Be specific and detailed: Include relevant context about your project, such as Python version, frameworks used, and existing patterns to follow. 2. Provide examples: When you need code that follows certain patterns or styles, provide examples. 3. Use iterative refinement: Start with a basic request, then refine the results. 4. Specify output format: Clarify how you want information presented. ### Python-Specific Prompting Strategies 1. Request specific Python versions or features: Clarify which Python version you're targeting. 2. Specify testing frameworks: When requesting tests, mention your preferred framework. 3. Ask for alternative approaches: Python often offers multiple solutions to problems. 4. Request educational explanations: For learning purposes, ask the AI to explain its reasoning. ### Using AI for Code Review When using AI to review your Python code, structured prompts yield better results: ### Troubleshooting with AI When debugging problems, provide context systematically: ## Ethical Considerations and Limitations As you integrate AI tools into your Python development workflow, consider these important ethical considerations and limitations: ### Ethical Considerations 1. Intellectual Property and Licensing - Code generated by AI may be influenced by training data with various licenses - For commercial projects, consult your legal team about AI code usage policies - Consider adding comments attributing AI-generated sections when substantial 2. Security Risks - Never blindly implement AI-suggested security-critical code without review - AI may recommend outdated or vulnerable patterns it learned from older code - Verify cryptographic implementations, authentication mechanisms, and input validation independently 3. Overreliance and Skill Development - Balance AI usage with developing personal understanding - For educational settings, consider policies on appropriate AI assistance - Use AI to enhance learning rather than bypass it 4. Bias and Fairness - AI may perpetuate biases present in training data - Review generated code for potential unfair treatment or assumptions - Be especially careful with user-facing features and data processing pipelines 5. Environmental Impact - Large AI models have significant computational and energy costs - Consider using more efficient, specialized code tools for routine tasks - Batch similar requests when possible instead of making many small queries ### Technical Limitations 1. Knowledge Cutoffs - AI assistants have knowledge cutoffs and may not be aware of recent Python developments - Verify suggestions for newer Python versions or recently updated libraries - Example: An AI might not know about features introduced in Python 3.11 or 3.12 if its training cutoff predates them 2. Context Length Restrictions - Most AI tools have limits on how much code they can process at once - For large files or complex projects, focus queries on specific components - Provide essential context rather than entire codebases 3. Hallucinations and Inaccuracies - AI can confidently suggest incorrect implementations or non-existent functions - Always verify generated code works as expected - Be especially wary of package import suggestions, API usage patterns, and framework-specific code 4. Understanding Project-Specific Context - AI lacks full understanding of your project architecture and requirements - Generated code may not align with your established patterns or constraints - Always review for compatibility with your broader codebase 5. Time-Sensitive Information - Best practices, dependencies, and security recommendations change over time - Verify suggestions against current Python community standards - Double-check deprecation warnings and avoid outdated patterns ### Practical Mitigation Strategies 1. Code Review Process - Establish clear guidelines for reviewing AI-generated code - Use the same quality standards for AI-generated and human-written code - Consider automated testing requirements for AI contributions 2. Attribution and Documentation - Document where and how AI tools were used in your development process - Consider noting substantial AI contributions in code comments - Example: `# Initial implementation generated by GitHub Copilot, modified to handle edge cases` 3. Verification Practices - Test AI-generated code thoroughly, especially edge cases - Verify performance characteristics claimed by AI suggestions - Cross-check security recommendations with trusted sources 4. Balanced Use Policy - Develop team guidelines for appropriate AI tool usage - Encourage use for boilerplate, documentation, and creative starting points - Emphasize human oversight for architecture, security, and critical algorithms 5. Continuous Learning - Use AI explanations as learning opportunities - Ask AI to explain its suggestions and verify understanding - Build knowledge to reduce dependency on AI for core concepts ## The Future of AI in Python Development AI tools for Python development are evolving rapidly. Current trends suggest these future directions: - More specialized Python-specific models: Trained specifically on Python codebases with deeper framework understanding - Enhanced IDE integration: More seamless AI assistance throughout the development workflow - Improved testing capabilities: AI generating more comprehensive test suites with higher coverage - Custom models for organisations: Trained on internal codebases to better match company standards - Agent-based development: AI systems that can execute multi-step development tasks with minimal guidance As these tools evolve, maintaining a balanced approach that leverages AI strengths while preserving human oversight will remain essential for quality Python development. ============================================================ SOURCE: appendices/checklist.qmd ============================================================ # Python Development Workflow Checklist This checklist provides a practical reference for setting up and maintaining Python projects of different scales. Choose the practices that match your project's complexity and team size. | Development Stage | Simple/Beginner Project | Intermediate/Large Project | |-------------------|-------------------------|----------------------------| | Project Setup | | | | Create project structure | ✅ Basic directory with code and tests | ✅ Full `src` layout with package under `src/` | | Initialize version control | ✅ `git init` and basic `.gitignore` | ✅ Advanced `.gitignore` with branch strategies | | Add essential files | ✅ README.md | ✅ README.md, LICENSE, CONTRIBUTING.md | | | | | | Environment Setup | | | | Create virtual environment | ✅ `python -m venv .venv` | ✅ `uv venv` or containerized environment | | Track dependencies | ✅ `pip freeze > requirements.txt` | ✅ `requirements.in` with `pip-compile` or `uv pip compile` | | Install dependencies | ✅ `pip install -r requirements.txt` | ✅ `pip-sync` or `uv pip sync` | | | | | | Code Quality | | | | Formatting | ✅ Basic PEP 8 adherence | ✅ Automated with Ruff (`ruff format`) | | Linting | ❌ or basic Flake8 | ✅ Ruff with multiple rule sets enabled | | Type checking | ❌ or basic annotations | ✅ mypy with increasing strictness | | Security scanning | ❌ | ✅ Bandit | | Dead code detection | ❌ | ✅ Vulture | | | | | | Testing | | | | Unit tests | ✅ Basic pytest | ✅ Comprehensive pytest with fixtures | | Test coverage | ❌ or basic | ✅ pytest-cov with coverage targets | | Mocking | ❌ | ✅ pytest-mock for external dependencies | | Integration tests | ❌ | ✅ For component interactions | | Functional tests | ❌ | ✅ For key user workflows | | | | | | Documentation | | | | Code documentation | ✅ Basic docstrings | ✅ Comprehensive docstrings (Google/NumPy style) | | API documentation | ✅ Generated with pydoc | ✅ MkDocs + mkdocstrings | | User guides | ✅ Basic README usage examples | ✅ Comprehensive MkDocs site with tutorials | | | | | | Version Control Practices | | | | Commit frequency | ✅ Regular commits | ✅ Atomic, focused commits | | Commit messages | ✅ Basic descriptive messages | ✅ Structured commit messages with context | | Branching | ❌ or basic feature branches | ✅ Git-flow or trunk-based with feature branches | | Code reviews | ❌ | ✅ Pull/Merge requests with review guidelines | | | | | | Automation | | | | Local automation | ❌ | ✅ pre-commit hooks | | CI pipeline | ❌ or basic | ✅ GitHub Actions with matrix testing | | CD pipeline | ❌ | ✅ Automated deployments/releases | | | | | | Packaging & Distribution | | | | Package configuration | ✅ Basic `pyproject.toml` | ✅ Comprehensive configuration with extras | | Build system | ✅ Basic setuptools | ✅ Modern build with PEP 517 support | | Release process | ✅ Manual versioning | ✅ Semantic versioning with automation | | Publication | ❌ or manual PyPI upload | ✅ Automated PyPI deployment via CI | | | | | | Maintenance | | | | Dependency updates | ✅ Manual updates | ✅ Scheduled updates with dependabot | | Security monitoring | ❌ | ✅ Vulnerability scanning | | Performance profiling | ❌ | ✅ Regular profiling and benchmarking | | User feedback channels | ❌ | ✅ Issue templates and contribution guidelines | ## Project Progression Path For projects that start simple but grow in complexity, follow this progression: 1. Start with the essentials: - Project structure and version control - Virtual environment - Basic testing - Clear README 2. Add code quality tools incrementally: - First add Ruff for formatting and basic linting - Then add mypy for critical modules - Finally add security scanning 3. Enhance testing as complexity increases: - Add coverage reporting - Implement mocking for external dependencies - Add integration tests for component interactions 4. Improve documentation with growth: - Start with good docstrings from day one - Transition to MkDocs when README becomes insufficient - Generate API documentation from docstrings 5. Automate processes as repetition increases: - Add pre-commit hooks for local checks - Implement CI for testing across environments - Add CD when deployment becomes routine Remember: Don't overengineer! Choose the practices that add value to your specific project and team. It's better to implement a few practices well than to poorly implement many. ============================================================ SOURCE: appendices/editors.qmd ============================================================ # Introduction to Python IDEs and Editors While this book focuses on Python development practices rather than specific tools, your choice of development environment can significantly impact your productivity and workflow. This appendix provides a brief overview of popular editors and IDEs for Python development, with particular attention to how they integrate with the tools and practices discussed throughout this book. ## Visual Studio Code Visual Studio Code (VS Code) has become one of the most popular editors for Python development due to its balance of lightweight design and powerful features. ### Key Features for Python Development - Python Extension: Microsoft's official Python extension provides IntelliSense, linting, debugging, code navigation, and Jupyter notebook support - Virtual Environment Detection: Automatically detects and allows switching between virtual environments - Integrated Terminal: Run Python scripts and commands without leaving the editor - Debugging: Full-featured debugging with variable inspection and breakpoints - Extensions Ecosystem: Rich marketplace with extensions for most Python tools ### Integration with Development Tools - Virtual Environments: Detects venv, conda, and other environment types; shows active environment in status bar - Linting/Formatting: Native integration with Ruff, Black, mypy, and other quality tools - Testing: Test Explorer UI for pytest, unittest - Package Management: Terminal integration for pip, Poetry, PDM, and other package managers - Git: Built-in Git support for commits, branches, and pull requests ### Configuration Example `.vscode/settings.json`: ### AI-Assistant Integration - GitHub Copilot: Code suggestions directly in the editor - IntelliCode: AI-enhanced code completions - Live Share: Collaborative coding sessions ## Neovim Neovim is a highly extensible text editor popular among developers who prefer keyboard-centric workflows and extensive customisation. ### Key Features for Python Development - Extensible Architecture: Lua-based configuration and plugin system - Terminal Integration: Built-in terminal emulator - Modal Editing: Efficient text editing with different modes - Performance: Fast startup and response, even for large files ### Integration with Development Tools - Language Server Protocol (LSP): Native support for Python language servers like Pyright and Jedi - Virtual Environments: Support through plugins and configuration - Code Completion: Various completion engines (nvim-cmp, COC) - Linting/Formatting: Integration with tools like Ruff, Black, and mypy - Testing: Run tests through plugins or terminal integration ### Configuration Example Simplified `init.lua` excerpt for Python development: ### AI-Assistant Integration - GitHub Copilot.vim: Code suggestions - Neural: Code completions powered by local models ## Emacs Emacs is a highly customisable text editor with a rich ecosystem of packages and a long history in the development community. ### Key Features for Python Development - Extensibility: customisable with Emacs Lisp - Org Mode: Literate programming and documentation - Multiple Modes: Specialized modes for different file types - Integrated Environment: Email, shell, and other tools integrated ### Integration with Development Tools - Python Mode: Syntax highlighting, indentation, and navigation for Python - Virtual Environments: Support through pyvenv, conda.el - Linting/Formatting: Integration with Flycheck, Black, Ruff - Testing: Run tests with pytest-emacs - Package Management: Manage dependencies through shell integration ### Configuration Example Excerpt from `.emacs` or `init.el`: ### AI-Assistant Integration - Copilot.el: GitHub Copilot integration - ChatGPT-shell: Interact with LLMs from within Emacs ## AI-Enhanced Editors ### Cursor Cursor (formerly Warp AI) is built on top of VS Code but focused on AI-assisted development. #### Key Features - AI Chat: Integrated chat interface for coding assistance - Code Explanation: Ask about selected code - Code Generation: Generate code from natural language descriptions - VS Code Base: All VS Code features and extensions available - customised for AI Interaction: UI designed around AI-assisted workflows #### Integration with Python Tools - Inherits VS Code's excellent Python ecosystem support - AI features that understand Python code context - Assistance with complex Python patterns and libraries ### Whisper (Anthropic) Claude Code (Whisper) from Anthropic is an AI-enhanced development environment: #### Key Features - Terminal-Based Assistant: AI-powered code generation from the command line - Task Automation: Natural language for development tasks - Context-Aware Assistance: Understands project structure and code - Code Explanation: In-depth explanations of complex code #### Integration with Python Tools - Works alongside existing development environments - Can assist with tool configuration and integration - Helps debug issues with Python tooling ## Choosing the Right Environment The best development environment depends on your specific needs: - VS Code: Excellent for most Python developers; balances ease of use with powerful features - Neovim: Ideal for keyboard-focused developers who value speed and customisation - Emacs: Great for developers who want an all-in-one environment with deep customisation - AI-Enhanced Editors: Valuable for those looking to leverage AI in their workflow Consider these factors when choosing: 1. Learning curve: VS Code has a gentle learning curve, while Neovim and Emacs require more investment 2. Performance needs: Neovim offers the best performance for large files 3. Extensibility importance: Emacs and Neovim offer the deepest customisation 4. Team standards: Consider what your team uses for easier collaboration 5. AI assistance: If AI-assisted development is important, specialized editors may offer better integration ## Editor-Agnostic Best Practices Regardless of your chosen editor, follow these best practices: 1. Learn keyboard shortcuts: They dramatically increase productivity 2. Use extensions for Python tools: Integrate the tools from this book 3. Set up consistent formatting: Configure your editor to use the same tools as your CI pipeline 4. customise for your workflow: Adapt your environment to your specific needs 5. Version control your configuration: Track editor settings in Git for consistency Remember that the editor is just a tool—the development practices in this book can be applied regardless of your chosen environment. The best editor is the one that helps you implement good development practices while staying out of your way during the creative process. ============================================================ SOURCE: appendices/tools.qmd ============================================================ # Python Development Tools Reference This reference provides brief descriptions of the development tools mentioned throughout the guide, organised by their primary function. ## Environment & Dependency Management - venv: Python's built-in tool for creating isolated virtual environments. - pip: The standard package installer for Python. - pip-tools: A set of tools for managing Python package dependencies with pinned versions via requirements.txt files. - uv: A Rust-based, high-performance Python package manager and environment manager compatible with pip. - pipx: A tool for installing and running Python applications in isolated environments. ## Code Quality & Formatting - Ruff: A fast, Rust-based Python linter and formatter that consolidates multiple tools. - Black: An opinionated Python code formatter that enforces a consistent style. - isort: A utility to sort Python imports alphabetically and automatically separate them into sections. - Flake8: A code linting tool that checks Python code for style and logical errors. - Pylint: A comprehensive Python static code analyzer that looks for errors and enforces coding standards. ## Testing - pytest: A powerful, flexible testing framework for Python that simplifies test writing and execution. - pytest-cov: A pytest plugin for measuring code coverage during test execution. - pytest-mock: A pytest plugin for creating and managing mock objects in tests. ## Type Checking - mypy: A static type checker for Python that helps catch type-related errors before runtime. - pydoc: Python's built-in documentation generator and help system. ## Security & Code Analysis - Bandit: A tool designed to find common security issues in Python code. - Vulture: A tool that detects unused code in Python programs. ## Documentation - MkDocs: A fast and simple static site generator for building project documentation from Markdown files. - mkdocs-material: A Material Design theme for MkDocs. - mkdocstrings: A MkDocs plugin that automatically generates documentation from docstrings. - Sphinx: A comprehensive documentation tool that supports multiple output formats. ## Package Building & Distribution - build: A simple, correct PEP 517 package builder for Python projects. - twine: A utility for publishing Python packages to PyPI securely. - setuptools: The standard library for packaging Python projects. - setuptools-scm: A tool that manages your Python package versions using git metadata. - wheel: A built-package format for Python that provides faster installation. ## Continuous Integration & Deployment - GitHub Actions: GitHub's built-in CI/CD platform for automating workflows. - pre-commit: A framework for managing and maintaining pre-commit hooks. - Codecov: A tool for measuring and reporting code coverage in CI pipelines. ## Version Control - Git: A distributed version control system for tracking changes in source code. - GitHub/GitLab: Web-based platforms for hosting Git repositories with collaboration features. ## Project Setup & Management - Cookiecutter: A command-line utility that creates projects from templates, enabling consistent project setup with predefined structure and configurations. It uses a templating system to generate files and directories based on user inputs. - GitHub Repository Templates: A GitHub feature that allows repositories to serve as templates for new projects. Users can generate new repositories with the same directory structure and files without needing to install additional tools. Unlike cookiecutter, GitHub templates don't support parameterization but offer a zero-installation approach to project scaffolding. ## Advanced Tools - Cython: A language that makes writing C extensions for Python as easy as writing Python. - Docker: A platform for developing, shipping, and running applications in containers. - Kubernetes: An open-source system for automating deployment, scaling, and management of containerized applications. - Pants/Bazel: Build systems designed for monorepos and large codebases. ============================================================ SOURCE: appendices/tool-comparison.qmd ============================================================ # Comparison of Python Environment and Package Management Tools This appendix provides a side-by-side comparison of the major Python environment and package management tools covered throughout this book. ## Comparison Table | Feature | venv | conda | uv | Hatch | Poetry | PDM | |---------|------|-------|----|----|--------|-----| | Core Focus | Virtual environments | Environments & packages across languages | Fast package installation | Project management | Dependency management & packaging | Standards-compliant packaging | | Implementation Language | Python | Python | Rust | Python | Python | Python | | Performance | Standard | Moderate | Very Fast | Standard | Moderate | Fast | | Virtual Environment Support | Built-in | Built-in | Built-in | Built-in | Built-in | Optional (PEP 582) | | Lock File | No (requires pip-tools) | No (uses explicit envs) | Yes | Yes | Yes | Yes | | Dependency Resolution | Basic (via pip) | Sophisticated | Efficient | Basic | Sophisticated | Sophisticated | | Non-Python Dependencies | No | Yes | No | No | No | No | | Project Config File | None | environment.yml | requirements.txt | pyproject.toml | pyproject.toml | pyproject.toml | | PEP 621 Compliance | N/A | No | N/A | Yes | Partial | Yes | | Multiple Environment Management | No (one env per directory) | Yes | No | Yes | No | Via configuration | | Dependency Groups | No | Via separate files | Via separate files | Yes | Yes | Yes | | Package Building | No | Limited | No | Yes | Yes | Yes | | Publishing to PyPI | No | Limited | No | Yes | Yes | Yes | | Cross-Platform Support | Yes | Yes | Yes | Yes | Yes | Yes | | Best For | Simple projects, teaching | Scientific/ML projects | Fast installations, CI environments | Dev workflow automation | Library development | Standards-focused projects | | Learning Curve | Low | Moderate | Low | Moderate | Moderate-High | Moderate | | Script/Task Running | No | Limited | No | Advanced | Basic | Advanced | | Community Size/Adoption | Very High | Very High | Growing | Moderate | High | Growing | | Plugin System | No | No | No | Yes | Limited | Yes | | Development Status | Stable/Mature | Stable/Mature | Active Development | Active Development | Stable/Mature | Active Development | ## Installation Methods | Tool | pip/pipx | Homebrew | Official Installer | Platform Package Managers | |------|----------|----------|-------------------|----------------------------| | venv | Built-in with Python | N/A | N/A | N/A | | conda | No | Yes | Yes (Miniconda/Anaconda) | Some | | uv | Yes | Yes | Yes (curl installer) | Growing | | Hatch | Yes | Yes | No | Some | | Poetry | Yes | Yes | Yes (custom installer) | Some | | PDM | Yes | Yes | No | Some | ## Typical Usage Patterns | Tool | Typical Command Sequence | |------|--------------------------| | venv | `python -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt` | | conda | `conda create -n myenv python=3.10 && conda activate myenv && conda install pandas numpy` | | uv | `uv venv && source .venv/bin/activate && uv pip sync requirements.txt` | | Hatch | `hatch init && hatch shell && hatch run test` | | Poetry | `poetry init && poetry add requests && poetry install && poetry run python script.py` | | PDM | `pdm init && pdm add requests pytest --dev && pdm install && pdm run pytest` | ## Use Case Recommendations ### For Beginners 1. venv + pip: Simplest to understand, built-in to Python 2. uv: Fast, familiar pip-like interface with modern features ### For Data Science/Scientific Computing 1. conda: Best support for scientific packages and non-Python dependencies 2. Poetry or PDM: When standard Python packages are sufficient ### For Library Development 1. Poetry: Great packaging and publishing workflows 2. Hatch: Excellent for multi-environment testing 3. PDM: Standards-compliant approach ### For Application Development 1. PDM: PEP 582 mode simplifies deployment 2. Poetry: Lock file ensures reproducible environments 3. Hatch: Task management features help automate workflows ### For CI/CD Environments 1. uv: Fastest installation speeds 2. Poetry/PDM: Reliable lock files ensure consistency ### For Teams with Mixed Experience Levels 1. Poetry: Opinionated approach enforces consistency 2. uv: Familiar interface with performance benefits 3. Hatch: Flexibility for different team workflows ## Migration Paths | From | To | Migration Approach | |------|----|--------------------| | pip + requirements.txt | uv | Use directly with existing requirements.txt | | pip + requirements.txt | Poetry | `poetry init` then `poetry add` packages | | pip + requirements.txt | PDM | `pdm import -f requirements requirements.txt` | | conda | Poetry/PDM | Export conda env to requirements, then import | | Pipenv | Poetry | `poetry init` + manual migration or conversion tools | | Pipenv | PDM | `pdm import -f pipenv Pipfile` | | Poetry | PDM | `pdm import -f poetry pyproject.toml` | ## When to Consider Multiple Tools Some projects benefit from using multiple tools for different purposes: - conda + pip: Use conda for complex dependencies, pip for Python-only packages - venv + uv: Use venv for environment isolation, uv for fast package installation - Hatch + uv: Use Hatch for project workflows, uv for faster installations ## Future Trends The Python packaging ecosystem continues to evolve toward: 1. Standards Compliance: Increasing adoption of PEPs 518, 517, 621 2. Performance optimisation: More Rust-based tools like uv 3. Simplified Workflows: Better integration between tools 4. Improved Lock Files: More secure and deterministic builds 5. Better Environment Management: Alternatives to traditional virtual environments By understanding the strengths and trade-offs of each tool, you can select the approach that best fits your specific project requirements and team preferences. ============================================================ SOURCE: appendices/bash-scaffold-script.qmd ============================================================ # Python Development Pipeline Scaffold Python Script ```bash #!/bin/bash # scaffold_python_project.sh - A simple script to create a Python project with best practices # Usage: ./scaffold_python_project.sh my_project if [ -z "$1" ]; then echo "Please provide a project name." echo "Usage: ./scaffold_python_project.sh my_project" exit 1 fi PROJECT_NAME=$1 # Convert hyphens to underscores for Python package naming conventions PACKAGE_NAME=$(echo $PROJECT_NAME | tr '-' '_') echo "Creating project: $PROJECT_NAME" echo "Package name will be: $PACKAGE_NAME" # Create project directory mkdir -p $PROJECT_NAME cd $PROJECT_NAME # Create basic structure following the recommended src layout # The src layout enforces proper package installation and creates clear boundaries mkdir -p src/$PACKAGE_NAME tests docs # Create package files # __init__.py makes the directory a Python package touch src/$PACKAGE_NAME/__init__.py touch src/$PACKAGE_NAME/main.py # Create test files - keeping tests separate but adjacent to the implementation # This follows the principle of separating implementation from tests touch tests/__init__.py touch tests/test_main.py # Create documentation placeholder - establishing documentation from the start # Even minimal docs are better than no docs echo "# $PROJECT_NAME Documentation" > docs/index.md # Create README.md with basic information # README is the first document anyone sees and should provide clear instructions echo "# $PROJECT_NAME A Python project created with best practices. ## Installation \`\`\`bash pip install $PROJECT_NAME \`\`\` ## Usage \`\`\`python from $PACKAGE_NAME import main \`\`\` ## Development \`\`\`bash # Create virtual environment python -m venv .venv source .venv/bin/activate # On Windows: .venv\\Scripts\\activate # Install in development mode pip install -e . # Run tests pytest \`\`\` " > README.md # Create .gitignore file to exclude unnecessary files from version control # This prevents committing files that should not be in the repository echo "# Python __pycache__/ *.py[cod] *$py.class *.so .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ *.egg-info/ .installed.cfg *.egg # Virtual environments # Never commit virtual environments to version control .venv/ venv/ ENV/ # Testing .pytest_cache/ .coverage htmlcov/ # Documentation docs/_build/ # IDE .idea/ .vscode/ *.swp *.swo " > .gitignore # Create pyproject.toml for modern Python packaging # This follows PEP 517/518 standards and centralizes project configuration echo "[build-system] requires = [\"setuptools>=61.0\", \"wheel\"] build-backend = \"setuptools.build_meta\" [project] name = \"$PROJECT_NAME\" version = \"0.1.0\" description = \"A Python project created with best practices\" readme = \"README.md\" requires-python = \">=3.8\" authors = [ {name = \"Your Name\", email = \"your.email@example.com\"} ] [project.urls] \"Homepage\" = \"https://github.com/yourusername/$PROJECT_NAME\" # Specify the src layout for better package isolation [tool.setuptools] package-dir = {\"\" = \"src\"} packages = [\"$PACKAGE_NAME\"] # Configure pytest to look in the tests directory [tool.pytest.ini_options] testpaths = [\"tests\"] " > pyproject.toml # Create requirements.in for direct dependencies # This approach is cleaner than freezing everything with pip freeze echo "# Project dependencies # Add your dependencies here, e.g.: # requests>=2.25.0 " > requirements.in # Create example main.py with docstrings and type hints # Starting with good documentation and typing practices from the beginning echo "\"\"\"Main module for $PROJECT_NAME.\"\"\" def example_function(text: str) -> str: \"\"\"Return a greeting message. Args: text: The text to include in the greeting. Returns: A greeting message. \"\"\" return f\"Hello, {text}!\" " > src/$PACKAGE_NAME/main.py # Create example test file # Tests verify that code works as expected and prevent regressions echo "\"\"\"Tests for the main module.\"\"\" from $PACKAGE_NAME.main import example_function def test_example_function(): \"\"\"Test the example function returns the expected greeting.\"\"\" result = example_function(\"World\") assert result == \"Hello, World!\" " > tests/test_main.py # Initialize git repository # Version control should be established from the very beginning git init git add . git commit -m "Initial project setup" echo "" echo "Project $PROJECT_NAME created successfully!" echo "" echo "Next steps:" echo "1. cd $PROJECT_NAME" echo "2. python -m venv .venv" echo "3. source .venv/bin/activate # On Windows: .venv\\Scripts\\activate" echo "4. pip install -e ." echo "5. pytest" echo "" echo "Happy coding!" ``` ============================================================ SOURCE: appendices/cookiecutter.qmd ============================================================ # Cookiecutter Template This appendix introduces and explains the companion cookiecutter template for the Python Development Pipeline described in this book. The template allows you to quickly scaffold new Python projects that follow all the recommended practices, saving you time and ensuring consistency across your projects. ## What is Cookiecutter? Cookiecutter is a command-line utility that creates projects from templates. It takes a template directory containing a `cookiecutter.json` file with template variables and replaces them with user-provided values, generating a project directory structure with all necessary files. ## Getting Started with the Template ### Prerequisites - Python 3.7 or later - pip package manager ### Installation First, install cookiecutter: ### Creating a New Project To create a new project using our Python Development Pipeline template: You'll be prompted to provide information about your project, such as: - Project name - Author information - Python version requirements - License type - Development level (basic or advanced) - Documentation preferences - CI/CD preferences - Package manager choice (pip-tools or uv) After answering these questions, cookiecutter will generate a complete project structure with all the configuration files and setup based on your choices. ## Template Features The template implements all the best practices discussed throughout this book: ### Project Structure - Uses the recommended `src` layout for better package isolation - Properly organised test directory - Documentation setup with MkDocs (if selected) - Clear separation of concerns across files and directories ### Development Environment - Configured virtual environment instructions - Dependency management using either pip-tools or uv - `requirements.in` and `requirements-dev.in` files for clean dependency specification ### Code Quality Tools - Ruff for formatting and linting - mypy for type checking - Bandit for security analysis (with advanced setup) - Pre-configured with sensible defaults in `pyproject.toml` ### Testing - pytest setup with example tests - Coverage configuration - Test helper fixtures ### Documentation - MkDocs with Material theme (if selected) - API documentation generation with mkdocstrings - Template pages for quickstart, examples, and API reference ### CI/CD - GitHub Actions workflows for testing, linting, and type checking - Publish workflow for PyPI deployment - Matrix testing across Python versions ## customisation Options The template offers several customisation options during generation: ### Basic vs. Advanced Setup - Basic: Lighter configuration focused on essential tools - Advanced: Full suite of tools including security scanning, stricter type checking, and comprehensive CI/CD ### Documentation Options - Choose whether to include MkDocs documentation setup - If included, get a complete documentation structure ready for content ### CI/CD Options - Include GitHub Actions workflows for automated testing and deployment - Configure publishing workflows for PyPI integration ## Template Structure The generated project follows this structure: ## Post-Generation Steps After creating your project, the template provides instructions for: 1. Creating and activating a virtual environment 2. Installing dependencies 3. Setting up version control 4. Running initial tests The generated `README.md` includes detailed development setup instructions specific to your configuration choices. ## Extending the Template You can extend or customise the template for your specific needs: ### Adding Custom Components Fork the template repository and add additional files or configurations specific to your organisation or preferences. ### Modifying Tool Configurations The `pyproject.toml` file contains all tool configurations and can be adjusted to match your coding standards and preferences. ### Creating Specialized Variants Create specialized variants of the template for different types of projects (e.g., web applications, data science, CLI tools) while maintaining the core best practices. ## Best Practices for Using the Template 1. Use for new projects: The template is designed for new projects rather than retrofitting existing ones. 2. Commit immediately after generation: Make an initial commit right after generating the project to establish a clean baseline. 3. Review and adjust configurations: While the defaults are sensible, review and adjust configurations to match your specific project needs. 4. Keep dependencies updated: Regularly update the `requirements.in` files as your project evolves. 5. Follow the workflow: The template sets up the infrastructure, but you still need to follow the development workflow described in this book. ## Conclusion The Python Development Pipeline cookiecutter template encapsulates the practices and principles discussed throughout this book, allowing you to rapidly bootstrap projects with best practices already in place. By using this template, you ensure consistency across projects and can focus more on solving problems rather than setting up infrastructure. Whether you're starting a small personal project or a larger team effort, this template provides a solid foundation that can scale with your needs while maintaining professional development standards. ============================================================ SOURCE: appendices/hatch.qmd ============================================================ # Hatch - Modern Python Project Management ## Introduction to Hatch Hatch is a modern, extensible Python project management tool designed to simplify the development workflow through standardization and automation. Created by Ofek Lev and first released in 2017, Hatch has undergone significant evolution to become a comprehensive solution that handles environment management, dependency resolution, building, and publishing. Unlike traditional tools that focus primarily on packaging or dependency management, Hatch takes a holistic approach to project management, addressing the entire development lifecycle. What sets Hatch apart is its flexibility, extensibility, and focus on developer experience through an intuitive CLI and plugin system. ## Key Features of Hatch ### Project Management Hatch provides comprehensive project management capabilities: - Project initialization: Quickly set up standardized project structures - Flexible configuration: Standardized configuration in `pyproject.toml` - Version management: Easily bumper version numbers - Script running: Execute defined project scripts ### Environment Management One of Hatch's standout features is its sophisticated environment handling: - Multiple environments per project: Define development, testing, documentation environments - Matrix environments: Test across Python versions and dependency sets - Isolated environments: Clean, reproducible development spaces - Environment synchronization: Keep environments updated ### Build and Packaging Hatch streamlines the packaging workflow: - Standards-compliant: Implements PEP 517/518 build system - Multiple build targets: Source distributions and wheels - Build hooks: customise the build process - Metadata standardization: PEP 621 compliant metadata ### Extensibility Hatch is designed for extensibility: - Plugin system: Extend functionality through plugins - Custom commands: Add project-specific commands - Environment customisation: Define environment-specific tools - Build customisation: Extend the build process ## Getting Started with Hatch ### Installation Hatch can be installed through several methods: Verify your installation: ### Creating a New Project Create a new project with Hatch: The project structure might look like: ### Basic Configuration Hatch uses `pyproject.toml` for configuration: ## Essential Hatch Commands ### Environment Management ### Dependency Management ### Building and Publishing ### Version Management ## Advanced Hatch Features ### Environment Matrix Hatch can manage testing across multiple Python versions: Run commands across all environments: ### Custom Scripts Define project-specific scripts: Run these scripts: ### Environment Features Enable specific tools in environments: ### Build Hooks customise the build process: ## Best Practices with Hatch ### Project Structure A recommended structure for Hatch projects: To use this source layout: ### Environment Management Strategies 1. Specialized Environments: Create purpose-specific environments 2. Matrix Testing: Test across Python versions 3. Feature Toggles: organise functionality by feature ### Version Control Practices 1. Configure version source: Use git tags or a version file 2. Automate version bumping: Use Hatch's version commands in your workflow ### Integration with Development Tools Configure tools like Black and Ruff directly in `pyproject.toml`: ## Integration with Development Workflows ### IDE Integration Hatch environments work with most Python IDEs: #### VS Code 1. Create environments: `hatch env create` 2. Find the environment path: `hatch env find default` 3. Select the interpreter from this path in VS Code #### PyCharm 1. Create environments: `hatch env create` 2. Find the environment path: `hatch env find default` 3. Add the interpreter in PyCharm settings ### CI/CD Integration #### GitHub Actions Example ## Troubleshooting Common Issues ### Environment Creation Failures If environments fail to create: ### Build Issues For build-related problems: ### Plugin Problems If plugins aren't working: ## Comparison with Other Tools ### Hatch vs. Poetry - Hatch: More flexible, multiple environments, standards-focused - Poetry: More opinionated, stronger dependency resolution - Key difference: Hatch's multiple environments per project vs. Poetry's single environment approach ### Hatch vs. PDM - Hatch: Focus on the entire development workflow - PDM: Stronger focus on dependency management with PEP 582 support - Key difference: Hatch's broader scope vs. PDM's emphasis on dependencies ### Hatch vs. pip + venv - Hatch: Integrated environment and project management - pip + venv: Separate tools requiring manual coordination - Key difference: Hatch's automation vs. traditional manual approach ## When to Use Hatch Hatch is particularly well-suited for: 1. Complex Development Workflows: Multiple environments, testing matrices 2. Teams with Diverse Projects: Standardization across different project types 3. Open Source Maintainers: Multiple environment testing and streamlined releases 4. Projects Requiring customisation: Plugin system for specialized needs Hatch might not be ideal for: 1. Very Simple Scripts: Might be overkill for trivial projects 2. Teams Heavily Invested in Poetry: Migration costs might outweigh benefits 3. Projects with Unusual Build Systems: Some specialized build needs might require additional customisation ## Conclusion Hatch represents a modern approach to Python project management that emphasizes flexibility, standards compliance, and developer experience. Its unique multi-environment capabilities, combined with comprehensive project lifecycle management, make it a powerful choice for both application and library development. While newer than some alternatives like Poetry, Hatch's strict adherence to Python packaging standards ensures compatibility with the broader ecosystem. Its plugin system and flexible configuration options allow it to adapt to a wide range of project needs, from simple libraries to complex applications. For developers looking for a tool that can grow with their projects and adapt to various workflows, Hatch provides a compelling combination of power and flexibility. Its focus on standardization and automation helps reduce the cognitive overhead of project management, allowing developers to focus more on writing code and less on managing tooling. ============================================================ SOURCE: appendices/conda.qmd ============================================================ # Using Conda for Environment Management ## Introduction to Conda Conda is a powerful open-source package and environment management system that runs on Windows, macOS, and Linux. While similar to the virtual environment tools covered in the main text, conda offers distinct advantages for certain Python workflows, particularly in data science, scientific computing, and research domains. Unlike tools that focus solely on Python packages, conda can package and distribute software for any language, making it especially valuable for projects with complex dependencies that extend beyond the Python ecosystem. ## When to Consider Conda Conda is particularly well-suited for: - Data science projects requiring scientific packages (NumPy, pandas, scikit-learn, etc.) - Research environments with mixed-language requirements (Python, R, C/C++ libraries) - Projects with complex binary dependencies that are difficult to compile - Cross-platform development where consistent environments across operating systems are crucial - GPU-accelerated computing requiring specific CUDA versions - Bioinformatics, computational physics, and other specialized scientific domains ## Conda vs. Other Environment Tools | Feature | Conda | venv + pip | uv | |---------|-------|------------|-----| | Focus | Any language packages | Python packages | Python packages | | Binary package distribution | Yes (pre-compiled) | Limited | Limited | | Dependency resolution | Environment-level solver | Package-level solver | Fast, improved solver | | Platform support | Windows, macOS, Linux | Windows, macOS, Linux | Windows, macOS, Linux | | Non-Python dependencies | Excellent | Limited | Limited | | Speed | Moderate | Moderate | Very fast | | Scientific package support | Excellent | Good | Good | ## Getting Started with Conda ### Installation Conda is available through several distributions: 1. Miniconda: Minimal installer containing just conda and its dependencies 2. Anaconda: Full distribution including conda and 250+ popular data science packages For most development purposes, Miniconda is recommended as it provides a minimal base that you can build upon as needed. To install Miniconda: ### Basic Conda Commands #### Creating Environments #### Activating and Deactivating Environments #### Managing Packages #### Environment Management ## Environment Files with Conda Conda uses YAML files to define environments, making them easily shareable and reproducible: This file defines: - The environment name (`datasci`) - Channels to search for packages (with preference order) - Conda packages with optional version constraints - Additional pip packages to install Create this environment with: ## Best Practices for Conda ### Channel Management Conda packages come from "channels." The main ones are: - defaults: Official Anaconda channel - conda-forge: Community-led channel with more up-to-date packages For consistent environments, specify channels explicitly in your environment files and consider adding channel priority: This prioritizes conda-forge packages over defaults when both are available. ### minimising Environment Size Conda environments can become large. Keep them streamlined by: 1. Only installing what you need 2. Using the `--no-deps` flag when appropriate 3. Considering a minimal base environment with `conda create --name myenv python` ### Managing Conflicting Dependencies When facing difficult dependency conflicts: ### Combining Conda with pip While conda can install most packages, some are only available on PyPI. The recommended approach: 1. Install all conda-available packages first using conda 2. Then install PyPI-only packages using pip This approach is implemented automatically when using an environment.yml file with a pip section. ### Environment Isolation from System Python Avoid using your system Python installation with conda. Instead: ## Integration with Development Workflows ### Using Conda with VS Code VS Code can automatically detect and use conda environments: 1. Install the Python extension 2. Open the Command Palette (Ctrl+Shift+P) 3. Select "Python: Select Interpreter" 4. Choose your conda environment from the list ### Using Conda with Jupyter Conda integrates well with Jupyter notebooks: ### CI/CD with Conda For GitHub Actions, you can use conda environments: ## Common Pitfalls and Solutions ### Slow Environment Creation Conda environments can take time to create due to dependency resolution: ### Conflicting Channels Mixing packages from different channels can cause conflicts: ### Large Environment Sizes Conda environments can grow large, especially with the Anaconda distribution: ## Mamba: A Faster Alternative For large or complex environments, consider mamba, a reimplementation of conda's package manager in C++: Mamba offers significant speed improvements for environment creation and package installation while maintaining compatibility with conda commands. ## Conclusion Conda provides a robust solution for environment management, particularly valuable for scientific computing, data science, and research applications. While more complex than venv, it solves specific problems that other tools cannot easily address, especially when dealing with non-Python dependencies or cross-platform binary distribution. For projects focusing purely on Python dependencies without complex binary requirements, the venv and uv approaches covered in the main text may provide simpler workflows. However, understanding conda remains valuable for many Python practitioners, especially those working in scientific domains. ============================================================ SOURCE: appendices/venv.qmd ============================================================ # Getting Started with venv ## Introduction to venv The `venv` module is Python's built-in tool for creating virtual environments. Introduced in Python 3.3 and standardized in PEP 405, it has become the official recommended way to create isolated Python environments. As a module in the standard library, `venv` is immediately available with any Python installation, requiring no additional installation step. Virtual environments created with `venv` provide isolated spaces where Python projects can have their own dependencies, regardless of what dependencies other projects may have. This solves the common problem of conflicting package requirements across different projects and prevents changes to one project from affecting others. ## Why Use venv? Virtual environments are essential in Python development for several reasons: 1. Dependency Isolation: Each project can have its own dependencies, regardless of other projects' requirements 2. Consistent Environments: Ensures reproducible development and deployment environments 3. Clean Testing: Test against specific package versions without affecting the system Python 4. Conflict Prevention: Avoids "dependency hell" where different projects need different versions of the same package 5. Project organisation: Clearly separates project dependencies from system or global packages ## Getting Started with venv ### Creating a Virtual Environment To create a virtual environment using `venv`, open a terminal and run: The command creates a directory containing: - A Python interpreter copy - The `pip` package manager - A basic set of installed libraries - Scripts to activate the environment ### Activating the Environment Before using the virtual environment, you need to activate it. The activation process adjusts your shell's PATH to prioritize the virtual environment's Python interpreter and tools. #### On Windows: #### On macOS and Linux: After activation, your shell prompt typically changes to indicate the active environment: All Python and pip commands now use the virtual environment's versions instead of the system ones. ### Deactivating the Environment When you're done working on the project, deactivate the environment: This restores your shell to its original state, using the system Python interpreter. ## Advanced venv Options ### Creating Environments with Specific Python Versions To create an environment with a specific Python version, use that version's interpreter: ### Creating Environments Without pip By default, `venv` installs pip in new environments. To create one without pip: ### Creating System Site-packages Access Normally, virtual environments are isolated from system site-packages. To allow access: This creates an environment that can see system packages, but newly installed packages still go into the virtual environment. ### Upgrading pip in a New Environment Virtual environments often include an older pip version. It's good practice to upgrade: ## Managing Dependencies with venv While `venv` creates the environment, you'll use `pip` to manage packages within it. ### Installing Packages With your environment activated: ### Tracking Dependencies To track installed packages: This creates a text file listing all installed packages and their versions. ### Installing from Requirements To recreate an environment elsewhere: ## Best Practices with venv ### Directory Naming Conventions Common virtual environment directory names include: - `.venv`: Hidden directory (less visible clutter) - `venv`: Explicit directory name - `env`: Shorter alternative The `.venv` name is increasingly popular as it: - Keeps it hidden in file browsers - Makes it easy to add to `.gitignore` - Is recognised by many IDEs and tools ### Version Control Integration Never commit virtual environment directories to version control. Add them to `.gitignore`: ### Environment Management Across Projects Create a new virtual environment for each project: ### IDE Integration Most Python IDEs integrate well with venv environments: #### VS Code 1. Open your project folder 2. Press Ctrl+Shift+P 3. Select "Python: Select Interpreter" 4. Choose the environment from the list #### PyCharm 1. Go to Settings → Project → Python Interpreter 2. Click the gear icon → Add 3. Select "Existing Environment" and navigate to the environment's Python ## Comparing venv with Other Tools ### venv vs. virtualenv `virtualenv` is a third-party package that inspired the creation of `venv`. - venv: Built into Python, no installation needed, slightly fewer features - virtualenv: Third-party package, more features, better backwards compatibility For most modern Python projects, `venv` is sufficient, but `virtualenv` offers some advanced options and supports older Python versions. ### venv vs. conda While both create isolated environments, they serve different purposes: - venv: Python-specific, lightweight, manages only Python packages - conda: Cross-language package manager, handles non-Python dependencies, preferred for scientific computing ### venv vs. Poetry/PDM These are newer tools that combine dependency management with virtual environments: - venv+pip: Separate tools for environments and package management - Poetry/PDM: All-in-one solutions with lock files, dependency resolution, packaging ## Troubleshooting Common Issues ### Activation Script Not Found If you can't find the activation script: Make sure the environment was created successfully and you're using the correct path. ### Packages Not Found After Installation If packages are installed but not importable: 1. Verify the environment is activated (check prompt prefix) 2. Check if you have multiple Python installations 3. Reinstall the package in the active environment ### Permission Issues If you encounter permission errors: ## Script Examples for venv Workflows ### Project Setup Script ### Environment Recreation Script ## Conclusion The `venv` module provides a simple, reliable way to create isolated Python environments directly from the standard library. While newer tools offer more features and automation, `venv` remains a fundamental building block of Python development workflows, offering an excellent balance of simplicity and utility. For most Python projects, the combination of `venv` and `pip` provides a solid foundation for environment management. As projects grow in complexity, you can build upon this foundation with additional tools while maintaining the same core principles of isolation and reproducibility. ============================================================ SOURCE: appendices/uv.qmd ============================================================ # UV - High-Performance Python Package Management ## Introduction to uv `uv` is a modern, high-performance Python package installer and resolver written in Rust. Developed by Astral, it represents a significant evolution in Python tooling, designed to address the performance limitations of traditional Python package management tools while maintaining compatibility with the existing Python packaging ecosystem. Unlike older tools that are written in Python itself, uv's implementation in Rust gives it exceptional speed advantages—often 10-100x faster than traditional tools for common operations. This performance boost is particularly noticeable in larger projects with complex dependency graphs. ## Key Features and Benefits ### Performance Performance is uv's most distinctive feature: - Parallel Downloads: Downloads and installs packages in parallel - optimised Dependency Resolution: Efficiently resolves dependencies with a modern algorithm - Cached Builds: Maintains a build artifact cache to avoid redundant work - Rust Implementation: Low memory usage and high computational efficiency In practical terms, this means environments that might take minutes to create with traditional tools can be ready in seconds with uv. ### Compatibility Despite its modern architecture, uv maintains compatibility with Python's ecosystem: - Standard Wheel Support: Installs standard Python wheel distributions - PEP Compliance: Follows relevant Python Enhancement Proposals for packaging - Requirements.txt Support: Works with traditional requirements files - pyproject.toml Support: Compatible with modern project configurations ### Unified Functionality uv combines features from several traditional tools: - Environment Management: Similar to venv but faster - Package Installation: Like pip but with parallel processing - Dependency Resolution: Similar to pip-tools but more efficient - Lockfile Generation: Creates deterministic environments like pip-compile ## Getting Started with uv ### Installation uv can be installed in several ways: ### Basic Commands uv has an intuitive command structure that will feel familiar to pip users: ### Working with Virtual Environments uv integrates environment management with package installation: ## Dependency Management with uv ### Compiling Requirements uv offers an efficient workflow for managing dependencies using a two-file approach similar to pip-tools: The generated requirements.txt will contain exact versions of all dependencies (including transitive ones), ensuring reproducible environments. ### Development Dependencies For more complex projects, you can separate production and development dependencies: The `-c requirements.txt` constraint ensures compatible versions between production and development dependencies. ### Updating Dependencies When you need to update packages: ## Advanced uv Features ### Offline Mode uv supports working in environments without internet access: ### Direct URLs and Git Dependencies uv can install packages from various sources: ### Configuration Options uv allows configuration through command-line options: ### Performance optimisation To maximise uv's performance: ## Integration with Workflows ### CI/CD Integration uv is particularly valuable in CI/CD pipelines where speed matters: ### IDE Integration While IDEs typically detect standard virtual environments, you can explicitly configure them: #### VS Code 1. Create an environment: `uv venv` 2. Select the interpreter at `.venv/bin/python` (Unix) or `.venv\Scripts\python.exe` (Windows) #### PyCharm 1. Create an environment: `uv venv` 2. In Settings → Project → Python Interpreter, add the interpreter from the .venv directory ## Comparing uv with Other Tools ### uv vs. pip | Feature | uv | pip | |---------|-----|-----| | Installation Speed | Very fast (parallel) | Slower (sequential) | | Dependency Resolution | Fast, efficient | Slower, sometimes problematic | | Environment Management | Built-in | Requires separate tool (venv) | | Lock Files | Native support | Requires pip-tools | | Caching | Global, efficient | More limited | | Compatibility | High with standard packages | Universal | ### uv vs. pip-tools | Feature | uv | pip-tools | |---------|-----|-----------| | Speed | Very fast | Moderate | | Implementation | Rust | Python | | Environment Management | Integrated | Separate (needs venv) | | Command Structure | `uv pip compile/sync` | `pip-compile/pip-sync` | | Hash Generation | Supported | Supported | ### uv vs. Poetry/PDM | Feature | uv | Poetry/PDM | |---------|-----|------------| | Focus | Performance | Project management | | Configuration | Minimal (uses standard files) | More extensive | | Learning Curve | Gentle (similar to pip) | Steeper | | Project Structure | Flexible | More opinionated | | Publishing to PyPI | Basic support | Comprehensive support | ## Best Practices with uv ### Dependency Management Workflow A recommended workflow using uv for dependency management: 1. Define direct dependencies in a `requirements.in` file with minimal version constraints 2. Compile locked requirements with `uv pip compile requirements.in -o requirements.txt` 3. Install dependencies with `uv pip sync requirements.txt` 4. Update dependencies periodically with `uv pip compile --upgrade requirements.in` ### Optimal Project Structure A simple project structure that works well with uv: ### Version Control Considerations When using version control with uv: - Commit both `.in` and `.txt` files to ensure reproducible builds - Add `.venv/` to your `.gitignore` - Consider committing hash-verified requirements for security ## Troubleshooting uv ### Common Issues and Solutions #### Missing Binary Wheels If you encounter issues with packages requiring compilation: #### Dependency Conflicts For dependency resolution issues: #### Environment Problems If environments aren't working properly: ## Conclusion uv represents an exciting advancement in Python tooling, offering significant performance improvements while maintaining compatibility with existing workflows. Its speed benefits are particularly valuable for: - CI/CD pipelines where build time matters - Large projects with many dependencies - Development environments with frequent updates - Teams looking to improve developer experience While newer than some traditional tools, uv's compatibility with standard Python packaging conventions makes it a relatively low-risk adoption with potentially high rewards in terms of productivity and performance. As it continues to mature, uv is positioned to become an increasingly important part of the Python development ecosystem. For most projects, uv can be a drop-in replacement for pip and pip-tools, offering an immediate performance boost without requiring significant workflow changes—a rare combination of revolutionary performance with evolutionary adoption requirements. ============================================================ SOURCE: appendices/poetry.qmd ============================================================ # Poetry - Modern Python Packaging and Dependency Management ## Introduction to Poetry Poetry is a modern Python package management tool designed to simplify dependency management and packaging in Python projects. Developed by Sébastien Eustace and released in 2018, Poetry aims to solve common problems in the Python ecosystem by providing a single tool to handle dependency installation, package building, and publishing. Poetry's core philosophy is to make Python packaging more deterministic and user-friendly through declarative dependency specification, lock files for reproducible environments, and simplified commands for common workflows. By combining capabilities that traditionally required multiple tools (pip, setuptools, twine, etc.), Poetry offers a more cohesive development experience. ## Key Features of Poetry ### Dependency Management Poetry's dependency resolution is one of its strongest features: - Deterministic builds: Poetry resolves dependencies considering the entire dependency graph, preventing many common conflicts - Lock file: The `poetry.lock` file ensures consistent installations across different environments - Easy version specification: Simple syntax for version constraints - Dependency groups: organise dependencies into development, testing, and other logical groups ### Project Setup and Configuration Poetry uses a single configuration file for project metadata and dependencies: - pyproject.toml: All project configuration lives in one standard-compliant file - Project scaffolding: `poetry new` command creates a standardized project structure - Environment management: Automatic handling of virtual environments ### Build and Publish Workflow Poetry streamlines the package distribution process: - Unified build command: `poetry build` creates both source and wheel distributions - Simplified publishing: `poetry publish` handles uploading to PyPI - Version management: Tools to bump version numbers according to semantic versioning ## Getting Started with Poetry ### Installation Poetry can be installed in several ways: After installation, verify that Poetry is working: ### Creating a New Project To create a new project with Poetry: Alternatively, initialize Poetry in an existing project: This interactive command helps you create a `pyproject.toml` file with your project's metadata and dependencies. ### Basic Configuration The `pyproject.toml` file is the heart of a Poetry project. Here's a sample: ## Essential Poetry Commands ### Managing Dependencies ### Environment Management ### Building and Publishing ### Running Scripts ## Advanced Poetry Features ### Dependency Groups Poetry allows organising dependencies into logical groups: Install specific groups: ### Version Constraints Poetry supports various version constraint syntaxes: - `^1.2.3`: Compatible with 1.2.3 =1.2.3,<1.5.0`: Version between 1.2.3 (inclusive) and 1.5.0 (exclusive) - `1.2.3`: Exactly version 1.2.3 - `*`: Any version ### Private Repositories Configure private package repositories: ### Script Commands Define custom commands in your `pyproject.toml`: These commands become available through `poetry run` or when the package is installed. ## Best Practices with Poetry ### Project Structure A recommended project structure for Poetry projects: To use the `src` layout with Poetry: ### Dependency Management Strategies 1. Minimal Version Specification: Use `^` (caret) constraint to allow compatible updates 2. Development vs. Production Dependencies: Use groups to separate dependencies 3. Update Strategy: Regularly update the lock file ### Version Control Practices 1. Always commit the lock file: The `poetry.lock` file ensures reproducible builds 2. Consider a CI step to verify lock file consistency: ### Integration with Development Tools #### Code Formatting and Linting Configure tools like Black and Ruff in `pyproject.toml`: #### Type Checking Configure mypy in `pyproject.toml`: ## Integration with Development Workflows ### IDE Integration Poetry integrates well with most Python IDEs: #### VS Code 1. Install the Python extension 2. Configure VS Code to use Poetry's environment: - It should detect the Poetry environment automatically - Or set `python.poetryPath` in settings #### PyCharm 1. Go to Settings → Project → Python Interpreter 2. Add the Poetry-created interpreter (typically in `~/.cache/pypoetry/virtualenvs/`) 3. Or use PyCharm's Poetry plugin ### CI/CD Integration #### GitHub Actions Example ## Troubleshooting Common Issues ### Dependency Resolution Errors If Poetry can't resolve dependencies: ### Virtual Environment Problems For environment-related issues: ### Package Publishing Issues When facing publishing problems: ## Comparison with Other Tools ### Poetry vs. pip + venv - Poetry: Single tool for environment, dependencies, and packaging - pip + venv: Separate tools for different aspects of the workflow - Key difference: Poetry adds dependency resolution and lock file ### Poetry vs. Pipenv - Poetry: Stronger focus on packaging and publishing - Pipenv: Primarily focused on application development - Key difference: Poetry's packaging capabilities make it more suitable for libraries ### Poetry vs. PDM - Poetry: More opinionated, integrated experience - PDM: More standards-compliant, supports PEP 582 - Key difference: Poetry's custom installer vs. PDM's closer adherence to PEP standards ### Poetry vs. Hatch - Poetry: Focus on dependency management and packaging - Hatch: Focus on project management and multi-environment workflows - Key difference: Poetry's stronger dependency resolution vs. Hatch's project lifecycle features ## When to Use Poetry Poetry is particularly well-suited for: 1. Library Development: Its packaging and publishing tools shine for creating distributable packages 2. Team Projects: The lock file ensures consistent environments across team members 3. Projects with Complex Dependencies: The resolver helps manage intricate dependency requirements 4. Developers Wanting an All-in-One Solution: The unified interface simplifies the development workflow Poetry might not be ideal for: 1. Simple Scripts: May be overkill for very small projects 2. Projects with Unusual Build Requirements: Complex custom build processes might need more specialized tools 3. Integration with Existing pip-Based Workflows: Requires adapting established processes ## Conclusion Poetry represents a significant evolution in Python package management, offering a more integrated and user-friendly approach to dependencies, environments, and packaging. Its focus on deterministic builds through the lock file mechanism and simplified workflow commands addresses many pain points in traditional Python development. While Poetry introduces its own conventions and may require adaptation for teams used to traditional tools, the benefits in terms of reproducibility and developer experience make it worth considering for both new and existing Python projects. As the tool continues to mature and the ecosystem around it grows, Poetry is establishing itself as a standard part of the modern Python development toolkit.