Introduction

#

This document explains how the dxaws unit and integration tests are structured and how to work with the test frameworks effectively.

Why we test (and why multiple layers matter)

#

The goal of testing in dxaws is not just correctness — it is confidence.

Each layer of testing captures a different kind of value:

• Unit tests give confidence that local behaviour is correct and refactor-safe
• Integration tests give confidence that module contracts compose correctly
• Acceptance tests give confidence that dxaws correctly and completely interfaces with AWS

No single test layer can provide all of this confidence on its own. Together, these layers allow dxaws modules to evolve internally while remaining externally trustworthy.

Testing philosophy

#

dxaws uses two distinct layers of tests, each with a clearly defined purpose:

• Unit tests protect local behaviour and enable aggressive refactoring
• Integration tests protect external contracts and composition

The distinction is intentional and enforced.

Unit tests

#

Definition
A unit test validates the behaviour of a single module or class in isolation.

Rules

• One-to-one mapping between source files and test files
• The test imports only the file under test
• All dependencies are mocked or stubbed
• Failures must be attributable to changes in the unit under test

Why this matters

• Test failures are immediately actionable
• Internal refactors are safe as long as behaviour is preserved
• Tests document intent, not implementation details

Examples

• dxaws_s3/manager.py → tests/test_manager.py
• dxaws_s3/planner.py → tests/test_planner.py

In manager unit tests specifically:

• planner, executor, and provider behaviour is mocked
• inputs may be duck-typed rather than real model objects
• tests validate orchestration and control-flow invariants
• tests should not fail due to changes in planner/models; those changes should be caught by their own unit tests or by integration tests

A note on mocking

#

Unit tests should mock or stub dependencies, but avoid mocking the unit under test.

• Prefer simple fakes/stubs for protocols where possible
• Use mocks for call assertions when the interaction itself is the invariant
• If a unit test passes only because the mock “agrees” with the test, consider adding or strengthening an integration test

Integration tests

#

Definition
An integration test validates that a module’s external interface behaves correctly when real components are composed together.

Rules

• Tests exercise the module as a consumer would
• Real planners, models, and executors are used
• Providers may be stubbed, but interfaces are real
• Tests validate contracts, not internal call structure

What integration tests protect

• Public APIs remain usable
• Components compose correctly
• Refactors do not silently break external behaviour

Scope Integration tests are:

• fast
• local
• deterministic
• AWS-free (unless explicitly stated otherwise)

They are not end-to-end or infrastructure tests.

What counts as the “external interface”

#

For dxaws modules, the external interface is intentionally small:

• the package exports (what from dxaws_<module> import ... exposes)
• the manager methods intended for consumers (e.g. plan(), apply(), converge())
• the return types and error semantics of those manager methods

Everything else (models, planners, executors, providers, helper functions) is considered internal implementation detail.

Relationship between unit and integration tests

#

Both layers are required. Neither replaces the other.

Pytest markers

#

Tests are explicitly categorized using pytest markers.

• Unit tests are the default (typically unmarked)
• @pytest.mark.integration — contract and composition tests

Recommended usage:

• pytest → runs unit tests (and anything unmarked)
• pytest -m integration → runs only integration tests

This keeps the default development loop fast while preserving confidence in module contracts.

Tip: register markers in pyproject.toml (or pytest.ini) to avoid warnings and keep pytest --strict-markers viable.

When to write which test (checklist)

#

Use this checklist when deciding where a test belongs.

Write a unit test if:

#

• You are testing a single class or function
• You want failures to be attributable to one file
• Dependencies can be mocked or stubbed
• You expect to refactor internals frequently

Write an integration test if:

#

• You are testing a module’s public interface
• Real planners / models / executors must work together
• You want to protect against accidental contract breakage
• A consumer could realistically depend on this behaviour

If unsure, start with a unit test. Add an integration test only when behaviour crosses module boundaries.

Practical implications for dxaws modules

#

• Managers are tested at all three layers:
  • unit tests validate orchestration logic
  • integration tests validate public behaviour
  • acceptance tests validate real AWS behaviour
• Planners, executors, and providers are unit-tested only
• The package root (e.g. dxaws_s3) is treated as a public contract and validated via integration tests

Test directory structure

#

Tests are organised by intent, not by module:

tests/
├── unit/
│   └── test_<file>.py
├── integration/
│   └── test_<module>_contract.py
└── acceptance/
    └── test_<module>_lifecycle.py

This structure makes it immediately obvious:

• what kind of confidence a test provides
• how expensive it is to run
• whether it is safe to run locally by default

Unit tests are the default and should be fast enough to run continuously. Integration tests are still local but exercise real composition. Acceptance tests are explicit, opt-in, and interact with AWS.

Acceptance tests (live AWS)

#

Acceptance tests validate dxaws modules against real AWS APIs by creating, verifying, and destroying actual resources.

These tests exist to answer a single question:

Does this dxaws module correctly and completely manage real AWS resources?

What acceptance tests cover

#

Acceptance tests verify that:

• AWS provider implementations are correct
• Required AWS parameters and options are fully implemented
• Desired state results in the expected real-world resources
• Cleanup paths work correctly and completely

This layer ensures dxaws is not just theoretically correct, but operationally sound.

What acceptance tests are not

#

Acceptance tests are:

• not unit tests
• not local integration tests
• not run as part of the default development loop

They are intentionally isolated due to cost, latency, and environmental requirements.

Location and naming

#

Acceptance tests should live in a clearly separated directory:

• tests/aws/ or tests/acceptance/

Recommended naming:

• tests/aws/test_<module>_lifecycle.py

Pytest markers

#

Acceptance tests must be explicitly marked:

• @pytest.mark.aws or @pytest.mark.acceptance

Recommended usage:

• pytest → unit tests
• pytest -m integration → integration tests
• pytest -m aws → acceptance (live AWS) tests

Safety guardrails (mandatory)

#

Acceptance tests must include the following safeguards:

1. Dedicated AWS account
   Tests should run in a dedicated sandbox or test account with constrained permissions and budgets.

2. Deterministic tagging
   All resources created by acceptance tests must be tagged, at minimum:

   • dxaws:test = true
   • dxaws:module = <module-name>
   • dxaws:run_id = <uuid>

   Optional but recommended:

   • dxaws:ttl = <timestamp>

3. Guaranteed cleanup
   Tests must clean up resources even on failure (e.g. using pytest fixtures with yield and finally).

4. Verification via AWS APIs
   Resource correctness must be verified by querying AWS APIs, not by assuming success.

5. Eventual consistency tolerance
   Acceptance tests should include bounded retries where AWS eventual consistency applies.

Execution controls

#

Acceptance tests must never run accidentally.

Recommended controls:

• Require explicit environment variables (e.g. DXAWS_AWS_TESTS=1)
• Require an explicit AWS profile or role
• Skip tests automatically if prerequisites are not met

Typical scope

#

Acceptance tests should focus on golden paths:

• Create → verify → delete lifecycle tests
• Idempotency tests (apply the same desired state twice)

They should avoid combinatorial explosions of configuration, which belong in unit or integration tests.

Language and spelling conventions

#

dxaws documentation and code comments follow Canadian English conventions.

Examples:

• behaviour (not behavior)
• behaviour-driven (not behavior-driven)
• organise / organised (not organize / organized)

This applies to:

• documentation
• comments in source code
• test descriptions and assertions

Consistency here improves readability and avoids unnecessary churn in reviews.

Why this matters

#

dxaws modules are intended to be:

• long-lived
• composable
• orchestrated by higher-level systems (Lambda, Step Functions, runbooks)

This testing strategy ensures:

• internal freedom to evolve
• stable external contracts
• confidence that primitives remain trustworthy over time