5.2Testing, CI/CD, and automation tools

Automation also carries the danger of "distributing a wrong configuration faster and wider." That is exactly why you need a mechanism to verify changes mechanically before they reach production (tests and a pipeline). This section organizes what each CI/CD stage protects when shipping network changes safely, the roles of the virtual lab CML and the test framework pyATS, and choosing among Ansible, Terraform, and NSO by "what each is good at"—as a match against requirements, not rote tool comparison.

5.2.1Stages of a CI/CD pipeline

• CI (Continuous Integration) runs an automatic build, check, and test on every commit/Pull Request, catching broken changes early. For networks, a lint (syntax/format check) stage (e.g., yamllint) first mechanically confirms "is the syntax even valid?" If it fails here, the change stops before reaching production.
• CD (Continuous Delivery/Deployment) is the stage that distributes passing changes to production automatically (or after approval). A typical order is lint -> test in a virtual lab -> approval -> deploy to production -> post-verification. Each stage is a gate that lets only what passed the previous one through, and which stage failed becomes the clue for isolating the cause.

5.2.2Test tools: CML and pyATS

• Cisco Modeling Labs (CML) is an environment that simulates a virtual network close to real hardware. In the pipeline's "test stage," you build the same topology as production in CML and try changes safely before production (a sandbox). You can validate without physical gear, and breaking it does not affect production.
• pyATS (+Genie) is Cisco's network test-automation framework. It parses show output into structured data and mechanically asserts expected state—"are BGP neighbors Established?", "is the interface up?" It is used to compare before/after state and detect regressions (unexpected degradation).

5.2.3Configuration/orchestration tools

• Ansible is an agentless (no resident software on targets; mainly SSH/API) configuration tool that writes the "desired state" in a YAML playbook and applies it push-style. Many modules are built to be idempotent, and it is widely used to configure/collect from network devices (ios_config, etc.). It is relatively easy to learn/adopt.
• Terraform is a provisioning tool that defines infrastructure declaratively, tracks "what is currently built" in a state file, and applies only the diff. It is strong at the create/destroy lifecycle of cloud resources and infrastructure; its core is diff management (plan/apply) between "desired" and "actual."
• Cisco NSO (Network Services Orchestrator) abstracts a multi-vendor network as a service model and translates/distributes/reconciles a higher-level intent like "provision one L3VPN" into each device's configuration. It is strong at multi-vendor, large-scale service delivery and continuous consistency.

A team's CI/CD pipeline runs, per Pull Request to Git, in this order: (1) yamllint checks the playbook format -> (2) spin up a production-like topology in CML and apply the change -> (3) pyATS validates "all BGP neighbors Established" and "target IFs up" -> (4) human approval -> (5) apply to production with Ansible -> (6) post-verify production with pyATS. One day a PR fails at stage (3) and stops. The log shows that after applying the change in CML, "one site's BGP neighbor stayed Idle." The key is to read this failure not as a pipeline malfunction but as proof the pipeline did its job: without this validation stage, the misconfiguration would have been distributed to every production site, noticed only after BGP actually dropped. Stage (3) (CML + pyATS) is "a safety net that mechanically validates expected state in a virtual environment before touching production," and which stage failed tells you where the cause lives—stage (1) means syntax, stage (3) means behavior, stage (6) means a production-specific difference. If this team thought "skip validation to ship faster," that would maximize automation's danger (distributing a wrong config faster and wider)—exactly backwards. Tool selection likewise follows strengths: the workhorse that applies config idempotently to devices is Ansible, expected-state validation is pyATS, the virtual test range is CML, and if you must abstract and run a multi-vendor service (L3VPN, etc.), NSO is the core.

5.2.4Section summary

• CI/CD is a chain of gates lint -> test in CML -> approval -> production -> post-verify; a failure at the test stage is the healthy act of stopping a misconfiguration before production
• CML is a virtual test range to try changes safely before production; pyATS structures show output and validates expected state (regression detection)
• Choose tools by strength: Ansible (agentless/idempotent config delivery), Terraform (declarative/state-managed provisioning), NSO (multi-vendor service orchestration)