Contents
The Cost of a Bad Firewall Change
A firewall change gone wrong is not just a technical problem — it's a business incident. If you accidentally block traffic to a payment processor at 11am on a business day, you have just created a P1 that affects revenue, triggers client SLAs, and generates an urgent post-mortem.
I have deployed firewall changes on Cisco ASA (single and multi-context), Cisco Firepower, and Palo Alto Networks platforms across production networks in healthcare, finance, and government sectors. This is the methodology I follow every time — regardless of whether the change is adding one rule or restructuring a zone.
Phase 1 — Planning
Understand What's Actually Being Asked
Engineers frequently receive firewall requests that are underspecified. "Allow server A to talk to server B" is not a firewall request — it's a sentence. Before writing a single access control entry, confirm:
- Source: exact IP or subnet (not "the app server")
- Destination: exact IP or subnet
- Protocol and port: TCP/UDP? Which specific port? Both directions?
- Direction: is this inbound to the firewall, outbound, or internal east-west?
- Application context: what application, what business purpose?
- Is this permanent or temporary? Temporary rules must have an expiry date in the ticket
Blast Radius Analysis
Before any change, ask: if this rule is wrong, what breaks?
Lab Test
If the change is anything beyond adding a straightforward permit rule, test it in Cisco CML or EVE-NG first. Build the same interface topology, apply the proposed rules, and verify with packet-level testing (ping, curl, Wireshark capture on the simulated host).
Phase 2 — Pre-Change
Backup Configuration
# Cisco ASA — backup running config before change
ASA# copy running-config tftp://10.1.1.10/asa-fw01-pre-change-2026-04-15.cfg
# Also save locally to flash as a second backup
ASA# copy running-config flash:pre-change-2026-04-15.cfg
# Palo Alto — create a named configuration snapshot
admin@PA-FW> save config to pre-change-2026-04-15.xml
# Also commit-lock if working in a team environment to prevent concurrent commits
admin@PA-FW> request config-lock add comment "CHG0012345 - firewall rule change by emman"
Snapshot Current State
Document the "before" state — this is your rollback baseline and your post-change comparison point.
# ASA — capture current connection table and access-list hit counts
ASA# show conn count
ASA# show access-list | include elements|hitcnt
ASA# show xlate count
# Palo Alto — capture current session count and security rule hit counts
admin@PA-FW> show session info
admin@PA-FW> show running security-policy
Phase 3 — Execution
The Golden Rule: One Rule at a Time
Never batch multiple rule changes into a single push. Apply one rule, test it, then apply the next. If something breaks, you know exactly which rule caused it.
Cisco ASA — Adding a Permit Rule
# Adding a specific application permit — principle of least privilege
ASA(config)# access-list INSIDE_TO_DMZ remark CHG0012345 - Allow app-server to DB (Apr 15 2026)
ASA(config)# access-list INSIDE_TO_DMZ extended permit tcp host 10.10.5.25 host 10.20.1.10 eq 5432
# TCP only, specific source host, specific destination host, specific port (PostgreSQL)
# The remark must include the change ticket number and date — required for audits
# Verify the rule is in the correct position (ACLs are first-match)
ASA# show access-list INSIDE_TO_DMZ | begin CHG0012345
# Confirm the new rule appears before any deny that would shadow it
# IMPORTANT: Do NOT write mem yet — test first
Palo Alto — Adding a Security Policy Rule
# In Palo Alto, changes go into candidate config — require explicit commit
# This is a safety advantage over ASA: changes don't take effect until you commit
# Add rule via CLI (candidate config)
admin@PA-FW# set rulebase security rules "Allow-AppServer-DB" from trust to dmz
admin@PA-FW# set rulebase security rules "Allow-AppServer-DB" source 10.10.5.25/32
admin@PA-FW# set rulebase security rules "Allow-AppServer-DB" destination 10.20.1.10/32
admin@PA-FW# set rulebase security rules "Allow-AppServer-DB" application postgresql
admin@PA-FW# set rulebase security rules "Allow-AppServer-DB" service application-default
admin@PA-FW# set rulebase security rules "Allow-AppServer-DB" action allow
admin@PA-FW# set rulebase security rules "Allow-AppServer-DB" log-end yes
# Preview what will change before committing
admin@PA-FW> show config diff
# Commit — traffic is affected only after this command
admin@PA-FW> commit description "CHG0012345 Allow app-server to DB port 5432"
Phase 4 — Verification
Traffic Test
Always confirm the change achieves its intent — don't assume a successful config push means traffic flows.
# From the source host — verify connectivity after rule application
$ nc -zv 10.20.1.10 5432
# Connection to 10.20.1.10 5432 port [tcp/postgresql] succeeded!
# If this fails despite the rule: check NAT, check routing, check other ACLs in path
# ASA — verify the new rule has hits after traffic test
ASA# show access-list INSIDE_TO_DMZ | begin CHG0012345
# hitcnt=1 (or more) — rule matched. Zero hits = traffic is not matching this rule
# Palo Alto — verify traffic in session table and logs
admin@PA-FW> show session all filter source 10.10.5.25 destination 10.20.1.10
admin@PA-FW> show log traffic direction equal forward src-addr equal 10.10.5.25
For Deny/Delete Changes — Confirm No Legitimate Traffic Was Blocked
# ASA — check syslog for deny hits after applying a new deny rule
ASA# show log | include Deny
# Any denies matching your new rule in the 5 minutes post-change?
# If yes and traffic is unexpected: investigate before proceeding
# If yes and traffic is expected (the deny is working): document and confirm with requester
# After verifying — save config
ASA# write memory
# Do NOT write memory until verification is complete
# Unsaved config can be reverted by reloading the ASA — your last defense
Rollback Procedure
If something goes wrong, execute rollback immediately — before spending time debugging.
# ASA — rollback via config restore
ASA# copy flash:pre-change-2026-04-15.cfg running-config
# Prompts for destination (accept default). Merges the backup config into running.
# For a clean rollback of ACL changes: remove the new rule manually
ASA(config)# no access-list INSIDE_TO_DMZ extended permit tcp host 10.10.5.25 host 10.20.1.10 eq 5432
# Palo Alto — rollback candidate config to last committed state
admin@PA-FW> revert config
admin@PA-FW> commit description "ROLLBACK CHG0012345"
# Or roll back to a named saved config
admin@PA-FW> load config from pre-change-2026-04-15.xml
admin@PA-FW> commit
The Full Checklist
- [ ] Change request ticket open with business justification
- [ ] Source, destination, protocol, and port confirmed with requester
- [ ] Blast radius assessed — understand what breaks if the rule is wrong
- [ ] Lab test completed for complex or high-risk changes
- [ ] Peer review by a second engineer (mandatory for PCI/critical zones)
- [ ] Pre-change backup: running config saved to TFTP and flash/XML
- [ ] Current state snapshot: hit counts, session counts documented
- [ ] Maintenance window open in SolarWinds (if device state changes expected)
- [ ] Team notified (NOC, app owner, client if applicable)
- [ ] Change applied — one rule at a time
- [ ] Traffic test passed from source host
- [ ] Hit counter verified (> 0 after traffic test)
- [ ] Syslog checked for unintended denies
- [ ] Configuration saved (write mem / commit)
- [ ] Change ticket updated with "verified and implemented" status
- [ ] Network diagram updated if topology changed
- [ ] Maintenance window cleared
- [ ] Any temporary rules have expiry date in ticket (and calendar reminder to remove)