Avatar of Pau Lo
Pau Lo
 asked on

firewall related standard operating procedures

Is it still fairly common best practice to document ‘standard operating procedures’ for the overall management and administration of critical data centre devices such as firewalls, and if so can you provide some examples of common maintenance or ‘day to day’ tasks that would benefit from documented standard operating procedures, for say overall management and administration of a firewall device?

From a risk management and support continuity perspective, it was always seen as a best practice as it allowed for standard administration of critical devices, that would reduce the likelihood of administrative error and subsequent downtime, security issues and disruption to the business. It also ensured if a key member of the support team was unavailable, there was some continuity, so another member of the team had a reference guide to use for specific tasks.

The kick back we have sometimes received when trying to encourage such documentation is the engineers are all experienced and qualified, so they don’t need such documentation to do their job – is this valid, or do you still work towards standard operating procedures and if so why types of firewall management/administration ‘procedures’ do you document? I don’t work on the operational support side so getting your views would be very interesting. Sone common things I have seen previously documented in a SOP (albeit not for firewalls) are rebuild and restore 'procedures' for disaster situations, such as hardware failures. There may be many more useful standard procedures to document though, but any that would allow restoration of service and prevent security breaches are obvious candidates for administration standards. 

I've also seen that some places uses the SOP as part of business continuity testing drills, e.g. can someone complete the mandatory daily/weekly/monthly administration and maintenance of a firewall based on the documented procedures available. Do you do this as well as part of skills transfer initiatives? 

SecurityHardwareSoftwareNetworkingHardware Firewalls

Avatar of undefined
Last Comment
David Favor

8/22/2022 - Mon
Keith Alabaster

View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
Pau Lo

Thanks Keith. aside from changes to the rule base SOP/standards, are there any other common maintenance tasks specific to firewalls where you had SOP's at your previous employers when you were involved in the day to day support and administration of perimeter security?
Keith Alabaster

Yes, quite a few. Hardware/firmware/software changes to the products. How and when they could/should be implemented  how they were tested and could be rolled back.

External penetration testing by approved organisations. This was to test generally against known threats but to also audit the effectiveness of the controls against the requirement of the agreed business security model and mandate.

The sop also listed what actions were to be taken in the event of a breach. These may sound obvious but differing situations drive different actions. For example, in the event of a suspected breach, do you shut down external access completely? Just one service? Who makes the decision? The engineer? A business director? What is the escalation and authorisation procedure? All part of the sop and all should be aware of them otherwise each engineer will make their own call.

Even simple tasks such as reports on traffic flow, protocol bottlenecks or bandwidth utilisation, unusual traffic hits on a particular services, all of these are standard maintenance activities which depending on the sop's may or may not drive an action. If not in the sop then any colleague will make their own individual judgement and your 'standard's is blown.

Was there a particular area you were thinking about or just generalities?
Pau Lo

It is just interesting to see what SOP would be common specific to firewall management, from engineers who have worked specifically in that field. All of those you have listed make logical sense. It is partly drive from a support continuity perspective, that has been tested with COVID-19, short-term loss of number of experienced engineers scenarios, does the organisation have SOP and knowledge transfer to be able to cope and maintain administration of such devices in such situations, and iron out single points of failure or opportunities for developing additional SOP/procedure notes, that have been overlooked.

The management for each area of IT support (e.g. DBA, Desktop Support, Server Support, Network Devices etc) should be documenting SOP/procedure notes for standard administrative duties, then their colleagues should be asked to achieve the relevant task using the SOP documentation, as part of routine BCP test drills. To identify skills gaps, single points of failure, inadequate SOP documentation etc, etc.
Your help has saved me hundreds of hours of internet surfing.
Keith Alabaster

Exactly so.

It is for the organisation to decide on how detailed it wants the SOP to be but regardless each entry should be focussed, clear and concise, achievable, be timebound so it is reviewed as still relevant and have a named owner (saying it is owned by the IT department is pathetic and pointless). For example at a simple level, a general item could be' NO firewall changes will be made in the production perimeter environment during normal working hours without prior written approval from the IT director'. At this level it is an abiding principle however, it is clear in its purpose. it does not tell the engineer how to be an engineer but it is absolute in what they can and cannot do. At the other end another item may be that all inbound ingress points will have a source and destination, a protocol and be bound to a specific user group. If another group want the same access to another inbound service then another rule must be made specifically for that access. Again, you are not telling them how to do their job but you are stating the level of protection you want. It is clear, has purpose and rather importantly it is auditable. It is ALSO understandable by qualified people who do NOT work for your organisation. A prime example is the Cyber Essentials audit (not sure if you have an equivalent) - you get asked questions (and there are a lot) and you need near identical answers that only having documented standards, policies and procedures can provide. Try the same exercise with a couple of engineers and you will get their personal viewpoints :)


Here is one sample, and my experience is more towards change management and managing the audit requirements on the execution. Without documentation, how would we know the practice are enforced and followed when any of the administrator denied responsibility of certain task. Disputes can and will happened. No verifiable common understanding is setting up for trouble. Knowing what is the benchmark help to train and settle down newbie who joined or have to come in event of role change or separation of duties. Use of common terms and boundaries are important hence documented too.

David Favor

If you're truly concerned about SOPs + reproducible/understandable setups...

You'll always avoid using dedicated "devices" with GUI.

Insteady you'll use dedicated Linux boxes for firewall devices, running iptables rules.

This way everything can be documented + converted to scripts which run each time the device reboots, ensuring 100% duplicated config steps 100% of the time.

And, yes, having SOPs + reproducible configs is essential to running rock solid infrastructure.
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.