firewall related standard operating procedures
Is it still fairly common best practice to document ‘standard operating procedures’ for the overall management and administration of critical data centre devices such as firewalls, and if so can you provide some examples of common maintenance or ‘day to day’ tasks that would benefit from documented standard operating procedures, for say overall management and administration of a firewall device?
From a risk management and support continuity perspective, it was always seen as a best practice as it allowed for standard administration of critical devices, that would reduce the likelihood of administrative error and subsequent downtime, security issues and disruption to the business. It also ensured if a key member of the support team was unavailable, there was some continuity, so another member of the team had a reference guide to use for specific tasks.
The kick back we have sometimes received when trying to encourage such documentation is the engineers are all experienced and qualified, so they don’t need such documentation to do their job – is this valid, or do you still work towards standard operating procedures and if so why types of firewall management/administration ‘procedures’ do you document? I don’t work on the operational support side so getting your views would be very interesting. Sone common things I have seen previously documented in a SOP (albeit not for firewalls) are rebuild and restore 'procedures' for disaster situations, such as hardware failures. There may be many more useful standard procedures to document though, but any that would allow restoration of service and prevent security breaches are obvious candidates for administration standards.
I've also seen that some places uses the SOP as part of business continuity testing drills, e.g. can someone complete the mandatory daily/weekly/monthly administration and maintenance of a firewall based on the documented procedures available. Do you do this as well as part of skills transfer initiatives?
External penetration testing by approved organisations. This was to test generally against known threats but to also audit the effectiveness of the controls against the requirement of the agreed business security model and mandate.
The sop also listed what actions were to be taken in the event of a breach. These may sound obvious but differing situations drive different actions. For example, in the event of a suspected breach, do you shut down external access completely? Just one service? Who makes the decision? The engineer? A business director? What is the escalation and authorisation procedure? All part of the sop and all should be aware of them otherwise each engineer will make their own call.
Even simple tasks such as reports on traffic flow, protocol bottlenecks or bandwidth utilisation, unusual traffic hits on a particular services, all of these are standard maintenance activities which depending on the sop's may or may not drive an action. If not in the sop then any colleague will make their own individual judgement and your 'standard's is blown.
Was there a particular area you were thinking about or just generalities?
The management for each area of IT support (e.g. DBA, Desktop Support, Server Support, Network Devices etc) should be documenting SOP/procedure notes for standard administrative duties, then their colleagues should be asked to achieve the relevant task using the SOP documentation, as part of routine BCP test drills. To identify skills gaps, single points of failure, inadequate SOP documentation etc, etc.
It is for the organisation to decide on how detailed it wants the SOP to be but regardless each entry should be focussed, clear and concise, achievable, be timebound so it is reviewed as still relevant and have a named owner (saying it is owned by the IT department is pathetic and pointless). For example at a simple level, a general item could be' NO firewall changes will be made in the production perimeter environment during normal working hours without prior written approval from the IT director'. At this level it is an abiding principle however, it is clear in its purpose. it does not tell the engineer how to be an engineer but it is absolute in what they can and cannot do. At the other end another item may be that all inbound ingress points will have a source and destination, a protocol and be bound to a specific user group. If another group want the same access to another inbound service then another rule must be made specifically for that access. Again, you are not telling them how to do their job but you are stating the level of protection you want. It is clear, has purpose and rather importantly it is auditable. It is ALSO understandable by qualified people who do NOT work for your organisation. A prime example is the Cyber Essentials audit (not sure if you have an equivalent) - you get asked questions (and there are a lot) and you need near identical answers that only having documented standards, policies and procedures can provide. Try the same exercise with a couple of engineers and you will get their personal viewpoints :)
https://wiki.sni.pitt.edu/images/9/9b/Firewall_Procedure.doc
You'll always avoid using dedicated "devices" with GUI.
Insteady you'll use dedicated Linux boxes for firewall devices, running iptables rules.
This way everything can be documented + converted to scripts which run each time the device reboots, ensuring 100% duplicated config steps 100% of the time.
And, yes, having SOPs + reproducible configs is essential to running rock solid infrastructure.