IT Support agencies and contingency plans

Extensive documentation and disaster/recovery plans on every single case.
It seems that performing on some intevals a disaster simulation helps a lot.
And as always redudancy is always the best failsafe option....
Now for the case...the cleaning lady pulled the cable...no salvation on this... :)

I suppose it also depends on what type of "Support" you are looking for.

Every "system" is unique and no one can walk in off the street and be expected to "fix" things no matter what they say.

Sure, a broken PC is a broken PC.  Many can get it to boot again. But need internal info get get it back on your network and setup to work with your systems.

If a database (I'm a database person) goes down, it takes specific knowledge of your internal system to get it back up.

1) Do you have any formal/informal arrangements with 3rd party IT Support companies, so you can get some sort of priority in case you lose your internal staff for short/long term? I was just wondering if such an agreement is common/mandatory to have something in place for emergency scenarios to get priority service, or is it relatively easy to source external IT Support at short notice in your area, reducing the need to have some sort of contract in place with a local IT Support providers in case you needed temporary cover for internal absentees?

My approach is to run everything on automatic.

Hourly backups run automatically.

Restores are manual.

No realtime data replication, as this is generally expensive (staff wise + coding retool wise) to maintain.

2) I'm not sure on the costs associated with having some sort of informal agreement with a local support provider and whether in your opinion such agreements are worthwhile as you may never require them/may be able to go to market and source staff quickly anyway from any provider in the local area.

This means you trust the competence of a 3rd party.

Hint: No one cares more about "you" than "you".

Tip: Any backup contingency, even a simple daily backup, is only as good as how often you do a restore test.

Backups may or may not work, from day to day.

Only a restore test can tell you if you can restore your backups to a working system.

3) What contingencies do you have at your company with external IT support providers in case you needed temporary/agency staff to make up for the shortfall in your permanent internal support staff in the short/long term, e.g. long term sickness, maternity leave etc. Do you just wait for the scenario to occur and then try and get temporary/agency cover from *wherever*, or do you have something a bit more formal in place with a specific local support provider, and are there costs incurred with such an agreement?

None.

I only trust myself with backups + restores.

4) What are the main pros/cons to either approach? Which is more common etc?

This is a long list related to competency + dependence.

There's nothing wrong with using a 3rd party, so long as you consider then 100% incompetent + have no dependence on them.

What I mean by this is, you run a daily/weekly/monthly restore test personally + your backups are accessible without any.... conversation or interaction with your 3rd party.

My personal system for projects.

1) I run hourly backups, which are only accessible to me.

2) I run daily backups, which all project stakeholders have access too.

3) If system dies, I restore the most recent hourly backup... which is good... derived from long coordinating stakeholder conversations to determine correct data to restore.

4) If system dies + I'm dead (someone always dies first, might be a stakeholder or me), customer has access to daily backups.

Daily backup access is provided via HTTPS + SSH, where most stakeholders run a nightly rsync job to pull down their own data.

In the case of large data, I provide specialized code for them to pull only file deltas, generating their own local zstd tarball each night.

I suppose it also depends on what type of "Support" you are looking for.

I cant really be specific as it depends on the person who could be off long term sick, but it could be anything from DBA cover, Desktop/End User Support cover, Infrastructure Support cover (e.g. network hardware, backups, central systems like AD etc). All the common roles you have in any larger IT section.

https://www.experts-exchange.com/questions/29063033/MySQL-Solution-for-Hot-backup-for-60GB-database.html provides a longer description of to run hourly backups for large amounts of data.

Some interesting responses. I wasn't really coming at it from a DR perspective, more general IT support staff to handle day to day support operations, e.g. fixing incidents, dealing with service requests, general health & maintenance operations. Things that cannot be automated.

>>I cant really be specific as it depends on the person who could be off long term sick, but it could be anything

You probably cannot rely out outsiders.  You need to have someone onsite to assist.

So, it sounds like you need some level of cross training so no single person has all the knowledge.

I should have perhaps added, IT support for the business is an in house function, made up of permanent full time employees of the company, and IT support is not outsourced to a 3rd party. It was just the risk scenario of losing IT support (internal people) in the short/long term, and what agreements could be sought with 3rd parties for temporary cover until they return to work. We are not looking to outsource IT support to a 3rd party, I was just looking into how others plan for the unavailability of their internal IT staff, and whether they just deal with that as it occurs, of it they have/should have something a bit more formal with a local 3rd party support provider.

Every person, even outside of IT, should have a backup person identified.  So, no single person knows everything about a single piece.

It's not realistic for the backup person to know EVERYTHING the primary does but should know enough that if you bring in an outside person with expertise in 'X', the backup should know your internal information enough to assist.

If you lose your lead mechanic on a Formula-1 racing team and bring in a new one from the outside, the new person won't know where the tools are in your shop.  Someone locally will need to know.

OK, now I understand your intension.
But at the end, the same calculation.
What slightwv said is absolutely correct, short term absense should be bridged by other people from your own stuff.
For longer abnsence you may need a temporary replacement. But there are a lot of freelancers out there you can hire for a limited time. Even IT Companies do the same.
As better the documentation, as faster you can get people productive, even if they are from your own IT-Staff. 

Every person, even outside of IT, should have a backup person identified.  So, no single person knows everything about a single piece.

That make sense. In your experience, are there any particular roles in IT Support teams whereby there is a greater likelihood individuals may pose single points of failure in terms of knowledge? I suppose the more specialized the role, the more likely there is not sufficient backup knowledge in a smaller team. Its how you would identify such single points of failure in your teams knowledge/skillset and how to then put plans in place for cross training/additional hires.

What slightwv said is absolutely correct, short term absense should be bridged by other people from your own stuff.

the risk I was thinking of though was with COVID-19 you could face the risk of losing a lot of staff short/long term at the same time. Making it harder to keep bridging a growing number of absentees within the same team (sometimes smaller numbers).

Hello Pao,
yes, Covid brought up a lot of lacks in companies organizations. But you realize it usually, if the disaster already has happened. People are working for years in a company, but sometimes you realize their worth first, if they are not there anymore. The other option is, that people keep their knowledge secret to make themselves unreplacable. 

To limit it a bit down, you should have some options in your backhand.
- A reliable IT company near you which can possible help
- Some freelancers of freelancer agencies where you can hire short term people.
- the option to work remote (this especially for Covid).
- forcing information / knowledge exchange inside the own IT staff (following slightvw aspects).

A precodition for the first two options is..
- a well documented infrastructure
- a good documentation for all your services (good = quality, not quantity)
- a desaster recovery plan for each service
- also spreading information inside the IT staff. 

Not complete, just a few basics.
External people can never replace internal people at once, but you have to give them a chance to fill the gap. As better the documentation, as faster they can do it. And also a good documentation gives the internal staff the chance to repair some things without having specialist knowledge of the topic.