What are the best immutable backup solutions to recover from a ransomware attack?

<opinion>
I/M/O, any automatic backup solution that does not require an administrator to enter a password before backing up can be subverted.  But even if a password is required to be entered, ransomware can pick up the password and use it to destroy your backups.  There is no perfect 100% reliable solution.

System:  Full backup, not incremental, done weekly to an external USB drive which is only plugged into the system when a backup is being done.

Data:  Keep it off the system drive.  Back it up locally to a RAID set which is brought online only during backups.
</opinion>

the key is backups should be incremental and performed on a write-only support, preferrably via api calls.

it is both primordial and sufficient to ensure the systems that need to be backed up cannot overwrite or delete existing backups.

obviously, the backup location and system should be secure. an old machine in a dedicated network segment might do. it does not require internet access not any connectivity besides the port used for the backup. sticking it in a domain is an absolute nogo.

Cohesity is one of the best.

I appreciate all the comments and advice. I like the many ideas of keeping the backup off the network or disconnected once it completes, but we are currently doing hourly backups and I don't know how I would be able to do that. We also replicate that data offsite on an hourly basis. These are, of course, incremental backups.

(This gives me the idea to only keep the system online during business hours, but I don't know how to automate that and sometimes if we have high rates of change, our replication can take hours and I don't really know when that will be.)

I have recently added tape backups back into our current network backups in order to have an untouchable backup once completed. We send then send the tapes offsite, but even with LTO8, the processing time is too long as we have 120 TB to maintain and I do full backups so as to not confuse our other backup system with different Incremental points. This only allows me one full backup per month and leaves me with weeks in between- but, at least, it is something to fall back on.

The cost of the Pure/Cohesity solution is coming in at (MSRP) $430, 000 per year on a subsciption basis or 1.3 million on a 5-year Capex model. That is an enormous increase and I am in the beginning stages of evaluating all of this to find the best solution. 

You could also look at

N-Able

https://www.n-able.com/

and Backup Direct to the Cloud ! (even with low bandwidth) it's very fast if backup times are an issue!

Replication = Garbage In is Garbage Out. (There's plenty of huge failures because of this).

Any image backup is Garbage In = Garbage Out.

We deploy Veeam with two tiers of Cloud:
 Tier 1: Not Immutable
 Tier 2: Immutable

We have key backups uploaded to Immutable as soon as the backup completes while others will be uploaded daily or weekly depending on what they are and the rate of change.

The destination for most of the big settings is a Scale-Out File Server standalone or cluster capable of the necessary ingest rates and safety resilience for the on-premises data.

The KEY in all of this: We bare-metal or hypervisor restore a full set of VM images on a quarterly basis.

We have Veeam set up to consolidate the incremental backups regularly and also to test the backup chains to make sure they are healthy. BTDT as far as going to restore and the one we needed was hooped because the chain was broken back to the parent though that was with StorageCraft ShadowProtect.

Set up correctly, Veeam is an excellent product to provide the three tiers of backup, immutability, and ease of use and monitoring over time.

Hi,
I know Veeam and Rubrik because using it for my customers, both good products.
Whatever product, look for immutable, and offsite copies, of all your main datas, that's the keys.
Rubrik has native immutable filesystem, and fine hardware, both included in the solution. And some optionnal features to help against ransomwares. 

One more important feature is portability: We are able to get to Veeam backups without having to install the Veeam server setup if it gets wiped. In a disaster recovery situation this is critical.

Some products require a full re-install before backups become accessible.

Thanks, everyone, for your replies. I found it very helpful to be reminded of completely taking backups offline and ensuring they can't be overwritten, while still having the convenience of the always-on backups whereby they are done more frequently and hourly, in my case.

I am also thankful to be reminded of the examples where the solution providers are not always mistake-free and the importance of garbage in/garbage out. I will be sure to ramp up my testing of backups and further develop a more thorough and efficient plan for doing so.

I will continue working on this as I further evaluate the Pure/Cohesity solution, as well as Dell, Veeam, and Rubrik solutions. Thanks again for taking the time to share your knowledge and further enable me to an informed decision.