What are some of the proven methods of archiving large quantities of files (e.g. millions of files, terabytes in size)? I'm looking for a way to retrieve/add files to an archive format VB.NET in such a way that it is still robust even if the archive is huge.
I have explored using ZIP files for archives, with Chilkat's libraries. There's a 4gb limit on zip archives though. I've thought about ISOs, TARs, maybe CAB files?
What other standardized formats exist out there specifically for archiving large quanties of files? Is there anything microsoft made, or reliable VB.NET classes that can manipulate them?