Advise on storage
Posted on 2015-02-11
Ive upgrade an application I worked on a long time ago, which is working fine but storage is running out of control.
When I took over a few years ago files were stored in a database but kept falling over when the database file hit 2Gb, so I stored the files in a folder instead (sub folder for each record as one record could hold 0-~ files). After a few years the storage has gone from 2Gb being an issue to now 780Gb, however its using 10.2Tb of disc space due to the cluster size of the server and the majority of the files being very small (1-3Kb in size).
I cant do anything about the cluster size on the server, so need to look into alternative ways.
So I'm revisiting the idea of storing the files in a database, and looking at using SQLite to store the files in however slightly worried about the write locking (when SQLite writes it locks the database). I've been looking at splitting the file into segments and over multiple database files. so I can keep the the SQLite files around 200Mb each which I think would be a good size and reducing the loss of storage in the clusters.
Does anyone have any advise or previous experience in storing files for web apps?