tonelm54
asked on
Advise on storage
Ive upgrade an application I worked on a long time ago, which is working fine but storage is running out of control.
When I took over a few years ago files were stored in a database but kept falling over when the database file hit 2Gb, so I stored the files in a folder instead (sub folder for each record as one record could hold 0-~ files). After a few years the storage has gone from 2Gb being an issue to now 780Gb, however its using 10.2Tb of disc space due to the cluster size of the server and the majority of the files being very small (1-3Kb in size).
I cant do anything about the cluster size on the server, so need to look into alternative ways.
So I'm revisiting the idea of storing the files in a database, and looking at using SQLite to store the files in however slightly worried about the write locking (when SQLite writes it locks the database). I've been looking at splitting the file into segments and over multiple database files. so I can keep the the SQLite files around 200Mb each which I think would be a good size and reducing the loss of storage in the clusters.
Does anyone have any advise or previous experience in storing files for web apps?
Thank you
When I took over a few years ago files were stored in a database but kept falling over when the database file hit 2Gb, so I stored the files in a folder instead (sub folder for each record as one record could hold 0-~ files). After a few years the storage has gone from 2Gb being an issue to now 780Gb, however its using 10.2Tb of disc space due to the cluster size of the server and the majority of the files being very small (1-3Kb in size).
I cant do anything about the cluster size on the server, so need to look into alternative ways.
So I'm revisiting the idea of storing the files in a database, and looking at using SQLite to store the files in however slightly worried about the write locking (when SQLite writes it locks the database). I've been looking at splitting the file into segments and over multiple database files. so I can keep the the SQLite files around 200Mb each which I think would be a good size and reducing the loss of storage in the clusters.
Does anyone have any advise or previous experience in storing files for web apps?
Thank you
ASKER
I've been reading up on this using this guide 'http://www.dummies.com/how-to/content/storing-data-with-php-flat-file-or-database.html' however there doesn't seem any down side to using the files in a database.
Any reason why I cant store in database
Thank you
Any reason why I cant store in database
Thank you
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
It sounds to me like you should try and use one of PHP file compression for storage.These can be fast for file collections of less than a thousand or so files. But with 780Gb o files, this might get less than easy to find a balance between speed and disk space?