[Last Call] Learn how to a build a cloud-first strategyRegister Now


Tiered storage in Linux?

Posted on 2011-10-11
Medium Priority
Last Modified: 2012-05-12
I've been looking at this for a while, and not come up with anything satisfactory.  I don't know if it can be done, but I thought I'd solicit some opinions on how people think it might be achieved.  It may prove to be impossible...

I have a Linux server that runs 24/7, with a very light load.  It has a software RAID6 set with everything except /boot on it.  The drives never spin down because there are always log entries, database updates etc.  One of the small tasks assigned to this box, for example, is to poll a solar-power inverter every 5 minutes and record the result to a MySQL database.

How can I allow the drives to spin down?  I'm thinking perhaps using a USB flash drive as a buffer, like having a tiered drive, so that if the power goes out it's not lost, but every few hours it can be flushed to the drives.  Is that doable?

Failing that, is it sane to just write to the flash drive and rsync back to the spinning drives occasionally (for folders with logfiles and for database files)?

Other than that, how long can I reasonably stall output of the data within the machine so I don't fire up the drives very often?

My ideal would just be to pop in an SSD or other flash device and have it perform block-level buffering for the main drives, allowing the main drives to be updated in big chunks rather than piecemeal as they are just now.
Question by:Filthy_Defiler

Accepted Solution

fcontrepois earned 2000 total points
ID: 36953766
why is it such a problem that your drives do not spin down ?
Good disks handle working 24/7 quite well

What you want is a filesystem cache on disk, AFAIK only ZFS can do that on linux (http://zfsonlinux.org/)

Another solution is to use the Unionfs

A bientot
LVL 19

Expert Comment

ID: 36953898
You might have an issue with latency, if you manage to spin all the disks down they may not all come up at the same time, not necessarily an issue for a single disk but a raid set? Not something I'd be happy relying on.

Is it worth spinning down on this system? You could move the log server to a system with ssd disks if you needed to conserve electricity etc but it really just might not ne worth it for this.
LVL 35

Expert Comment

by:Duncan Roe
ID: 36954644
I have been told more than once that the majority of wear of drive bearings occurs during spin-up and spin-down. So I would not worry that they never stop.
NEW Veeam Backup for Microsoft Office 365 1.5

With Office 365, it’s your data and your responsibility to protect it. NEW Veeam Backup for Microsoft Office 365 eliminates the risk of losing access to your Office 365 data.


Author Comment

ID: 36955688
To clarify, this "server" is a box at home with a stack of SATA drives in it.  It's my home filestore, runs the databases for my Squeezeboxes, XBMC and to monitor the inverter for my solar panels.  Latency during spin-up shouldn't be an issue (so long as Linux is happy that drives will take a few seconds to reappear).

The drives are just sitting there wasting power 99% of the time, and if I can spin them down then that's a few Wh saved.  They're also sitting there at 50+ degrees C, which I'd rather see cooled off a little.

The drives should be happy for 300,000 spin-ups, so if it's only happening every few hours it's not going to be an issue within the lifetime of the drives.

I'm not using ZFS (ext3 and XFS only), though it did look interesting.  Maybe next time!  I also stumbled across FlashCache and bcache, but they don't look terribly mature.

Any other good ideas?  I intend to gradually swap the drives out for WD Caviar Greens to save a little power, but that's longer term.
LVL 41

Expert Comment

ID: 36958508
Ok definitely consider SSD disk instead.

Author Closing Comment

ID: 36960627
Well, looks like I'm out of luck just now.  I'll have a much closer look at ZFS and see if it fits the bill.  Could be a candidate for next time that box gets rebuilt.


Featured Post

Free recovery tool for Microsoft Active Directory

Veeam Explorer for Microsoft Active Directory provides fast and reliable object-level recovery for Active Directory from a single-pass, agentless backup or storage snapshot — without the need to restore an entire virtual machine or use third-party tools.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
Article by: evilrix
Looking for a way to avoid searching through large data sets for data that doesn't exist? A Bloom Filter might be what you need. This data structure is a probabilistic filter that allows you to avoid unnecessary searches when you know the data defin…
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
Suggested Courses
Course of the Month18 days, 10 hours left to enroll

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question