Any feedback about desktop grid storage (storage cloud on desktop) ?

Dear experts,

I was wondering if any of you had plan or are already using some storage solution on a desktop grid ?

Desktop grid are known since the SETI project but this project was a computational desktop grid (CPU share) where I am looking for a storage one (HDD share).

Any experience related to a "centralized use" based on your managed desktops available storage space and availability ?

Thanks for reading anyway
Best regards
LVL 18
BigSchmuhAsked:
Who is Participating?
 
Gerald ConnollyCommented:
Although there are several distributed NAS systems available such as Panasas and Gluster, none that i am aware of will make use of the spare capacity on a bunch of workstations.

Even if you had the physical security of multiple copies of the data which would seriously restrict the available space, you would still have the problem of the security of the data.
0
 
Brian PiercePhotographerCommented:
Are you talking about something like DropBox ? http://www.dropbox.com/
0
 
BigSchmuhAuthor Commented:
No, I am not looking for a centralized online unsafe storage like Dropbox, neither for a centralized backup like BackBlaze.
I am looking for a distributed safe long-term storage, I thought of this storage to be distributed on a Desktop grid.
0
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

 
Gerald ConnollyCommented:
Are you looking to use the spare storage on a bunch of existing workstations or just that you need some storage on your LAN.

If its is the former then it will not be reliable, if its the latter then just buy some NAS boxes
0
 
BigSchmuhAuthor Commented:
I am looking to ise spare storage on workstation...and yes it will be more reliable than a centralized solution. That's exactly what Internet was designed for...
0
 
Gerald ConnollyCommented:
Well even if there were some software out there to allow you to harvest the spare capacity on workstations it actually less reliable than using centralised storage, mainly because you do not have any control over it. What happens if the user of the workstation decides to reboot, or switch it off and who would be responsible for virus checking and the security of the data stored? Who is backing it up etc etc.

And using a NAS solution doesnt necessarily need to be centralised.
0
 
BigSchmuhAuthor Commented:
Desktop grid storage obviously requires 3 copy minimum, 5 may be more than enough if the dispatch allows to avoid using correlated nodes.
 
Do you know any software able to dispatch on many NAS ?
0
 
BigSchmuhAuthor Commented:
Data security is not a problem :
-all files should be splitted in small blocs and send all around using bittorrent capability
-all files are encrypted with the main catalog unaware of their key

Restricting the space ? 100GB per workstation and 1000 workstations means about 100TB which are 20TB useable...
0
 
Gerald ConnollyCommented:
You seemed to have got all your own answers!

What software are you going to use to run this farm?
0
 
marcustechCommented:
Sounds like a nightmare!  I would expect that the overhead and duplication on this sort of distributed file system would run the network ragged - have you got any ideas on what the overhead will be like?
0
 
BigSchmuhAuthor Commented:
I don't have any package ready for that...I am wondering if all those possible features (bittorrent, encryption, catalog, ...) make "storage desktop grid" some sense or if this is not a good idea at all.

Regarding the network, P2P exchange aren't that bad and appears "flat" on the grid (vs centralized exchanges). No new bottleneck are expected, it takes the available bandwidth and that's all.
0
 
Gerald ConnollyCommented:
@marcustech, i agree with that, and it is going to be real difficult to manage as well. Its probably going to be more trouble than its worth.

I know it sounds like a good idea, but the fact that there isnt any software around to actually do this might be a clue!

Another problem might be that if a users workstation starts slowing down because of all the extra disk and network traffic, i suspect that the user will start killing processes to stop it happening! Once the user community find out what's going on you might have a revolt on your hands!  :-)
0
 
marcustechCommented:
Dropbox use what they call 'Lan Sync' when you have shared folders in the cloud they replicate changes peer to peer between workstations on the same subnet - http://www.dropbox.com/help/137  That sounds similar to what you're after, but there's not a lot of detail on how it's actually implemented.
0
 
BigSchmuhAuthor Commented:
Any more feedbacks, bottlenecks, open-source project or software that you could suggest ?
0
 
BigSchmuhAuthor Commented:
Thanks for trying !
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.