I have a spidering program that indexes data from a database server.
This spider is made up of the following components:
Colony - Windows Service
Spider - Individual Indexer limited to a specific company.
The Colony is aware of all of its Spiders, the spiders are only aware of the company they are indexing.
I would like the Colony to spawn Spiders in separate processes for the following reasons:
- Crash safety: if one spider crashes it should not take down the whole colony of spiders.
- Threading: Each spider should have access to its own thread pool (and not have to share with others), the main reason behind this is that I do not want one company to be able block other spiders from spawning threads to do there indexing.
I did consider AppDomain however I am unclear if running each spider instance in its own AppDomain would satisfy threading point above.