query regarding Dataset storage?

If we create a Dataset for each user, where will be these datasets stored.And for instance if there are 1 million users active then each user will have a separate dataset.Will this affect the performance or how will it be handled by framework?
LVL 4
pml_sivaAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

jnhorstCommented:
Wow, 1 million users and a dataset for each!

But seriously, I would think the only way to deal with that would be a database back end that would keep each user's data.  The other option would be saving the dataset as an XML file (the dataset class has WriteXml() and ReadXml() methods for this), naming the file according to the user's id.  If the dataset is large (lots of data), though, the disk IO cost might hit performance in an unacceptable way.

Datasets are xml abstractions of the database objects like tables and relationships, so if the technology at hand includes or can include a database like SQL Server or Oracle, it would be best to store the data in a database, and then load up/update the dataset from that data.

John
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
mmarinovCommented:
Hi pml_siva,

I'm totaly agree with jnhorst. Also why do you have to hold all users' data in dataset in the same time ? to answer you first question - DataSet is stored in the memory - it gets all the data and stays there. For example and see the difference - the datareader object read one recod at a time and does not cosume a lot of memory resources, but it keeps the connection to the database open

Regards!
B..M
0
RejojohnyCommented:

>>If we create a Dataset for each user
Is it just create .. i mean created in a page? do u plan to store it too? If s, how? is the data common between the users? what r u planning to use to store? cache?

>>Will this affect the performance
it will surely affect performance if u have 1 million dataset in any storage like cache, session, application etc ..what u could think of is having a common storage for all users and then querrying from it whenever required .. so this will be like a temporary storage on ur web server (memory) ...
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

pml_sivaAuthor Commented:
How to design for this type of requirement?
For ex I'm loading Dataset in pageload of my application when the user logs in.
If 1 million user logs in at the same time,what will happen?Whether 1 million datasets will be created or is there any?If 1 million datasets created then for ex if each datasets holds a size of 1Kb then for 1 million it would occupy 1 million Kb size in the web server..Definetly this will affect the performance of the web server.So how to implement this.......
0
mmarinovCommented:
pml_siva,

if you have 1 million user of your web site you must have a really good server :)
so yes, because you create dataset in the page_load - there will be 1 million datasets in the server's memory
so if you have a good configuration of your server ( with muc memory - may 2 GB ) it won't be a problem
also for operationg with the request of 1 million users you must have at least dual-processor machine

B..M
0
pml_sivaAuthor Commented:
Ok.How the datasets will be stored internally in the Cache ie.In which format?
0
pml_sivaAuthor Commented:
Ok For this case how to design the Implementations without affecting the performance..
When the User Logs in I need to load the Datas in the DB into the DataGrid and i need  to perform actions such as sorting etc...So if 1 million users logs in at the same time.Then it would be difficult ...and it creates 1 million datasets in the server.
So which is the right way to implement this?
Can u help me regarding this....

Thanks in Advance.
Pml_siva
0
mmarinovCommented:
pml_siva,

because of this you can use reader object - it reads a record at a time and does not keep the records in the memory
the problem is with reader object that keeps the connection open
so you have to choose which scenario to use
for cashing user's dataset you can see this article :http://weblogs.asp.net/tatochip/archive/2004/02/09/70105.aspx

B..M
0
RejojohnyCommented:
r u sure u want to look at this scenario ..  i mean as "mmarinov" menioned abvove u really need a very good server for implementing such an application .. normally for a page to excute, the time taken is less than a second .. so u r basically taking about about 1 million connection for every 1 second !!! higly impossoble ... what kind of a application r u taking about .. now the question arises what kind of data is to be shown? is there anything common between the data shown to the user .. if s, then u could store the dataset (which will be a select * from the table) in ur application cache and then filter from here based on requirement . so basically u have just one dataset in the memory ...
0
AerosSagaCommented:
only query the records you need, use the top function heavily.  Storing all the datasets in memory pretty much kills any advantage of having one.  I agree with all the other experts this is not a good idea.

Aeros
0
RejojohnyCommented:
I would recommend a split of points ... we have all tried to explain the cons of such an approach and also suggested alternatives ..
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.