|
Subject:
|
Large Data
|
|
Posted By:
|
paul_singh
|
Post Date:
|
2/15/2004 12:31:55 PM
|
Hi Guys,
I hope someone can enlighten me here. We have a windows based application which we want to convert to ASP.NET. Initially when the windows based app fires, it downloads a large amount of data (approx 2 to 3 meg) based on user profile from SQL Server. Then app uses this data to populate GUI and do some selection work and other stuff. Data retrieved is in the form of XML. How should i handle this scenario for web based app. Obviously its not a good idea to store all this data in Session state since number of users could get large and saving this user profile data could get messy in session. I was thinking maybe i could get data the way its done in windows based app and still use XML files with each file named based on user profile or i could make use of temporary tables in database to handle each user specific data. But this could get ugly also since number of users could be large and storing data for each user profile in temp tables might not be a good idea. XML files could work but then there is overhead of navigating that XML file even with using XPath. Any ideas? I am kindda stuck and look for feedbacks from you guys if you have encountered this and how did you handle it? Thanks a lot!
Paul
|
|
Reply By:
|
planoie
|
Reply Date:
|
2/15/2004 11:50:54 PM
|
I imagine that you aren't going to need all 3 meg of the user's information at one point (i.e. one page) during the web application. So you should probably just build the web application to only get what it needs for each page. Even though you may eventually get all 3 meg worth of information, getting it piece by piece as you use it would be much more efficient.
Using temp tables won't accomplish much if the source data is already in the database because that will just move the data to a different place in the database and increase clutter by duplicating it.
If you get the data directly from SQL, you don't need to worry about dealing with it in XML form. Even if you need to persist the data do some user variable (like in the session) you can store the ADO.net objects. (Although, ADO.net saves all it's data structures as XML so you are kind of getting the same thing. But you might as well let ADO.net deal with the XML.)
Is the data stored in a manner that it's difficult to get at in small chunks? Perhaps you need to reconfigure the schema.
Peter
------------------------------------------------------
Work smarter, not harder.
|
|
Reply By:
|
paul_singh
|
Reply Date:
|
2/16/2004 8:04:47 AM
|
Hi Pete,
Thanks for your feedback. The problem is something like this. Datainfo stored in SQL gets pulled out based on user security
(what user can see) e.g. what regions they can see (geography, their keys etc) and its used for GUI display which user selects for filtering from page to page(It could take a while). If we dont do this upfront then each page(geography, product etc) user sees will have to be configured to add the security parameter on top to get database info. These records could go well upto 10,000 rows or more (e.g. left and right side list box selections will vary based on what user selects from a dropdown box. If user selects top item from dropdownlist box, left hand side list box will have 2 items but if he selects the last item then left side listbox could very well go upto 6000 items. This is done so user could select items from left side listbox and move them to right side to store his keys). If i dont get all data at one shot and get it in steps, it will cause delay but maybe its more efficient design in long run. Currently when the program runs next time it compares last pulled down data's timestamp and replaces with new ones on client machine.
You are right about duplicating data in SQL. Unfortunately, I dont have any control over schema of DB and company won't agree to change it since this would break other clients DB they deal with.
Paul.
|
