Alright, I've been reading tons of books our company's upcoming project and I am pushing for .NET. Basically i've been granted this request, yay for me! anyways here is my questions.

What is the best way to store/access data.

Scenario:
We have An articles mainpage. It displays 12 articles on the mainpage. This mainpage will receive over 10,000 hits a day, and will have over 1000 unique visitors on at one time.


I've seen about 3 distinct ways of doing this.

1. Basic, always read from the database via datareader and bind to controls.

2. If dataSet doesn't exist, retreive data one time, store in cache and grab from cache every other time.

3. Store information in an XML file, and set the varybyparam to the xml file, so every time the xml file is updated, we update the cached dataSet.

4. Simply store the data in an xml file, and just read the xml data to a dataset and then bind to the controls.

All seem like they have their advantages.

#1 seems like it still is the fastest, but with our site, we will have over thousands of users on at one time so i want to be able to free up system resources for other things.

#2, Seems alright, but for some reasons performance wise doesn't perform as well as the other 3.

#3 seems pretty cool, because it uses all of the .net technology features that are just pretty sweet and does perform better than #2 because in #2 everytime a datasource is updated, then it has to re-update cache and re-query the database instead of just re-reading from an xml file.

#4, is sweet, but i seem to have locking isssues with the xml file. If one thread is updating, and 1 is trying to read, it seems to all crap out. so that's not an option. #3, locks the cache and updates it approprietly.

Is there any other options out there to optimize the performance, scalability, and throughput? If not, which of these do you suggest? all opinions are welcome, if you can please give a good description on why you chose what you did. Thanks alot! I hope this thread will help all people out.

Hi there,

I am not sure I understand the situation completely. You expect about 10,000 hits, but with a 1,000 concurrent users. So, they all hit the system at the same time, generating 1,000 hits. And this repeats itself 10 times a day? Seems strange....
Or are you talking about a potential of 1,000 users?

Anyway, I'd go for option 2. Storing your data in a database has loads of advantages, including querying, updating, and accessibility by other applications.
Future versions of ASP.NET will allow cache invalidating from SQL Server too. Just point to a table, and your dataset will be invalidated whenever the table changes.
Method two is fast, while you still have a scalable solution. Just invalidate the cache every x minutes, where x is the number of minutes you allow for "stale data" (differences between the cached version and the live database).

If you want, you can even mix 2 and 3 a little. Cache the dataset from the database, and setup a FileDependency. Whenever the backend changes the database, change / touch the XML file as well to invalidate the cache. Not pretty, but it may work.....

Depending on your scenario, option 1 is good too. 10,000 hits, querying 12 articles is not too much work for a database server. It's a query every 5 seconds if you consider a 12 hour use of the application.

But personally, I'd go for option 2. It will be fast, easy to create and deploy, will work in a Web farm scenario, and basically "feels like the right way to this".


I can see you're a good programmer: You even think zero-based: ;)

Cheers,

Imar




---------------------------------------
Imar Spaanjaars
Everyone is unique, except for me.

thank you very much for the compliments Imar. yes sorry for the confusion about the users and what not. Basically we have about 2500 Unique visitors on per hour or so. Yes i figured that option 2 is the best way to go and most commonly used. Next is question is page level caching, i'll create a new post for that one, hope you get a chance to view it!

I second Imar's suggestions. That is how I have done it. Cache the data. If you can tie a cache dependency to something like a file, then even better, otherwise a timed expiration is usually suitable. (I have heard that ASP.NET resets itself within IIS on a somewhat regular basis, so you might find that ASP.NET will reset, thus wiping the cache clean, before your cache timeouts pass.)

Peter
------------------------------------------------------
Work smarter, not harder.

Ah yes, Mr. Peter, a person who has helped me alot. Yes I agree with both you and Imar. Not sure yet if i want to do the absolute expiration or the dendency via an xml file. I am thinking XML file because then i won't have to re-create the RSS feed for the main page and kill two stones with one bird.... or the other way around.:D

Ah, that is a very good idea. Make the server cache dependent on the XML file that you use for the RSS feed. You get a file based RSS feed that is built when you update the data that it contains (so you don't have to query the DB every time), but you also get a file dependency for the web cache. That's very, very clever!

Yes Sir. Peter that works well. Especially for those sites who have rss feeds, you can basically automate that system to drive off of 1 rss feed. I am thinking smarter not harder! The problem i run into randomly is when a process is trying to read the rss feed while another process is writing it. Ever ran into this? I get the run-time error or "This file is being used by another process" and it's just a pain to get it up and running again. any thoughts?

Any locking methods to the extent of this psuedo code:

Lock(MyFile.rss)
' Write stuff to MyFile.rss
Unlock(MyFile.rss)

I just wanted to drop a side note of thanks alot for your opinions Peter and Imar. You two seem to have a really good handle on .Net, this makes my adventures much more easier!