Wrox Programmer Forums
| Search | Today's Posts | Mark Forums Read
General .NET For general discussion of MICROSOFT .NET topics that don't fall within any of the other .NET forum subcategories or .NET language forums.  If your question is specific to a language (C# or Visual Basic) or type of application (Windows Forms or ASP.Net) try an applicable forum category. ** PLEASE BE SPECIFIC WITH YOUR QUESTION ** When posting here, provide details regarding the Microsoft .NET language you are using and/or what type of application (Windows/Web Forms, etc) you are working in, if applicable to the question. This will help others answer the question without having to ask.
Welcome to the p2p.wrox.com Forums.

You are currently viewing the General .NET section of the Wrox Programmer to Programmer discussions. This is a community of software programmers and website developers including Wrox book authors and readers. New member registration was closed in 2019. New posts were shut off and the site was archived into this static format as of October 1, 2020. If you require technical support for a Wrox book please contact http://hub.wiley.com
 
Old July 21st, 2004, 08:31 AM
Authorized User
 
Join Date: Sep 2003
Location: , , .
Posts: 93
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via AIM to flyin
Default Design Question

Hello all. I have a design question for all you developers.

Lets take a simple scenario: I have 1 master table that tells me what Topics an author has posted under and then a bunch of tables representing each Topic.

Needless to say these tables aren't related and i have to query the master table for the TableName Field to see what tables an author has posted under. Well now to the question.

This query can get very intense with the multiple unions i am doing and i do not want to hit the database everytime for this query so i figure hey, i'll cache the resulting datatable and cache it for 1 minute to save the DB. That works fine, but once that expires in the minute the DB gets hit up again and i can just see a ton of overhead there. My next solution was to run a job initially that would do this intense query for the most popular authors and in the process of the query, get the resulting set, store that datatable in a dataset and write that dataset out to an xml file using dataset.WriteXML, so that every subsequent request will simply read from the XML file if exists, else write a fresh one.

My main question is that once the xml file starts to get large like lets say about 10,000 records, does filling and querying that dataset resulting from the xml file start to get slow? Does the indexing in the Dataset start to cause troubles? I can do this test myself, but i was just wondering if you guys have encountered this and if you have a better solution perhaps?

 
Old July 21st, 2004, 12:54 PM
planoie's Avatar
Friend of Wrox
Points: 16,481, Level: 55
Points: 16,481, Level: 55 Points: 16,481, Level: 55 Points: 16,481, Level: 55
Activity: 0%
Activity: 0% Activity: 0% Activity: 0%
 
Join Date: Aug 2003
Location: Clifton Park, New York, USA.
Posts: 5,407
Thanks: 0
Thanked 16 Times in 16 Posts
Default

Why can't you change the cache timeout to something longer?

And why do you have tables that sound to be very much alike instead of having a single table?
 
Old July 21st, 2004, 01:03 PM
Authorized User
 
Join Date: May 2004
Location: , , USA.
Posts: 24
Thanks: 0
Thanked 0 Times in 0 Posts
Default

Cache the entire 10000 rows. Use ADO (.net ofcourse) to filter and
display 500 rows at a time. Remove the cache when any table related
to this query changes.

SS
 
Old July 21st, 2004, 03:53 PM
Authorized User
 
Join Date: Sep 2003
Location: , , .
Posts: 93
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via AIM to flyin
Default

Peter, The table structure has been in use for quite some time now and there is alot of uniqueness to each table, that's why i had to make a compatable table that stores what tables the author has posted within.

to answer you and sundar_revathi question, yes i can make the cache time longer as far as expiring cache when when any of the tables change, can i do this through a query? or when a person uploads a new article will i have to know the cachekey in order to expire it there?




 
Old July 21st, 2004, 03:59 PM
planoie's Avatar
Friend of Wrox
Points: 16,481, Level: 55
Points: 16,481, Level: 55 Points: 16,481, Level: 55 Points: 16,481, Level: 55
Activity: 0%
Activity: 0% Activity: 0% Activity: 0%
 
Join Date: Aug 2003
Location: Clifton Park, New York, USA.
Posts: 5,407
Thanks: 0
Thanked 16 Times in 16 Posts
Default

Thats' usually how you would do it. When any code runs that affects the contents of the table, you re-cache it or expire the cached item so it automatically reloads and caches the next time. This way you never have to expire the cache until you actually change/add data.




Similar Threads
Thread Thread Starter Forum Replies Last Post
design question androoo ADO.NET 1 November 13th, 2004 09:41 AM
design question androoo Classic ASP Databases 1 November 13th, 2004 09:41 AM
design question androoo General .NET 1 November 13th, 2004 09:40 AM
design question androoo .NET Web Services 1 November 13th, 2004 09:39 AM





Powered by vBulletin®
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright (c) 2020 John Wiley & Sons, Inc.