Persistant web process

Newbie_71 · October 30th, 2011, 05:02 PM

Is there a way to make a web based process that loads one tiny record from a database into memory and continues running indefinitely? Then, when another process calls it, it doesn't have to re-query the database or retrieve the data from anywhere other than RAM? Forget about usefulness, how does one accomplish this?

I want to make one persistant instance of a web process that is called by various instances of other processes.

Imar · October 30th, 2011, 05:08 PM

Hi there,

Is this about a persistent web process or about getting access to (semi) persistent data you don't want to fetch from the database on each request? In the latter case you can use the .NET Cache which was designed for exactly this purpose.

Cheers,

Imar

Newbie_71 · October 30th, 2011, 11:57 PM

I'm thinking about something along the lines of a service that runs on a web server and has the entire web server all to itself. In particular, all the RAM being the most important. It would load a handful of tables with a large number of records in them. It would load every record in the three tables, except would only load a couple fields in each table, which amounts to less than 10% of the fields. The objects that are instantiated would be stripped down to the point of being classes that conist of a small number of public variables and absolutely nothing else to keep memory consumption to a minimum.
All three tables have a primary key that is an integer and have always had a rule in place on the database server which does not allow deletions. You can search every object based on its primary key - 1 being its index in the array. This should make retrieval quick.

The process would be called by another process running on another computer on a network. It would be passed three arrays of objects with the same signature as the three types of objects being retrieved. The only part already filled in is the primary key fields. It takes each object in the array and sets it's other variables equal to those of the object whose index equals the primary key -1. The array is then returned to the calling process.

The process also has a low priority thread that sleeps for 6 hours, simultaneously re-fetches the lists, swaps the pointers to the first element in the contiguos array afterwards, and releases the memory for the old list and then goes to sleep for another 6 hours. Swapping would be handled by setting a flag for queing incoming calls, waiting for existing calls to finish, swapping, re-allowing call processing, then freeing memory, and then going back to sleep.

I've implemented this all inside the same process and it resulted in EXTREMELY fast data retrieval, except consumed a lot of RAM. However, that was simply a program accessing a cache object which was functioning as this process inside the same program. I don't know if the overhead of calling a seperate process is going to make it run slower than standard DB access yet or not. You would call it once with a couple thousand queries in your list, it would query all three tables once for each entry, and return the results as a big array.

Imar · October 31st, 2011, 10:17 AM

I guess it all depends on how large the data sets are that you are returning and how frequently your code is called.

Did you do a performance analysis to see if SQL Server is indeed the bottleneck? You may be optimizing something that doesn't need to be optimized.

That said, you could still use the ASP.NET cahce (depending on the size of the dataset) or go the hard way and build something like a Windows Service with an embedded WCF service: http://msdn.microsoft.com/en-us/library/ms733069.aspx That's a little beyond the scope of what I can help you with over a forum post though.

Cheers.

Imar