I'm using .NET, and by and large all of the light configuration files are XML. It runs much faster than hitting a database. There are even cases when I've seen guys take advantage of XML to do some display or light processing using XSLT. However, anytime someone has a really large chunk of data, it doesn't go into an XML file, it goes into a database.

Why does XML slow down and lose it's edge compared to a database as size increases? And what size gives you roughly the same performance? ie if you've less data than that, better to use XML; but if you've got more than store it in a database?

-------------------------

Whatever you can do or dream you can, begin it. Boldness has genius, power and magic in it. Begin it now.
-Johann von Goethe

When Two Hearts Race... Both Win.
-Dove Chocolate Wrapper

Chroniclemaster1, Founder of www.EarthChronicle.com
A Growing History of our Planet, by our Planet, for our Planet.

__________________
-------------------------

Whatever you can do or dream you can, begin it. Boldness has genius, power and magic in it. Begin it now.
-Johann von Goethe

When Two Hearts Race... Both Win.
-Dove Chocolate Wrapper

Chroniclemaster1, Founder of www.EarthChronicle.com
A Growing History of our Planet, by our Planet, for our Planet.

Why does XML slow down and lose it's edge compared to a database as size increases?

XML is stored as a text file, and to extract any information you need to read the whole file and parse it. Most applications will also allocate enough memory to hold a representation of the XML file in memory (and this will often be five to ten times the size of the raw data on disk). Incidentally, this is no different from the way a spreadsheet works. If you've got a hundred thousand employees and you just want to know one person's salary, then this obviously imposes a lot of overhead. A database, by contrast, is designed so that you can answer many queries by only reading those blocks on the disk that contain the information you are actually looking for.

>And what size gives you roughly the same performance?

That's very dependent on the actual use you are making of the data. If your application needs all the data, and if it fits in memory, then there's no reason why the XML solution should be slower.

Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference

*the lightbulb turns on* So a login system for instance would clearly make a more scalable solution as a database because you are typically logging in one person at a time. But a company directory would make sense as XML because you're showing all the data.

What about this case? Does it make sense to use XML if you have an application where each user request is hitting only a fraction of the data, but "as an application" all the data is being used during the day. ie is the important point whether each user request is using all the data or whether the "application" is using all the data?

ie if you were displaying products for a website in several different product lines. Each visitor would probably be looking only at one product line, but throughout the day different visitors are looking at all the various product lines and using all the data.

In .NET, would this depend on whether you had caching turned on?

If you were serving the xml file at each user request, I would think that pretty much the whole thing would have to be used by each user for it to be preferable to use from a performance standpoint. But if you were using XMLReader in conjunction with caching, it seems like the aggragate of all user requests might be more relevant. Would that be a situation where XML would make more sense than using a database?

-------------------------

Whatever you can do or dream you can, begin it. Boldness has genius, power and magic in it. Begin it now.
-Johann von Goethe

When Two Hearts Race... Both Win.
-Dove Chocolate Wrapper

Chroniclemaster1, Founder of www.EarthChronicle.com
A Growing History of our Planet, by our Planet, for our Planet.

>So a login system for instance would clearly make a more scalable solution as a database because you are typically logging in one person at a time.

If you've got a few thousand users (rather than a few million) then it could make sense to load the basic authentication data from XML into shared memory at start of day. But the other problem you have to consider is how to handle updates; one of the advantages of a database is that it provides fine-grained updates, with concurrency control and recovery.

Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference

Cool. So from a performance standpoint, a mid-sized XML login system would make sense. You can load it, cache it, and use it all day. But the first time two people forget their password at the same and need to reset them... bad news.

I just got Professional ASP.NET 2.0 Databases, and I'm still working on the earlier chapters. However, I was skimming ahead and one of the chapters is a case study in using the new XML data types in SQL Server 2005. I understand the basic principle of serializing and deserializing XML back and forth between DB and XML file. Do you see this as being the future of XML, something that's useful in certain cases, or a big snore? I'm having a hard time understanding how "cross breeding" XML and databases impacts application development.

And thank you for the insights!

-------------------------

Whatever you can do or dream you can, begin it. Boldness has genius, power and magic in it. Begin it now.
-Johann von Goethe

When Two Hearts Race... Both Win.
-Dove Chocolate Wrapper

Chroniclemaster1, Founder of www.EarthChronicle.com
A Growing History of our Planet, by our Planet, for our Planet.

XML files can be very big. There is no performance issue with XML file size along.

If you use DOM to process XML, you have to load the whole XML file into memory, the size of file IS an issue. If you use SAX(XMLReader in C#) to process XML, you go through the XML, so the size of XML doesn't matter with performance.

Solution for this bad news is another application to update this memory.
For real-time change of password or new userID, you need another web application to update that information in your shared server memory. It's not hard at all. :-)

Hope helps,