Does not really matter how crazy the data is since no matter what method you
use you must at sometime parse the data to make sense of it. It is ALWAYS
better to do this at the earliest poaaible point. Therefore the only
solution that makes sense to me, of course I could be missing something,
would be to read the data from the flat files, parse it, and save it to a
relational database (MSSQLServer, Oracle, Sybase, etc... ). Then as soon as
you have parsed it and saved it, purge it from the flat files to keep them
small. Once this process is complete SQL can do the reast and you can query,
organize and report on the data in every way imaginable, and the size of
your data no longer matters.






-----Original Message-----
From: helga@k... [mailto:helga@k...]
Sent: Monday, October 21, 2002 10:48
To: professional vb
Subject: [pro_vb] Memory Streaming & writing


Hi all

I think I've got a tricky one - but who knows?

I have a file wich to any normal human been is really insane (timm file).
It contains data of customers (statements, invoices, handsets, calls,
etc). and this data is scattered thus meaning it will have data of client
xyz and then some other client 'abc' the maybe 5 or 6 more and then again
client xyz.  A real mess you can say.

Now, currently I'm taking one client on a time, read his data, updates the
database, closes the file, then does the next client.

In the beginning this was 'OK' - now its a problem.  The files is
something like 100+MB and I've got already about 500 clients.  The time it
takes to filter my data is thus enourmous.

What I want to do is this:  Stream the file and whilst streaming, generate
files for each customer found, write his/her data to the file and when
finished, close all the files and then start doing my filtering.

Does this make sense?

Can someone assist please?  Maybe a starting point, etc.

Regards
Helga


---
Visual C# - A Guide for VB6 Developers
This book will make it easy to transfer your skills
from Visual Basic 6 to C#, the language of choice
of the .NET Framework.
http://www.wrox.com/ACON11.asp?ISBN=1861007175&p2p0059