Hi,
I have a program which regularly scans a large number of text files, the files are stored in HTML. My script loads the file- line by line, each line is stripped of HTML and then parsed for the sought phrase or word. the problem is that there are over 6000 such files with size (100kb-150kb). i have timed my operations and right now, it takes 1 second to fully scan a file, therefore to scan the 6000 files will take about 2hrs which will be most unacceptable to my users. is there any way to speed up the search process. i am looking at XML but i don't quite understand the concept and i do not know if it is relevant.

please help

__________________
Everything takes longer than you expect!

I think you can use Index server for this purpose. The following link gives an article which will give you the introductory knowledge on index server.

http://askasp.com/articles.asp?ArtID=6

Can you post your current code - we may be able to optimise it.

Cheers,

Chris

Hi guys,
I lost my internet connection for a while....i hope it is not too late to still get some help. i'll really appreciate it

Thanks - please see the code below


        ' assume sought string is in a variable called session("strsoughtdetail")

        intnumfinds=0

        do

            strcasetextfile= 'get a text file name


            SET filescan = Server.CreateObject("Scripting.FileSystemObject")
            SET scanfile = filescan.OpenTextFile(server.mappath(strcasetextfi le),1)

            while (not(scanfile.AtEndOfStream))

                strscannedline=scanfile.readline

                if instr(1,strscannedline,session("strsoughtdetail"), 1)>0 then

                        intnumfinds=intnumfinds+1

                end if
            wend

            scanfile.close
            SET scanfile=nothing
            SET filescan=nothing

        loop 'until 6000 text files of approx. 100kb are proccessed

I got a marginal increase in performance (about 10%) by Dim'ing all variables before use & creating the filesystem object outside the do loop so that you only create it once.

As madhukp suggests, you may want to use some kind of indexing server to get decent increases in performance.

You could also just load the data into a db and query against that.

Cheers,

Chris

I had an experience that the performance of filesystemobject is very bad on files larger than 60 KB. (that which cannot be opened with notepad). Later I used Index Server to solve it.