Well you could write a PHP Bot. Can it be done?? Absolutely.
What you need is a regular expression <html> parser to extract URL's from a webpage, and then a mechanism to put those URL's in queue. Finding a balance between excess resource consumption and a healthy robot is quite a challenge. Doing so with cronjobs would be the likely solution, schedule the robot to run on intervals, then sleep, then run...
The question is do you how many TERABYTES of free space do you have for all the information that such a bot would gather? If you're doing a Google-like search, then your free space will be very quickly consumed.
My suggestion is look at dmoz.org, if you're determined to have your own search, they make their directory freely available to anyone who wants it. Aside from Google's bot, Dmoz is one of Google's primary sources of information.
If you do go to all the trouble of writing a bot, be very careful with how quickly it parses information from one website, because, one man's bot can quickly become another's denial of service attack!
And then there's always Google, run a search for PHP Robot, chances are someone's already written it. Writing a robot is a complex application, I don't think anyone here has the time or energy to go into the details of what that would require.
: )
Rich
:::::::::::::::::::::::::::::::::
Smiling Souls
http://www.smilingsouls.net
:::::::::::::::::::::::::::::::::
Search Engine in PHP
