i'm trying to screen scrape a webpage developed using .Net. user enters a zipcode in the first page and submits which goes to second page which displays if the zipcode is valid/invalid. So the second page should be screen scraped. so all the zipcodes to be entered in first page will be picked from database en the screenscrape results should be stored back into database(i.e valid/invalid). My problem is how to get the actual URL of first page because zipcode should be appended as a parameter to the URL. i tried doing this but the page is not getting submitted along with zipcode....so it's remaining in the first page only.Any help would be appreciated. thanks

What part of the scrape are you having problems with?? Just a problem appending the user entered zipcode to the URL?

In its simplest form you could do:

WebClient objWebClient = new WebClient();
string strURL = "http://www.secondpage.com?url=" + txtZipCode.Text ;
UTF8Encoding objUTF8 = new UTF8Encoding();
lblOutput.Text = objUTF8.GetString(objWebClient.DownloadData(strURL ))


-------------------------
I will only tell you how to do it, not do it for you.
Unless, of course, you want to hire me to do work for you.

^^Thats my signature

thanks for your quick response, Actually i'm using VBScript for this screen scrape process. The script works fine for any other website like yahoo,cingular etc but the site which i'm trying for is http://www.t-mobile.com/locator.aspx...2fDefault.aspx
Here you enter ZIP Code and submit page

  If valid, redirects to http://www.t-mobile.com/Default.aspx

  If invalid, redirects to http://www.t-mobile.com/locator.aspx? referer=http%3a%2f%2fwww.t-mobile.com%2fDefault.aspx

so when i append zipcode to first URL it should be submitted to
second URL and here i do the screen scraping. but when i try appending zipcode=21071 to first URL it's not submitting to second page instead displays the first page only. Please help me solve this problem.

What are you trying to do, the store locator or the coverage area? For the coverage area you would use the below URL to pull up the map:

http://www.t-mobile.com/coverage/default.aspx?zip=44301

Where 44301 is the zipcode from your application.

If I go to the page: http://www.t-mobile.com/locator.aspx...2fDefault.aspx there is a text box there to provide my Zipcode but there isnt a zip code passed via the querystring when you enter or are redirected to the index. It looks like the page is setting a cookie or setting a variable in session to hold that zip code when the user posts the form.

To that end, what you need to do is write a bot that goes to that page, inputs the zip code, and posts the form back to T-Mobiles server THEN scrape the screen. What you are doing isn't causing the form to post or the zip code to be passed in, all it is doing is sending a zip code ot the page which the application is ignoring.

-------------------------
I will only tell you how to do it, not do it for you.
Unless, of course, you want to hire me to do work for you.

^^Thats my signature