screen scraper issue

chiu08820 · March 20th, 2006, 09:32 AM

Hi All,

I had a problem to scrape information from a public web page, which has two options for user selection. Following are the source codes for the two option parts of this web page:

(1)part1
<TABLE class="formBody1u">
    <TR>
        <TD ALIGN=RIGHT class="LightNormalText"><span class="componentNameText">Market Date:</span></TD>

        <TD>
            <SELECT NAME="marketdate" onChange="submit()">

                        <OPTION VALUE="2006-03-26">26-MAR-2006

                        <OPTION VALUE="2006-03-25">25-MAR-2006

                        <OPTION VALUE="2006-03-24">24-MAR-2006

                        <OPTION VALUE="2006-03-23">23-MAR-2006

                        <OPTION VALUE="2006-03-22">22-MAR-2006

                        <OPTION VALUE="2006-03-21">21-MAR-2006

                        <OPTION VALUE="2006-03-20" SELECTED>20-MAR-2006

                        <OPTION VALUE="2006-03-19">19-MAR-2006

                        <OPTION VALUE="2006-03-18">18-MAR-2006

                        <OPTION VALUE="2006-03-17">17-MAR-2006

                        <OPTION VALUE="2006-03-16">16-MAR-2006
            </SELECT>
        </TD>
    </TR>
...
======================
(2)part2
<TABLE class="formBody3">
    <TR>
        <TD ALIGN=LEFT COLSPAN=4><span class="componentNameText">Display Size:</span>
          <SELECT NAME="scrollsize" onChange="submit()">

            <OPTION VALUE="2">2 rows
            <OPTION VALUE="3">3 rows
            <OPTION VALUE="4">4 rows
            <OPTION VALUE="5">5 rows
            <OPTION VALUE="6">6 rows
            <OPTION VALUE="7">7 rows
            <OPTION VALUE="8">8 rows
            <OPTION VALUE="9">9 rows
            <OPTION VALUE="10">10 rows
            <OPTION VALUE="11">11 rows
            <OPTION VALUE="12">12 rows
            <OPTION VALUE="13">13 rows
            <OPTION VALUE="14">14 rows
            <OPTION VALUE="15">15 rows
            <OPTION VALUE="16">16 rows
            <OPTION VALUE="17">17 rows
            <OPTION VALUE="18">18 rows
            <OPTION VALUE="19">19 rows
            <OPTION VALUE="20">20 rows
            <OPTION VALUE="21">21 rows
            <OPTION VALUE="22">22 rows
            <OPTION VALUE="23">23 rows
            <OPTION VALUE="24" SELECTED>24 rows
            <OPTION VALUE="25">25 rows
          </SELECT>
        </TD>
...
==============
My code is following:

String sUrl = "http://mospublic.xxx.com/xxx/jsp/load_forecast.jsp";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(sUrl);
UTF8Encoding encoding = new UTF8Encoding();
string postData="";
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
HttpWebResponse response;

postData = "marketdate=2006-03-21&scrollsize=19";
byte[] data = encoding.GetBytes(postData);
request.ContentLength = data.Length;
Stream newStream = request.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();

response = (HttpWebResponse)request.GetResponse();

===============

I can change various row numbers to get the right information. However I changed the market date to other date(2006-03-21), but always got the 2006-03-20 information. I think there is a issue for parsing the marketdate "2006-03-21", because there is a special character "-". Do I need to take care of it specially? Have anybody can solve this problem.