Wrox Home  
Search P2P Archive for: Go

  Return to Index  

aspx_professional thread: Screen Scraping


Message #1 by bob@s... on Wed, 5 Mar 2003 22:37:06
Why do you only want to do 256 lines at a time?
Try getting all of it at once and then breaking it once you have it all:

http://www.learnasp.com/freebook/learn/utilitybelt_lib.aspx

webRequest1 = WebRequest1.Create(strURL);
webResponse1 = webRequest1.GetResponse();
sr1=New StreamReader(webResponse1.GetResponseStream());
strResult = sr1.ReadToEnd();

Console.Writeline(strResult);

Rahul Singh
Anant Systems, Inc.


-----Original Message-----
From: bob@s... [mailto:bob@s...] 
Sent: Wednesday, March 05, 2003 10:37 PM
To: ASPX_Professional
Subject: [aspx_professional] Screen Scraping

I am trying to do some screen scraping using HTTPWebResponse.  However,
I 
don't seem to be getting all of the data back.  It stops part way
through 
the page.  Is there a limit to the amount of data I can get?  The code I

am using is below:

HttpWebRequest myHttpWebRequest 
(HttpWebRequest)WebRequest.Create(sURL);
// Sends the HttpWebRequest and waits for the response.
HttpWebResponse myHttpWebResponse = (HttpWebResponse)
myHttpWebRequest.GetResponse();
// Gets the stream associated with the response.
Stream receiveStream = myHttpWebResponse.GetResponseStream();
Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
// Pipes the stream to a higher level stream reader with the required 
encoding format.
StreamReader readStream = new StreamReader( receiveStream, encode );
Console.WriteLine("\r\nResponse stream received.");
Char[] read = new Char[256];  
int count = readStream.Read( read, 0, 256 );
Console.WriteLine("HTML...\r\n");
while (count > 0)
{
String str = new String(read, 0, count);
Console.Write(str);
count = readStream.Read(read, 0, 256);
}
Console.WriteLine("");
// Releases the resources of the response.
myHttpWebResponse.Close();
// Releases the resources of the Stream.
readStream.Close();



  Return to Index