Screen Scraping Site

[email protected] · April 23rd, 2006, 12:06 PM

Hello Everyone and thanks for your help in advance. I am workign on an application that screen scrapes a heavily javascripted site. I am using VB.Net to do the screen scrape, however, the output that is returned tells me to "enable scripting in my browser" (obviously, there is no browser involved). So my primary question is, can javascript be run in anything other than a web page (for example in a Windows form) to manipulate the output? Or does anyone have a better suggestion? Any help on this topic would be greatly appreciated. Thanks.

crmpicco · April 24th, 2006, 04:57 AM

Here is a full working example of ASP.NET screen scraping, written in ASP.NET (VB.NET):

Code:

<%@ Import Namespace="System" %>
<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>

<script language="VB" runat="server">

Sub Page_Load(Sender as Object, E as EventArgs)

Dim oRequest As WebRequest = WebRequest.Create("http://www.aspdev.org/asp.net/")
Dim oResponse As WebResponse = oRequest.GetResponse()

Dim oStream As Stream = oResponse.GetResponseStream()

Dim oStreamReader As New StreamReader(oStream, Encoding.UTF8)

Response.Write(oStreamReader.ReadToEnd())
oResponse.Close()
oStreamReader.Close()


End Sub


</script>

www.crmpicco.co.uk
www.ie7.com

[email protected] · April 24th, 2006, 11:14 AM

Thanks for the resposne. Perhaps I am missing something, but I am aware of the WebRequest and WebResponse objects in Vb.Net. However, the problem I am running into is that there are scripts that appear to have to run to render the page correctly. I'll play around with the sample code (the only thing I can see different is the encoding) but I'm not sure this will work. Am I missing the point?

Once again, thanks for the help.

crmpicco · April 25th, 2006, 09:00 AM

AFAIK WebRequest and WebResponse will work in VB.NET, just as it does above notice the Script Language is Visual Basic. Have you got any further forward?

www.crmpicco.co.uk
www.ie7.com

crmpicco · April 25th, 2006, 09:01 AM

also, Hugh, you may get more response on the VB/VB.NET board.

www.crmpicco.co.uk
www.ie7.com