Wrox Programmer Forums
| Search | Today's Posts | Mark Forums Read
Javascript How-To Ask your "How do I do this with Javascript?" questions here.
Welcome to the p2p.wrox.com Forums.

You are currently viewing the Javascript How-To section of the Wrox Programmer to Programmer discussions. This is a community of software programmers and website developers including Wrox book authors and readers. New member registration was closed in 2019. New posts were shut off and the site was archived into this static format as of October 1, 2020. If you require technical support for a Wrox book please contact http://hub.wiley.com
 
Old April 23rd, 2006, 12:06 PM
Authorized User
 
Join Date: Jul 2003
Location: , , .
Posts: 98
Thanks: 0
Thanked 0 Times in 0 Posts
Default Screen Scraping Site

Hello Everyone and thanks for your help in advance. I am workign on an application that screen scrapes a heavily javascripted site. I am using VB.Net to do the screen scrape, however, the output that is returned tells me to "enable scripting in my browser" (obviously, there is no browser involved). So my primary question is, can javascript be run in anything other than a web page (for example in a Windows form) to manipulate the output? Or does anyone have a better suggestion? Any help on this topic would be greatly appreciated. Thanks.

 
Old April 24th, 2006, 04:57 AM
Friend of Wrox
Points: 8,019, Level: 38
Points: 8,019, Level: 38 Points: 8,019, Level: 38 Points: 8,019, Level: 38
Activity: 0%
Activity: 0% Activity: 0% Activity: 0%
 
Join Date: Jan 2005
Location: Mauchline, East Ayrshire, Scotland
Posts: 1,525
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to crmpicco Send a message via AIM to crmpicco Send a message via MSN to crmpicco Send a message via Yahoo to crmpicco
Default

Here is a full working example of ASP.NET screen scraping, written in ASP.NET (VB.NET):
Code:
<%@ Import Namespace="System" %>
<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>

<script language="VB" runat="server">

Sub Page_Load(Sender as Object, E as EventArgs)

Dim oRequest As WebRequest = WebRequest.Create("http://www.aspdev.org/asp.net/")
Dim oResponse As WebResponse = oRequest.GetResponse()

Dim oStream As Stream = oResponse.GetResponseStream()

Dim oStreamReader As New StreamReader(oStream, Encoding.UTF8)

Response.Write(oStreamReader.ReadToEnd())
oResponse.Close()
oStreamReader.Close()


End Sub


</script>

www.crmpicco.co.uk
www.ie7.com
 
Old April 24th, 2006, 11:14 AM
Authorized User
 
Join Date: Jul 2003
Location: , , .
Posts: 98
Thanks: 0
Thanked 0 Times in 0 Posts
Default

Thanks for the resposne. Perhaps I am missing something, but I am aware of the WebRequest and WebResponse objects in Vb.Net. However, the problem I am running into is that there are scripts that appear to have to run to render the page correctly. I'll play around with the sample code (the only thing I can see different is the encoding) but I'm not sure this will work. Am I missing the point?

Once again, thanks for the help.

 
Old April 25th, 2006, 09:00 AM
Friend of Wrox
Points: 8,019, Level: 38
Points: 8,019, Level: 38 Points: 8,019, Level: 38 Points: 8,019, Level: 38
Activity: 0%
Activity: 0% Activity: 0% Activity: 0%
 
Join Date: Jan 2005
Location: Mauchline, East Ayrshire, Scotland
Posts: 1,525
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to crmpicco Send a message via AIM to crmpicco Send a message via MSN to crmpicco Send a message via Yahoo to crmpicco
Default

AFAIK WebRequest and WebResponse will work in VB.NET, just as it does above notice the Script Language is Visual Basic. Have you got any further forward?

www.crmpicco.co.uk
www.ie7.com
 
Old April 25th, 2006, 09:01 AM
Friend of Wrox
Points: 8,019, Level: 38
Points: 8,019, Level: 38 Points: 8,019, Level: 38 Points: 8,019, Level: 38
Activity: 0%
Activity: 0% Activity: 0% Activity: 0%
 
Join Date: Jan 2005
Location: Mauchline, East Ayrshire, Scotland
Posts: 1,525
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to crmpicco Send a message via AIM to crmpicco Send a message via MSN to crmpicco Send a message via Yahoo to crmpicco
Default

also, Hugh, you may get more response on the VB/VB.NET board.

www.crmpicco.co.uk
www.ie7.com




Similar Threads
Thread Thread Starter Forum Replies Last Post
how can clear the screen the screen malli_kv2 BOOK: Beginning Java 2, JDK 5 Edition 4 January 17th, 2009 12:43 AM
Publish Web Site & Copy Web Site angshujit ASP.NET 2.0 Basics 1 June 15th, 2007 09:21 AM
Draw on the screen without clearing the screen pu132 Visual Basic 2005 Basics 0 August 25th, 2006 09:03 PM
how do i change the site icon on a geocitiies site uprocker2 HTML Code Clinic 2 March 4th, 2005 07:20 AM





Powered by vBulletin®
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright (c) 2020 John Wiley & Sons, Inc.