Re: capturing a webpage

Hi!

> Question 1: How can I capture the contents being displayed (the raw html
> data is fine - I can parse later) into memory?

The best way to get the contents of a web page is not through a browser,
but to actually just make the request yourself. You can do this several
ways...

 * If you're not too proud to shell out (;-) then read a temporary file
   back into RAM, you can use the "wget" program. You can find this at
   http://www.gnu.org/software/wget/wget.html. It's a like a swiss army
   knife for dealing with HTTP. Worth getting even if you don't use it
   with your program. Note that even though it's GNU, if you shell out to
   call it, you don't have to worry about any "license dilution" issues.

 * If you want to do it the fun way, just make a socket connection to
   your target's port 80 and send the following data "GET /\r\n\r\n". Then
   just read bytes until there aren't any more to read. Now you have
   captured the web page! Replace the "/" with the directory path to
   whatever page you want. Note that this won't work on password
   protected pages, you'll have to get more involved.

You can test the second method by doing a "telnet www.google.com 80" from
your DOS prompt. Then just type "GET /" and hit [Enter] twice. You should
be inundated with Google-ized webbery!

I would include Forth code, but I'm at work and my shop is among the
unenlightened -- we just have crude ironmongery like Python and C++... ;-)

I would highly recommend you grab the HTTP RFC's and give them a quick
read. The protocol is dirt simple, and since a lot of modern stuff
speaks it, you can do a lot of nifty things.
http://www.w3.org/Protocols/rfc1945/rfc1945 is probably the canonical
starting point if you're interested.

I hope this helps!

Jos'h
----------------------------------------------------------------------
sftalk_at_forth.com The SwiftForth programming discussion email list
To unsubscribe, send subject "unsubscribe" to sftalk-request_at_forth.com
For list command help, send subject "help" to sftalk-request_at_forth.com
Message archives are located at http://www.forth.com/archive/sftalk
----------------------------------------------------------------------
This list is a forum for SwiftForth users. For product support and bug
reports, please send email to support_at_forth.com
----------------------------------------------------------------------
Received on Mon Oct 24 2005 - 07:08:55 PDT