[vox-tech] Perl question - How do I use an HTTP header to download a file [SOLVED]?

Wed Mar 2 04:41:18 PST 2005

Jay Strauss wrote:
> Hi,
> 
> Using LWP::UserAgent and its pals, I get back the response (below via 
> Data::Dumper).  If I use the same $request object and feed it to 
> www::mechanize it does the right thing and the data file is in the 
> ->content.  I can't see www::mech does it, or how to do it properly with 
>  LWP::UserAgent
> 
> Basically the response is saying, go to this location with this cookie 
> and download the file.  How am I supposed to use this header?  Am I 
> suppose to feed this header into some other type of object and do a 
> request on it?
> 
> Any suggestions?
> 
> Thanks
> Jay
> 
> $VAR1 = bless( {
>                  '_protocol' => 'HTTP/1.1',
>                  '_content' => '<html><head><title>Object 
> moved</title></head><body>^M
> <h2>Object moved to <a 
> href=\'/DelayedQuote/QuoteData.dat\'>here</a>.</h2>^M
> </body></html>^M
> ',
>                  '_rc' => '302',
>                  '_headers' => bless( {
>                                         'client-response-num' => 1,
>                                         'cache-control' => 'private',
>                                         'set-cookie' => [
> 
> 'CBOESiteTrackingID=d58134de-4d88-4d13-9d5e-c831599506a6; expires=Tue, 
> 17-Feb-2105 12:17:19 GMT; path=/',
> 
> 'WEBTRENDS_ID=; expires=Wed, 16-Feb-2005 12:17:19 GMT; path=/',
> 
> 'DownLoadError=; expires=Wed, 16-Feb-2005 12:17:19 GMT; path=/',
> 
> 'QueryData=79D55F9BFFCA04C80D64EFC35FDECCEC487BB6D44EF9493E2214453402DC21A172A00E496D6348D6C2B9CD6E60C0BC172ACE78326718524614957081569C6A566053D27798472A2A5F90F1CBC849A20ED2FDB94956EFC33E; 
> path=/'
>                                                         ],
>                                         'location' => 
> '/DelayedQuote/QuoteData.dat',
>                                         'date' => 'Thu, 17 Feb 2005 
> 12:17:19 GMT',
>                                         'client-peer' => 
> '198.160.148.116:80',
>                                         'content-length' => '144',
>                                         'x-aspnet-version' => '1.1.4322',
>                                         'client-date' => 'Thu, 17 Feb 
> 2005 12:14:20 GMT',
>                                         'content-type' => 'text/html; 
> charset=iso-8859-1',
>                                         'title' => 'Object moved',
>                                         'server' => 'Microsoft-IIS/6.0'
>                                       }, 'HTTP::Headers' ),
>                  '_msg' => 'Found',
>                  '_request' => bless( {
>                                         '_content' => 
> '__EVENTTARGET=&__EVENTARGUMENT=&NOVIEWSTATE=dDwtODc5MTM4MTUwOzs%2B6wDeQv0gEC342sUPKG%2B3n6ZStJ8%3D&ucHeader%3AucCBOEHeaderLinks%3AucCBOEHeaderSearch%3Asearchtext=&txtTicker=QQQQ&cmdSubmit=Download', 
> 
>                                         '_uri' => bless( do{\(my $o = 
> 'http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx')}, 
> 'URI::http' ),
>                                         '_headers' => bless( {
> 
> 'user-agent' => 'Windows IE 6',
> 
> 'content-type' => 'application/x-www-form-urlencoded',
> 
> 'content-length' => 196
>                                                              }, 
> 'HTTP::Headers' ),
>                                         '_method' => 'POST'
>                                       }, 'HTTP::Request' )
>                }, 'HTTP::Response' );
> _______________________________________________
> vox-tech mailing list
> vox-tech at lists.lugod.org
> http://lists.lugod.org/mailman/listinfo/vox-tech
> 
> 

By adding the "referer" things worked out:

#!/usr/bin/perl

use WWW::Mechanize;
use LWP::UserAgent;
use HTML::Form;
use Data::Dumper;

$url    = "http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx";
$agent  = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)';
$symbol = 'QQQQ';
$ua     = LWP::UserAgent->new(cookie_jar=>{}, agent=>$agent);

push @{ $ua->requests_redirectable }, 'POST';
$resp = $ua->get($url);

$form = HTML::Form->parse($resp->content,$resp->base);
$form->find_input('__VIEWSTATE')->name('NOVIEWSTATE');
$form->value(txtTicker => $symbol);

my $req = $form->click;
$req->header('referer' => $url);

my $resp = $ua->request($req);
print $resp->content;

Jay