On Mon, 23 Nov 2009, Ken Irwin wrote: > Hi all, > > I'm moving to a new web server and struggling to get it configured properly. The problem of the moment: having a Perl CGI script call another web page in the background and make decisions based on its content. On the old server I used an antique Perl script called "hcat" (from the Pelican book<http://oreilly.com/openbook/webclient/ch04.html>); I've also tried curl and LWP::Simple. > > In all three cases, I get the same behavior: it works just fine on the command line, but when called by the web server through a CGI script, the LWP (or other socket connection) gets no results. It sounds like a permissions thing, but I don't know what kind of permissions setting to tinker with. In the test script below, my command line outputs: > > Content-type: text/plain > Getting URL: http://www.npr.org > 885 lines > > Whereas the web output just says "Getting URL: http://www.npr.org" - and doesn't even get to the "Couldn't get" error message. > > Any clue how I can make use of a web page's contents from w/in a CGI script? (The actual application has to do with exporting data from our catalog, but I need to work out the basic mechanism first.) > > Here's the script I'm using. > > #!/bin/perl > use LWP::Simple; > print "Content-type: text/plain\n\n"; > my $url = "http://www.npr.org"; > print "Getting URL: $url\n"; > my $content = get $url; > die "Couldn't get $url" unless defined $content; > @lines = split (/\n/, $content); > foreach (@lines) { $i++; } > print "\n\n$i lines\n\n"; > > Any ideas? I'd suggest testing the results of the call, rather than just looking for content, as an empty response could be a result of the server you're connecting to. (unlikely in this case, but it happens once in a while, particularly if you turn off redirection, or support caching). Unfortunately, you might have to use LWP::UserAgent, rather than LWP::Simple: #!/bin/perl -- use strict; use warnings; use LWP::UserAgent; my $ua = LWP::UserAgent->new( timeout => 60 ); my $response = $ua->get('http://www.npr.org/'); if ( $response->is_success() ) { my $content = $response->decoded_content(); ... } else { print "HTTP Error : ",$response->status_line(),"\n"; } __END__ (and changing the shebang line for my location of perl, your version worked via both CGI and command line) oh ... and you don't need the foreach loop: my $i = @lines; -Joe