On Mon, 23 Nov 2009, Ken Irwin wrote:

> Hi all,
>
> I'm moving to a new web server and struggling to get it configured properly. The problem of the moment: having a Perl CGI script call another web page in the background and make decisions based on its content. On the old server I used an antique Perl script called "hcat" (from the Pelican book<http://oreilly.com/openbook/webclient/ch04.html>); I've also tried curl and LWP::Simple.
>
> In all three cases, I get the same behavior: it works just fine on the command line, but when called by the web server through a CGI script, the LWP (or other socket connection) gets no results. It sounds like a permissions thing, but I don't know what kind of permissions setting to tinker with. In the test script below, my command line outputs:
>
> Content-type: text/plain
> Getting URL: http://www.npr.org
> 885 lines
>
> Whereas the web output just says "Getting URL: http://www.npr.org" - and doesn't even get to the "Couldn't get" error message.
>
> Any clue how I can make use of a web page's contents from w/in a CGI script? (The actual application has to do with exporting data from our catalog, but I need to work out the basic mechanism first.)
>
> Here's the script I'm using.
>
> #!/bin/perl
> use LWP::Simple;
> print "Content-type: text/plain\n\n";
> my $url = "http://www.npr.org";
> print "Getting URL: $url\n";
> my $content = get $url;
> die "Couldn't get $url" unless defined $content;
> @lines = split (/\n/, $content);
> foreach (@lines) { $i++; }
> print "\n\n$i lines\n\n";
>
> Any ideas?

I'd suggest testing the results of the call, rather than just looking for 
content, as an empty response could be a result of the server you're 
connecting to.  (unlikely in this case, but it happens once in a while, 
particularly if you turn off redirection, or support caching). 
Unfortunately, you might have to use LWP::UserAgent, rather than 
LWP::Simple:

 	#!/bin/perl --

 	use strict; use warnings;
 	use LWP::UserAgent;

 	my $ua = LWP::UserAgent->new( timeout => 60 );

 	my $response = $ua->get('http://www.npr.org/');
 	if ( $response->is_success() ) {
 		my $content = $response->decoded_content();
 		...
 	} else {
 		print "HTTP Error : ",$response->status_line(),"\n";
 	}

 	__END__

(and changing the shebang line for my location of perl, your version 
worked via both CGI and command line)


oh ... and you don't need the foreach loop:

 	my $i = @lines;

-Joe