Features of the Robot module include:
The robot engine obeys the Robot Exclusion protocol, as well as a proposed addition. See SEE ALSO for references to documents describing the Robot Exclusion protocol and web robots.
I.e. a method for setting and getting each attribute, depending on whether you passed an argument?
Each of the hook procedures and functions is described below. A robot must provide a follow-url-test hook, and at least one of the following:
.
Create a new robot engine instance.
If the constructor fails for any reason, a warning message will be printed,
and undef
will be returned.
Having created a new robot, it should be configured using the methods described below. Certain attributes of the Robot can be set during creation; they can be (re)set after creation, using the setAttribute() method.
The attributes of the Robot are described below, in the Robot Attributes section.
Invokes the robot, initially traversing the root URLs provided in LIST,
and any which have been provided with the
addUrl()
method before
invoking
run()
.
If you have not correctly configured the robot, the method will
return undef
.
The initial set of URLs can either be passed as arguments to the run() method, or with the addUrl() method before you invoke run() . Each URL can be specified either as a string, or as a URI::URL object.
Before invoking this method, you should have provided at least some of the hook functions. See the example given in the EXAMPLES section below.
By default the run() method will iterate until there are no more URLs in the queue. You can override this behavior by providing a continue-test hook function, which checks for the termination conditions. This particular hook function, and use of hook functions in general, are described below.
Change the value of one or more robot attributes. Attributes are identified using a string, and take scalar values. For example, to specify the name of your robot, you set the NAME attribute:
The supported attributes for the Robot module are listed below, in the ROBOT ATTRIBUTES section.
Queries a Robot for the value of an attribute. For example, to query the version number of your robot, you would get the VERSION attribute:
The supported attributes for the Robot module are listed below, in the ROBOT ATTRIBUTES section.
Used to add one or more URLs to the queue for the robot. Each URL can be passed as a simple string, or as a URI::URL object.
Returns True (non-zero) if all URLs were successfully added, False (zero) if at least one of the URLs could not be added.
Register a hook function which should be invoked by the robot at a specific point in the control flow. There are a number of hook points in the robot, which are identified by a string. For a list of hook points, see the SUPPORTED HOOKS section below.
If you provide more than one function for a particular hook, then the hook functions will be invoked in the order they were added. I.e. the first hook function called will be the first hook function you added.
Used to specify a proxy for the given scheme. The protocol argument can be a reference to a list of protocols.
Specifies that proxies should not be used for the specified domains or hosts.
Load proxy settings from protocol_proxy environment variables:
ftp_proxy
, http_proxy
, no_proxy
, etc.
Some of the attributes must be set before you start the Robot (with the run() method). These are marked as mandatory in the list below.
User-Agent
field of HTTP requests,
and so will appear in server logs.
mandatory
mandatory
mandatory
Default: 0 (false)
Default: depth
Default: 1
Default: 1 (true)
Wherever a hook function is passed a $url
argument,
this will be a URI::URL object, with the URL fully specified.
I.e. even if the URL was seen in a relative link,
it will be passed as an absolute URL.
This hook is invoked just before entering the main iterative loop of the robot. The intention is that the hook will be used to restore state, if such an operation is required.
This can be helpful if the robot is running in an incremental mode, where state is saved between each run of the robot.
This hook is invoked on all URLs seen by the robot,
regardless of whether the URL is actually traversed.
In addition to the standard $robot
and $hook
arguments,
the third argument is $url
, which is the URL being travered by
the robot.
For a given URL, the hook function will be invoked at most once, regardless of how many times the URL is seen by the Robot. If you are interested in seeing the URL every time, you can use the invoke-on-link hook.
This hook is invoked to determine whether the robot should traverse the given URL. If the hook function returns 0 (zero), then the robot will do nothing further with the URL. If the hook function returns non-zero, then the robot will get the contents of the URL, invoke further hooks, and extract links if the contents are HTML.
This hook is invoked on URLs which are about to be traversed by the robot; i.e. URLs which have passed the follow-url-test hook.
This hook is invoked if the Robot ever fails to get the contents
of a URL.
The $response
argument is an object of type HTTP::Response.
This hook function is invoked for all URLs for which the contents are successfully retrieved.
The $url
argument is a URI::URL object for the URL currently being
processed by the Robot engine.
The $response
argument is an HTTP::Response object,
the result of the GET request on the URL.
The $structure
argument is an
HTML::Element object which is the root of a tree structure constructed
from the contents of the URL.
You can set the
IGNORE_TEXT
attribute to specify whether the structure
passed includes the textual content of the page,
or just the HTML elements.
The $filename
argument is
the path to a local temporary file which contains
a local copy of the URL contents.
You cannot assume that the file will exist after control has returned
from your hook function.
This hook function is invoked for all links seen as the robot traverses. When the robot is parsing a page ($from_url) for links, for every link seen the invoke-on-link hook is invoked with the URL of the source page, and the destination URL. The destination URL is in canonical form.
This hook is invoked at the end of the robot's main iterative loop. If the hook function returns non-zero, then the robot will continue execution with the next URL. If the hook function returns zero, then the Robot will terminate the main loop, and close down after invoking the following two hooks.
If no continue-test hook function is provided, then the robot will always loop.
This hook is used to save any state information required by the robot application.
This hook is used to generate a report for the run of the robot, if such is desired.
Your hook function should be declared as follows:
If your function returns anything other than undef
,
then a If-Modified-Since: field will be added to the request header.
$url
we tried to GET,
and the $response
which resulted.
If you provided a the modified-since manpage hook, then provide an invoke-after-get function, and look for error code 304 (or RC_NOT_MODIFIED if you are using HTTP::Status, which you should be :-):
If you are behind a firewall, then you will have to add something like the following, just before calling the run() method:
This is the home page of the Group at Canon Research Centre Europe who are responsible for Robot.pm.
This is a de facto standard which defines how a `well behaved' Robot client should interact with web servers and web pages.
Guidelines and suggestions for those who are (considering) developing a web robot.
Weblint is a perl script which is used to check HTML for syntax errors and stylistic problems, in the same way lint is used to check C.
This is a well-organized collection of Perl resources, such as modules, documents, and scripts. CPAN is mirrored at FTP sites around the world.
<neilb@cre.canon.co.uk>
This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.