Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.datamodel
Class RobotsHonoringPolicy

    java.lang.Object
      javax.management.Attribute
          org.archive.crawler.settings.Type
              org.archive.crawler.settings.ComplexType
                  org.archive.crawler.settings.ModuleType
                      org.archive.crawler.datamodel.RobotsHonoringPolicy

All Implemented Interfaces:
  java.io.Serializable, javax.management.DynamicMBean

    public class RobotsHonoringPolicy

extends ModuleType

RobotsHonoringPolicy represent the strategy used by the crawler for determining how robots.txt files will be honored. Five kinds of policies exist:

classic:
  obey the first set of robots.txt directives that apply to your current user-agent
ignore:
  ignore robots.txt directives entirely
custom:
  obey a specific operator-entered set of robots.txt directives for a given host
most-favored:
  obey the most liberal restrictions offered (if *any* crawler is allowed to get a page, get it)
most-favored-set:
  given some set of user-agent patterns, obey the most liberal restriction offered to any

The two last ones has the opportunity of adopting a different user-agent to reflect the restrictions we've opted to use.

Author:
  John Erik Halse
See Also:
  Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType
`ComplexType.MBeanAttributeInfoIterator`

Field Summary
`static java.lang.String` | `ATTR_CUSTOM_ROBOTS`
`static java.lang.String` | `ATTR_MASQUERADE`
`static java.lang.String` | `ATTR_NAME`
`static java.lang.String` | `ATTR_TYPE`
`static java.lang.String` | `ATTR_USER_AGENTS`
`static int` | `CLASSIC`
`static int` | `CUSTOM`
`static int` | `IGNORE`
`static int` | `MOST_FAVORED`
`static int` | `MOST_FAVORED_SET`

Fields inherited from class org.archive.crawler.settings.ComplexType
`definition, definitionMap`

Constructor Summary
`RobotsHonoringPolicy()`
`RobotsHonoringPolicy(java.lang.String name)` Creates a new instance of RobotsHonoringPolicy.

Method Summary
`java.lang.String` | `getCustomRobots(CrawlerSettings settings)` Get the supplied custom robots.txt
`int` | `getType(java.lang.Object context)` Get the policy-type.
`StringList` | `getUserAgents(CrawlerSettings settings)` If policy-type is most favored crawler of set, then this method gets a list of all useragents in that set.
`boolean` | `isType(java.lang.Object o, int type)` Check if policy is of a certain type.
`boolean` | `shouldMasquerade(CrawlURI curi)` This method returns true if the crawler should masquerade as the user agent which restrictions it opted to use.

Methods inherited from class org.archive.crawler.settings.ModuleType
`addElement, listUsedFiles`

Methods inherited from class org.archive.crawler.settings.ComplexType
`addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute`

Methods inherited from class org.archive.crawler.settings.Type
`addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient`

Methods inherited from class javax.management.Attribute
`getName`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Field Detail

### CLASSIC

    public static final int CLASSIC

See Also:
  Constant Field Values

### IGNORE

    public static final int IGNORE

See Also:
  Constant Field Values

### CUSTOM

    public static final int CUSTOM

See Also:
  Constant Field Values

### MOST_FAVORED

    public static final int MOST_FAVORED

See Also:
  Constant Field Values

### MOST_FAVORED_SET

    public static final int MOST_FAVORED_SET

See Also:
  Constant Field Values

### ATTR_NAME

    public static final java.lang.String ATTR_NAME

See Also:
  Constant Field Values

### ATTR_TYPE

    public static final java.lang.String ATTR_TYPE

See Also:
  Constant Field Values

### ATTR_MASQUERADE

    public static final java.lang.String ATTR_MASQUERADE

See Also:
  Constant Field Values

### ATTR_CUSTOM_ROBOTS

    public static final java.lang.String ATTR_CUSTOM_ROBOTS

See Also:
  Constant Field Values

### ATTR_USER_AGENTS

    public static final java.lang.String ATTR_USER_AGENTS

See Also:
  Constant Field Values

Constructor Detail

### RobotsHonoringPolicy

    public RobotsHonoringPolicy(java.lang.String name)

  Creates a new instance of RobotsHonoringPolicy.

  Parameters:
    `name` - the name of the RobotsHonoringPolicy attirubte.

### RobotsHonoringPolicy

    public RobotsHonoringPolicy()

Method Detail

### getUserAgents

    public StringList getUserAgents(CrawlerSettings settings)

  If policy-type is most favored crawler of set, then this method gets a list of all useragents in that set.

  Returns:
    List of Strings with user agents

### shouldMasquerade

    public boolean shouldMasquerade(CrawlURI curi)

  This method returns true if the crawler should masquerade as the user agent which restrictions it opted to use. (Only relevant for policy-types: most-favored and most-favored-set).

  Returns:
    true if we should masquerade

### getCustomRobots

    public java.lang.String getCustomRobots(CrawlerSettings settings)

  Get the supplied custom robots.txt

  Returns:
    String with content of alternate robots.txt

### getType

    public int getType(java.lang.Object context)

  Get the policy-type.

  Returns:
    policy type
  See Also:
    `CLASSIC`, `IGNORE`, `CUSTOM`, `MOST_FAVORED`, `MOST_FAVORED_SET`

### isType

    public boolean isType(java.lang.Object o,
                          int type)

  Check if policy is of a certain type.

  Parameters:
    `o` - An object that can be resolved into a settings object.
    `type` - the type to check against.
  Returns:
    true if the policy is of the submitted type

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.