Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.datamodel
Class CrawlOrder

    java.lang.Object
      javax.management.Attribute
          org.archive.crawler.settings.Type
              org.archive.crawler.settings.ComplexType
                  org.archive.crawler.settings.ModuleType
                      org.archive.crawler.datamodel.CrawlOrder

All Implemented Interfaces:
  java.io.Serializable, javax.management.DynamicMBean

    public class CrawlOrder

extends ModuleType
implements java.io.Serializable

Represents the 'root' of the settings hierarchy. Contains those settings that do not belong to any specific module, but rather relate to the crawl as a whole (much of this is used by the CrawlController directly or indirectly).

See Also:
  `ModuleType`, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType
`ComplexType.MBeanAttributeInfoIterator`

Field Summary
`static java.lang.String` | `ATTR_BDB_CACHE_PERCENT` Percentage of heap to allocate to bdb cache
`static java.lang.String` | `ATTR_CHECKPOINT_COPY_BDBJE_LOGS` When checkpointing, copy the bdb logs.
`static java.lang.String` | `ATTR_CHECKPOINTS_PATH`
`static java.lang.String` | `ATTR_DISK_PATH`
`static java.lang.String` | `ATTR_EXTRACT_PROCESSORS`
`static java.lang.String` | `ATTR_FETCH_PROCESSORS`
`static java.lang.String` | `ATTR_FROM`
`static java.lang.String` | `ATTR_HTTP_HEADERS`
`static java.lang.String` | `ATTR_LOGGERS`
`static java.lang.String` | `ATTR_LOGS_PATH`
`static java.lang.String` | `ATTR_MAX_BYTES_DOWNLOAD`
`static java.lang.String` | `ATTR_MAX_DOCUMENT_DOWNLOAD`
`static java.lang.String` | `ATTR_MAX_TIME_SEC`
`static java.lang.String` | `ATTR_MAX_TOE_THREADS`
`static java.lang.String` | `ATTR_NAME`
`static java.lang.String` | `ATTR_POST_PROCESSORS`
`static java.lang.String` | `ATTR_PRE_FETCH_PROCESSORS`
`static java.lang.String` | `ATTR_RECORDER_IN_BUFFER`
`static java.lang.String` | `ATTR_RECORDER_OUT_BUFFER`
`static java.lang.String` | `ATTR_RECOVER_PATH`
`static java.lang.String` | `ATTR_RECOVER_RETAIN_FAILURES`
`static java.lang.String` | `ATTR_RECOVER_SCOPE_ENQUEUES`
`static java.lang.String` | `ATTR_RECOVER_SCOPE_INCLUDES`
`static java.lang.String` | `ATTR_RULES`
`static java.lang.String` | `ATTR_SCRATCH_PATH`
`static java.lang.String` | `ATTR_SETTINGS_DIRECTORY`
`static java.lang.String` | `ATTR_STATE_PATH`
`static java.lang.String` | `ATTR_USER_AGENT`
`static java.lang.String` | `ATTR_WRITE_PROCESSORS`
`static java.lang.Boolean` | `DEFAULT_CHECKPOINT_COPY_BDBJE_LOGS`

Fields inherited from class org.archive.crawler.settings.ComplexType
`definition, definitionMap`

Constructor Summary
`CrawlOrder()` Construct a CrawlOrder.

Method Summary
`void` | `checkUserAgentAndFrom()` Checks if the User Agent and From field are set 'correctly' in the specified Crawl Order.
`java.io.File` | `getCheckpointsDirectory()`
`CrawlController` | `getController()`
`java.lang.String` | `getCrawlOrderName()` Get the name of the order file.
`java.lang.String` | `getFrom(CrawlURI curi)`
`MapType` | `getLoggers()` Returns the Map of the StatisticsTracking modules that are included in the configuration that the current instance of this class is representing.
`int` | `getMaxToes()` Returns the set number of maximum toe threads.
`RobotsHonoringPolicy` | `getRobotsHonoringPolicy()` This method gets the RobotsHonoringPolicy object from the orders file.
`java.io.File` | `getSettingsDir(java.lang.String key)` Return fullpath to the directory named by `key` in settings.
`java.lang.String` | `getUserAgent(CrawlURI curi)`
`void` | `setController(CrawlController controller)`

Methods inherited from class org.archive.crawler.settings.ModuleType
`addElement, listUsedFiles`

Methods inherited from class org.archive.crawler.settings.ComplexType
`addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute`

Methods inherited from class org.archive.crawler.settings.Type
`addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient`

Methods inherited from class javax.management.Attribute
`getName`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Field Detail

### ATTR_NAME

    public static final java.lang.String ATTR_NAME

See Also:
  Constant Field Values

### ATTR_SETTINGS_DIRECTORY

    public static final java.lang.String ATTR_SETTINGS_DIRECTORY

See Also:
  Constant Field Values

### ATTR_DISK_PATH

    public static final java.lang.String ATTR_DISK_PATH

See Also:
  Constant Field Values

### ATTR_LOGS_PATH

    public static final java.lang.String ATTR_LOGS_PATH

See Also:
  Constant Field Values

### ATTR_CHECKPOINTS_PATH

    public static final java.lang.String ATTR_CHECKPOINTS_PATH

See Also:
  Constant Field Values

### ATTR_STATE_PATH

    public static final java.lang.String ATTR_STATE_PATH

See Also:
  Constant Field Values

### ATTR_SCRATCH_PATH

    public static final java.lang.String ATTR_SCRATCH_PATH

See Also:
  Constant Field Values

### ATTR_RECOVER_PATH

    public static final java.lang.String ATTR_RECOVER_PATH

See Also:
  Constant Field Values

### ATTR_RECOVER_RETAIN_FAILURES

    public static final java.lang.String ATTR_RECOVER_RETAIN_FAILURES

See Also:
  Constant Field Values

### ATTR_RECOVER_SCOPE_INCLUDES

    public static final java.lang.String ATTR_RECOVER_SCOPE_INCLUDES

See Also:
  Constant Field Values

### ATTR_RECOVER_SCOPE_ENQUEUES

    public static final java.lang.String ATTR_RECOVER_SCOPE_ENQUEUES

See Also:
  Constant Field Values

### ATTR_MAX_BYTES_DOWNLOAD

    public static final java.lang.String ATTR_MAX_BYTES_DOWNLOAD

See Also:
  Constant Field Values

### ATTR_MAX_DOCUMENT_DOWNLOAD

    public static final java.lang.String ATTR_MAX_DOCUMENT_DOWNLOAD

See Also:
  Constant Field Values

### ATTR_MAX_TIME_SEC

    public static final java.lang.String ATTR_MAX_TIME_SEC

See Also:
  Constant Field Values

### ATTR_MAX_TOE_THREADS

    public static final java.lang.String ATTR_MAX_TOE_THREADS

See Also:
  Constant Field Values

### ATTR_HTTP_HEADERS

    public static final java.lang.String ATTR_HTTP_HEADERS

See Also:
  Constant Field Values

### ATTR_USER_AGENT

    public static final java.lang.String ATTR_USER_AGENT

See Also:
  Constant Field Values

### ATTR_FROM

    public static final java.lang.String ATTR_FROM

See Also:
  Constant Field Values

### ATTR_PRE_FETCH_PROCESSORS

    public static final java.lang.String ATTR_PRE_FETCH_PROCESSORS

See Also:
  Constant Field Values

### ATTR_FETCH_PROCESSORS

    public static final java.lang.String ATTR_FETCH_PROCESSORS

See Also:
  Constant Field Values

### ATTR_EXTRACT_PROCESSORS

    public static final java.lang.String ATTR_EXTRACT_PROCESSORS

See Also:
  Constant Field Values

### ATTR_WRITE_PROCESSORS

    public static final java.lang.String ATTR_WRITE_PROCESSORS

See Also:
  Constant Field Values

### ATTR_POST_PROCESSORS

    public static final java.lang.String ATTR_POST_PROCESSORS

See Also:
  Constant Field Values

### ATTR_LOGGERS

    public static final java.lang.String ATTR_LOGGERS

See Also:
  Constant Field Values

### ATTR_RULES

    public static final java.lang.String ATTR_RULES

See Also:
  Constant Field Values

### ATTR_RECORDER_OUT_BUFFER

    public static final java.lang.String ATTR_RECORDER_OUT_BUFFER

See Also:
  Constant Field Values

### ATTR_RECORDER_IN_BUFFER

    public static final java.lang.String ATTR_RECORDER_IN_BUFFER

See Also:
  Constant Field Values

### ATTR_BDB_CACHE_PERCENT

    public static final java.lang.String ATTR_BDB_CACHE_PERCENT

  Percentage of heap to allocate to bdb cache

  See Also:
    Constant Field Values

### ATTR_CHECKPOINT_COPY_BDBJE_LOGS

    public static final java.lang.String ATTR_CHECKPOINT_COPY_BDBJE_LOGS

  When checkpointing, copy the bdb logs. Default is true. If false, then we do not copy logs on checkpoint AND we tell bdbje never to delete log files; instead it renames files-to-delete with a '.del' extension. Assumption is that when this setting is false, an external process is managing the removing of bdbje log files and that come time to recover from a checkpoint, the files that comprise a checkpoint are manually assembled.

  See Also:
    Constant Field Values

### DEFAULT_CHECKPOINT_COPY_BDBJE_LOGS

    public static final java.lang.Boolean DEFAULT_CHECKPOINT_COPY_BDBJE_LOGS

Constructor Detail

### CrawlOrder

    public CrawlOrder()

  Construct a CrawlOrder.

Method Detail

### getUserAgent

    public java.lang.String getUserAgent(CrawlURI curi)

  Parameters:
    `curi` -
  Returns:
    user-agent header value to use

### getFrom

    public java.lang.String getFrom(CrawlURI curi)

  Parameters:
    `curi` -
  Returns:
    from header value to use

### getMaxToes

    public int getMaxToes()

  Returns the set number of maximum toe threads.

  Returns:
    Number of maximum toe threads

### getRobotsHonoringPolicy

    public RobotsHonoringPolicy getRobotsHonoringPolicy()

  This method gets the RobotsHonoringPolicy object from the orders file.

  Returns:
    the new RobotsHonoringPolicy

### getCrawlOrderName

    public java.lang.String getCrawlOrderName()

  Get the name of the order file.

  Returns:
    the name of the order file.

### getController

    public CrawlController getController()

  Returns:
    The crawl controller.

### setController

    public void setController(CrawlController controller)

  Parameters:
    `controller` -

### getLoggers

    public MapType getLoggers()

  Returns the Map of the StatisticsTracking modules that are included in the configuration that the current instance of this class is representing.

  Returns:
    Map of the StatisticsTracking modules

### checkUserAgentAndFrom

    public void checkUserAgentAndFrom()
                               throws FatalConfigurationException

  Checks if the User Agent and From field are set 'correctly' in the specified Crawl Order.

  Throws:
    `FatalConfigurationException`

### getCheckpointsDirectory

    public java.io.File getCheckpointsDirectory()

  Returns:
    Checkpoint directory.

### getSettingsDir

    public java.io.File getSettingsDir(java.lang.String key)
                                throws javax.management.AttributeNotFoundException

  Return fullpath to the directory named by `key` in settings. If directory does not exist, it and all intermediary dirs will be created.

  Parameters:
    `key` - Key to use going to settings.
  Returns:
    Full path to directory named by `key`.
  Throws:
    `javax.management.AttributeNotFoundException`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.