Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.processor
Class CrawlMapper

    java.lang.Object
      javax.management.Attribute
          org.archive.crawler.settings.Type
              org.archive.crawler.settings.ComplexType
                  org.archive.crawler.settings.ModuleType
                      org.archive.crawler.framework.Processor
                          org.archive.crawler.processor.CrawlMapper

All Implemented Interfaces:
  java.io.Serializable, javax.management.DynamicMBean, FetchStatusCodes

Direct Known Subclasses:
  HashCrawlMapper, LexicalCrawlMapper

    public abstract class CrawlMapper

extends Processor
implements FetchStatusCodes

A simple crawl splitter/mapper, dividing up CandidateURIs/CrawlURIs between crawlers by diverting some range of URIs to local log files (which can then be imported to other crawlers). May operate on a CrawlURI (typically early in the processing chain) or its CandidateURI outlinks (late in the processing chain, after LinksScoper), or both (if inserted and configured in both places).

Applies a map() method, supplied by a concrete subclass, to classKeys to map URIs to crawlers by name.

One crawler name is distinguished as the 'local name'; URIs mapped to this name are not diverted, but continue to be processed normally.

If using the JMX importUris operation importing URLs dropped by a `CrawlMapper` instance, use `recoveryLog` style.

Version:
  $Date: 2007-06-07 21:34:56 +0000 (Thu, 07 Jun 2007) $, $Revision: 5199 $
Author:
  gojomo
See Also:
  Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType
`ComplexType.MBeanAttributeInfoIterator`

Field Summary
`static java.lang.String` | `ATTR_CHECK_OUTLINKS` whether to map CrawlURI's outlinks (if CandidateURIs)
`static java.lang.String` | `ATTR_CHECK_URI` whether to map CrawlURI itself (if status nonpositive)
`static java.lang.String` | `ATTR_DIVERSION_DIR` where to log diversions
`static java.lang.String` | `ATTR_LOCAL_NAME` name of local crawler (URIs mapped to here are not diverted)
`static java.lang.String` | `ATTR_MAP_OUTLINK_DECIDE_RULES` decide rules to determine if an outlink is subject to mapping
`static java.lang.String` | `ATTR_ROTATION_DIGITS` rotate logs when change occurs within this # of digits of timestamp
`protected ArrayLongFPCache` | `cache`
`static java.lang.Boolean` | `DEFAULT_CHECK_OUTLINKS`
`static java.lang.Boolean` | `DEFAULT_CHECK_URI`
`static java.lang.String` | `DEFAULT_DIVERSION_DIR`
`static java.lang.String` | `DEFAULT_LOCAL_NAME`
`static java.lang.Integer` | `DEFAULT_ROTATION_DIGITS`
`(package private) java.util.HashMap<java.lang.String,java.io.PrintWriter>` | `diversionLogs` Mapping of target crawlers to logs (PrintWriters)
`protected java.lang.String` | `localName` name of the enclosing crawler (URIs mapped here stay put)
`(package private) java.lang.String` | `logGeneration` Truncated timestamp prefix for diversion logs; when current time doesn't match, it's time to close all current logs.

Fields inherited from class org.archive.crawler.framework.Processor
`ATTR_DECIDE_RULES, ATTR_ENABLED, attrDecideRules`

Fields inherited from class org.archive.crawler.settings.ComplexType
`definition, definitionMap`

Fields inherited from interface org.archive.crawler.datamodel.FetchStatusCodes
`S_BLOCKED_BY_CUSTOM_PROCESSOR, S_BLOCKED_BY_QUOTA, S_BLOCKED_BY_RUNTIME_LIMIT, S_BLOCKED_BY_USER, S_CONNECT_FAILED, S_CONNECT_LOST, S_DEEMED_CHAFF, S_DEFERRED, S_DELETED_BY_USER, S_DNS_SUCCESS, S_DOMAIN_PREREQUISITE_FAILURE, S_DOMAIN_UNRESOLVABLE, S_GETBYNAME_SUCCESS, S_OTHER_PREREQUISITE_FAILURE, S_OUT_OF_SCOPE, S_PREREQUISITE_UNSCHEDULABLE_FAILURE, S_PROCESSING_THREAD_KILLED, S_ROBOTS_PRECLUDED, S_ROBOTS_PREREQUISITE_FAILURE, S_RUNTIME_EXCEPTION, S_SERIOUS_ERROR, S_TIMEOUT, S_TOO_MANY_EMBED_HOPS, S_TOO_MANY_LINK_HOPS, S_TOO_MANY_RETRIES, S_UNATTEMPTED, S_UNFETCHABLE_URI, S_UNQUEUEABLE`

Constructor Summary
`CrawlMapper(java.lang.String name, java.lang.String description)` Constructor.

Method Summary
`protected boolean` | `decideToMapOutlink(CandidateURI cauri)`
`protected void` | `divertLog(CandidateURI cauri, java.lang.String target)` Note the given CandidateURI in the appropriate diversion log.
`protected java.io.PrintWriter` | `getDiversionLog(java.lang.String target)` Get the diversion log for a given target crawler node node.
`protected DecideRule` | `getMapOutlinkDecideRule(java.lang.Object o)`
`protected void` | `initialTasks()` Classes subclassing this one should override this method to perform processor specific actions.
`protected void` | `innerProcess(CrawlURI curi)` Classes subclassing this one should override this method to perform their custom actions on the CrawlURI.
`protected abstract java.lang.String` | `map(CandidateURI cauri)` Look up the crawler node name to which the given CandidateURI should be mapped.
`protected void` | `updateGeneration(java.lang.String nowGeneration)` Close and mark as finished all existing diversion logs, and arrange for new logs to use the new generation prefix.

Methods inherited from class org.archive.crawler.framework.Processor
`checkForInterrupt, finalTasks, getController, getDecideRule, getDefaultNextProcessor, innerRejectProcess, isContentToProcess, isExpectedMimeType, isHttpTransactionContentToProcess, kickUpdate, process, report, rulesAccept, rulesAccept, setDefaultNextProcessor, spawn`

Methods inherited from class org.archive.crawler.settings.ModuleType
`addElement, listUsedFiles`

Methods inherited from class org.archive.crawler.settings.ComplexType
`addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute`

Methods inherited from class org.archive.crawler.settings.Type
`addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient`

Methods inherited from class javax.management.Attribute
`getName`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Field Detail

### ATTR_CHECK_URI

    public static final java.lang.String ATTR_CHECK_URI

  whether to map CrawlURI itself (if status nonpositive)

  See Also:
    Constant Field Values

### DEFAULT_CHECK_URI

    public static final java.lang.Boolean DEFAULT_CHECK_URI

### ATTR_CHECK_OUTLINKS

    public static final java.lang.String ATTR_CHECK_OUTLINKS

  whether to map CrawlURI's outlinks (if CandidateURIs)

  See Also:
    Constant Field Values

### DEFAULT_CHECK_OUTLINKS

    public static final java.lang.Boolean DEFAULT_CHECK_OUTLINKS

### ATTR_MAP_OUTLINK_DECIDE_RULES

    public static final java.lang.String ATTR_MAP_OUTLINK_DECIDE_RULES

  decide rules to determine if an outlink is subject to mapping

  See Also:
    Constant Field Values

### ATTR_LOCAL_NAME

    public static final java.lang.String ATTR_LOCAL_NAME

  name of local crawler (URIs mapped to here are not diverted)

  See Also:
    Constant Field Values

### DEFAULT_LOCAL_NAME

    public static final java.lang.String DEFAULT_LOCAL_NAME

See Also:
  Constant Field Values

### ATTR_DIVERSION_DIR

    public static final java.lang.String ATTR_DIVERSION_DIR

  where to log diversions

  See Also:
    Constant Field Values

### DEFAULT_DIVERSION_DIR

    public static final java.lang.String DEFAULT_DIVERSION_DIR

See Also:
  Constant Field Values

### ATTR_ROTATION_DIGITS

    public static final java.lang.String ATTR_ROTATION_DIGITS

  rotate logs when change occurs within this # of digits of timestamp

  See Also:
    Constant Field Values

### DEFAULT_ROTATION_DIGITS

    public static final java.lang.Integer DEFAULT_ROTATION_DIGITS

### diversionLogs

    java.util.HashMap<java.lang.String,java.io.PrintWriter> diversionLogs

  Mapping of target crawlers to logs (PrintWriters)

### logGeneration

    java.lang.String logGeneration

  Truncated timestamp prefix for diversion logs; when current time doesn't match, it's time to close all current logs.

### localName

    protected java.lang.String localName

  name of the enclosing crawler (URIs mapped here stay put)

### cache

    protected ArrayLongFPCache cache

Constructor Detail

### CrawlMapper

    public CrawlMapper(java.lang.String name,
                       java.lang.String description)

  Constructor.

  Parameters:
    `name` - Name of this processor.

Method Detail

### innerProcess

    protected void innerProcess(CrawlURI curi)

  Description copied from class: `Processor`
  Classes subclassing this one should override this method to perform their custom actions on the CrawlURI.

  Overrides:
    `innerProcess` in class `Processor`
  Parameters:
    `curi` - The CrawlURI being processed.

### decideToMapOutlink

    protected boolean decideToMapOutlink(CandidateURI cauri)

### getMapOutlinkDecideRule

    protected DecideRule getMapOutlinkDecideRule(java.lang.Object o)

### updateGeneration

    protected void updateGeneration(java.lang.String nowGeneration)

  Close and mark as finished all existing diversion logs, and arrange for new logs to use the new generation prefix.

  Parameters:
    `nowGeneration` - new generation (timestamp prefix) to use

### map

    protected abstract java.lang.String map(CandidateURI cauri)

  Look up the crawler node name to which the given CandidateURI should be mapped.

  Parameters:
    `cauri` - CandidateURI to consider
  Returns:
    String node name which should handle URI

### divertLog

    protected void divertLog(CandidateURI cauri,
                             java.lang.String target)

  Note the given CandidateURI in the appropriate diversion log.

  Parameters:
    `cauri` - CandidateURI to append to a diversion log
    `target` - String node name (log name) to receive URI

### getDiversionLog

    protected java.io.PrintWriter getDiversionLog(java.lang.String target)

  Get the diversion log for a given target crawler node node.

  Parameters:
    `target` - crawler node name of requested log
  Returns:
    PrintWriter open on an appropriately-named log file

### initialTasks

    protected void initialTasks()

  Description copied from class: `Processor`
  Classes subclassing this one should override this method to perform processor specific actions.

  This method is garanteed to be called after the crawl is set up, but before any URI-processing has occured.

  Overrides:
    `initialTasks` in class `Processor`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.