Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.extractor
Class ExtractorImpliedURI

    java.lang.Object
      javax.management.Attribute
          org.archive.crawler.settings.Type
              org.archive.crawler.settings.ComplexType
                  org.archive.crawler.settings.ModuleType
                      org.archive.crawler.framework.Processor
                          org.archive.crawler.extractor.Extractor
                              org.archive.crawler.extractor.ExtractorImpliedURI

All Implemented Interfaces:
  java.io.Serializable, javax.management.DynamicMBean, CoreAttributeConstants

    public class ExtractorImpliedURI

extends Extractor
implements CoreAttributeConstants

An extractor for finding 'implied' URIs inside other URIs. If the 'trigger' regex is matched, a new URI will be constructed from the 'build' replacement pattern. Unlike most other extractors, this works on URIs discovered by previous extractors. Thus it should appear near the end of any set of extractors. Initially, only finds absolute HTTP(S) URIs in query-string or its parameters. TODO: extend to find URIs in path-info

Author:
  Gordon Mohr
See Also:
  Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType
`ComplexType.MBeanAttributeInfoIterator`

Field Summary
`static java.lang.String` | `ATTR_BUILD_PATTERN` replacement pattern used to build 'implied' URI
`static java.lang.String` | `ATTR_REMOVE_TRIGGER_URIS` whether to remove URIs that trigger addition of 'implied' URI; default false
`static java.lang.String` | `ATTR_TRIGGER_REGEXP` regex which when matched triggers addition of 'implied' URI

Fields inherited from class org.archive.crawler.framework.Processor
`ATTR_DECIDE_RULES, ATTR_ENABLED, attrDecideRules`

Fields inherited from class org.archive.crawler.settings.ComplexType
`definition, definitionMap`

Fields inherited from interface org.archive.crawler.datamodel.CoreAttributeConstants
`A_ANNOTATIONS, A_CONTENT_DIGEST, A_CONTENT_TYPE, A_CREDENTIAL_AVATARS_KEY, A_DELAY_FACTOR, A_DISTANCE_FROM_SEED, A_DNS_FETCH_TIME, A_DNS_SERVER_IP_LABEL, A_ETAG_HEADER, A_FETCH_BEGAN_TIME, A_FETCH_COMPLETED_TIME, A_FETCH_HISTORY, A_FORCE_RETIRE, A_HERITABLE_KEYS, A_HTML_BASE, A_HTTP_BIND_ADDRESS, A_HTTP_PROXY_HOST, A_HTTP_PROXY_PORT, A_HTTP_TRANSACTION, A_LAST_MODIFIED_HEADER, A_LOCALIZED_ERRORS, A_META_ROBOTS, A_MINIMUM_DELAY, A_MIRROR_PATH, A_PREREQUISITE_URI, A_REFERENCE_LENGTH, A_RETRY_DELAY, A_RRECORD_SET_LABEL, A_RUNTIME_EXCEPTION, A_SOURCE_TAG, A_STATUS, HEADER_TRUNC, LENGTH_TRUNC, TIMER_TRUNC, TRUNC_SUFFIX`

Constructor Summary
`ExtractorImpliedURI(java.lang.String name)` Constructor

Method Summary
`void` | `extract(CrawlURI curi)` Perform usual extraction on a CrawlURI
`protected static java.lang.String` | `extractImplied(java.lang.CharSequence uri, java.lang.String trigger, java.lang.String build)` Utility method for extracting 'implied' URI given a source uri, trigger pattern, and build pattern.
`java.lang.String` | `report()` Compiles and returns a report (in human readable form) about the status of the processor.

Methods inherited from class org.archive.crawler.extractor.Extractor
`innerProcess`

Methods inherited from class org.archive.crawler.framework.Processor
`checkForInterrupt, finalTasks, getController, getDecideRule, getDefaultNextProcessor, initialTasks, innerRejectProcess, isContentToProcess, isExpectedMimeType, isHttpTransactionContentToProcess, kickUpdate, process, rulesAccept, rulesAccept, setDefaultNextProcessor, spawn`

Methods inherited from class org.archive.crawler.settings.ModuleType
`addElement, listUsedFiles`

Methods inherited from class org.archive.crawler.settings.ComplexType
`addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute`

Methods inherited from class org.archive.crawler.settings.Type
`addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient`

Methods inherited from class javax.management.Attribute
`getName`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Field Detail

### ATTR_TRIGGER_REGEXP

    public static final java.lang.String ATTR_TRIGGER_REGEXP

  regex which when matched triggers addition of 'implied' URI

  See Also:
    Constant Field Values

### ATTR_BUILD_PATTERN

    public static final java.lang.String ATTR_BUILD_PATTERN

  replacement pattern used to build 'implied' URI

  See Also:
    Constant Field Values

### ATTR_REMOVE_TRIGGER_URIS

    public static final java.lang.String ATTR_REMOVE_TRIGGER_URIS

  whether to remove URIs that trigger addition of 'implied' URI; default false

  See Also:
    Constant Field Values

Constructor Detail

### ExtractorImpliedURI

    public ExtractorImpliedURI(java.lang.String name)

  Constructor

  Parameters:
    `name` -

Method Detail

### extract

    public void extract(CrawlURI curi)

  Perform usual extraction on a CrawlURI

  Specified by:
    `extract` in class `Extractor`
  Parameters:
    `curi` - Crawl URI to process.

### extractImplied

    protected static java.lang.String extractImplied(java.lang.CharSequence uri,
                                                     java.lang.String trigger,
                                                     java.lang.String build)

  Utility method for extracting 'implied' URI given a source uri, trigger pattern, and build pattern.

  Parameters:
    `uri` - source to check for implied URI
    `trigger` - regex pattern which if matched implies another URI
    `build` - replacement pattern to build the implied URI
  Returns:
    implied URI, or null if none

### report

    public java.lang.String report()

  Description copied from class: `Processor`
  Compiles and returns a report (in human readable form) about the status of the processor. The processor's name (of implementing class) should always be included.

  Examples of stats declared would include:
  * Number of CrawlURIs handled.
  * Number of links extracted (for link extractors)
  etc.

  Overrides:
    `report` in class `Processor`
  Returns:
    A human readable report on the processor's state.

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.