Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.extractor
Class ExtractorHTML

    java.lang.Object
      javax.management.Attribute
          org.archive.crawler.settings.Type
              org.archive.crawler.settings.ComplexType
                  org.archive.crawler.settings.ModuleType
                      org.archive.crawler.framework.Processor
                          org.archive.crawler.extractor.Extractor
                              org.archive.crawler.extractor.ExtractorHTML

All Implemented Interfaces:
  java.io.Serializable, javax.management.DynamicMBean, CoreAttributeConstants

Direct Known Subclasses:
  AggressiveExtractorHTML, JerichoExtractorHTML

    public class ExtractorHTML

extends Extractor
implements CoreAttributeConstants

Basic link-extraction, from an HTML content-body, using regular expressions.

Author:
  gojomo
See Also:
  Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType
`ComplexType.MBeanAttributeInfoIterator`

Field Summary
`(package private) static java.lang.String` | `APPLET`
`static java.lang.String` | `ATTR_EXTRACT_JAVASCRIPT` whether to try finding links in Javscript; default true
`static java.lang.String` | `ATTR_EXTRACT_ONLY_FORM_GETS`
`static java.lang.String` | `ATTR_IGNORE_FORM_ACTION_URLS`
`static java.lang.String` | `ATTR_IGNORE_UNEXPECTED_HTML`
`static java.lang.String` | `ATTR_TREAT_FRAMES_AS_EMBED_LINKS`
`(package private) static java.lang.String` | `BASE`
`(package private) static java.lang.String` | `CLASSEXT`
`(package private) static java.lang.String` | `EACH_ATTRIBUTE_EXTRACTOR`
`static java.lang.String` | `EXTRACT_VALUE_ATTRIBUTES`
`(package private) static java.lang.String` | `FRAME`
`(package private) static java.lang.String` | `IFRAME`
`(package private) static java.lang.String` | `JAVASCRIPT`
`(package private) static java.lang.String` | `LIKELY_URI_PATH`
`(package private) static java.lang.String` | `LINK`
`(package private) static int` | `MAX_ATTR_VAL_LENGTH`
`(package private) static java.lang.String` | `NON_HTML_PATH_EXTENSION`
`protected long` | `numberOfCURIsHandled`
`protected long` | `numberOfLinksExtracted`
`(package private) static java.lang.String` | `RELEVANT_TAG_EXTRACTOR`
`(package private) static java.lang.String` | `WHITESPACE`

Fields inherited from class org.archive.crawler.framework.Processor
`ATTR_DECIDE_RULES, ATTR_ENABLED, attrDecideRules`

Fields inherited from class org.archive.crawler.settings.ComplexType
`definition, definitionMap`

Fields inherited from interface org.archive.crawler.datamodel.CoreAttributeConstants
`A_ANNOTATIONS, A_CONTENT_DIGEST, A_CONTENT_TYPE, A_CREDENTIAL_AVATARS_KEY, A_DELAY_FACTOR, A_DISTANCE_FROM_SEED, A_DNS_FETCH_TIME, A_DNS_SERVER_IP_LABEL, A_ETAG_HEADER, A_FETCH_BEGAN_TIME, A_FETCH_COMPLETED_TIME, A_FETCH_HISTORY, A_FORCE_RETIRE, A_HERITABLE_KEYS, A_HTML_BASE, A_HTTP_BIND_ADDRESS, A_HTTP_PROXY_HOST, A_HTTP_PROXY_PORT, A_HTTP_TRANSACTION, A_LAST_MODIFIED_HEADER, A_LOCALIZED_ERRORS, A_META_ROBOTS, A_MINIMUM_DELAY, A_MIRROR_PATH, A_PREREQUISITE_URI, A_REFERENCE_LENGTH, A_RETRY_DELAY, A_RRECORD_SET_LABEL, A_RUNTIME_EXCEPTION, A_SOURCE_TAG, A_STATUS, HEADER_TRUNC, LENGTH_TRUNC, TIMER_TRUNC, TRUNC_SUFFIX`

Constructor Summary
`ExtractorHTML(java.lang.String name)`
`ExtractorHTML(java.lang.String name, java.lang.String description)`

Method Summary
`void` | `extract(CrawlURI curi)`
`(package private) void` | `extract(CrawlURI curi, java.lang.CharSequence cs)` Run extractor.
`protected boolean` | `isHtmlExpectedHere(CrawlURI curi)` Test whether this HTML is so unexpected (eg in place of a GIF URI) that it shouldn't be scanned for links.
`protected void` | `processEmbed(CrawlURI curi, java.lang.CharSequence value, java.lang.CharSequence context)`
`protected void` | `processEmbed(CrawlURI curi, java.lang.CharSequence value, java.lang.CharSequence context, char hopType)`
`protected void` | `processGeneralTag(CrawlURI curi, java.lang.CharSequence element, java.lang.CharSequence cs)`
`protected void` | `processLink(CrawlURI curi, java.lang.CharSequence value, java.lang.CharSequence context)` Handle generic HREF cases.
`protected boolean` | `processMeta(CrawlURI curi, java.lang.CharSequence cs)` Process metadata tags.
`protected void` | `processScript(CrawlURI curi, java.lang.CharSequence sequence, int endOfOpenTag)`
`protected void` | `processScriptCode(CrawlURI curi, java.lang.CharSequence cs)` Extract the (java)script source in the given CharSequence.
`protected void` | `processStyle(CrawlURI curi, java.lang.CharSequence sequence, int endOfOpenTag)` Process style text.
`java.lang.String` | `report()` Compiles and returns a report (in human readable form) about the status of the processor.

Methods inherited from class org.archive.crawler.extractor.Extractor
`innerProcess`

Methods inherited from class org.archive.crawler.framework.Processor
`checkForInterrupt, finalTasks, getController, getDecideRule, getDefaultNextProcessor, initialTasks, innerRejectProcess, isContentToProcess, isExpectedMimeType, isHttpTransactionContentToProcess, kickUpdate, process, rulesAccept, rulesAccept, setDefaultNextProcessor, spawn`

Methods inherited from class org.archive.crawler.settings.ModuleType
`addElement, listUsedFiles`

Methods inherited from class org.archive.crawler.settings.ComplexType
`addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute`

Methods inherited from class org.archive.crawler.settings.Type
`addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient`

Methods inherited from class javax.management.Attribute
`getName`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Field Detail

### RELEVANT_TAG_EXTRACTOR

    static final java.lang.String RELEVANT_TAG_EXTRACTOR

### MAX_ATTR_VAL_LENGTH

    static final int MAX_ATTR_VAL_LENGTH

### EACH_ATTRIBUTE_EXTRACTOR

    static final java.lang.String EACH_ATTRIBUTE_EXTRACTOR

### LIKELY_URI_PATH

    static final java.lang.String LIKELY_URI_PATH

See Also:
  Constant Field Values

### WHITESPACE

    static final java.lang.String WHITESPACE

See Also:
  Constant Field Values

### CLASSEXT

    static final java.lang.String CLASSEXT

See Also:
  Constant Field Values

### APPLET

    static final java.lang.String APPLET

See Also:
  Constant Field Values

### BASE

    static final java.lang.String BASE

See Also:
  Constant Field Values

### LINK

    static final java.lang.String LINK

See Also:
  Constant Field Values

### FRAME

    static final java.lang.String FRAME

See Also:
  Constant Field Values

### IFRAME

    static final java.lang.String IFRAME

See Also:
  Constant Field Values

### ATTR_TREAT_FRAMES_AS_EMBED_LINKS

    public static final java.lang.String ATTR_TREAT_FRAMES_AS_EMBED_LINKS

See Also:
  Constant Field Values

### ATTR_IGNORE_FORM_ACTION_URLS

    public static final java.lang.String ATTR_IGNORE_FORM_ACTION_URLS

See Also:
  Constant Field Values

### ATTR_EXTRACT_ONLY_FORM_GETS

    public static final java.lang.String ATTR_EXTRACT_ONLY_FORM_GETS

See Also:
  Constant Field Values

### ATTR_EXTRACT_JAVASCRIPT

    public static final java.lang.String ATTR_EXTRACT_JAVASCRIPT

  whether to try finding links in Javscript; default true

  See Also:
    Constant Field Values

### EXTRACT_VALUE_ATTRIBUTES

    public static final java.lang.String EXTRACT_VALUE_ATTRIBUTES

See Also:
  Constant Field Values

### ATTR_IGNORE_UNEXPECTED_HTML

    public static final java.lang.String ATTR_IGNORE_UNEXPECTED_HTML

See Also:
  Constant Field Values

### numberOfCURIsHandled

    protected long numberOfCURIsHandled

### numberOfLinksExtracted

    protected long numberOfLinksExtracted

### JAVASCRIPT

    static final java.lang.String JAVASCRIPT

See Also:
  Constant Field Values

### NON_HTML_PATH_EXTENSION

    static final java.lang.String NON_HTML_PATH_EXTENSION

See Also:
  Constant Field Values

Constructor Detail

### ExtractorHTML

    public ExtractorHTML(java.lang.String name)

### ExtractorHTML

    public ExtractorHTML(java.lang.String name,
                         java.lang.String description)

Method Detail

### processGeneralTag

    protected void processGeneralTag(CrawlURI curi,
                                     java.lang.CharSequence element,
                                     java.lang.CharSequence cs)

### processScriptCode

    protected void processScriptCode(CrawlURI curi,
                                     java.lang.CharSequence cs)

  Extract the (java)script source in the given CharSequence.

  Parameters:
    `curi` - source CrawlURI
    `cs` - CharSequence of javascript code

### processLink

    protected void processLink(CrawlURI curi,
                               java.lang.CharSequence value,
                               java.lang.CharSequence context)

  Handle generic HREF cases.

  Parameters:
    `curi` -
    `value` -
    `context` -

### processEmbed

    protected final void processEmbed(CrawlURI curi,
                                      java.lang.CharSequence value,
                                      java.lang.CharSequence context)

### processEmbed

    protected void processEmbed(CrawlURI curi,
                                java.lang.CharSequence value,
                                java.lang.CharSequence context,
                                char hopType)

### extract

    public void extract(CrawlURI curi)

  Specified by:
    `extract` in class `Extractor`

### extract

    void extract(CrawlURI curi,
                 java.lang.CharSequence cs)

  Run extractor. This method is package visible to ease testing.

  Parameters:
    `curi` - CrawlURI we're processing.
    `cs` - Sequence from underlying ReplayCharSequence. This is TRANSIENT data. Make a copy if you want the data to live outside of this extractors' lifetime.

### isHtmlExpectedHere

    protected boolean isHtmlExpectedHere(CrawlURI curi)
                                  throws org.apache.commons.httpclient.URIException

  Test whether this HTML is so unexpected (eg in place of a GIF URI) that it shouldn't be scanned for links.

  Parameters:
    `curi` - CrawlURI to examine.
  Returns:
    True if HTML is acceptable/expected here
  Throws:
    `org.apache.commons.httpclient.URIException`

### processScript

    protected void processScript(CrawlURI curi,
                                 java.lang.CharSequence sequence,
                                 int endOfOpenTag)

### processMeta

    protected boolean processMeta(CrawlURI curi,
                                  java.lang.CharSequence cs)

  Process metadata tags.

  Parameters:
    `curi` - CrawlURI we're processing.
    `cs` - Sequence from underlying ReplayCharSequence. This is TRANSIENT data. Make a copy if you want the data to live outside of this extractors' lifetime.
  Returns:
    True robots exclusion metatag.

### processStyle

    protected void processStyle(CrawlURI curi,
                                java.lang.CharSequence sequence,
                                int endOfOpenTag)

  Process style text.

  Parameters:
    `curi` - CrawlURI we're processing.
    `sequence` - Sequence from underlying ReplayCharSequence. This is TRANSIENT data. Make a copy if you want the data to live outside of this extractors' lifetime.
    `endOfOpenTag` -

### report

    public java.lang.String report()

  Description copied from class: `Processor`
  Compiles and returns a report (in human readable form) about the status of the processor. The processor's name (of implementing class) should always be included.

  Examples of stats declared would include:
  * Number of CrawlURIs handled.
  * Number of links extracted (for link extractors)
  etc.

  Overrides:
    `report` in class `Processor`
  Returns:
    A human readable report on the processor's state.

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.