Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.framework
Class WriterPoolProcessor

    java.lang.Object
      javax.management.Attribute
          org.archive.crawler.settings.Type
              org.archive.crawler.settings.ComplexType
                  org.archive.crawler.settings.ModuleType
                      org.archive.crawler.framework.Processor
                          org.archive.crawler.framework.WriterPoolProcessor

All Implemented Interfaces:
  java.io.Serializable, javax.management.DynamicMBean, CoreAttributeConstants, FetchStatusCodes, CrawlStatusListener

Direct Known Subclasses:
  ARCWriterProcessor, WARCWriterProcessor

    public abstract class WriterPoolProcessor

extends Processor
implements CoreAttributeConstants, CrawlStatusListener, FetchStatusCodes

Abstract implementation of a file pool processor. Subclass to implement for a particular `WriterPoolMember` instance.

Author:
  Parker Thompson, stack
See Also:
  Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType
`ComplexType.MBeanAttributeInfoIterator`

Field Summary
`protected static java.lang.String` | `ANNOTATION_UNWRITTEN` CrawlURI annotation indicating no record was written
`static java.lang.String` | `ATTR_COMPRESS` Key to use asking settings for file compression value.
`static java.lang.String` | `ATTR_MAX_BYTES_WRITTEN` Key for the maximum bytes to write attribute.
`static java.lang.String` | `ATTR_MAX_SIZE_BYTES` Key to use asking settings for file max size value.
`static java.lang.String` | `ATTR_PATH` Key to use asking settings for arc path value.
`static java.lang.String` | `ATTR_POOL_MAX_ACTIVE` Key to get maximum pool size.
`static java.lang.String` | `ATTR_POOL_MAX_WAIT` Key to get maximum wait on pool object before we give up and throw IOException.
`static java.lang.String` | `ATTR_PREFIX` Key to use asking settings for file prefix value.
`static java.lang.String` | `ATTR_SKIP_IDENTICAL_DIGESTS` Key for whether to skip writing records of content-digest repeats
`static java.lang.String` | `ATTR_SUFFIX` Key to use asking settings for file suffix value.
`static boolean` | `DEFAULT_COMPRESS` Default as to whether we do compression of files.

Fields inherited from class org.archive.crawler.framework.Processor
`ATTR_DECIDE_RULES, ATTR_ENABLED, attrDecideRules`

Fields inherited from class org.archive.crawler.settings.ComplexType
`definition, definitionMap`

Fields inherited from interface org.archive.crawler.datamodel.CoreAttributeConstants
`A_ANNOTATIONS, A_CONTENT_DIGEST, A_CONTENT_TYPE, A_CREDENTIAL_AVATARS_KEY, A_DELAY_FACTOR, A_DISTANCE_FROM_SEED, A_DNS_FETCH_TIME, A_DNS_SERVER_IP_LABEL, A_ETAG_HEADER, A_FETCH_BEGAN_TIME, A_FETCH_COMPLETED_TIME, A_FETCH_HISTORY, A_FORCE_RETIRE, A_HERITABLE_KEYS, A_HTML_BASE, A_HTTP_BIND_ADDRESS, A_HTTP_PROXY_HOST, A_HTTP_PROXY_PORT, A_HTTP_TRANSACTION, A_LAST_MODIFIED_HEADER, A_LOCALIZED_ERRORS, A_META_ROBOTS, A_MINIMUM_DELAY, A_MIRROR_PATH, A_PREREQUISITE_URI, A_REFERENCE_LENGTH, A_RETRY_DELAY, A_RRECORD_SET_LABEL, A_RUNTIME_EXCEPTION, A_SOURCE_TAG, A_STATUS, HEADER_TRUNC, LENGTH_TRUNC, TIMER_TRUNC, TRUNC_SUFFIX`

Fields inherited from interface org.archive.crawler.datamodel.FetchStatusCodes
`S_BLOCKED_BY_CUSTOM_PROCESSOR, S_BLOCKED_BY_QUOTA, S_BLOCKED_BY_RUNTIME_LIMIT, S_BLOCKED_BY_USER, S_CONNECT_FAILED, S_CONNECT_LOST, S_DEEMED_CHAFF, S_DEFERRED, S_DELETED_BY_USER, S_DNS_SUCCESS, S_DOMAIN_PREREQUISITE_FAILURE, S_DOMAIN_UNRESOLVABLE, S_GETBYNAME_SUCCESS, S_OTHER_PREREQUISITE_FAILURE, S_OUT_OF_SCOPE, S_PREREQUISITE_UNSCHEDULABLE_FAILURE, S_PROCESSING_THREAD_KILLED, S_ROBOTS_PRECLUDED, S_ROBOTS_PREREQUISITE_FAILURE, S_RUNTIME_EXCEPTION, S_SERIOUS_ERROR, S_TIMEOUT, S_TOO_MANY_EMBED_HOPS, S_TOO_MANY_LINK_HOPS, S_TOO_MANY_RETRIES, S_UNATTEMPTED, S_UNFETCHABLE_URI, S_UNQUEUEABLE`

Constructor Summary
`WriterPoolProcessor(java.lang.String name)`
`WriterPoolProcessor(java.lang.String name, java.lang.String description)`

Method Summary
`protected java.util.List<java.lang.String>` | `cacheMetadata()`
`protected void` | `checkBytesWritten()`
`protected void` | `checkpointRecover()` Called out of `initialTasks()` when recovering a checkpoint.
`void` | `crawlCheckpoint(java.io.File checkpointDir)` Called by `CrawlController` when checkpointing.
`void` | `crawlEnded(java.lang.String sExitMessage)` Called when a CrawlController has ended a crawl and is about to exit.
`void` | `crawlEnding(java.lang.String sExitMessage)` Called when a CrawlController is ending a crawl (for any reason)
`void` | `crawlPaused(java.lang.String statusMessage)` Called when a CrawlController is actually paused (all threads are idle).
`void` | `crawlPausing(java.lang.String statusMessage)` Called when a CrawlController is going to be paused.
`void` | `crawlResuming(java.lang.String statusMessage)` Called when a CrawlController is resuming a crawl that had been paused.
`void` | `crawlStarted(java.lang.String message)` Called on crawl start.
`java.lang.Object` | `getAttributeUnchecked(java.lang.String name)` Version of getAttributes that catches and logs exceptions and returns null if failure to fetch the attribute.
`protected java.lang.String` | `getCheckpointStateFile()`
`protected java.lang.String[]` | `getDefaultPath()`
`protected java.lang.String` | `getFirstrecordBody(java.io.File orderFile)` Write the arc metadata body content.
`protected java.lang.String` | `getFirstrecordStylesheet()`
`protected java.lang.String` | `getHostAddress(CrawlURI curi)` Return IP address of given URI suitable for recording (as in a classic ARC 5-field header line).
`long` | `getMaxSize()` Max size we want files to be (bytes).
`long` | `getMaxToWrite()`
`java.util.List<java.lang.String>` | `getMetadata()` Return list of metadatas to add to first arc file metadata record.
`java.util.List<java.io.File>` | `getOutputDirs()`
`protected WriterPool` | `getPool()`
`int` | `getPoolMaximumActive()`
`int` | `getPoolMaximumWait()`
`java.lang.String` | `getPrefix()`
`protected java.util.concurrent.atomic.AtomicInteger` | `getSerialNo()`
`java.lang.String` | `getSuffix()`
`protected long` | `getTotalBytesWritten()`
`void` | `initialTasks()` Classes subclassing this one should override this method to perform processor specific actions.
`protected abstract void` | `innerProcess(CrawlURI curi)` Writes a CrawlURI and its associated data to store file.
`boolean` | `isCompressed()`
`protected int` | `loadCheckpointSerialNumber()`
`protected void` | `saveCheckpointSerialNumber(java.io.File checkpointDir, int serialNo)`
`protected void` | `setPool(WriterPool pool)`
`protected void` | `setTotalBytesWritten(long totalBytesWritten)`
`protected abstract void` | `setupPool(java.util.concurrent.atomic.AtomicInteger serialNo)` Set up pool of files.
`protected boolean` | `shouldWrite(CrawlURI curi)` Whether the given CrawlURI should be written to archive files.

Methods inherited from class org.archive.crawler.framework.Processor
`checkForInterrupt, finalTasks, getController, getDecideRule, getDefaultNextProcessor, innerRejectProcess, isContentToProcess, isExpectedMimeType, isHttpTransactionContentToProcess, kickUpdate, process, report, rulesAccept, rulesAccept, setDefaultNextProcessor, spawn`

Methods inherited from class org.archive.crawler.settings.ModuleType
`addElement, listUsedFiles`

Methods inherited from class org.archive.crawler.settings.ComplexType
`addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute`

Methods inherited from class org.archive.crawler.settings.Type
`addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient`

Methods inherited from class javax.management.Attribute
`getName`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Field Detail

### ATTR_COMPRESS

    public static final java.lang.String ATTR_COMPRESS

  Key to use asking settings for file compression value.

  See Also:
    Constant Field Values

### DEFAULT_COMPRESS

    public static final boolean DEFAULT_COMPRESS

  Default as to whether we do compression of files.

  See Also:
    Constant Field Values

### ATTR_PREFIX

    public static final java.lang.String ATTR_PREFIX

  Key to use asking settings for file prefix value.

  See Also:
    Constant Field Values

### ATTR_PATH

    public static final java.lang.String ATTR_PATH

  Key to use asking settings for arc path value.

  See Also:
    Constant Field Values

### ATTR_SUFFIX

    public static final java.lang.String ATTR_SUFFIX

  Key to use asking settings for file suffix value.

  See Also:
    Constant Field Values

### ATTR_MAX_SIZE_BYTES

    public static final java.lang.String ATTR_MAX_SIZE_BYTES

  Key to use asking settings for file max size value.

  See Also:
    Constant Field Values

### ATTR_POOL_MAX_ACTIVE

    public static final java.lang.String ATTR_POOL_MAX_ACTIVE

  Key to get maximum pool size. This key is for maximum files active in the pool.

  See Also:
    Constant Field Values

### ATTR_POOL_MAX_WAIT

    public static final java.lang.String ATTR_POOL_MAX_WAIT

  Key to get maximum wait on pool object before we give up and throw IOException.

  See Also:
    Constant Field Values

### ATTR_MAX_BYTES_WRITTEN

    public static final java.lang.String ATTR_MAX_BYTES_WRITTEN

  Key for the maximum bytes to write attribute.

  See Also:
    Constant Field Values

### ATTR_SKIP_IDENTICAL_DIGESTS

    public static final java.lang.String ATTR_SKIP_IDENTICAL_DIGESTS

  Key for whether to skip writing records of content-digest repeats

  See Also:
    Constant Field Values

### ANNOTATION_UNWRITTEN

    protected static final java.lang.String ANNOTATION_UNWRITTEN

  CrawlURI annotation indicating no record was written

  See Also:
    Constant Field Values

Constructor Detail

### WriterPoolProcessor

    public WriterPoolProcessor(java.lang.String name)

Parameters:
  `name` - Name of this processor.

### WriterPoolProcessor

    public WriterPoolProcessor(java.lang.String name,
                               java.lang.String description)

Parameters:
  `name` - Name of this processor.
  `description` - Description for this processor.

Method Detail

### getDefaultPath

    protected java.lang.String[] getDefaultPath()

### initialTasks

    public void initialTasks()

  Description copied from class: `Processor`
  Classes subclassing this one should override this method to perform processor specific actions.

  This method is garanteed to be called after the crawl is set up, but before any URI-processing has occured.

  Overrides:
    `initialTasks` in class `Processor`

### getSerialNo

    protected java.util.concurrent.atomic.AtomicInteger getSerialNo()

### setupPool

    protected abstract void setupPool(java.util.concurrent.atomic.AtomicInteger serialNo)

  Set up pool of files.

### innerProcess

    protected abstract void innerProcess(CrawlURI curi)

  Writes a CrawlURI and its associated data to store file. Currently this method understands the following uri types: dns, http, and https.

  Overrides:
    `innerProcess` in class `Processor`
  Parameters:
    `curi` - CrawlURI to process.

### checkBytesWritten

    protected void checkBytesWritten()

### shouldWrite

    protected boolean shouldWrite(CrawlURI curi)

  Whether the given CrawlURI should be written to archive files. Annotates CrawlURI with a reason for any negative answer.

  Parameters:
    `curi` - CrawlURI
  Returns:
    true if URI should be written; false otherwise

### getHostAddress

    protected java.lang.String getHostAddress(CrawlURI curi)

  Return IP address of given URI suitable for recording (as in a classic ARC 5-field header line).

  Parameters:
    `curi` - CrawlURI
  Returns:
    String of IP address

### getAttributeUnchecked

    public java.lang.Object getAttributeUnchecked(java.lang.String name)

  Version of getAttributes that catches and logs exceptions and returns null if failure to fetch the attribute.

  Parameters:
    `name` - Attribute name.
  Returns:
    Attribute or null.

### getMaxSize

    public long getMaxSize()

  Max size we want files to be (bytes). Default is ARCConstants.DEFAULT_MAX_ARC_FILE_SIZE. Note that ARC files will usually be bigger than maxSize; they'll be maxSize + length to next boundary.

  Returns:
    ARC maximum size.

### getPrefix

    public java.lang.String getPrefix()

### getOutputDirs

    public java.util.List<java.io.File> getOutputDirs()

### isCompressed

    public boolean isCompressed()

### getPoolMaximumActive

    public int getPoolMaximumActive()

  Returns:
    Returns the poolMaximumActive.

### getPoolMaximumWait

    public int getPoolMaximumWait()

  Returns:
    Returns the poolMaximumWait.

### getSuffix

    public java.lang.String getSuffix()

### getMaxToWrite

    public long getMaxToWrite()

### crawlEnding

    public void crawlEnding(java.lang.String sExitMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController is ending a crawl (for any reason)

  Specified by:
    `crawlEnding` in interface `CrawlStatusListener`
  Parameters:
    `sExitMessage` - Type of exit. Should be one of the STATUS constants in defined in CrawlJob.
  See Also:
    `CrawlJob`

### crawlEnded

    public void crawlEnded(java.lang.String sExitMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController has ended a crawl and is about to exit.

  Specified by:
    `crawlEnded` in interface `CrawlStatusListener`
  Parameters:
    `sExitMessage` - Type of exit. Should be one of the STATUS constants in defined in CrawlJob.
  See Also:
    `CrawlJob`

### crawlStarted

    public void crawlStarted(java.lang.String message)

  Description copied from interface: `CrawlStatusListener`
  Called on crawl start.

  Specified by:
    `crawlStarted` in interface `CrawlStatusListener`
  Parameters:
    `message` - Start message.

### getCheckpointStateFile

    protected java.lang.String getCheckpointStateFile()

### crawlCheckpoint

    public void crawlCheckpoint(java.io.File checkpointDir)
                         throws java.io.IOException

  Description copied from interface: `CrawlStatusListener`
  Called by `CrawlController` when checkpointing.

  Specified by:
    `crawlCheckpoint` in interface `CrawlStatusListener`
  Parameters:
    `checkpointDir` - Checkpoint dir. Write checkpoint state here.
  Throws:
    `java.io.IOException`

### crawlPausing

    public void crawlPausing(java.lang.String statusMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController is going to be paused.

  Specified by:
    `crawlPausing` in interface `CrawlStatusListener`
  Parameters:
    `statusMessage` - Should be `STATUS_WAITING_FOR_PAUSE`. Passed for convenience

### crawlPaused

    public void crawlPaused(java.lang.String statusMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController is actually paused (all threads are idle).

  Specified by:
    `crawlPaused` in interface `CrawlStatusListener`
  Parameters:
    `statusMessage` - Should be `CrawlJob.STATUS_PAUSED`. Passed for convenience

### crawlResuming

    public void crawlResuming(java.lang.String statusMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController is resuming a crawl that had been paused.

  Specified by:
    `crawlResuming` in interface `CrawlStatusListener`
  Parameters:
    `statusMessage` - Should be `CrawlJob.STATUS_RUNNING`. Passed for convenience

### getPool

    protected WriterPool getPool()

### setPool

    protected void setPool(WriterPool pool)

### getTotalBytesWritten

    protected long getTotalBytesWritten()

### setTotalBytesWritten

    protected void setTotalBytesWritten(long totalBytesWritten)

### checkpointRecover

    protected void checkpointRecover()

  Called out of `initialTasks()` when recovering a checkpoint. Restore state.

### loadCheckpointSerialNumber

    protected int loadCheckpointSerialNumber()

  Returns:
    Serial number from checkpoint state file or if unreadable, -1 (Client should check for -1).

### saveCheckpointSerialNumber

    protected void saveCheckpointSerialNumber(java.io.File checkpointDir,
                                              int serialNo)
                                       throws java.io.IOException

  Throws:
    `java.io.IOException`

### getMetadata

    public java.util.List<java.lang.String> getMetadata()

  Return list of metadatas to add to first arc file metadata record. Default is to stylesheet the order file. To specify stylesheet, override `getFirstrecordStylesheet()`. Get xml files from settingshandler. Currently order file is the only xml file. We're NOT adding seeds to meta data.

  Returns:
    List of strings and/or files to add to arc file as metadata or null.

### cacheMetadata

    protected java.util.List<java.lang.String> cacheMetadata()

### getFirstrecordStylesheet

    protected java.lang.String getFirstrecordStylesheet()

### getFirstrecordBody

    protected java.lang.String getFirstrecordBody(java.io.File orderFile)

  Write the arc metadata body content. Its based on the order xml file but into this base we'll add other info such as machine ip.

  Parameters:
    `orderFile` - Order file.
  Returns:
    String that holds the arc metaheader body.

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.