Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.prefetch
Class QuotaEnforcer

    java.lang.Object
      javax.management.Attribute
          org.archive.crawler.settings.Type
              org.archive.crawler.settings.ComplexType
                  org.archive.crawler.settings.ModuleType
                      org.archive.crawler.framework.Processor
                          org.archive.crawler.prefetch.QuotaEnforcer

All Implemented Interfaces:
  java.io.Serializable, javax.management.DynamicMBean, FetchStatusCodes

    public class QuotaEnforcer

extends Processor
implements FetchStatusCodes

A simple quota enforcer. If the host, server, or frontier group associated with the current CrawlURI is already over its quotas, blocks the current URI's processing with S_BLOCKED_BY_QUOTA.

Version:
  $Date: 2007-04-06 00:40:50 +0000 (Fri, 06 Apr 2007) $, $Revision: 5040 $
Author:
  gojomo
See Also:
  Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType
`ComplexType.MBeanAttributeInfoIterator`

Field Summary
`protected static java.lang.String` | `ATTR_FORCE_RETIRE` whether to force-retire when over-quote detected
`protected static java.lang.String` | `ATTR_GROUP_MAX_ALL_KB` group max all fetch bytes (including error responses)
`protected static java.lang.String` | `ATTR_GROUP_MAX_FETCH_RESPONSES` group max fetch responses (including error codes)
`protected static java.lang.String` | `ATTR_GROUP_MAX_FETCH_SUCCESSES` group max successful fetches
`protected static java.lang.String` | `ATTR_GROUP_MAX_SUCCESS_KB` group max successful fetch bytes
`protected static java.lang.String` | `ATTR_HOST_MAX_ALL_KB` host max all fetch bytes (including error responses)
`protected static java.lang.String` | `ATTR_HOST_MAX_FETCH_RESPONSES` host max fetch responses (including error codes)
`protected static java.lang.String` | `ATTR_HOST_MAX_FETCH_SUCCESSES` host max successful fetches
`protected static java.lang.String` | `ATTR_HOST_MAX_SUCCESS_KB` host max successful fetch bytes
`protected static java.lang.String` | `ATTR_SERVER_MAX_ALL_KB` server max all fetch bytes (including error responses)
`protected static java.lang.String` | `ATTR_SERVER_MAX_FETCH_RESPONSES` server max fetch responses (including error codes)
`protected static java.lang.String` | `ATTR_SERVER_MAX_FETCH_SUCCESSES` server max successful fetches
`protected static java.lang.String` | `ATTR_SERVER_MAX_SUCCESS_KB` server max successful fetch bytes
`protected static java.lang.Boolean` | `DEFAULT_FORCE_RETIRE`
`protected static java.lang.Long` | `DEFAULT_GROUP_MAX_ALL_KB`
`protected static java.lang.Long` | `DEFAULT_GROUP_MAX_FETCH_RESPONSES`
`protected static java.lang.Long` | `DEFAULT_GROUP_MAX_FETCH_SUCCESSES`
`protected static java.lang.Long` | `DEFAULT_GROUP_MAX_SUCCESS_KB`
`protected static java.lang.Long` | `DEFAULT_HOST_MAX_ALL_KB`
`protected static java.lang.Long` | `DEFAULT_HOST_MAX_FETCH_RESPONSES`
`protected static java.lang.Long` | `DEFAULT_HOST_MAX_FETCH_SUCCESSES`
`protected static java.lang.Long` | `DEFAULT_HOST_MAX_SUCCESS_KB`
`protected static java.lang.Long` | `DEFAULT_SERVER_MAX_ALL_KB`
`protected static java.lang.Long` | `DEFAULT_SERVER_MAX_FETCH_RESPONSES`
`protected static java.lang.Long` | `DEFAULT_SERVER_MAX_FETCH_SUCCESSES`
`protected static java.lang.Long` | `DEFAULT_SERVER_MAX_SUCCESS_KB`
`protected static int` | `GROUP`
`protected static int` | `HOST`
`protected static java.lang.String[][]` | `keys`
`protected static int` | `NAME`
`protected static int` | `RESPONSE_KB`
`protected static int` | `RESPONSES`
`protected static int` | `SERVER`
`protected static int` | `SUCCESS_KB`
`protected static int` | `SUCCESSES`

Fields inherited from class org.archive.crawler.framework.Processor
`ATTR_DECIDE_RULES, ATTR_ENABLED, attrDecideRules`

Fields inherited from class org.archive.crawler.settings.ComplexType
`definition, definitionMap`

Fields inherited from interface org.archive.crawler.datamodel.FetchStatusCodes
`S_BLOCKED_BY_CUSTOM_PROCESSOR, S_BLOCKED_BY_QUOTA, S_BLOCKED_BY_RUNTIME_LIMIT, S_BLOCKED_BY_USER, S_CONNECT_FAILED, S_CONNECT_LOST, S_DEEMED_CHAFF, S_DEFERRED, S_DELETED_BY_USER, S_DNS_SUCCESS, S_DOMAIN_PREREQUISITE_FAILURE, S_DOMAIN_UNRESOLVABLE, S_GETBYNAME_SUCCESS, S_OTHER_PREREQUISITE_FAILURE, S_OUT_OF_SCOPE, S_PREREQUISITE_UNSCHEDULABLE_FAILURE, S_PROCESSING_THREAD_KILLED, S_ROBOTS_PRECLUDED, S_ROBOTS_PREREQUISITE_FAILURE, S_RUNTIME_EXCEPTION, S_SERIOUS_ERROR, S_TIMEOUT, S_TOO_MANY_EMBED_HOPS, S_TOO_MANY_LINK_HOPS, S_TOO_MANY_RETRIES, S_UNATTEMPTED, S_UNFETCHABLE_URI, S_UNQUEUEABLE`

Constructor Summary
`QuotaEnforcer(java.lang.String name)` Constructor.

Method Summary
`protected boolean` | `applyQuota(CrawlURI curi, java.lang.String quotaKey, long actual)` Apply the quota specified by the given key against the actual value provided.
`protected boolean` | `checkQuotas(CrawlURI curi, CrawlSubstats.HasCrawlSubstats hasStats, int CAT)` Check all quotas for the given substats and category (server, host, or group).
`protected void` | `innerProcess(CrawlURI curi)` Classes subclassing this one should override this method to perform their custom actions on the CrawlURI.

Methods inherited from class org.archive.crawler.framework.Processor
`checkForInterrupt, finalTasks, getController, getDecideRule, getDefaultNextProcessor, initialTasks, innerRejectProcess, isContentToProcess, isExpectedMimeType, isHttpTransactionContentToProcess, kickUpdate, process, report, rulesAccept, rulesAccept, setDefaultNextProcessor, spawn`

Methods inherited from class org.archive.crawler.settings.ModuleType
`addElement, listUsedFiles`

Methods inherited from class org.archive.crawler.settings.ComplexType
`addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute`

Methods inherited from class org.archive.crawler.settings.Type
`addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient`

Methods inherited from class javax.management.Attribute
`getName`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Field Detail

### SERVER

    protected static final int SERVER

See Also:
  Constant Field Values

### HOST

    protected static final int HOST

See Also:
  Constant Field Values

### GROUP

    protected static final int GROUP

See Also:
  Constant Field Values

### NAME

    protected static final int NAME

See Also:
  Constant Field Values

### SUCCESSES

    protected static final int SUCCESSES

See Also:
  Constant Field Values

### SUCCESS_KB

    protected static final int SUCCESS_KB

See Also:
  Constant Field Values

### RESPONSES

    protected static final int RESPONSES

See Also:
  Constant Field Values

### RESPONSE_KB

    protected static final int RESPONSE_KB

See Also:
  Constant Field Values

### keys

    protected static final java.lang.String[][] keys

### ATTR_SERVER_MAX_FETCH_SUCCESSES

    protected static final java.lang.String ATTR_SERVER_MAX_FETCH_SUCCESSES

  server max successful fetches

### DEFAULT_SERVER_MAX_FETCH_SUCCESSES

    protected static final java.lang.Long DEFAULT_SERVER_MAX_FETCH_SUCCESSES

### ATTR_SERVER_MAX_SUCCESS_KB

    protected static final java.lang.String ATTR_SERVER_MAX_SUCCESS_KB

  server max successful fetch bytes

### DEFAULT_SERVER_MAX_SUCCESS_KB

    protected static final java.lang.Long DEFAULT_SERVER_MAX_SUCCESS_KB

### ATTR_SERVER_MAX_FETCH_RESPONSES

    protected static final java.lang.String ATTR_SERVER_MAX_FETCH_RESPONSES

  server max fetch responses (including error codes)

### DEFAULT_SERVER_MAX_FETCH_RESPONSES

    protected static final java.lang.Long DEFAULT_SERVER_MAX_FETCH_RESPONSES

### ATTR_SERVER_MAX_ALL_KB

    protected static final java.lang.String ATTR_SERVER_MAX_ALL_KB

  server max all fetch bytes (including error responses)

### DEFAULT_SERVER_MAX_ALL_KB

    protected static final java.lang.Long DEFAULT_SERVER_MAX_ALL_KB

### ATTR_HOST_MAX_FETCH_SUCCESSES

    protected static final java.lang.String ATTR_HOST_MAX_FETCH_SUCCESSES

  host max successful fetches

### DEFAULT_HOST_MAX_FETCH_SUCCESSES

    protected static final java.lang.Long DEFAULT_HOST_MAX_FETCH_SUCCESSES

### ATTR_HOST_MAX_SUCCESS_KB

    protected static final java.lang.String ATTR_HOST_MAX_SUCCESS_KB

  host max successful fetch bytes

### DEFAULT_HOST_MAX_SUCCESS_KB

    protected static final java.lang.Long DEFAULT_HOST_MAX_SUCCESS_KB

### ATTR_HOST_MAX_FETCH_RESPONSES

    protected static final java.lang.String ATTR_HOST_MAX_FETCH_RESPONSES

  host max fetch responses (including error codes)

### DEFAULT_HOST_MAX_FETCH_RESPONSES

    protected static final java.lang.Long DEFAULT_HOST_MAX_FETCH_RESPONSES

### ATTR_HOST_MAX_ALL_KB

    protected static final java.lang.String ATTR_HOST_MAX_ALL_KB

  host max all fetch bytes (including error responses)

### DEFAULT_HOST_MAX_ALL_KB

    protected static final java.lang.Long DEFAULT_HOST_MAX_ALL_KB

### ATTR_GROUP_MAX_FETCH_SUCCESSES

    protected static final java.lang.String ATTR_GROUP_MAX_FETCH_SUCCESSES

  group max successful fetches

### DEFAULT_GROUP_MAX_FETCH_SUCCESSES

    protected static final java.lang.Long DEFAULT_GROUP_MAX_FETCH_SUCCESSES

### ATTR_GROUP_MAX_SUCCESS_KB

    protected static final java.lang.String ATTR_GROUP_MAX_SUCCESS_KB

  group max successful fetch bytes

### DEFAULT_GROUP_MAX_SUCCESS_KB

    protected static final java.lang.Long DEFAULT_GROUP_MAX_SUCCESS_KB

### ATTR_GROUP_MAX_FETCH_RESPONSES

    protected static final java.lang.String ATTR_GROUP_MAX_FETCH_RESPONSES

  group max fetch responses (including error codes)

### DEFAULT_GROUP_MAX_FETCH_RESPONSES

    protected static final java.lang.Long DEFAULT_GROUP_MAX_FETCH_RESPONSES

### ATTR_GROUP_MAX_ALL_KB

    protected static final java.lang.String ATTR_GROUP_MAX_ALL_KB

  group max all fetch bytes (including error responses)

### DEFAULT_GROUP_MAX_ALL_KB

    protected static final java.lang.Long DEFAULT_GROUP_MAX_ALL_KB

### ATTR_FORCE_RETIRE

    protected static final java.lang.String ATTR_FORCE_RETIRE

  whether to force-retire when over-quote detected

  See Also:
    Constant Field Values

### DEFAULT_FORCE_RETIRE

    protected static final java.lang.Boolean DEFAULT_FORCE_RETIRE

Constructor Detail

### QuotaEnforcer

    public QuotaEnforcer(java.lang.String name)

  Constructor.

  Parameters:
    `name` - Name of this processor.

Method Detail

### innerProcess

    protected void innerProcess(CrawlURI curi)

  Description copied from class: `Processor`
  Classes subclassing this one should override this method to perform their custom actions on the CrawlURI.

  Overrides:
    `innerProcess` in class `Processor`
  Parameters:
    `curi` - The CrawlURI being processed.

### checkQuotas

    protected boolean checkQuotas(CrawlURI curi,
                                  CrawlSubstats.HasCrawlSubstats hasStats,
                                  int CAT)

  Check all quotas for the given substats and category (server, host, or group).

  Parameters:
    `curi` - CrawlURI to mark up with results
    `hasStats` - holds CrawlSubstats with actual values to test
    `CAT` - category index (SERVER, HOST, GROUP) to quota settings keys
  Returns:
    true if quota precludes fetching of CrawlURI

### applyQuota

    protected boolean applyQuota(CrawlURI curi,
                                 java.lang.String quotaKey,
                                 long actual)

  Apply the quota specified by the given key against the actual value provided. If the quota and actual values rule out processing the given CrawlURI, mark up the CrawlURI appropriately.

  Parameters:
    `curi` - CrawlURI whose processing is subject to a potential quota limitation
    `quotaKey` - settings key to get applicable quota
    `actual` - current value to compare to quota
  Returns:
    true is CrawlURI is blocked by a quota, false otherwise

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.