Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.admin
Class StatisticsSummary

    java.lang.Object
      org.archive.crawler.admin.StatisticsSummary

    public class StatisticsSummary

extends java.lang.Object

This class provides descriptive statistics of a finished crawl job by using the crawl report files generated by StatisticsTracker. Any formatting changes to the way StatisticsTracker writes to the summary crawl reports will require changes to this class.

The following statistics are accessible from this class:

- Successfully downloaded documents per fetch status code
- Successfully downloaded documents per document mime type
- Amount of data per mime type
- Successfully downloaded documents per host
- Amount of data per host
- Successfully downloaded documents per top-level domain name (TLD)
- Disposition of all seeds
- Successfully downloaded documents per host per source

TODO: Make it so summarizing is not done all in RAM so we avoid OOME.

Author:
  Frank McCown
See Also:
  `StatisticsTracker`

Field Summary
`protected java.lang.String` | `bandwidthKbytesPerSec`
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `dnsStatusCodeDistribution`
`protected java.lang.String` | `durationTime`
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `hostsBytes`
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `hostsDistribution` Keep track of hosts
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `hostsDnsBytes`
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `hostsDnsDistribution`
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `mimeTypeBytes`
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `mimeTypeDistribution` Keep track of the file types we see (mime type -> count)
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `mimeTypeDnsBytes`
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `mimeTypeDnsDistribution`
`protected java.lang.String` | `processedDocsPerSec`
`protected java.util.Map<java.lang.String,SeedRecord>` | `processedSeedsRecords` Keep track of processed seeds
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `statusCodeDistribution` Keep track of status codes
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `tldBytes`
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `tldDistribution` Keep track of TLDs
`protected java.util.Hashtable<java.lang.String,LongWrapper>` | `tldHostDistribution`
`protected java.lang.String` | `totalDataWritten`
`protected long` | `totalDnsHostDocuments`
`protected long` | `totalDnsHostSize`
`protected long` | `totalDnsMimeSize`
`protected long` | `totalDnsMimeTypeDocuments`
`protected long` | `totalDnsStatusCodeDocuments`
`protected long` | `totalFileTypeDocuments`
`protected long` | `totalHostDocuments`
`protected long` | `totalHosts`
`protected long` | `totalHostSize`
`protected long` | `totalMimeSize`
`protected long` | `totalMimeTypeDocuments`
`protected long` | `totalStatusCodeDocuments`
`protected long` | `totalTldDocuments`
`protected long` | `totalTldSize`

Constructor Summary
`StatisticsSummary(CrawlJob cjob)` Constructor

Method Summary
`java.lang.String` | `getBandwidthKbytesPerSec()`
`long` | `getBytesPerHost(java.lang.String host)` Returns the accumulated number of bytes downloaded from a given host.
`long` | `getBytesPerMimeType(java.lang.String filetype)` Returns the accumulated number of bytes from files of a given file type.
`long` | `getBytesPerTld(java.lang.String tld)` Returns the total number of bytes downloaded for a given TLD.
`java.util.Hashtable` | `getDnsMimeDistribution()`
`java.util.Hashtable` | `getDnsStatusCodeDistribution()` Return a HashMap representing the distribution of DNS status codes for successfully fetched curis, as represented by a hashmap where key -> val represents (string)code -> (integer)count.
`java.lang.String` | `getDurationTime()`
`java.util.Hashtable` | `getHostsDnsDistribution()`
`long` | `getHostsPerTld(java.lang.String tld)` Get the number of hosts with a particular TLD.
`java.util.Hashtable` | `getMimeDistribution()` Returns a HashMap that contains information about distributions of encountered mime types.
`java.lang.String` | `getProcessedDocsPerSec()`
`java.util.TreeMap<java.lang.String,LongWrapper>` | `getReverseSortedCopy(java.util.Map<java.lang.String,LongWrapper> mapOfLongWrapperValues)` Sort the entries of the given HashMap in descending order by their values, which must be longs wrapped with `LongWrapper`.
`java.util.SortedMap` | `getReverseSortedHostsDistribution()` Return a copy of the hosts distribution in reverse-sorted (largest first) order.
`java.util.Iterator<SeedRecord>` | `getSeedRecordsSortedByStatusCode()` Returns sorted Iterator of seeds records based on status code.
`java.util.Hashtable` | `getStatusCodeDistribution()` Return a HashMap representing the distribution of HTTP status codes for successfully fetched curis, as represented by a hashmap where key -> val represents (string)code -> (integer)count.
`java.util.Hashtable` | `getTldBytes()`
`java.util.Hashtable` | `getTldDistribution()`
`java.util.Hashtable` | `getTldHostDistribution()`
`java.lang.String` | `getTotalDataWritten()`
`long` | `getTotalDnsHostDocuments()`
`long` | `getTotalDnsHostSize()`
`long` | `getTotalDnsMimeSize()`
`long` | `getTotalDnsMimeTypeDocuments()`
`long` | `getTotalDnsStatusCodeDocuments()`
`long` | `getTotalHostDnsDocuments()`
`long` | `getTotalHostDocuments()`
`long` | `getTotalHosts()`
`long` | `getTotalHostSize()`
`long` | `getTotalMimeSize()`
`long` | `getTotalMimeTypeDocuments()`
`long` | `getTotalStatusCodeDocuments()`
`long` | `getTotalTldDocuments()`
`long` | `getTotalTldSize()`
`protected static void` | `incrementMapCount(java.util.Map<java.lang.String,LongWrapper> map, java.lang.String key)` Increment a counter for a key in a given HashMap.
`protected static void` | `incrementMapCount(java.util.Map<java.lang.String,LongWrapper> map, java.lang.String key, long increment)` Increment a counter for a key in a given HashMap by an arbitrary amount.
`boolean` | `isStats()`
`boolean` | `readCrawlReport()` Reads duration time, processed docs/sec, bandwidth, and total size of crawl from crawl-report.txt.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### totalDnsStatusCodeDocuments

    protected long totalDnsStatusCodeDocuments

### totalStatusCodeDocuments

    protected long totalStatusCodeDocuments

### totalFileTypeDocuments

    protected long totalFileTypeDocuments

### totalMimeTypeDocuments

    protected long totalMimeTypeDocuments

### totalDnsMimeTypeDocuments

    protected long totalDnsMimeTypeDocuments

### totalDnsHostDocuments

    protected long totalDnsHostDocuments

### totalHostDocuments

    protected long totalHostDocuments

### totalMimeSize

    protected long totalMimeSize

### totalDnsMimeSize

    protected long totalDnsMimeSize

### totalHostSize

    protected long totalHostSize

### totalDnsHostSize

    protected long totalDnsHostSize

### totalTldDocuments

    protected long totalTldDocuments

### totalTldSize

    protected long totalTldSize

### totalHosts

    protected long totalHosts

### durationTime

    protected java.lang.String durationTime

### processedDocsPerSec

    protected java.lang.String processedDocsPerSec

### bandwidthKbytesPerSec

    protected java.lang.String bandwidthKbytesPerSec

### totalDataWritten

    protected java.lang.String totalDataWritten

### mimeTypeDistribution

    protected java.util.Hashtable<java.lang.String,LongWrapper> mimeTypeDistribution

  Keep track of the file types we see (mime type -> count)

### mimeTypeBytes

    protected java.util.Hashtable<java.lang.String,LongWrapper> mimeTypeBytes

### mimeTypeDnsDistribution

    protected java.util.Hashtable<java.lang.String,LongWrapper> mimeTypeDnsDistribution

### mimeTypeDnsBytes

    protected java.util.Hashtable<java.lang.String,LongWrapper> mimeTypeDnsBytes

### statusCodeDistribution

    protected java.util.Hashtable<java.lang.String,LongWrapper> statusCodeDistribution

  Keep track of status codes

### dnsStatusCodeDistribution

    protected java.util.Hashtable<java.lang.String,LongWrapper> dnsStatusCodeDistribution

### hostsDistribution

    protected java.util.Hashtable<java.lang.String,LongWrapper> hostsDistribution

  Keep track of hosts

### hostsBytes

    protected java.util.Hashtable<java.lang.String,LongWrapper> hostsBytes

### hostsDnsDistribution

    protected java.util.Hashtable<java.lang.String,LongWrapper> hostsDnsDistribution

### hostsDnsBytes

    protected java.util.Hashtable<java.lang.String,LongWrapper> hostsDnsBytes

### tldDistribution

    protected java.util.Hashtable<java.lang.String,LongWrapper> tldDistribution

  Keep track of TLDs

### tldBytes

    protected java.util.Hashtable<java.lang.String,LongWrapper> tldBytes

### tldHostDistribution

    protected java.util.Hashtable<java.lang.String,LongWrapper> tldHostDistribution

### processedSeedsRecords

    protected transient java.util.Map<java.lang.String,SeedRecord> processedSeedsRecords

  Keep track of processed seeds

Constructor Detail

### StatisticsSummary

    public StatisticsSummary(CrawlJob cjob)

  Constructor

  Parameters:
    `cjob` - Completed crawl job

Method Detail

### incrementMapCount

    protected static void incrementMapCount(java.util.Map<java.lang.String,LongWrapper> map,
                                            java.lang.String key)

  Increment a counter for a key in a given HashMap. Used for various aggregate data.

  Parameters:
    `map` - The HashMap
    `key` - The key for the counter to be incremented, if it does not exist it will be added (set to 1). If null it will increment the counter "unknown".

### incrementMapCount

    protected static void incrementMapCount(java.util.Map<java.lang.String,LongWrapper> map,
                                            java.lang.String key,
                                            long increment)

  Increment a counter for a key in a given HashMap by an arbitrary amount. Used for various aggregate data. The increment amount can be negative.

  Parameters:
    `map` - The HashMap
    `key` - The key for the counter to be incremented, if it does not exist it will be added (set to equal to `increment`). If null it will increment the counter "unknown".
    `increment` - The amount to increment counter related to the `key`.

### getMimeDistribution

    public java.util.Hashtable getMimeDistribution()

  Returns a HashMap that contains information about distributions of encountered mime types. Key/value pairs represent mime type -> count.

  Note: All the values are wrapped with a `LongWrapper`

  Returns:
    mimeTypeDistribution

### getTotalMimeTypeDocuments

    public long getTotalMimeTypeDocuments()

### getTotalDnsMimeTypeDocuments

    public long getTotalDnsMimeTypeDocuments()

### getTotalMimeSize

    public long getTotalMimeSize()

### getTotalDnsMimeSize

    public long getTotalDnsMimeSize()

### getStatusCodeDistribution

    public java.util.Hashtable getStatusCodeDistribution()

  Return a HashMap representing the distribution of HTTP status codes for successfully fetched curis, as represented by a hashmap where key -> val represents (string)code -> (integer)count. Note: All the values are wrapped with a `LongWrapper`

  Returns:
    statusCodeDistribution

### getDnsStatusCodeDistribution

    public java.util.Hashtable getDnsStatusCodeDistribution()

  Return a HashMap representing the distribution of DNS status codes for successfully fetched curis, as represented by a hashmap where key -> val represents (string)code -> (integer)count. Note: All the values are wrapped with a `LongWrapper`

  Returns:
    dnsStatusCodeDistribution

### getDnsMimeDistribution

    public java.util.Hashtable getDnsMimeDistribution()

### getTotalDnsStatusCodeDocuments

    public long getTotalDnsStatusCodeDocuments()

### getTotalStatusCodeDocuments

    public long getTotalStatusCodeDocuments()

### getTotalHostDocuments

    public long getTotalHostDocuments()

### getTotalDnsHostDocuments

    public long getTotalDnsHostDocuments()

### getHostsDnsDistribution

    public java.util.Hashtable getHostsDnsDistribution()

### getTotalHostDnsDocuments

    public long getTotalHostDnsDocuments()

### getTotalHostSize

    public long getTotalHostSize()

### getTotalDnsHostSize

    public long getTotalDnsHostSize()

### getTldDistribution

    public java.util.Hashtable getTldDistribution()

### getTldBytes

    public java.util.Hashtable getTldBytes()

### getTotalTldDocuments

    public long getTotalTldDocuments()

### getTotalTldSize

    public long getTotalTldSize()

### getTldHostDistribution

    public java.util.Hashtable getTldHostDistribution()

### getTotalHosts

    public long getTotalHosts()

### getDurationTime

    public java.lang.String getDurationTime()

### getProcessedDocsPerSec

    public java.lang.String getProcessedDocsPerSec()

### getBandwidthKbytesPerSec

    public java.lang.String getBandwidthKbytesPerSec()

### getTotalDataWritten

    public java.lang.String getTotalDataWritten()

### getReverseSortedCopy

    public java.util.TreeMap<java.lang.String,LongWrapper> getReverseSortedCopy(java.util.Map<java.lang.String,LongWrapper> mapOfLongWrapperValues)

  Sort the entries of the given HashMap in descending order by their values, which must be longs wrapped with `LongWrapper`.

  Elements are sorted by value from largest to smallest. Equal values are sorted in an arbitrary, but consistent manner by their keys. Only items with identical value and key are considered equal. If the passed-in map requires access to be synchronized, the caller should ensure this synchronization.

  Parameters:
    `mapOfLongWrapperValues` - Assumes values are wrapped with LongWrapper.
  Returns:
    a sorted set containing the same elements as the map.

### getHostsPerTld

    public long getHostsPerTld(java.lang.String tld)

  Get the number of hosts with a particular TLD.

  Parameters:
    `tld` - top-level domain name
  Returns:
    Total crawled hosts

### getBytesPerHost

    public long getBytesPerHost(java.lang.String host)

  Returns the accumulated number of bytes downloaded from a given host.

  Parameters:
    `host` - name of the host
  Returns:
    the accumulated number of bytes downloaded from a given host

### getBytesPerTld

    public long getBytesPerTld(java.lang.String tld)

  Returns the total number of bytes downloaded for a given TLD.

  Parameters:
    `tld` - TLD
  Returns:
    the total number of bytes downloaded for a given TLD

### getBytesPerMimeType

    public long getBytesPerMimeType(java.lang.String filetype)

  Returns the accumulated number of bytes from files of a given file type.

  Parameters:
    `filetype` - Filetype to check.
  Returns:
    the accumulated number of bytes from files of a given mime type

### readCrawlReport

    public boolean readCrawlReport()

  Reads duration time, processed docs/sec, bandwidth, and total size of crawl from crawl-report.txt.

  Returns:
    true if stats found.

### getSeedRecordsSortedByStatusCode

    public java.util.Iterator<SeedRecord> getSeedRecordsSortedByStatusCode()

  Returns sorted Iterator of seeds records based on status code.

  Returns:
    sorted Iterator of seeds records

### getReverseSortedHostsDistribution

    public java.util.SortedMap getReverseSortedHostsDistribution()

  Return a copy of the hosts distribution in reverse-sorted (largest first) order.

  Returns:
    SortedMap of hosts distribution

### isStats

    public boolean isStats()

  Returns:
    True if we compiled stats, false if none to compile (e.g. there are no reports files on disk).

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.