Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.util
Class MemFPMergeUriUniqFilter

    java.lang.Object
      org.archive.crawler.util.FPMergeUriUniqFilter
          org.archive.crawler.util.MemFPMergeUriUniqFilter

All Implemented Interfaces:
  UriUniqFilter

    public class MemFPMergeUriUniqFilter

extends FPMergeUriUniqFilter

Crude all-in-memory FP-merging UriUniqFilter.

Author:
  gojomo

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.util.FPMergeUriUniqFilter
`FPMergeUriUniqFilter.PendingItem`

Nested classes/interfaces inherited from interface org.archive.crawler.datamodel.UriUniqFilter
`UriUniqFilter.HasUriReceiver`

Field Summary
`protected it.unimi.dsi.fastutil.longs.LongArrayList` | `allFps`
`protected it.unimi.dsi.fastutil.longs.LongArrayList` | `newFps`

Fields inherited from class org.archive.crawler.util.FPMergeUriUniqFilter
`DEFAULT_MAX_PENDING, FLUSH_DELAY_FACTOR, maxPending, mergeDupAtLast, mergeDuplicateCount, nextFlushAllowableAfter, pendDupAtLast, pendDuplicateCount, pendingSet, profileLog, quickCache, quickDupAtLast, quickDuplicateCount, receiver`

Constructor Summary
`MemFPMergeUriUniqFilter()`

Method Summary
`protected void` | `addNewFp(long currFp)` Add an FP (which may be an old or new FP) to the new complete list.
`protected it.unimi.dsi.fastutil.longs.LongIterator` | `beginFpMerge()` Begin merging pending candidates with complete list.
`long` | `count()`
`protected void` | `finishFpMerge()` Complete the merge of candidate and previously-known FPs (closing files/iterators as appropriate).

Methods inherited from class org.archive.crawler.util.FPMergeUriUniqFilter
`add, addForce, addNow, close, createFp, flush, forget, note, pend, pending, profileLog, requestFlush, setDestination, setMaxPending, setProfileLog`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### allFps

    protected it.unimi.dsi.fastutil.longs.LongArrayList allFps

### newFps

    protected it.unimi.dsi.fastutil.longs.LongArrayList newFps

Constructor Detail

### MemFPMergeUriUniqFilter

    public MemFPMergeUriUniqFilter()

Method Detail

### beginFpMerge

    protected it.unimi.dsi.fastutil.longs.LongIterator beginFpMerge()

  Description copied from class: `FPMergeUriUniqFilter`
  Begin merging pending candidates with complete list. Return an Iterator which will return all previously-known FPs in turn.

  Specified by:
    `beginFpMerge` in class `FPMergeUriUniqFilter`
  Returns:
    Iterator over all previously-known FPs

### addNewFp

    protected void addNewFp(long currFp)

  Description copied from class: `FPMergeUriUniqFilter`
  Add an FP (which may be an old or new FP) to the new complete list. Should only be called after beginFpMerge() and before finishFpMerge().

  Specified by:
    `addNewFp` in class `FPMergeUriUniqFilter`
  Parameters:
    `currFp` - the FP to add

### finishFpMerge

    protected void finishFpMerge()

  Description copied from class: `FPMergeUriUniqFilter`
  Complete the merge of candidate and previously-known FPs (closing files/iterators as appropriate).

  Specified by:
    `finishFpMerge` in class `FPMergeUriUniqFilter`

### count

    public long count()

  Returns:
    Count of already seen URIs.

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.