Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.io
Class CrawlerJournal

    java.lang.Object
      org.archive.crawler.io.CrawlerJournal

Direct Known Subclasses:
  RecoveryJournal

    public class CrawlerJournal

extends java.lang.Object

Utility class for a crawler journal/log that is compressed and rotates by serial number at checkpoints.

Author:
  gojomo

Field Summary
`protected it.unimi.dsi.mg4j.util.MutableString` | `accumulatingBuffer` Allocate a buffer for accumulating lines to write and reuse it.
`static java.lang.String` | `GZIP_SUFFIX` suffix to recognize gzipped files
`protected java.io.File` | `gzipFile` File we're writing journal to.
`protected long` | `lines` line count
`static java.lang.String` | `LOG_ERROR` prefix for error lines
`static java.lang.String` | `LOG_TIMESTAMP` prefix for timestamp lines
`protected java.io.Writer` | `out` Stream on which we record frontier events.
`protected int` | `timestamp_interval` number of lines between timestamps

Constructor Summary
`CrawlerJournal(java.io.File file)` Create a new crawler journal at the given location
`CrawlerJournal(java.lang.String path, java.lang.String filename)` Create a new crawler journal at the given location

Method Summary
`void` | `checkpoint(java.io.File checkpointDir)` Handle a checkpoint by rotating the current log to a checkpoint-named file and starting a new log.
`void` | `close()` Flush and close the underlying IO objects.
`protected void` | `considerTimestamp()` Write a timestamp line if appropriate
`static java.io.BufferedInputStream` | `getBufferedInput(java.io.File source)` Get a BufferedInputStream on the recovery file given.
`static java.io.BufferedReader` | `getBufferedReader(java.io.File source)` Get a BufferedReader on the crawler journal given
`protected java.io.Writer` | `initialize(java.io.File f)`
`protected void` | `noteLine()` Count and note a line
`void` | `seriousError(java.lang.String err)` Note a serious error vioa a special log line
`void` | `writeLine(it.unimi.dsi.mg4j.util.MutableString mstring)` Write a line.
`void` | `writeLine(java.lang.String string)` Write a line
`void` | `writeLine(java.lang.String s1, java.lang.String s2)` Write a line of two strings
`void` | `writeLine(java.lang.String s1, java.lang.String s2, java.lang.String s3)` Write a line of three strings

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### LOG_ERROR

    public static final java.lang.String LOG_ERROR

  prefix for error lines

  See Also:
    Constant Field Values

### LOG_TIMESTAMP

    public static final java.lang.String LOG_TIMESTAMP

  prefix for timestamp lines

  See Also:
    Constant Field Values

### out

    protected java.io.Writer out

  Stream on which we record frontier events.

### lines

    protected long lines

  line count

### timestamp_interval

    protected int timestamp_interval

  number of lines between timestamps

### GZIP_SUFFIX

    public static final java.lang.String GZIP_SUFFIX

  suffix to recognize gzipped files

  See Also:
    Constant Field Values

### gzipFile

    protected java.io.File gzipFile

  File we're writing journal to. Keep a reference in case we want to rotate it off.

### accumulatingBuffer

    protected it.unimi.dsi.mg4j.util.MutableString accumulatingBuffer

  Allocate a buffer for accumulating lines to write and reuse it.

Constructor Detail

### CrawlerJournal

    public CrawlerJournal(java.lang.String path,
                          java.lang.String filename)
                   throws java.io.IOException

  Create a new crawler journal at the given location

  Parameters:
    `path` - Directory to make thejournal in.
    `filename` - Name to use for journal file.
  Throws:
    `java.io.IOException`

### CrawlerJournal

    public CrawlerJournal(java.io.File file)
                   throws java.io.IOException

  Create a new crawler journal at the given location

  Parameters:
    `file` - path at which to make journal
  Throws:
    `java.io.IOException`

Method Detail

### getBufferedReader

    public static java.io.BufferedReader getBufferedReader(java.io.File source)
                                                    throws java.io.IOException

  Get a BufferedReader on the crawler journal given

  Parameters:
    `source` - File journal
  Returns:
    journal buffered reader.
  Throws:
    `java.io.IOException`

### getBufferedInput

    public static java.io.BufferedInputStream getBufferedInput(java.io.File source)
                                                        throws java.io.IOException

  Get a BufferedInputStream on the recovery file given.

  Parameters:
    `source` - file to open
  Returns:
    journal buffered input stream.
  Throws:
    `java.io.IOException`

### initialize

    protected java.io.Writer initialize(java.io.File f)
                                 throws java.io.FileNotFoundException,
                                        java.io.IOException

  Throws:
    `java.io.FileNotFoundException`
    `java.io.IOException`

### writeLine

    public void writeLine(java.lang.String string)

  Write a line

  Parameters:
    `string` - String

### writeLine

    public void writeLine(java.lang.String s1,
                          java.lang.String s2)

  Write a line of two strings

  Parameters:
    `s1` - String
    `s2` - String

### writeLine

    public void writeLine(java.lang.String s1,
                          java.lang.String s2,
                          java.lang.String s3)

  Write a line of three strings

  Parameters:
    `s1` - String
    `s2` - String
    `s3` - String

### writeLine

    public void writeLine(it.unimi.dsi.mg4j.util.MutableString mstring)

  Write a line.

  Parameters:
    `mstring` - MutableString to write

### noteLine

    protected void noteLine()
                     throws java.io.IOException

  Count and note a line

  Throws:
    `java.io.IOException`

### considerTimestamp

    protected void considerTimestamp()
                              throws java.io.IOException

  Write a timestamp line if appropriate

  Throws:
    `java.io.IOException`

### close

    public void close()

  Flush and close the underlying IO objects.

### seriousError

    public void seriousError(java.lang.String err)

  Note a serious error vioa a special log line

  Parameters:
    `err` -

### checkpoint

    public void checkpoint(java.io.File checkpointDir)
                    throws java.io.IOException

  Handle a checkpoint by rotating the current log to a checkpoint-named file and starting a new log.

  Parameters:
    `checkpointDir` -
  Throws:
    `java.io.IOException`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.