Class OnDiskArray

  • All Implemented Interfaces:
    java.lang.AutoCloseable

    public class OnDiskArray
    extends java.lang.Object
    implements java.lang.AutoCloseable
    On Disc Array storage for records of a given size.

    This can be used to implement various fixed size record-based data structures. The file format is designed to have a fixed-size header followed by the actual data.

    Since:
    0.2
    Author:
    Erich Schubert
    • Field Summary

      Fields 
      Modifier and Type Field Description
      private java.nio.channels.FileChannel file
      Random Access File object.
      private java.nio.file.Path filename
      File name.
      private static int HEADER_POS_SIZE
      Position of file size (in records).
      private int headersize
      Size of the header in the file.
      private static int INTERNAL_HEADER_SIZE
      Size of the classes header size.
      private java.nio.channels.FileLock lock
      Lock for the file that will be kept while writing.
      protected int magic
      Magic number used to identify files.
      private java.nio.MappedByteBuffer map
      The memory mapped buffer.
      private int numrecs
      Number of records in the file.
      private int recordsize
      Size of the records in the file.
      private static long serialVersionUID
      Serial version.
      private boolean writable
      Writable or read-only object.
    • Constructor Summary

      Constructors 
      Constructor Description
      OnDiskArray​(java.nio.file.Path filename, int magicseed, int extraheadersize, boolean writable)
      Constructor to open an existing file.
      OnDiskArray​(java.nio.file.Path filename, int magicseed, int extraheadersize, int recordsize, boolean writable)
      Constructor to open an existing file.
      OnDiskArray​(java.nio.file.Path filename, int magicseed, int extraheadersize, int recordsize, int initialsize)
      Constructor to write a new file.
    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()
      Explicitly close the file.
      void ensureSize​(int size)
      Ensure that the file can fit the given number of records.
      java.nio.ByteBuffer getExtraHeader()
      Read the extra header data.
      protected int getExtraHeaderSize()
      Return the size of the extra header.
      java.nio.file.Path getFilename()
      Get the file name.
      int getNumRecords()
      Get number of records in file.
      java.nio.ByteBuffer getRecordBuffer​(int index)
      Get a record buffer
      protected int getRecordsize()
      Get the size of a single record.
      private long indexToFileposition​(long index)
      Compute file position from index number
      boolean isWritable()
      Check if the file is writable.
      private void mapArray()
      (Re-) map the data array.
      static int mixMagic​(int magic1, int magic2)
      Mix two magic numbers into one, to obtain a combined magic.
      void resizeFile​(int newsize)
      Resize file to the intended size
      private void validateHeader​(boolean validateRecordSize)
      Validates the header and throws an IOException if the header is invalid.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • serialVersionUID

        private static final long serialVersionUID
        Serial version.

        NOTE: Change this version whenever the file structure is changed in an incompatible way: This will modify the file magic, and thus prevent applications from reading incompatible files.

        See Also:
        Constant Field Values
      • magic

        protected int magic
        Magic number used to identify files.
      • headersize

        private int headersize
        Size of the header in the file. Note that the internal header is four integers already.
      • recordsize

        private int recordsize
        Size of the records in the file.
      • numrecs

        private int numrecs
        Number of records in the file.
      • filename

        private java.nio.file.Path filename
        File name.
      • file

        private final java.nio.channels.FileChannel file
        Random Access File object.
      • lock

        private java.nio.channels.FileLock lock
        Lock for the file that will be kept while writing.
      • writable

        private boolean writable
        Writable or read-only object.
      • map

        private java.nio.MappedByteBuffer map
        The memory mapped buffer.
      • INTERNAL_HEADER_SIZE

        private static final int INTERNAL_HEADER_SIZE
        Size of the classes header size.
        See Also:
        Constant Field Values
      • HEADER_POS_SIZE

        private static final int HEADER_POS_SIZE
        Position of file size (in records).
        See Also:
        Constant Field Values
    • Constructor Detail

      • OnDiskArray

        public OnDiskArray​(java.nio.file.Path filename,
                           int magicseed,
                           int extraheadersize,
                           int recordsize,
                           int initialsize)
                    throws java.io.IOException
        Constructor to write a new file.
        Parameters:
        filename - File name to be opened.
        magicseed - Magic number to derive real magic from.
        extraheadersize - header size NOT including the internal header
        recordsize - Record size
        initialsize - Initial file size (in records)
        Throws:
        java.io.IOException - on IO errors
      • OnDiskArray

        public OnDiskArray​(java.nio.file.Path filename,
                           int magicseed,
                           int extraheadersize,
                           int recordsize,
                           boolean writable)
                    throws java.io.IOException
        Constructor to open an existing file. The provided record size must match the record size stored within the files header. If you don't know this size yet and/or need to access the extra header you should use the other constructor below
        Parameters:
        filename - File name to be opened.
        magicseed - Magic number to derive real magic from.
        extraheadersize - header size NOT including the internal header
        recordsize - Record size
        writable - flag to open the file writable
        Throws:
        java.io.IOException - on IO errors
      • OnDiskArray

        public OnDiskArray​(java.nio.file.Path filename,
                           int magicseed,
                           int extraheadersize,
                           boolean writable)
                    throws java.io.IOException
        Constructor to open an existing file. The record size is read from the file's header and can be obtained by getRecordsize()
        Parameters:
        filename - File name to be opened.
        magicseed - Magic number to derive real magic from.
        extraheadersize - header size NOT including the internal header
        writable - flag to open the file writable
        Throws:
        java.io.IOException - on IO errors
    • Method Detail

      • mapArray

        private void mapArray()
                       throws java.io.IOException
        (Re-) map the data array.
        Throws:
        java.io.IOException - on mapping error.
      • validateHeader

        private void validateHeader​(boolean validateRecordSize)
                             throws java.io.IOException
        Validates the header and throws an IOException if the header is invalid. If validateRecordSize is set to true the record size must match exactly the stored record size within the files header, else the record size is read from the header and used.
        Parameters:
        validateRecordSize -
        Throws:
        java.io.IOException
      • mixMagic

        public static final int mixMagic​(int magic1,
                                         int magic2)
        Mix two magic numbers into one, to obtain a combined magic. Note: mixMagic(a,b) != mixMagic(b,a) usually.
        Parameters:
        magic1 - Magic number to mix.
        magic2 - Magic number to mix.
        Returns:
        Mixed magic number.
      • indexToFileposition

        private long indexToFileposition​(long index)
        Compute file position from index number
        Parameters:
        index - Index offset
        Returns:
        file position
      • resizeFile

        public void resizeFile​(int newsize)
                        throws java.io.IOException
        Resize file to the intended size
        Parameters:
        newsize - New file size.
        Throws:
        java.io.IOException - on IO errors
      • getRecordBuffer

        public java.nio.ByteBuffer getRecordBuffer​(int index)
                                            throws java.io.IOException
        Get a record buffer
        Parameters:
        index - Record index
        Returns:
        Byte buffer for the record
        Throws:
        java.io.IOException - on IO errors
      • getExtraHeaderSize

        protected int getExtraHeaderSize()
        Return the size of the extra header. Accessor.
        Returns:
        Extra header size
      • getExtraHeader

        public java.nio.ByteBuffer getExtraHeader()
                                           throws java.io.IOException
        Read the extra header data.
        Returns:
        additional header data
        Throws:
        java.io.IOException - on IO errors
      • getRecordsize

        protected int getRecordsize()
        Get the size of a single record.
        Returns:
        Record size.
      • getFilename

        public java.nio.file.Path getFilename()
        Get the file name.
        Returns:
        File name
      • isWritable

        public boolean isWritable()
        Check if the file is writable.
        Returns:
        true if the file is writable.
      • close

        public void close()
                   throws java.io.IOException
        Explicitly close the file. Note: following operations will likely cause IOExceptions.
        Specified by:
        close in interface java.lang.AutoCloseable
        Throws:
        java.io.IOException - on IO errors
      • getNumRecords

        public int getNumRecords()
        Get number of records in file.
        Returns:
        Number of records in the file.
      • ensureSize

        public void ensureSize​(int size)
                        throws java.io.IOException
        Ensure that the file can fit the given number of records.
        Parameters:
        size - Size
        Throws:
        java.io.IOException