de.lmu.ifi.dbs.elki.datasource.parser

Class ArffParser

java.lang.Object

de.lmu.ifi.dbs.elki.datasource.parser.ArffParser

All Implemented Interfaces:

Parser

@Title(value="ARFF File Format Parser")
public class ArffParser
extends java.lang.Object
implements Parser

Parser to load WEKA .arff files into ELKI.

This parser is quite hackish, and contains lots of not yet configurable magic.

TODO: Allow configuration of the vector types (double, float)

TODO: when encountering integer columns, produce integer vectors.

TODO: allow optional class labels.

Since:

0.4.0

Author:

Erich Schubert

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ArffParser.Parameterizer Parameterization class.

Field Summary

Fields

Modifier and Type	Field and Description
static java.util.regex.Matcher	ARFF_COMMENT Comment pattern.
static java.util.regex.Matcher	ARFF_HEADER_ATTRIBUTE Arff attribute declaration marker.
static java.util.regex.Matcher	ARFF_HEADER_DATA Arff data marker.
static java.util.regex.Matcher	ARFF_HEADER_RELATION Arff file marker.
static java.util.regex.Matcher	ARFF_NUMERIC Pattern for numeric columns.
static java.lang.String	DEFAULT_ARFF_MAGIC_CLASS Pattern to auto-convert columns to class labels.
static java.lang.String	DEFAULT_ARFF_MAGIC_EID Pattern to auto-convert columns to external ids.
(package private) NumberVector.Factory<?>	denseFactory Factory for dense vectors.
static java.util.regex.Matcher	EMPTY Empty line pattern.
(package private) java.util.ArrayList<java.lang.String>	labels (Reused) buffer for building label lists.
private static Logging	LOG Logger.
(package private) java.util.regex.Matcher	magic_class Pattern to recognize class label columns.
(package private) java.util.regex.Matcher	magic_eid Pattern to recognize external ids.

Constructor Summary

Constructors

Constructor and Description
ArffParser(java.util.regex.Pattern magic_eid, java.util.regex.Pattern magic_class) Constructor.
ArffParser(java.lang.String magic_eid, java.lang.String magic_class) Constructor.

Method Summary

Modifier and Type	Method and Description
void	cleanup() Perform cleanup operations after parsing.
private java.lang.Object[]	loadDenseInstance(java.io.StreamTokenizer tokenizer, int[] dimsize, TypeInformation[] etyp, int outdim)
private java.lang.Object[]	loadSparseInstance(java.io.StreamTokenizer tokenizer, int[] targ, int[] dimsize, TypeInformation[] elkitypes, int metaLength)
private java.io.StreamTokenizer	makeArffTokenizer(java.io.BufferedReader br) Make a StreamTokenizer for the ARFF format.
private void	nextToken(java.io.StreamTokenizer tokenizer) Helper function for token handling.
MultipleObjectsBundle	parse(java.io.InputStream instream) Returns a list of the objects parsed from the specified input stream.
private void	parseAttributeStatements(java.io.BufferedReader br, java.util.ArrayList<java.lang.String> names, java.util.ArrayList<java.lang.String> types) Parse the "@attribute" section of the ARFF file.
private void	processColumnTypes(java.util.ArrayList<java.lang.String> names, java.util.ArrayList<java.lang.String> types, int[] targ, TypeInformation[] etyp, int[] dims) Process the column types (and names!)
private void	readHeader(java.io.BufferedReader br) Read the dataset header part of the ARFF file, to ensure consistency.
private void	setupBundleHeaders(java.util.ArrayList<java.lang.String> names, int[] targ, TypeInformation[] etyp, int[] dimsize, MultipleObjectsBundle bundle, boolean sparse) Setup the headers for the object bundle.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final Logging LOG

Logger.

ARFF_HEADER_RELATION

public static final java.util.regex.Matcher ARFF_HEADER_RELATION

Arff file marker.

ARFF_HEADER_ATTRIBUTE

public static final java.util.regex.Matcher ARFF_HEADER_ATTRIBUTE

Arff attribute declaration marker.

ARFF_HEADER_DATA

public static final java.util.regex.Matcher ARFF_HEADER_DATA

Arff data marker.

ARFF_COMMENT

public static final java.util.regex.Matcher ARFF_COMMENT

Comment pattern.

DEFAULT_ARFF_MAGIC_EID

public static final java.lang.String DEFAULT_ARFF_MAGIC_EID

Pattern to auto-convert columns to external ids.

See Also:

Constant Field Values

DEFAULT_ARFF_MAGIC_CLASS

public static final java.lang.String DEFAULT_ARFF_MAGIC_CLASS

Pattern to auto-convert columns to class labels.

See Also:

Constant Field Values

ARFF_NUMERIC

public static final java.util.regex.Matcher ARFF_NUMERIC

Pattern for numeric columns.

EMPTY

public static final java.util.regex.Matcher EMPTY

Empty line pattern.

magic_eid

java.util.regex.Matcher magic_eid

Pattern to recognize external ids.

magic_class

java.util.regex.Matcher magic_class

Pattern to recognize class label columns.

labels

java.util.ArrayList<java.lang.String> labels

(Reused) buffer for building label lists.

denseFactory

NumberVector.Factory<?> denseFactory

Factory for dense vectors. TODO: Make parameterizable

Constructor Detail

ArffParser

public ArffParser(java.util.regex.Pattern magic_eid,
                  java.util.regex.Pattern magic_class)

Constructor.

Parameters:

magic_eid - Magic to recognize external IDs

magic_class - Magic to recognize class labels

ArffParser

public ArffParser(java.lang.String magic_eid,
                  java.lang.String magic_class)

Constructor.

Parameters:

magic_eid - Magic to recognize external IDs

magic_class - Magic to recognize class labels

Method Detail

parse

public MultipleObjectsBundle parse(java.io.InputStream instream)

Description copied from interface: Parser

Returns a list of the objects parsed from the specified input stream.

Specified by:

parse in interface Parser

Parameters:

instream - the stream to parse objects from

Returns:

a list containing those objects parsed from the input stream

loadSparseInstance

private java.lang.Object[] loadSparseInstance(java.io.StreamTokenizer tokenizer,
                                              int[] targ,
                                              int[] dimsize,
                                              TypeInformation[] elkitypes,
                                              int metaLength)
                                       throws java.io.IOException

Throws:

java.io.IOException

loadDenseInstance

private java.lang.Object[] loadDenseInstance(java.io.StreamTokenizer tokenizer,
                                             int[] dimsize,
                                             TypeInformation[] etyp,
                                             int outdim)
                                      throws java.io.IOException

Throws:

java.io.IOException

makeArffTokenizer

private java.io.StreamTokenizer makeArffTokenizer(java.io.BufferedReader br)

Make a StreamTokenizer for the ARFF format.

Parameters:

br - Buffered reader

Returns:

Tokenizer

setupBundleHeaders

private void setupBundleHeaders(java.util.ArrayList<java.lang.String> names,
                                int[] targ,
                                TypeInformation[] etyp,
                                int[] dimsize,
                                MultipleObjectsBundle bundle,
                                boolean sparse)

Setup the headers for the object bundle.

Parameters:

names - Attribute names

targ - Target columns

etyp - ELKI type information

dimsize - Number of dimensions in the individual types

bundle - Output bundle

sparse - Flag to create sparse vectors

readHeader

private void readHeader(java.io.BufferedReader br)
                 throws java.io.IOException

Read the dataset header part of the ARFF file, to ensure consistency.

Parameters:

br - Buffered Reader

Throws:

java.io.IOException

parseAttributeStatements

private void parseAttributeStatements(java.io.BufferedReader br,
                                      java.util.ArrayList<java.lang.String> names,
                                      java.util.ArrayList<java.lang.String> types)
                               throws java.io.IOException

Parse the "@attribute" section of the ARFF file.

Parameters:

br - Input

names - List (to fill) of attribute names

types - List (to fill) of attribute types

Throws:

java.io.IOException

processColumnTypes

private void processColumnTypes(java.util.ArrayList<java.lang.String> names,
                                java.util.ArrayList<java.lang.String> types,
                                int[] targ,
                                TypeInformation[] etyp,
                                int[] dims)

Process the column types (and names!) into ELKI relation style. Note that this will for example merge successive numerical columns into a single vector.

Parameters:

names - Attribute names

types - Attribute types

targ - Target dimension mapping (ARFF to ELKI), return value

etyp - ELKI type information, return value

dims - Number of successive dimensions, return value

nextToken

private void nextToken(java.io.StreamTokenizer tokenizer)
                throws java.io.IOException

Helper function for token handling.

Parameters:

tokenizer - Tokenizer

Throws:

java.io.IOException

cleanup

public void cleanup()

Description copied from interface: Parser

Perform cleanup operations after parsing.

Specified by:

cleanup in interface Parser