ELKI: Environment for DeveLoping KDD-Applications Supported by Index-Structures.
ELKI is a generic framework for a broad range of KDD-applications and their development. For background, contact-information, and contributors see https://elki-project.github.io/.
This is the documentation for version 0.8.0, published as:
Erich Schubert:
Automatic Indexing for Similarity Search in ELKI
Int. Conf. Similarity Search and Applications
https://doi.org/10.1007/978-3-031-17849-8_16
The ELKI website contains additional documentation. A Tutorial exported is included with this documentation and a good place to start.
To use the KDD-Framework we recommend an executable .jar-file:
elki.jar. Since release 0.3 it will by default invoke a minimalistic GUI called MiniGUI when
you call java -jar elki.jar
. For command line use (for example for batch processing and scripted operation),
you can get a description of usage by calling java -jar elki.jar KDDCLIApplication -h
.
The MiniGUI can also serve as a utility for building command lines, as it will print the full command line to the log window.
For more information on using files and available formats as data input see {@link elki.datasource.parser}. ELKI uses a whitespace separated vector format by default, but there also is a parser for ARFF files included that can read most ARFF files (mixing sparse and dense vectors is currently not allowed).
An extensive list of parameters can be browsed sorted by class or sorted by option ID.
Some examples of completely parameterized calls for different algorithms are described at example calls.
A list of related publications, giving details on many implemented algorithms, can be found in the class article references list.
The database connection manages reading of input files or databases and provides a {@link elki.database.Database Database}-Object - including index structures - as a virtual database to the KDDTask. The KDDTask applies a specified algorithm on this database and collects the result from the algorithm. Finally, KDDTask hands on the obtained result to a {@link elki.result.ResultHandler ResultHandler}. The default-handler is {@link elki.result.ResultWriter ResultWriter}, writing the result to STDOUT or, if specified, into a file.
The database and indexing layer is a key component of ELKI.
This is not just a storage for double[]
, as with many other frameworks.
It can store various types of objects, and the integrated index structures provide access to fast
{@link elki.database.query.distance.DistanceQuery distance},
{@link elki.database.query.similarity.SimilarityQuery similarity},
{@link elki.database.query.knn.KNNSearcher kNN},
{@link elki.database.query.rknn.RKNNSearcher RkNN} and
{@link elki.database.query.range.RangeSearcher range} query methods
for a variety of distance functions.
The standard flow for initializing a database is as depicted here:
The standard stream-based data sources such as
{@link elki.datasource.FileBasedDatabaseConnection FileBasedDatabaseConnection}
will open the stream, feed the contents through a
{@link elki.datasource.parser.Parser Parser} to obtain an initial
{@link elki.datasource.bundle.MultipleObjectsBundle MultipleObjectsBundle}. This is
a temporary container for the data, which can then be modified by arbitrary
{@link elki.datasource.filter.ObjectFilter ObjectFilter}s.
In the end, the
{@link elki.datasource.bundle.MultipleObjectsBundle MultipleObjectsBundle}
is bulk-inserted into a {@link elki.database.Database Database}, which will then
invoke its {@link elki.index.IndexFactory IndexFactory}s to add
{@link elki.index.Index Index} instances to the appropriate relations.
When a request for a {@link elki.database.query.distance.DistanceQuery distance}, {@link elki.database.query.similarity.SimilarityQuery similarity}, {@link elki.database.query.knn.KNNSearcher kNN}, {@link elki.database.query.rknn.RKNNSearcher RkNN} or {@link elki.database.query.range.RangeSearcher range} query is received by the database, it queries all indexes if they have support for this query. If so, an optimized query is returned, otherwise a linear scan query can be returned unless {@link elki.database.query.QueryBuilder#optimizedOnly} was given.
For this optimization to work, you should be using the proper APIs of the {@link elki.database.query.QueryBuilder} where possible, instead of initializing low level classes such as an explicit linear scan query.
For efficiency, try to instantiate the query only once per algorithm run, and avoid running the optimization step for every object.
A good place to get started is to have a look at some of the existing algorithms, and see how they are implemented.
Visit the ELKI Wiki, which has a growing amount of documentation. You are also welcome to contribute, of course!
ELKI is designed for command-line, GUI and Java operation. For command-line and GUI, an extensive help functionality is provided along with input assistance. Therefore, you should also support the parameterizable API. The requirements are quite different from regular Java constructors, and cannot be expressed in terms of a Java API.
For useful error reporting and input assistance in the GUI we need to have more extensive typing than Java uses (for example we might need numerical constraints) and we also want to be able to report more than one error at a time. In ELKI 0.4, much of the parameterization was refactored to static helper classes usually found as a public static class Par and instances of {@link elki.utilities.optionhandling.Parameterizer Parameterizer}.
Keep the complexity of Parameterizer classes and constructors invoked by these classes low, since these may be heavily used during the parameterization step. Postpone any extensive initialization to the main algorithm invocation step!