esis package


esis.cli module

Command Line Interface.


Remove all indexed documents.


Configure logging based on command line argument.

Parameters:log_level (int) – Log level passed form the command line

Print indexed documents information.


Index database information into elasticsearch.


Entry point for the script.


Parse command line arguments.

Returns:Parsed arguments
Return type:argparse.Namespace[source]

Send query to elasticsearch.


Directory validation.

esis.db module

Database related tools.

class esis.db.DBReader(database)[source]

Bases: object

Iterate through all db tables and rows easily.

Parameters:database (esis.db.Database) – Database to traverse
FTS_SUFFIXES = ('content', 'segdir', 'segments', 'stat', 'docsize')

Generator that traverses all tables in a database.

Returns:Table name
Return type:str
class esis.db.Database(db_filename)[source]

Bases: object

Generic database object.

Parameters:db_filename (str) – Path to the sqlite database file

Create connection.


Close connection.


Get table metadata through reflection.

sqlalchemy already provides a reflect method, but it will stop at the first failure, while this method will try to get as much as possible.

Parameters:table_names (list(str)) – Table names to inspect

Check database integrity.

Some files, especially those files created after carving, might not contain completely valid data.

class esis.db.DatetimeDecorator(*args, **kwargs)[source]

Bases: sqlalchemy.sql.type_api.TypeDecorator

A datetime class that translates data to ISO strings.

The reason ISO strings are used instead of datetime objects or integer timestamps is because is what elasticsearch handles as a datetime value. Internally it seems to store it as an integer timestamp, but that’s transparent to the user.


alias of TEXT

process_result_value(value, _dialect)[source]

Translate datetime/timestamp to ISO string.

class esis.db.IntegerDecorator(*args, **kwargs)[source]

Bases: sqlalchemy.sql.type_api.TypeDecorator

An integer class that translates ‘null’ values to None.

This is needed because some tables use ‘null’ instead of NULL and elastic search fails to index documents with strings where integers should be found.


alias of INTEGER

process_result_value(value, _dialect)[source]

Translate ‘null’ to None if needed.

class esis.db.TableReader(database, table_name)[source]

Bases: esis.db.TypeCoercionMixin

Iterate over all rows easily.

  • database (esis.db.Database) – Database being explored
  • table (sqlalchemy.sql.schema.Table) – Database table

Return table schema.

Returns:Column names and their type
Return type:dict(str, sqlalchemy.types.*)

Generator that traverses all rows in a table.

Returns:All rows in the table
Return type:generator(sqlalchemy.engine.result.RowProxy)
class esis.db.TypeCoercionMixin[source]

Bases: object

A mixin to transform database values.

This is useful to get safe values from sqlalchemy when data types are not very well defined in SQLite.

COERCIONS = {<class 'sqlalchemy.sql.sqltypes.BOOLEAN'>: <class 'esis.db.IntegerDecorator'>, <class 'sqlalchemy.sql.sqltypes.TIMESTAMP'>: <class 'esis.db.DatetimeDecorator'>, <class 'sqlalchemy.sql.sqltypes.NUMERIC'>: <class 'sqlalchemy.sql.sqltypes.TEXT'>, <class 'sqlalchemy.sql.sqltypes.DATE'>: <class 'esis.db.DatetimeDecorator'>, <class 'sqlalchemy.sql.sqltypes.BIGINT'>: <class 'esis.db.IntegerDecorator'>, <class 'sqlalchemy.sql.sqltypes.INTEGER'>: <class 'esis.db.IntegerDecorator'>, <class 'sqlalchemy.sql.sqltypes.DATETIME'>: <class 'esis.db.DatetimeDecorator'>, <class 'sqlalchemy.sql.sqltypes.SMALLINT'>: <class 'esis.db.IntegerDecorator'>} module

Elasticsearch related funcionality.

class, port)[source]

Bases: object

Elasticsearch client wrapper.

  • host (str) – Elasticsearch host
  • port (int) – Elasticsearch port
INDEX_NAME = 'sqlite'

Remove all indexed documents.


Return indexed documents information.

Returns:Indexed documents information
Return type:dict

Index all the information available in a directory.

In elasticsearch there will be an index for each database and a document type for each table in the database.

Parameters:directory (str) – Directory that should be indexed

Yield all documents that match a given query.

Parameters:query (str) – A simple query with data to search in elasticsearch
Returns:Records that matched the query as returned by elasticsearch
Return type:list(dict)
class, table_schema)[source]

Bases: object

ElasticSearch mapping.

  • document_type (str) – Document type user for the database table
  • table_schema (dict(str, sqlalchemy.types.*)) – Database table schema from sqlalchemy
SQL_TYPE_MAPPING = {<class 'sqlalchemy.sql.sqltypes.SMALLINT'>: 'integer', <class 'sqlalchemy.sql.sqltypes.DATETIME'>: 'date', <class 'sqlalchemy.sql.sqltypes.VARCHAR'>: 'string', <class 'sqlalchemy.sql.sqltypes.CHAR'>: 'string', <class 'sqlalchemy.sql.sqltypes.CLOB'>: 'string', <class 'sqlalchemy.sql.sqltypes.NullType'>: None, <class 'sqlalchemy.sql.sqltypes.BIGINT'>: 'long', <class 'sqlalchemy.sql.sqltypes.NCHAR'>: 'string', <class 'sqlalchemy.sql.sqltypes.INTEGER'>: 'long', <class 'sqlalchemy.sql.sqltypes.REAL'>: 'double', <class 'sqlalchemy.sql.sqltypes.BOOLEAN'>: 'boolean', <class 'sqlalchemy.sql.sqltypes.TIMESTAMP'>: 'date', <class 'sqlalchemy.sql.sqltypes.NUMERIC'>: None, <class 'sqlalchemy.sql.sqltypes.DECIMAL'>: None, <class 'sqlalchemy.sql.sqltypes.NVARCHAR'>: 'string', <class 'sqlalchemy.sql.sqltypes.FLOAT'>: 'float', <class 'sqlalchemy.sql.sqltypes.TIME'>: 'date', <class 'sqlalchemy.sql.sqltypes.DATE'>: None, <class 'sqlalchemy.sql.sqltypes.TEXT'>: 'string'}, table_name, row)[source]

Get document to be indexed from row.

  • db_filename (str) – Path to the database file
  • table_name – Database table name
  • row (sqlalchemy.engine.result.RowProxy) – Database row, document_type, document)[source]

Generate index action for a given document.

  • index_name (str) – Elasticsearch index to use
  • document_type – Elasticsearch document type to use
  • document – Document to be indexed

Action to be passed in bulk request

Return type:


esis.fs module

Filesystem functionality.

class esis.fs.TreeExplorer(directory, blacklist=None)[source]

Bases: object

Look for sqlite files in a tree and return the valid ones.

  • directory (str) – Base directory for the tree to be explored.
  • blacklist (list(str)) – List of relative directories to skip

Return paths to valid databases found under directory.

Returns:Paths to valid databases
Return type:list(str)

esis.util module

Utility functionality.


Return a timestamp for the given datetime object.

Parameters:datetime_obj (datetime.datetime) – datetime object to be converted
Returns:timestamp from the passed datetime object
Return type:int

Module contents

Elastic Search Index & Search.