OpenLSD ENOpenLSD

Download at SourceForge.net
ConceptLogic Technical Aspects Future Plan Howto Benchmarks
Home OpenLSD OpenLSM OpenR66

HowTo-Use-API

This document makes a review of some internal and external API. It is intend for developers that would like to extend or include some functions in their own development. You can find here a PDF version.

STATUS from Network API

The kinds of status returned from the Network API are as follow.

 

In general:

  • Bad Legacy : KOBADREQUEST
  • Bad Document : KOBADREQUEST if first call or KOSYSTEM if second or next call
  • For all kind of request : BADREQUEST if a bad request is done (bad command or bad index)

Admin:

  • KOSYSTEM if error not specified
  • SHUTDOWNIMMEDIATE | SHUTDOWN | STOP : OK
  • RESTART | START : CONTINUE
  • ENABLE | CREATEDIR | WIPEDIR | WIPERECURDIR : KOUNAVAIL if not exist else O

Del:

  • LEGACY unavailable : KOUNAVAIL
  • OK if deleted, KOSYSTEM if not deleted, KOBADREQ if MD5 or IDs are KO

Get | GetNAck :

  • KOUNAVAIL if first block in error due to no file exists
  • KOBADREQUEST if first block in error
  • CONTINUE if there is more block to come
  • OK if this is the last block

Info:

  • SIZE : OK if size > 0, or KOUNAVAIL if not available
  • EXIST : OK if it exist or KOSYSTEM if not exist or KOUNAVAIL if MD5 is wrong
  • COPY : OK if copy or KOUNAVAIL if not
  • LIST, LISTMD5, AEXIST, AEXISTMD5 : OK
  • SESSIONINFO : OK, BADREQUEST if no session are returned (and should be at least 1)

Move:

  • MOVE | MOVELEG : OK or BADREQUEST

PutFile | Put | PutNAck:

  • Argument from Client are incorrect : KOBADCLIENT
  • LEGACY unavailable : KOUNAVAIL
  • File already exist : BADREQUEST
  • Put OK : CONTINUE if there is more block to come or OK if this is the last block
  • Put KO : KOSYSTEM
  • Put KO but cannot restore to previous state (cannot delete) : KONOPREVIOUS

Info Handler API

SIZE

  • Returns the size of one file in LSD Server (true size so possibly different from the real size if the LSD Server Legacy is crypted)

EXIST

  • Returns the existence of one file in LSD Server and optionally test the MD5 if given

COPY

  • Returns the path of file exported (copy) from LSD Server in the out path of the Legacy

LIST

  • Returns the list of files in the LSD Server Legacy in the form of (“L1 S1 D1 L2 S2 D2 …”) by block (LSDConstants.MAXIMUM_FILELIST)

LISTMD5

  • Returns the list of files in the LSD Server Legacy in the form of (“L1 S1 D1 MD51 L2 S2 D2 MD52 …”) by block (LSDConstants.MAXIMUM_FILELIST)

AEXIST

  • Returns the non existence of a list of files (“L S D …”) in LSD Server in format (“L S D …”).

AEXISTMD5

  • Returns the non existence of a list of files (“L S D MD5 …”) in LSD Server testing the MD5 in format (“L S D …”).

SESSIONINFO

Returns some information from session and status inside the String as

  • First 2 data are long for previous free memory and next free memory (after gc) = “free1 free2”

Then a list of

  • “name remoteAddress CreationTimeMs LastReadTimeMs LastWriteTimeMs NbReadMsg NbWriteMsg isIdle isConnected isClosing Command”

If (Command != LSDConstants.LSD_UNAVAILABLE), then the previous information is immediately followed for the current session by

  • “Status Running isClosed Unique Legacy Storage Document”

The separator is ‘#’ and not a blank character in contrary to other lists. It is defined in LSDConstants.(s)separator and can be changed here.

 

Admin Handler API

SHUTDOWNIMMEDIATE

  • Shutdown immediately all services

SHUTDOWN

  • Shutdown all services

RESTART

  • Stop then start user service (legacy = 1 => not compressed, 2 => compressed, 3 => Protected)

START

  • Start user service (legacy = 1 => not compressed, 2 => compressed, 3 => Protected)

STOP

  • Stop user service (legacy = 1 => not compressed, 2 => compressed, 3 => Protected)

ENABLE

  • Enable the Legacy dynamically

DISABLE

  • Disable the Legacy dynamically

CREATEDIR

  • Create the directory associated

WIPEDIR

  • Delete the directory associated; if Storage is given, start from it.

WIPERECURDIR

  • Delete recursively the directory associated; if Storage is given, start from it.

 

Commands interface

LSDServer side Commands

Starting LSDServer

Run openlsd.network.server.LSDServer with the openlsd.xml configuration file as unique argument.

Initialize the Database data

Run openlsd.server.admin.LSDInitDBFromConfigFiles to initialize the necessary data in the database with the openlsddb.xml client config file, the openlsd.xml server config file and the IP (or DNS entry) of LSD Server as arguments.

Check the consistency and repair

Run openlsd.server.admin.LSDCheckInDBThreaded to check the consistency of files in OpenLSD Server and their index in the database:

    • The configuration filename (openlsddb.xml config) as first arg,
    • The LSD configuration filename (openlsd.xml) as the second arg,
    • Follow by options as:
      • '-nbthread' follow by nb of thread,
      • '-nbstep' follow by the minimal nb of directories by thread,
      • '-nomd5' to not check md5,
      • '-noout' to only have a summary of checking,
      • '-nodate' to ignore the last time a check were done else check from that date,
      • '-cfunc' to use native C procedure to check the files on the filesystem (getFileFromPath),
      • '-nonio' to use no Nio function as Java procedure to check the file one the filesystem,
      • '-repair' to repair all problems (optional),
      • '-repairdb' to repair DB references,
      • '-repairlsd' to repair LSD references
      • or '-repairupdb' to repair DB references including MD5.
    • Finally, the last argument can be the legacy on which the check will be applied; else all legacies will be used (optional).

Run openlsd.server.admin.LSDCheckInDBThreadedDualLimit to check the consistency of files in OpenLSD Server and their index in the database with two dates as limits:

    • The configuration filename (openlsddb.xml config) as first arg,
    • The LSD configuration filename (openlsd.xml) as the second arg,
    • Follow by options as for LSDCheckInDBThreaded, except the last argument which is mandatory
    • A fourth argument must be the legacy on which the check will be applied, since in dual mode, the principle is to check one Legacy during a short time where any action is disabling on it. For instance no import or delete should be active between 1 minute before and after the check is launch. Doing this way, the production is allowed to be running during check except this short time. '-nodate' is allow as only the upper date can be set (the second date is always the current date). One should however not allow delete operation during the check, since they can not be taken correctly into account.

Run openlsd.server.admin.LSDCheckDBConsistency to check the consistency of files in the database: (could be run out of LSD Server)

    • The configuration filename (openlsddb.xml config) as first arg,
    • Follow by options as:
      • '-nbthread' follow by nb of thread,
      • '-repair' to repair all problems (optional),
      • or '-repairdb' to repair DB references.
    • Finally, the last argument can be the legacy on which the check will be applied; else all legacies will be used (optional).

Document handling

Run openlsd.server.admin.LSDGetPaths with the config filename openlsddb.xml, the LSD configuration filename (openlsd.xml), the Legacy where to get the real paths of documents and the filename containing the list of business entry line by line.

Run openlsd.server.admin.LSDCheckSimilar with the config filename openlsddb.xml, the LSD configuration filename (openlsd.xml), the Legacy where to get the real paths of documents and the filename to test for the binary existence (similarity) in the specified Legacy.

Http Service from LSD Server

Using a simple HTTP browser gives access to specific functions as :

  • Info on Sessions : shows some numbers statistics on global sessions by Handler
  • Info on Storage : shows some statistics on Physical Storages (short description)
  • Full info on Storage : show more detailed statistics on Physical Storages but could be very long (depend on disk speed access and number of files)
  • Clean : force the JVM to run garbage collection dynamically
  • Shutdown : shutdown the OpenLSD Server (use the password of server Admin to check the authorization)

With the option CSV, the data are returned in CSV format.

LSDClient side Commands

Administrate LSDServer

Run openlsd.appli.admin.LSDAdmin to shutdown or other similar operations one or many LSD server instance:

  • The configuration filename openlsddb.xml as first argument
  • The commande as second args
    • -alterindex, to recompute important indexes
    • -shutdown, to shutdown gracefully the OpenLSD Server
    • -shutdownimmediate, to shutdown immediately the OpenLSD Server
    • -enable lid, -disable lid, to enable or disable one Legacy dynamically
    • -stopx, -startx, -restartx where x = (u)ser | (c)omp |(p)rotected, to stop, start or restart one Mina Service from OpenLSD Server side
    • -createdir lid sid, -deletedir lid sid, to create or delete one specific Storage inside the given Legacy
    • -deleterecurdir lid sid, to delete recursively directories from one Storage in one Legacy
    • -createbetweendir lid sid1 sid2, to create new Storages inside one Legacy between two indexes
    • -createmoredir lid nb, to create nb new Storages inside one Legacy

Where lid must be ok and sid, sid1, sid2 must be ok for createdir and createbetweendir.

  • If two more args are given, they are the hostname and port on which the operation will be applied, else all admin services will be used.

Session Check

Run openlsd.appli.admin.LSDSessionCheck with the config filename openlsddb.xml and optionally the Legacy on which the session check will be done with the associated OpenLSD Server.

Delete Files

Run openlsd.appli.admin.LSDDelete with the config filename openlsddb.xml, optionnaly a number of threads ('-nbthread nb'), the Legacy where to delete the document and the filename containing the list of business entry line by line.

Document handling

Run openlsd.server.admin.LSDGetCopy with the config filename openlsddb.xml, the Legacy where to get the copy of documents and the filename containing the list of business entry line by line. This function should be run from the LSD Server since filepath are local to the server.

Import Files

Two kinds of import exist: one using an auto import, as a permanent processes that runs in background, and another one using import as one runs only which exit after finishing. Each kind can work using a file by file schema of import or a group of files schema of import (block mode).

According to the method, it can be local or not local to the LSD Server.

An optional method exists for local versions only where each document is first tested in a binary comparison with previously inserted documents.

 

1)  AUTOIMPORT

Run openlsd.appli.autoimport.LSDAutoImport.java or openlsd.appli.autoimport.LSDAutoImportBlock.java to run AutoImport where:

    • The configuration filename openlsddb.xml as first arg.
    • As second group of args:
      • -nbthread nbthread (set the number of threads)
      • -nbblock nbblock (set the number of files in one block for Block version only)
      • -dbconn (set that each thread will have its own db connexion)
      • -nodelete (don't delete file after import but save one file with same name and extension=".ok" or ".ko" according to the import status)
      • -delete (default option) delete file when correctly imported or save one file with the same name and extension=".ko"
      • -samelid lid where it specifies that it will be the same lid for all files (optimization) (by default, if not specified, not the same lid)
      • –delay nbseconds to set how long the autoimporter will wait between two scans in the directory, sets in seconds
      • –async to use asynchronous connection to OpenLSD Server with non blocking command
      • –compressed to use compressed connection with LSD Server
      • –testing for testing purpose only (add the timestamp to the business index)
      • -2out set the out status to 2 files in the import source directory, one ".ok", another one ".ko" according to the import status
      • -compressed set the connection mode to OpenLSD Server as using a compressed mode (useful only in Net mode where files can be compressed)
    • As third group of args:
      • -dir directory extension stopfile (each file has one file with same name and the extension containing "lid idb" where lid is not there if -samelid is set, and the existence of stopfile is tested every delay and if it exists, the process stops)

2)  IMPORT

Run openlsd.appli.autoimport.LSDImport.java or openlsd.appli.autoimport.LSDImportNet.java or openlsd.appli.autoimport.LSDImportBlock.java to run import through the filesystem or network support where:

    • The configuration filename openlsddb.xml as first arg.
    • As second group of args:
      • -nbthread nbthread (set the number of threads)
      • -nbblock nbblock (set the number of files in one block for Block version only)
      • -dbconn (set that each thread will have its own db connexion)
      • -nodelete (don't delete file after import but save one file with same name and extension=".ok" or ".ko" according to the import status)
      • -delete (default option) delete file when correctly imported or save one file with the same name and extension=".ko"
      • -samelid lid where it specifies that it will be the same lid for all files (optimization) (by default, if not specified, not the same lid)
      • –delay nbseconds to set how long the autoimporter will wait between two scans in the directory, sets in seconds
      • –async to use asynchronous connection to OpenLSD Server with non blocking command
      • –compressed to use compressed connection with LSD Server
      • –testing for testing purpose only (add the timestamp to the business index)
      • -2out set the out status to 2 files in the import source directory, one ".ok", another one ".ko" according to the import status
    • As third group of args:
      • -dir directory extension (each file has one file with same name and the extension containing "lid idb" where lid is not there if -samelid is set)
      • -file file lid idb
      • -source sourcefile (containing a list of "file lid idb" where lid is not there if -samelid is set)

3)  IMPORT on Server with Check Similar

Run openlsd.server.autoimport.LSDServerImportCheckSimilar.java or openlsd.server.autoimport.LSDServerImportBlockCheckSimilar.java to run import through the filesystem support and from the OpenLSD Server which hosts the Legacy where:

    • The configuration filename openlsddb.xml as first arg.

    • The LSD configuration filename (openlsd.xml) as the second arg

    • As second group of args:

      • -nbthread nbthread (set the number of threads)

      • -nbblock nbblock (set the number of files in one block for Block version only)

      • -dbconn (set that each thread will have its own db connexion)

      • -nodelete (don't delete file after import but save one file with same name and extension=".ok" or ".ko" according to the import status)

      • -delete (default option) delete file when correctly imported or save one file with the same name and extension=".ko"

      • -samelid lid where it specifies that it will be the same lid for all files (optimization) (by default, if not specified, not the same lid)

      • –delay nbseconds to set how long the autoimporter will wait between two scans in the directory, sets in seconds

      • –async to use asynchronous connection to OpenLSD Server with non blocking command

      • –compressed to use compressed connection with LSD Server

      • –testing for testing purpose only (add the timestamp to the business index)

      • -2out set the out status to 2 files in the import source directory, one ".ok", another one ".ko" according to the import status

    • As third group of args:

      • -dir directory extension (each file has one file with same name and the extension containing "lid idb" where lid is not there if -samelid is set)

      • -file file lid idb

      • -source sourcefile (containing a list of "file lid idb" where lid is not there if -samelid is set)

 

LSD in Multiple Legacies side Commands

Starting LSDOpHandler

Run openlsd.multiple.ophandler.LSDOpHandler to activate the process of Multiple Legacies support (clone of import and delete functions):

  • The LSD configuration filename (openlsd.xml) as the first arg,
  • Follow by options as:
    • '-nbthread' follow by nb of thread,
    • '-stop' follow by filepath (mandatory) to specify what is the stop file (the existence of this file stops the process),
    • '-delay' time in ms to specify the delay between two scans of database,
    • '-exportop' filepath to specify the path/filename where rank.export will be added as export of operation logs to enable redo operation on copied databases if any for ML support with database replication. The file format is "idip;op;lid;sid;did" where idip is the id of one ML LSD Server, op is 2 for insert, 3 for delete
    • '-nbstep' follow by a number of steps to set a limit on number of files referenced in exportop file where each file will contain up to numberStep*190*3*nbThread (190 comes from a limit of SQL VARCHAR2)
    • '-portctl' follow by a port on which the OpHandler will listen with its HTTP Administration interface 
    • '-passwd' follow by a password which will be used to check the shutdown order through the HTTP Admin interface

Http Service from LSDOpHandler

Using a simple HTTP browser gives access to specific functions as :

  • Info on Sessions : shows some numbers statistics on global sessions by Handler

  • Clean : force the JVM to run garbage collection dynamically and clean also the pool of connections

  • Shutdown : shutdown the LSDOpHandler Server (use the password the authorization)

With the option CSV, the data are returned in CSV format.

Initialize the Database data for ML Support

Run openlsd.multiple.admin.LSDInitDBMLFromConfigFiles to initialize the necessary data in the database with the openlsddb.xml client config file, the openlsd.xml server config file and the IP (or DNS entry) of the new target LSD Server as arguments.

Check the consistency and repair a clone of Legacy

Run openlsd.multiple.admin.LSDCheckInDBThreadedML to check the consistency of files in OpenLSD Server and their index in the database.

It only checks LSD consistency with database, but can only correct LSD storage since another ML can have the missing files. So no action is taken into the database. However, it will say if anything goes wrong too from the database.

  • The configuration filename (openlsddb.xml config) as first arg,
  • The LSD configuration filename (openlsd.xml) as the second arg,
  • Follow by options as:
    • '-nbthread' following by nb of thread,
    • '-nbstep' following by the minimal nb of directories by thread,
    • '-nomd5' to not check md5,
    • '-noout' to only have a summary of checking,
    • '-update' to update the last time check in no repair mode and without any error found,
    • '-nodate' to ignore the last time a check were done else check from that date,
    • '-cfunc' to use native C procedure to check the files on the filesystem (getFileFromPath),
    • '-nonio' to use no Nio function as Java procedure to check the file one the filesystem,
    • '-idip' follow by the Id of the IP used as target OpenLSD Server to check,
    • '-opfix' follow by the Id of the IP used as source OpenLSD Server to use as valid source (source for import) when the fix of check should be done through the OpHandler instead of directly,
    • '-repair' to repair all problems (optional),
    • or '-repairlsd' to repair LSD references
  • Finally, the last argument can be the legacy on which the check will be applied; else all legacies will be used (optional).

Run openlsd.multiple.admin.LSDCheckInDBThreadedDualLimitML to check the consistency of files in OpenLSD Server and their index in the database with two dates as limits:

  • The configuration filename (openlsddb.xml config) as first arg,
  • The LSD configuration filename (openlsd.xml) as the second arg,
  • Follow by options as for LSDCheckInDBThreadedML,
  • A fourth argument must be the legacy on which the check will be applied, since in dual mode, the principle is to check one Legacy during a short time where any action is disabling on it. For instance no import or delete should be active between 1 minute before and after the check is launch. Doing this way, the production is allowed to be running during check except this short time. '-nodate' is allow as only the upper date can be set (the second date is always the current date). One should however not allow delete operation during the check, since they can not be taken correctly into account.

Run openlsd.multiple.admin.LSDInitOpFromDB or openlsd.multiple.admin.LSDInitOpFromDBForStorage to synchronize all copies either in concurrent mode (slower since each update is done with synchronization) or in no concurrency mode (no import or delete are done during the procedure is running). You can use also two other versions openlsd.multiple.admin.LSDInitOpFromDBPL or openlsd.multiple.admin.LSDInitOpFromDBPLForStorage which use PL/SQL procedures to enable 2.5 faster updates. Those functions populates the Op table, so the Op Handler will take those entries to synchronize all OpenLSD Servers.

  • The configuration filename (openlsddb.xml config) as first arg,
  • Follow by options as:
    • '-lid' follow by the legacy to resynchronize
    • '-idip' follow by the Id for IpPort of the LSD Server source
    • '-concurrent' specify that each reference has to be commited (less performance)
    • '-sid' follow by the storage to resynchronize in the given legacy (only for ForStorage versions)

Delete Files

Run openlsd.multiple.admin.LSDDelete with the config filename openlsddb.xml, optionally the option ‘-idip’ follow by the idip of the OpenLSD Server Id port (Protected port) where to validate the first delete, optionnaly a number of threads ('-nbthread nb'), the Legacy where to delete the document and the filename containing the list of business entry line by line.

Import Files in ML mode

Two kinds of import exist: one using an auto import, as a permanent processes that runs in background, and another one using import as one runs only which exit after finishing. Each kind can work using a file by file schema of import or a group of files schema of import (block mode).

According to the method, it can be local or not local to the LSD Server.

An optional method exists for local versions only where each document is first tested in a binary comparison with previously inserted documents.

 

1)  AUTOIMPORT in ML mode

Run openlsd.multiple.autoimport.LSDAutoImport.java or openlsd.multiple.autoimport.LSDAutoImportBlock.java to run AutoImport where:

    • The configuration filename openlsddb.xml as first arg.

    • As second group of args:

      • -nbthread nbthread (set the number of threads)

      • -nbblock nbblock (set the number of files in one block for Block version only)

      • -dbconn (set that each thread will have its own db connexion)

      • -nodelete (don't delete file after import but save one file with same name and extension=".ok" or ".ko" according to the import status)

      • -delete (default option) delete file when correctly imported or save one file with the same name and extension=".ko"

      • -samelid lid where it specifies that it will be the same lid for all files (optimization) (by default, if not specified, not the same lid)

      • –delay nbseconds to set how long the autoimporter will wait between two scans in the directory, sets in seconds

      • –async to use asynchronous connection to OpenLSD Server with non blocking command

      • –compressed to use compressed connection with LSD Server

      • –testing for testing purpose only (add the timestamp to the business index)

      • -2out set the out status to 2 files in the import source directory, one ".ok", another one ".ko" according to the import status

      • -compressed set the connection mode to OpenLSD Server as using a compressed mode (useful only in Net mode where files can be compressed)

      • -idip set the IdIp of the target OpenLSD Server to use (should be either Protected or Compressed port according to -compressed option) (optional)

    • As third group of args:

      • -dir directory extension stopfile (each file has one file with same name and the extension containing "lid idb" where lid is not there if -samelid is set, and the existence of stopfile is tested every delay and if it exists, the process stops)

2)  IMPORT in ML mode

Run openlsd.multiple.autoimport.LSDImport.java or openlsd.multiple.autoimport.LSDImportNet.java or openlsd.multiple.autoimport.LSDImportBlock.java to run import through the filesystem or network support where:

    • The configuration filename openlsddb.xml as first arg.

    • As second group of args:

      • -nbthread nbthread (set the number of threads)

      • -nbblock nbblock (set the number of files in one block for Block version only)

      • -dbconn (set that each thread will have its own db connexion)

      • -nodelete (don't delete file after import but save one file with same name and extension=".ok" or ".ko" according to the import status)

      • -delete (default option) delete file when correctly imported or save one file with the same name and extension=".ko"

      • -samelid lid where it specifies that it will be the same lid for all files (optimization) (by default, if not specified, not the same lid)

      • –delay nbseconds to set how long the autoimporter will wait between two scans in the directory, sets in seconds

      • –async to use asynchronous connection to OpenLSD Server with non blocking command

      • –compressed to use compressed connection with LSD Server

      • –testing for testing purpose only (add the timestamp to the business index)

      • -2out set the out status to 2 files in the import source directory, one ".ok", another one ".ko" according to the import status

      • -idip set the IdIp of the target OpenLSD Server to use (should be either Protected or Compressed port according to -compressed option) (optional)

    • As third group of args:

      • -dir directory extension (each file has one file with same name and the extension containing "lid idb" where lid is not there if -samelid is set)

      • -file file lid idb

      • -source sourcefile (containing a list of "file lid idb" where lid is not there if -samelid is set)

3)  IMPORT on Server with Check Similar in ML mode

Run openlsd.multiple.server.autoimport.LSDServerImportCheckSimilar.java or openlsd.multiple.server.autoimport.LSDServerImportBlockCheckSimilar.java to run import through the filesystem support and from the OpenLSD Server which hosts the Legacy where:

    • The configuration filename openlsddb.xml as first arg.

    • The LSD configuration filename (openlsd.xml) as the second arg

    • As second group of args:

      • -nbthread nbthread (set the number of threads)

      • -nbblock nbblock (set the number of files in one block for Block version only)

      • -dbconn (set that each thread will have its own db connexion)

      • -nodelete (don't delete file after import but save one file with same name and extension=".ok" or ".ko" according to the import status)

      • -delete (default option) delete file when correctly imported or save one file with the same name and extension=".ko"

      • -samelid lid where it specifies that it will be the same lid for all files (optimization) (by default, if not specified, not the same lid)

      • –delay nbseconds to set how long the autoimporter will wait between two scans in the directory, sets in seconds

      • –async to use asynchronous connection to OpenLSD Server with non blocking command

      • –compressed to use compressed connection with LSD Server

      • –testing for testing purpose only (add the timestamp to the business index)

      • -2out set the out status to 2 files in the import source directory, one ".ok", another one ".ko" according to the import status

      • -idip set the IdIp of the target OpenLSD Server to use (should be either Protected or Compressed port according to -compressed option) (optional)

    • As third group of args:

      • -dir directory extension (each file has one file with same name and the extension containing "lid idb" where lid is not there if -samelid is set)

      • -file file lid idb

      • -source sourcefile (containing a list of "file lid idb" where lid is not there if -samelid is set)