OpenLSD ENOpenLSD

Download at SourceForge.net

ConceptLogic Technical Aspects Future Plan Howto Benchmarks
Home OpenLSD OpenLSM OpenR66

OpenLSD

About

 

You will find in this web site documentations, howto, API and download for OpenLSD, OpenLSM and OpenR66 projects.

 

Global web site (general information) is here (English) or ici (Français).

 

Any comments or contributions are welcomed!

 

Contact: openlsd @ free.fr or better use the Mailing list from Sourceforge: openlsd-devel

 

openlsd-small

OpenLSD : Document Archiving : Open Legacy Storage Document

 

Frédéric Brégier

 

Here you will find Documentation or web links on OpenLSD:

Well, as all open source projects, in order to make some progress to this software - framework, please give me your advice, your comment and your fix in order to help this project to go higher. Any developpers or users are welcomed ! Join us at use the Mailing list from Sourceforge: openlsd-devel

Thank you for your interest!

OpenLSD 09/05/2008 : V1.0.4

Web retrieve improvement for big file to prevent OOM

 

OpenLSD 02/05/2008 : V1.0.3

Bug fixed introduced in 1.0.2

OpenLSD 01/05/2008 : V1.0.2

Small bug and documentation fix

OpenLSD 26/04/2008 : V1.0.0

- Add more benchmarks and documentations

- Add functions to enable clean shutdown of Tomcat servlet (shutdown of pool of connections)

- Allow multi threaded version for Delete/ML operations

- Change PL/SQL procedures (and relative functions when no procedures) for commit at the right time

- Add KeepAlive feature to keep some connections opened (in pool or long terme connections)

- Correct order of execution in OpHandler when a Delete or Move operations occurs (sequential execution)

- Add two new Ops: Replace and Move

- Add '-opfix' option to CheckInDbThreadedOra...ML to correct problems through OpHandler and not directly (try to update from other source)

- Add Web support for Admin interface for OpHandler

- Correct bugs in OpHandler pool of connection

- Add '-idip' option on ML functions to specify prefered main target

- Add a PL procedure for InitOpFromDBPL(ForStorage) and for update for ophandler

For Import/Delete => create one PL with findAvailableIdOp and newOp at once

- LSDInitOpFromDB(ForStorage) improve performance with no concurrent option i.e. no action on LSD can be done (no import, no delete, no ophandler) while its running. If "-concurrent" option is given, all actions can take place in concurrency but at the price of lower performance (around 2.5 time slower from 500 reference by second to 200 reference by second)

- Create CheckSimilar function support (test before import of binary similarity with already imported documents)

- New index have been added into database (import, checkSimilar)

- Add '-2out' option to output ok and ko status in two separated files in Import functions. This option is the default in Delete functions so no '-2out' option is proposed there.

- LimCon => delete from class names since it is not explicit and clear, so remove

- Ophandler : create log of database request or at least log of changes:

Create LSDOpHandlerExport that takes one file of log of changes and save one file with full information on document (MD5, Size, Business indexes).

Create LSDOpHandlerImport that takes the result of LSDOpHandlerExport to import those data into replicated database.

Add a toCSV function in Document and Business class

- CheckInDBThreadedDualLimit: add limitade as upper bound, so that freezing Import/Delete is very short

- Lock db in LSDDbStorage.checkStoragexx called from CheckInDBThreaded and CheckDBConsistency

- Add an update option in CheckInDbThreaded to enable the update of the last date of check even in non repair mode but only if no error is found.

- Fix performance issue in ML support from OpHandler and Import and Delete function in ML mode

- CheckInDBThreaded using directory access : try to compute a better dynamic nb of threads

- InitDBxxFromConfigFiles include init of OpHandler basic reference

- Improve performance in LSDInitOpFromDb(FromStorage) by reducing lock in database

- Create a new LSDInitOpFromDbPL(FromStorage) by doing work inside the database directly.

- Correct Block Import where samelid is Mandatory and improve performance

- Check in ML does not delete Database reference since its objective is to only correct concistency of ML Legacy and not the database. One have to use the "no ML" Check function to get the consistency from database too. However, the ML version will say if anything goes wrong too from the database.

- Make a difference between CLIENT_THREAD (Pool in MINA) and CLIENT_INSTANCE_THREAD (Pool of client)

- Adapt to Mina-V2-Pre-M2

- Change LOG4J to SLF4J as MINA does, using logback as example

- Create openlsddu to replace Unix command "du -ks" too slow and not portable

- Create an equivalent in Java of "df -kP" and "du -ks" and optimize getFileFromPath in Java. However, C functions are 3 times faster than Java version (except under MS Vista).

The equivalent of "df -kP" is only valid in JDK6, so the code is in comment to enable JDK5 usage. If one needs this version, it has to enable in LSDConstants the ISJAVA6 to true and to uncomment the corresponding code in the same class as getGlobalUsedFreeSpaceJava6.

- Fix some bugs in import by block, ML support

- Introduce a Web support to control the LSD Server (check, info session, info storage, shutdown with password check)

- Improve the shutdown process with several ways (System Hook on exit, Signal INT and USR1, HTTP, message)

- Change order in ByteBuffer initialization according to Mina's List

- Change some functions as abstract (those that need to be defined in extension)

- Add status in LSDStreamGobbler in start() form

OpenLSD 08/06/2007 : V0.9.3

- Fix getDocument in LSDDemuxingIoHandler which coult let open a Document when opening a new one

- Adapt to last Mina Trunk version

 

OpenLSD 07/01/2007 : V0.9.2

- Create LSDInitOpFromDBForStorage that takes 1 Legacy 1 Storage and 1 IDIP id

- Include in LSDDbIpPort the rank of order for Multiple Legacies (For one Legacy, the rank of OpenLSD Server that contributes to it) as info in context

- Correct the function getIpPort(LSDDbAdmin admin,int function,long legacy) adding the legacy argument to get the list of IpPort assigned to one Legacy on this function (and not all IpPort assigned to this function on any Legacy). The original function is kept for some services that needs to know all IpPort for all Legacies for one particular function.

- Add PostGreSQL support (fully tested and supported)

- Web Common services return now an int to distinguish which kind of error occurs.

- Correct PostGreSQL Storage check since '/' gives an integer division so we introduce a double for blocksize

- Add test of existence of each file (source and destination) during work of Import in Op Handler

- For ML, admin port needs the info and get support in order to check the existence of files

OpenLSD 06/18/2007 : V0.9.1

- Create AutoImport in ML mode

- Create LSDInitOpFromDB to populate Op table after a crash to resynchronize two or more Legacies

- Correct Op Handler such that if a document is trying to be inserted twice, it is removed from the Op table

- Concept and logic PDF take the number of the version as number.

OpenLSD 06/17/2007 : V0.9.0
- Bug corrected in OpenLSDWebImpl LSDAdminLocalServletAction
- Correct Import from Web with a similar interface than Get from Web: Put Servlets from OpenLSDWebImpl and LSDWebImport, LSDWebImportNet in WebSupport from OpenLSD
- Correct Servlet and improve common setup in init part of the servlet through LSDWebCommon.
- Add the LSDPoolConnectionAbstract in client session side to take care of pooled connections (in close method)
- Add the LSDPoolConnectionAbstract support in web client LSDWebClientGetPool
- Add Delete support at OpenLSDWebImpl in LSDDeleteServletAction and corresponding at WebSupport in LSDWebDelete
- Prepare Multiple Legacies support by adapting classes from openlsd.appli.autoimport.commonimport LSDImportFile, LSDImportFileBlock and LSDImportFileNet in extended classes
- Put LSDBusinessImpl class name in LSD DB Config file so as to remain generic as possible. Simplify the process of extension of the Framework by only extending LSDBusiness class. Clean the code accordingly.
- Create LSDGetCopy (copy of LSD files in outpath), LSDGetPaths (get paths of files in LSD)
- Create LSDCheckDBConsistency (verify correctness of Storage from Documents in database)
- Factorize some functions in LSDDbStorage for check consistency
- Change "Empty" class to "Abstract" class (delete, import)
- Change size handling in Storage by taking into account size of one block in the filesystem (LSDConstants.sizeblockfs = 4k and LSDConstants.computeSize) => later could be in Legacy
//XXX SIZE signs where it is taken into account.
JFS2 supports multiple file system block sizes of 512, 1024, 2048, and 4096, with 4 KBytes as standard.
EXT3 supports multiple file system block sizes of 1024, 2048 and 4096, with 4 KBytes as standard.
NTFS supports sizes of clusters from 512 bytes up to 64 KBytes, with 4 KBytes as standard.
- Change stopSession in Server for write call in DemuxIoHandler by sessionStopNoClose which not closes the conn
- Remove JDK5/6 version of jars, only left JDK6 but gives source of external dependant projects (MINA was missing)
- Correct index in LSDOP table
- finish multiple legacies support with one OpHandler with one database, extending Op with MD5 support //XXX REUSE
- Remove from multiple Legacies, Get from multiple Legacies
- Create support for Doxygen documentation generator

OpenLSD 05/29/2007 : V0.8.1
Very small update but nevertheless usefull - I hope ;-)
- Small refactorization on LSDCheckInDBAbstract from LSDCheckInDBImpl
- Add some error logging
- Create a separate package containing the same example of extension of LSDBusinessImpl in Example in a separate project OpenLSDImpl with the correct class name openlsd.business instead of openlsd.newbusiness
- Improve documentations


OpenLSD 05/28/2007 : V0.8.0

- Refactorization of Import, Check, Delete, Business information in LSDExtDbDocument

- Simplify the process of extension by providing explicit Implementation classes

that should be modify and adapt to the business logic (BusinessImpl)

LSDBusinessImpl: implement LSDBusiness that contains all data for Business

LSDCheckInDbImpl: implement LSCheckInDB that contains all data for Business

LSDDeleteEmptyImpl: implement LSDDeleteEmpty that contains all data for Business

LSDImportEmptyImpl: implement LSDImportEmpty that contains all data for Business

LSDInitExtDbDocumentImpl: implement initialization of internal LSDBusiness representation in LSDExtDbDocument

that contains all data for Business

- Correct Bug in LSDDelete

- Refactorize Directories and jars in Common - Common.jar, Server and ServerApp - Server.jar,

ClientSupport and ClientApp - Client.jar , BusinessImpl - Impl.jar, WebSupport - Web.jar

- Update docs and provide new Howtos

OpenLSD 05/22/2007 : V0.7.4
Clean Import functions:
- LSDImportFileBlock.java (clean comment)
- LSDCommonFunctions (delete some unused functions and create Specific support for the 1-file procedure)
- LSDSpecific, LSDMySQL, LSDOracle delete unused functions and create specific support for the 1-file procedure
- LSDDbDocument, LSDExtDbocument remove unused functions (find_first)
- Procedure InsertDocNoCommit : delete of idm (not useful)

OpenLSD 05/19/2007 : V0.7.3
- Improve documentations (Logic and Concept, Howto)
- Refactorize internal functions for Server configuration file reading.
- Create a new function to initialize database information based on the xml configuration files.


OpenLSD 05/14/2007 : V0.7.2
- Add again Unix scripts version and include documentations.

OpenLSD 05/13/2007 : V0.7.1
- Remove Unix scripts version and correct MSDOS scripts. If I find the time I will publish a new version with correct Unix script but anyone can do this as they are very simple.

 

OpenLSD 05/12/2007 : V0.7
This is the first public version. This version is not complete but works under the following conditions:
- Oracle support release
- DOS (Windows from XP to Vista tested) release
- Import, AutoImport, Admin, Delete, Init, Check Support
- Tomcat 5.5 support and Tomcat 6 support
- JVM 5 or 6 support

Done in this release:
- Fix bug regarding to Oracle procedures
- Remove some bug from MySQL version but not finished (:-(

Todo:
- Fix MySQL version (as was Oracle in 0.7) (in progress)
- Add Cache support (read and write) (in progress)
- Add Multiple location support for one Legacy (mirroring) (in progress)
- Add Spring support (to do)
- Improve documentation (in progress)
- Gui Admin (in progress)