ParallelVarFiler 1.25

[aminer10]

Hello,

I have updated ParallelVarFiler to version 1.25, i have added carefully {Try Except}  in all the methods so that to
catch exceptions and release memory etc. so the user don't need to manage the exceptions from outside ParallelVarFiler
cause it's innefficient to do it like that, hence ParallelVarFiler has become more proffessional and more stable, so if you need to use ParallelVarFiler i advice you to upgrade to version 1.25 that is more stable now.

As you have noticed you can save manually the parallel hashtable to the harddisk by using the SaveToStream() or SaveToString() or SaveToFile() methods, but to call those methods when you are running multiple threads, it's mandatory that you pass a file name to the constructor, if you don't pass a file name to the constructor that means you are using an in-memory parallel hashtable, so if you are using an in-memory parallel hashtable you have to stop your threads before calling  SaveToStream() or SaveToString() or SaveToFile() methods, and restart after that your threads, but if you are saving automaticly your parallel hashtable to the hardisk by passing a filename to the constructor, then you can safetly call the SaveToStream() or SaveToString() or SaveToFile() methods even if you are using mutiple threads.


Author: Amine Moulay Ramdane


This is Parallel Variable filer and streamer for Delphi and Freepascal that can be used also as a parallel hashtable and that uses ParallelHashList (Parallel Hashtable) with O(1) best case and O(log(n)) worst case access that uses lock striping and lightweight MREWs(multiple-readers-exclusive-writer) , this allows multiple threads to write and read concurently. also ParallelHashList maintains an independant counter , that counts the number of entries , for each segment of the hashtable and uses a lock for each counter, this is also for better scalability.

Description:

ParallelVarFiler is a Parallel HashTable that can be saved automaticly or manually to a file or to a stream or to a string and it can be restored from  a file or from a stream or from a string back to the Hashtable in memory, and it's fault tolerant to power failures etc.

You can collect different types of variables and save them into one file or stream. TParallelVarFiler reads and writes on files, streams and Strings. You can collect Strings, Integers and Dates, in fact everything that can be stored in a variant. In addition you can collect Streams.

You can use also ParallelVarFiler to send parameters over network. It can be used for example to share data by any IPC mechanism.

And please look at test.pas and test1.pas a parallel variable filer examples - compile and execute them...

Now you can use ParallelVarFiler as a small to medium database, i have added a Clean() and DeletedItems() methods and i have changed the constructor, now you can pass a file name to the constructor where the data will be wrote and updated and deleted... if you pass and empty file name the data will be updated and wrote and deleted only in memory. When you use a file name in the constructor, many readers and one writer can proceed concurrently. But when you pass an empty file name to the constructor, many writers and many readers can proceed concurrently in memory. and when you read a data item it only use the parallel hashtable in memory, hardisk is not used.

And when you pass a file name to the constructor and delete or update the items, those items will be marked deleted from the file, and you can use DeletedItems() to see how many data items are marked deleted in the file, and use after that the Clean() method to delete completly those data items from the file.

How can you store mutiple data inside a database row and map it to a key ?

You can simply use Parallel Variable Filer for that and stream your mutiple data variants and map them to a key...

As you know ParallelVarFiler is using ParallelHashList (a parallel hashtable that scales on multicores), so when you are using the GetKeys() method , and you want to get the data of those keys don't forget to test if the variant is not empty by using the VarIsEmpty() function, and when you are using GetStream() you can test if the data exists by testing the return boolean value from GetStream().

ParallelVarFiler is easy to learn, i have documented all the methods , and please read inside ParallelVarFiler.pas about
them.

Now ParallelVarFiler is Fault tolerant to power failures etc. i have done a simulation of power failures and data file damages and ParallelVarFiler is recovering from power failures and damages of the data file ...

As you know ParallelVarFiler is using ParallelHashList (a parallel hashtable), but as you know ParallelHashList is cache unfriendly since it's using a hash mechanism etc. and as you know ParallelVarFiler and ParallelHashlist  are memory bound, so since ParallelHashList is cache unfriendly and  it's memory bound so i don't think ParallelVarFiler or ParallelHahsList will scale very well, to avoid this problem you have to use something like replication across computers using TCP/IP and using your database in each computer and load balance between computers, but even if you are using replication and load balancing this can make the memory and hardisk truly parallel but this will not avoid the problem with the network bandwidth limitation. But ParallelVarFiler and ParallelHashlist are scaling to a certain point and they are fast and they are still usefull.

Please see the test.pas example inside the zip file to see how i am using it...

But please read the following:

This software is provided on an "as-is" basis, with no warranties, express or implied. The entire risk and liability of using it is yours. Any damages resulting from the use or misuse of this software will be the responsibility of the user.

please look at the test.pas and test1.pas example inside the zip file to see how to use ParallelVarFiler...


You can download ParallelVarFiler from:

http://pages.videotron.com/aminer/


Here is the methods that have been implemented:

PUBLIC METHODS:

constructor Create(file1:string,size,mrews:integer;casesensitive:boolean);

- Creates a new VarFiler ready to use, with size and with mrews number of mrewa and casesensitive

for case sensitive keys,the number of MREWS(multiple-readers-exclusive-writer) must be less or

equal to the Hashtable size and file1 is the the file to write to, if file1 is an empty string

the data will be wrote only to memory and use SaveToFile() to write to a file.

destructor Destroy;

- Destroys the ParallelVarFiler object and cleans up.

procedure Clear;

- Deletes all contents.

function Clean:boolean

- Cleans the deleted items from the file.

function DeletedItems:integer

- Returns the number of items marked deleted.

function LoadData:boolean

- Loads the data from the file passed to the constructor.

function Delete(Name : String):boolean;

- Deletes the variable with Name.

function Exists(Name : String) : Boolean;

- Returns True if a variable with Name exists

procedure GetKeys(Strings : TstringList);

- Fills up a TStringList descendant with all the keys.

function Count : Integer;

- Returns the number of variables

function Add(Name : String; Value : Variant ):boolean;

- Adds a new variable , given the Name

function AddStream( Name : String;Stream : TStream ):boolean;

- Adds a new Stream, given the Name

function Update(Name : String; Value : Variant):boolean;

- Updates the new variable , given the Name

function UpdateStream(Name : String; Stream : TStream ):boolean;

- Updates a new Stream , given the Name

function GetStream(Name : String;Stream : TStream):boolean;

- Fills up the stream with data

procedure SaveToStream(Stream : TStream);

- Saves all variables, streams and Graphics to a stream.

procedure SaveToFile(FileName : String);

- Saves all variables, streams and Graphics to a file.

procedure SaveToString(out S : String);
- Saves all variables, streams and Graphics to a string.

 procedure SaveToStreamAndDelete(Stream : TStream);

   - Saves all variables, streams and Graphics to a stream
     and delete after that all the data inside the hashtable
     and inside the file.
  
   procedure SaveToFileAndDelete(FileName : String);

   - Saves all variables, streams and Graphics to a file.
     and delete after that all the data inside the hashtable
     and inside the file.

   procedure SaveToStringAndDelete(out S : String);

   - Saves all variables, streams and Graphics to a string.
     and delete after that all the data inside the hashtable
     and inside the file.

function LoadFromStream(Stream : TStream):boolean;

- Loads all variables, streams and Graphics from a stream.

function LoadFromFile(FileName : String):boolean;

- Loads all variables, streams and Graphics from a File.

function LoadFromString(S : String):boolean;

- Loads all variables, streams and Graphics from a string.

function Stream2String(Stream1: TStream;var Mystring:string): boolean;

procedure String2Stream(var String2BeSaved:string; Stream1: TStream);

procedure Variant2Stream(var VR:variant; Stream1: TStream);

function Stream2Variant(Stream1: TStream;var VR:variant): boolean;


PUBLIC PROPERTIES:

Items : Variant

- Gets the value (indexed).


Language: FPC Pascal v2.2.0+ and Lazarus / Delphi 7 to 2007: http://www.freepascal.org/

Operating Systems: Win , Linux and Mac (x86).

Required FPC switches: -O3 -Sd -dFPC -dWin32 -dFreePascal

-Sd for delphi mode....

Required Delphi switches: -DMSWINDOWS -$H+

For Delphi use -DDelphi


Thank you,
Amine Moulay Ramdane.