Top Banner

ZCachePro Installation and Programming

The following is developer support documentation. This documentation reflects the latest information available. We intend to expand and improve this document over time, so you may want to mark this page and return often if you are a developer using ZCachePro.


Platform Installation Options

ZCachePro Core is a headless Java application designed to run as an application or as a service on a wide variety of support OS's. You need to use the proper installation file for your specific operating system. A generic version without the java service wrapper is available for currently unsupported operating systems. Currently supported OS's include the following:

  • Aix
  • Freebsd
  • HP-ux
  • HP-ux64
  • Linux-i386 (32 bit)
  • Linux (64 bit)
  • MacOSx
  • Solaris
  • Windows
  • Generic, no wrapper services.

Using the Java Service Wrapper allows you to start, stop, and restart ZCachePro Core. The wrapper allows you to run ZCachePro Core as a daemon process. The wrapper also provides nice logging services for System messages.

JDBC Driver Configuration

In the working directory where ZCachePro is located is a configuration file called "ZCacheProJDBCDrivers.cfg". This file is where JDBC driver interfaces are described so that ZCachePro can connect. Your startup of ZCachePro should include the classpath information for your JDBC drivers. The default ZCacheProJDBCDrivers.cfg file looks like this:

 # Modification of this file may be required to support different JDBC
 # drivers.  
 # These settings were tested with Oracle's and with
 # mysql-connector-java-5.0.7-bin.jar for MySQL.  
 # C4N is the Class.forName string used to instantiate the service. 
 # CS is the connect string
 # DMGC is an indication of the parameters being passed to DriverManager.getConnection()
 # call.  cs = connection string, unm = username, pwd = password
 # These three examples should provide all the information you need to make required
 # modifications.  Backup this file before making changes.

ZCachePro only requires a small subset of the JDBC driver's features, so there is more tolerance of variation in drivers than one might expect. If you need to change these default settings, remember that Java is case sensitive.

All substitutable run time parameters are bracketed by $'s. They are:

$IPADDRESS$ = IP address of the target database server
$PORT$ = Port number of the target database server
$SID$ = Database ID
$UNAME$ = username
$PWD$ = password of user

Not all these parameters are required for every driver, as you can see in the configuration file listing. DO NOT HARDCODE RUNTIME PARAMETERS!! Information will be inserted at these locations at runtime with information you have provided in ZCachePro configuration.

Data normalization requires modification to ZCachePro Core, so adding support for a currently unsupported database needs to be done by ZeroPoint. Contact us if you need support for a currently unsupported database. This file might require modification if a jdbc driver specification changes, but that is not likely to affect currently supported databases. Since ZCachePro Core only requires only a very standard subset of the jdbc driver feature set, changing this file is most likely not going to be required.

Pre configuration

With this new version ZCachePro Core analyses how data is accessed, so that ZCachePro Core can configure itself to provide data requests as quickly as possible, while using memory efficiently. As each new request is analyzed, and entry is stored in ZCacheProPreCfgInfo.cfg.

Pre configuration (ZCacheProPreCfgInfo.cfg) entries take the following form:

 T:DEMO2~I:[ID]~F:[FNAME, LNAME]~C:51296~S:2 These entries can be deleted before startup. You can also add
		  definitions in advance if you follow the precise form as shown.
 S: is for internal use only. If you add a new record then you can put '0' in place of the '2'.
 C: is for internal use only, but refers to the database size when the original search was requested. 
          If you are manually creating an entry you can replace '51296' with '0'.
 T: is the table alias.
 I: and F: are the column lists used for the index and the fields you are returning respectively. 
          The form is [column1Name, column2Name, ...]

As a developer you can use precfg information to determine whether you can optimize queries to reduce memory requirements. If two or three queries a identical except that there are slightly different columns in the fields specification, you can consolidate queries into a single query form. This will significantly reduce the amount of memory you need for caching, and reduce the amount of time required to load data from the SQL server. Studying the precfg file can be very helpful.

ZCachePro Core Configuration Info

Here is an example ZCachePro.cfg file:

#ZCacheProDataServer Configuration File
#Fri May 5 12:06:11 MDT 2000
SecurityPrefix=NoNe            ; This token is used as a simple request authentication
Threads=100                    ; The number of concurrent requests you want to handle.  
                               ; 100-500 is a typical range of values
IPAddress=          ; This is the IP address you want to bind ZCachePro to.  
Port=50104                     ; This is the port you want to use
ZipTmpFilesPath=zipTmpFiles    ; A temporary directory for zip temporary files 
FilesRoot=filesRoot            ; The root directory for file transfers
RemoteControlPort=9987         ; The IP address of the server, and this port defined remote access (0 disables)
RemoteSecurityPrefix=NoNe      ; This token is used as a simple remote request authentication
WriteLockTimeoutMinutes        ; This establishes the max length of time in minutes that a record can be locked

The correct number of threads is variable and application dependent. Each thread uses memory, and as the number of threads increase there is a reduction in efficiency. If you have a lot of memory, and the ratio of reads to writes is very high then increasing the number of threads might be a good idea. We have tested with 1000 threads, but 100 to 500 threads will be best in most cases. Once your server has reached maximum utilization, increasing threads will not give you much of an improvement in throughput, but you will be able to service more concurrent requests. Too many threads can actually degrade system performance.

As you gain experience using ZCachePro Core and after monitoring system parameters with ZCacheProRC you can get a 'feel' for adjusting the number of threads. Requests are handles so quickly that a hundred threads will likely be a lot more than you need. A hundred concurrent requests can be handled in 1 ms, so you don't need to be super aggressive unless your doing something unusual in your application.

Cache memory requirements are very application sensitive.

Maximum tested heap is 1.6GB on 32 bit systems and >6GB on 64 bit operating systems. Be sure to use the 64 bit version of java (if you have a 64 bit server) for the server. If you specify too large a heap, java will not create a JVM. The heap size is specified in the wrapper.conf file in conf directory of your working directory. If you are running Solaris 10 with a Sun UltraSparc processor(s) you can set the heap to 3.5GB. Maximum heap will not be used unless needed, but if you have the memory you can allocate it and guarantee that performance will be maximized. In general, se as large a heap value for Java as you can for bets results. If heap is too small performance is hampered, but will still give increased performance. But for maximum benefit use a large heap.

If you watch ZCacheProRC you can see how the heap is being utilized over time.

Database Configuration

Configuring a database is fairly simple. The first thing to do is to assign a name. This database name is a name that will be used to identify this specific database engine.

 cfg image   The Database Name

The reason for this is to simplify identification of two databases with similar or the same table names and structures. This name can be OldSystemData or whatever will help you to identify which database you are referring to.

If you wish to modify an existing name, you can use your mouse to 'select' the name you wish to modify, and hit <enter>.

The existing name will pop into an edit window, and you can modify it and hit <enter> to save the new name.

To insert a new database alias, all you do is highlight the field and hit <insert>. An edit window will pop up and you can type the new database name, (an alias) and it will be added to the database name list.

DBE Internal User Names

These names are the database user names that in this case are created in Oracle to establish data access rights.

These names should be assigned read/write privileges and have been previously defined in Oracle. To add a name to the list click on the list with your mouse and hit the <insert> key. This will pop up an edit window. Just type in the name and hit <enter>. This will add the user to the list.

To delete a name, select a name in the list with your mouse and hit the delete key.

The Password

This password is the password used by the DBE Internal User Names you have just defined. When configuring Oracle security for the users you want to use, be sure that they all have the same password. This is the password you enter into the password field.

The Database Engine

This is a drop down list of supported database engines. In the future this list will be expanded. For now, the list is limited to Oracle, and Other.

After filling in the rest of the fields, click the <Save Changes> button. This will save your changes.

Tables Configuration

The table configuration tool works in a very similar way to the database configuration tool. Editing options are the same.

Tables are a subset of the database. The 'Table Alias Name' is the name you wish to use in referring to this specific table. The 'Table Reference Name' is the table name reference required to access the table. In the example on the left, the table needs to be referenced as MATEX.DBUSER but we can just use DBUSER to refer to the table.

The Unique Field Name is the name of a unique (no duplicates) field that ZCachePro can use to identify a specific record. All tables are required to have a numeric column that will used as a unique id field. This is required for proper operation of ZCachePro. The uniqueid column can be named as you prefer. The "Enable Auto-Increment checkbox is no longer an option.

After the fields are filled in, be sure to select 'Save Changes' so that the changes can be made to the configuration file.

EXIT AND RESTART ZCachePro so that these changes will be in effect. In this version a restart is required.

Sample Request Structure Format

The request structure is as simple as it can be. More exotic searches can use stored procedures, aliases, etc, but for what we need this format will do the job.

Any extra intelligence can (and will) be applied on the server side. This will keep the request easy to implement.

The following request will be used as an example:

AccessKey_$%#@!##             ; The first line is actually a key, required to gain access to ZCachePro services
                              ; The next symbol will be W: for a write operation, or R: for a read operation.  
                              ; These are mutually exclusive terms.  Only one or the other may be used.R: or
                              ; R: is a read operation 
                              ; RL: is a read that locks the record
                              ; [W:I, W:U, W:D, W:CI] ; The initial read request (R:) has no ID assigned
                              ; W:D signals a Delete write operation
                              ; W:I signals an Insert write operation
                              ; W:U signals an Update write operation         
                              ; W:CI signals a Cached Insert write operation
                              ; WL: same as a normal write but clears a lock
                              ; WLC: (after v1.15) signals a Write Lock Clear
U:GenericServices, Inc.       ; Company info for the log.  In case there is a user with issues.
T:ADAPTERS                    ; T=type I=index F=FieldName  Type is the normally the database name.
I:COMPANYNAME;Mag             ; This lines starts with I: indicating that this is an index key
                              ; The field to use as an index is COMPANYNAME.
                              ; If there is a ':' the text following is a complete key value
                              ; A ';' indicates that the text following is a partial key
I:BUSTYPE;ISA                 ; This is a secondary index key, with "ISA" as a partial key
F:PRODUCTID                   ; This is a return field, and is returned as the first field
F:DESCRIPTION                 ; This field's content will sit in the second position
                              ; FRC: (after v1.15) specifies a Field Relative Change request which is valid for
                              ; for integer or real values. This is ideal for incrementing or decrementing values.
O:0                           ; Offset into the list  
                              ; NOTE: O:R:0 means that the offset is relative to current position information
                              ; which was supplied as part of the R: token listed above.  If the request returns a 
                              ; 'no records found' error the most likely , then the request must be made with a 
                              ; specific offset specified here (i.e. O:342) and when the request is returned new 
                              ; extended position information that is valid will be returned. The next request
                              ; then can be relative.
N:50	                        ; Number of lines (records) to return 
                              ; Offset and Number can be used to implement extremely fast 
                              ; list boxes in real time. Ask if you want to know more...

Each line of the request is terminated with a '\n'. The last line of the request always ends with "\n\n".

All requests are in clear text. There are no empty lines. All transmitted requests will have every line filled. If you are using the ZCacheProClient bean and also using contexts, the results are sent in encrypted form, which the bean unencrypts the data for you.

Any field can be used as an index.

All searches are case insensitive, which makes things easier.

The "R:" line can be used to search an existing cached data object. This feature is only useful for block data transactions only! This extra information is used for record style navigation through your data. Partial searches and exact searches do not use this information at this time.

If you have already made a request with a large number of available records you can use this number (that is returned from every request) to get another chunk of data much faster. If you do not specify the extra 'R:' information, the server will look to see if a search with the same parameters has been made and is in cache. By using the extended information, you start right where you left off. which dramatically decreases response time for relational navigation through the data set.

A typical example might look like this: R:3:5:20:2115150400 The first 3 parameters (3:5:20) give the position of the last returned row of data. Future accesses can the start where the previous request left off. The last portion of the string is the timestamp reflecting server start time. This string is in the following format "ssmmHHddMMyy". This portion is critical because the assigned object numbers change when the ZCachePro is restarted. Using this information, a check is made by ZCachePro to verify that the reference information is still valid.

The "T:" references the alias you defined when you configured ZCachePro and refers to a database table name.

The "N:" should contain the number of records you want back. The reply will contain how many matching records are available. If you want more than you requested you can set your offset and request the difference between what you already got and what is there and get the rest. You can use the search id to get what you want immediately from cache. N:All will return all matching records.

The text of this request is order specific. The data returned will also be sorted by the order of the fields automatically. In this example the product Id's would be in ascending order.

Reply Format

The following is an example of the reply you would get from the server:

ZCachePro_v1.15          ; This tells the client what syntax is being used for the reply
R:43:100:40:2115150400   ; As you can see positional data was returned from the request
ERRMSG:                  ; A string that is an error message, only when an error occurs
N:1                      ; 1 item or record is being returned
A:1                      ; There is only 1 record available

17542~CNE2000.LAN~14250~11/25/97 ;The data returned looks like this

A tilde '~' is used as a delimiter between fields so that data can be returned with important white space preserved, but with each field trimmed as much as possible. All data is returned in clear text.

If a request was made properly the only time you would get an error returned is when there is no match for the request. If there is an error, an error string will indicate the nature of the error.

There is a blank line between header info and data, so you can know where the header
ends and the data begins. There is no returned data on a failed request.


Dates will be handled in a standardized format. ZCachePro will convert the date properly to make the request, so that you can always search for dates by the same mechanism. Data returned will be in the same form as the request. You may want to reformat the reply to suit your application, but consistency simplifies things.

Additionally, partial requests are automatically in the proper order of precedence. The dashes are important. Slashes will not work.

I would recommend that all date index requests should be partial. In a case where tenths of a second are used you could fail a search. If you can be sure that this is NOT the case then go ahead and use an exact key if you want. It's up to you....

The date format is YYYY-MM-DD HH:MM:SS (24 Hrs)

The date records used in Oracle contain more than just the dates, but also include hours, minutes, and seconds. If you don't want seconds, you can easily crop that information from the returned string. There will be cases where seconds could be important, so they are there if you need them.

If there is a case where this data format will not work, please let me know. We can always extend this spec. We want to keep this protocol simple as possible.

The database field name must include the string "DATE". For now this is how we will identify whether the field is indeed a data field. I have a better solution but I don't want to invest the time required (for now) to implement it.


There is bean that makes this feature easy to implement, but for those who want to know, here is the process:

Files that are loaded on to the server, should be placed in a directory called FILES, in the working directory of ZCachePro and will be the ROOT directory for file transfers. Any directory structure can be used beneath this folder, but will have to specify an accurate path at download time. For security purposes no facility will be provided to search or expose the contents of the directory.

File dates and times will be preserved by using Zip compression methods. Each file will be zipped at transfer time, and an API will be provided to unzip the file on the users system. Files previously downloaded will be in cache and already zipped. Maximum file size to cache is a settable configuration item and is already handled by ZCachePro.

File transfers that are aborted can be restarted where the last transfer left off automatically. All the user has to do is reconnect to the Internet.

At this time I'm adding the Server-side capabilities required, and then I'll get the client side methods written. The file will arrive in binary to the client, posted to the disk as file$$$$.zip . After decompression this temporary file will be deleted.

The ability to get a specific file is being added to the protocol. This will require a reserved Type. The type will be FILEXFR. To request a specific file you would format a request as follows:

AccessKey_$%#@!##                   ; The access key
R:                                  ; Initial request has no ID assigned
U:GenericServices, Inc.             ; Company info for the log
T:FILEXFR                           ; T=type I=index
F:test/filename.ext                 ; request
N:1000000                           ; Number of bytes to return
O:0                                 ; Offset into file


ZCachePro_v1.15                        ;  	
R:44:56:61:2115150400               ; If transfer fails, you can optionally reuse the cached copy. 
N:14786                             ; If the file is large you can do a partial download.  Usually this will be the
                                    ; same as A:  
A:14786                             ; The file size

A binary data stream of the file will follow after the empty line that terminates the header. It will be possible for you to reload on a failed transfer from where you left off. Perhaps you won't need this often, but it's a built-in feature that will be there if you need it.

You can wait for me to complete the APIs or you can write your own. As you can see the protocol is simple.



AccessKey_$%#@!##                   ; The access key
R:                                  ; Initial request has no ID assigned
U:GenericServices, Inc.             ; Company info for the log
T:ADAPTERS                          ; T=type I=index
I:COMPANYNAME;Mag                   ; We want lancards by company w/srch string
I:BUSTYPE;ISA                       ; This is a good Idea not currently using
F:PRODUCTID                         ; This sets the field order for the
F:DESCRIPTION                       ; request
O:0                                 ; Offset into the list
N:50                                ; Number of lines to return



14000~MAG200 Warp Core Adapter
271~EXOS 205
273~EXOS 215
272~EXOS 215T
264~EtherLink (3C501)



AccessKey_$%#@!##                   ; The access key
R:                                  ; Initial request has no ID assigned
U:GenericServices, Inc.             ; Company info for the log
T:CPUS                              ; T=type I=index
I:COMPANYNAME;Intel                 ; Starts with partial key query
F:PRODUCTID                         ; This sets the field order for the 
F:DESCRIPTION                       ;
O:0                                 ; Offset into the list
N:50                                ; Number of lines to return



14406~Pentium (tm) 266
14407~Pentium (tm) 100
14408~Pentium (tm) 200
14410~Pentium (tm) 120
14411~Pentium (tm) 133
14412~Pentium (tm) 166
14413~Pentium (tm) 150
23654~Pentium Pro (tm) 200
16852~Pentium(r) II 233
22363~Pentium(r) II 300
23291~Pentium(r) II 200



AccessKey_$%#@!##                   ; The access key
R:                                  ; Initial request has no ID assigned
U:GenericServices, Inc.             ; Company info for the log
T:BUSTYPES                          ; T=type I=index 
F:PRODUCTID                         ; This sets the field order for the 
F:DESCRIPTION                       ;
O:0                                 ; Offset into the list
N:50                                ; Number of lines to return



5139~32-Bit PCI Hot Plug
5138~64-bit PCI Hot Plug
4657~AGP II
1413~Apple Nu-Bus
3958~Bus Expansion Slot
3956~Expansion Slot



AccessKey_$%#@!##                   ; The access key 
R:                                  ; Initial request has no ID assigned
U:GenericServices, Inc.             ; Company/User info for the log
T:STORAGEDEVICES                    ; T=type I=index 
I:COMPANYNAME;Seagate               ; 
F:PRODUCTID                         ; This sets the field order for the 
F:DESCRIPTION                       ;
O:0                                 ; Offset into the list
N:50                                ; Number of lines to return


ZCachePro_v1.15                        ;
R:42:111:2:2115150400               ;
N:2                                 ;
A:2                                 ;


Writing Data

The "W:" is an alternate choice for "R:" where writing changes to a table is required. Immediately after the colon the specific write operation is specified. U for update, I for insert, and U for update. Unlike a read request, the indexes are not specified. Only the fields are described, and after a colon is the specified data. (i.e. CUSTOMER:Thompson) In the case of a delete operation, the key field is specified in the same way. (i.e. CUST_ID:3442) Later in this document a specific example will be shown.

First Example "Delete"


AccessKey_$%#@!##                   ; The access key 
W:D                                 ; A write operation. In this case DELETE
U:GenericServices, Inc.             ; Company/User info for the log
T:STORAGEDEVICES                    ; T=type (table alias)
I:PRODUCTID:310                     ; This specifies the specific record to be deleted 


Deleted OK                          ;

Second Example "Insert"


AccessKey_$%#@!##                   ; The access key 
W:I                                 ; A write operation. In this case INSERT
U:GenericServices, Inc.             ; Company/User info for the log
T:STORAGEDEVICES                    ; T=type (table alias)
F:PRODNAME:WIDGET                   ; Product Name 


Inserted OK                         ;
W:ID:564                            ; One record inserted, new autoincremented entry 564

Third Example "Update"


AccessKey_$%#@!##                   ; The access key 
W:U                                 ; A write operation. In this case UPDATE
U:GenericServices, Inc.             ; Company/User info for the log
T:STORAGEDEVICES                    ; T=type (table alias)
I:PRODUCTID:54663                   ; This is the id to be modified 
F:PRODNAME:WAM WIDGET               ; Product Name 


Updated OK                          ;

Record Locking

Since ZCachePro is designed for internet remote-enabled database applications, locking has been integrated with a slightly different focus than most traditional database applications.

There are currently three different record locking options:

  • Read after locking.
  • Write and clear the lock.
  • Clear the lock

A background process clears locks after the lock exceeds the user specified timeout. The ZCachePro administrator should specify a reasonable value and the developer of the application should take into consideration the fact that the locks only last for a finite time, so that locking errors can be dealt with properly.

RL: is in all respects the same as the R: the only difference is that the record is locked and then read.

WL: is in all respects the same as the W: the only difference is that after a successful write the record is unlocked.

WLC: is a new tag that clears the lock on the record.

The request for WLC: looks like this:

AccessKey_$%#@!##                   ; The access key 
WLC:                                ; A write lock clear operation. 
U:GenericServices, Inc.             ; Company/User info for the log
T:STORAGEDEVICES                    ; T=type (table alias)
I:PRODUCTID:310                     ; This unique id specifies the specific record cleared 

Sample Java Source

This code snippet illustrates how specifically a request can be made from ZCachePro directly without the use of a bean. This code could easily be used on any non-Java platform that supports IP.

	public final void sendRequest() 
      String s = null;
         sock = new Socket(ipAddress, portNum);
         streamout  = sock.getOutputStream();
         streamin   = sock.getInputStream();
         BufferedReader in = new BufferedReader( new InputStreamReader(streamin) );
         String outstr = 
         byte[] baray = new byte[outstr.length()]; 
         baray = outstr.getBytes();
         streamout.write( baray, 0, baray.length );
         // in the test client a 'resultList' box displayed the results of the search
         resultList.add(" ");   
	         s = in.readLine();
	         while( s != null )
	            s = in.readLine();
	      catch( IOException ioe )
	   catch(Exception e)

In this example you can see both how to make a query, and how to get the results. The first blank line in the result marks the beginning of the data being returned. Please look at the Reply section and the examples for further information.

Cached Writes

Some write operation do not need to be completed before continuing to the next step. Cached Writes allows these transactions to be queued safely, until database activity is reduced. Then the write is completed. Usually there is no delay, but if there is it will only be as long as required for database activity to slow to roughly half capacity. The user sees no wait, which improves the feel of your site.

If the server is taken down without the write being completed, it will be completed the next time the server is brought up. Cached Write operations are stored on disk until completed.

Cached Write operations are exactly like any other write operation, except that the operation is signalled by W:CI (insert only).

©Copyright 2017, ZeroPoint, LLC.      Questions? Comments? Email Us.