GB2447273A - Data compression and handshaking protocol - Google Patents

Abstract

A data compression communications protocol encodes smaller integers using fewer bits (eg. integers -4096 to +4096 in 16 bits rather than in a heavily zero-populated 64 bits). This "OFTE" protocol supports 14 different types comprising four signed/unsigned integer types: I8, U8 (8 bits each), I16, U16 (16 bits each), I32, U32, I64 and U64, and six other types: Float (32 bits), Double (64 bits), String (an array of U8's), Enumeration (an array of I32's), DateTime (an I64) and OTFEPrice (a Double), with the type being indicated by the least (or most) significant two bits of the first byte. High and low priority OFTE messages are formed (according to given templates) from a header and a record field, the header indicating package length, database ID, sequence number, table ID and record Count, and more bytes are added if a message does not fit into a predefined buffer. A handshaking protocol establishes communication between clients and servers using a one byte bitmask which allows encryption and verification.

Description

The following sections describe the OTFE protocol.

1.1. OTFE

1.1.1. Introduction

OTFE is a communicational network protocol. The idea is that the transported messages should fit in as few bytes as possible -we make this by merging data, or by omitting the bytes, with bits set to 0, in the integers and using templates, bitmasks and floating arrays. OTFE is designed for transporting data, where information changes every second. This protocol helps us respond to these fast changes by giving us the opportunity to transmit messages with less length, thus making the transmission over the network much faster than in the ordinary way.

1.1.2. What types does OTFE support? OTFE defines 14 types that can be transmitted in OTFE-messages: * 18 -signed integers, fitted in 1 byte (value range from -2 to 2 -1) * 116-signed integers, fitted in 2 bytes (value range form 213 to - 2-1 and from 2 to 213 1) * 132 -signed inteers, fitted in 4 bytes (value range from 229 to 2131 andfrom23to229-1) * 164 -signed integers, fitted in 8 bytes (value range from 261 to - 229 -1 and form 2 to 261 -1) * Float -4 bytes, IEEE float * Double -8 bytes, IEEE double * String -an array of chars (stored as an array of U8) * Enum -enumeration -array of 132 * U8 -unsigned integers, fitted in 1 byte (value range from 0 to 261) * U16 -unsigned integers, fitted in 2 bytes (value range from 26 to 214 -1) * U32 -unsigned integers, fitted in 4 bytes (value range from 214 to 2 -1) * U64 -unsigned integers, fitted in 8 bytes (value range from 2 to 262 -1) * DateTime -an 164, indicating the difference (in milliseconds) between the current time and midnight, January 1, 1970 UTC.

* OTFEPrice -a Double number, indicating the Price; it can be represented as a decimal floating point number or as a fractional number.

The OTFEPrice can be of type Decimal or Fractional. If OTFEPrice is Decimal, then the number is represented as a Double value, followed by the minimum and the maximum length of the numbers after the floating point. For example, if we have a Decimal number 12.1, with minLength = 2, the number will be stored as 12.10; if we have a Decimal number 4.235431, with maxLength = 4, the number will be stored as 4.2354.

If OTFEPrice is Fractional, then the number is represented as an integer, containing the integer part of the number and a fraction with a denominator, stated in advance. So if we have a number 12.125 and a Denominator = 8, it will be represented as 121/8.

1.1.3. from 1NT64 to OTFE By default an integer is represented in 64 bits. But when we have numbers of a short value (e.g. 2, 48, 3456), many bits remain "unused" (set to 0). OTFE defines 4 different types of both signed and unsigned integers, to shorten the bytes used to represent the value of a number. For unsigned integers, we use the least significant two bits (little endian) or the most significant two bits (big endian) of the first byte to determine the type of the OTFE-encoding.

Value of bits 7-6 OTFE-encoding type 00 The value of the integer is stored in the rest 6 bits of the byte -OTFE type U8 01 The value of the integer is stored in the rest 6 bits of the byte ______________ and 1 additional byte -OTFE type U16 The value of the integer is stored in the rest 6 bits of the byte _______________ and 3 additional bytes -OTFE type U32 11 The value of the integer is stored in the rest 6 bits of the byte ______________ and 7 additional bytes -OTFE type U64

Table 1

For signed integers: in little endian we use the first least significant bit to determine the sign of the number and the following two, to tell the type of the encoding; in big endian we use the first two most significant bits to tell the type of the encoding and the following 1 bit to determine the sign of the number.

Value of bits 6-5 (or 7-6) OTFE-encoding type 00 The unsigned value of the integer is stored in the rest 5 _______________ bits of the byte -OTFE type 18 01 The unsigned value of the integer is stored in the rest 5 ______________ bits of the byte and 1 additional byte -OTFE type 116 The unsigned value of the integer is stored in the rest 5 _______________ bits of the byte and 3 additional bytes -OTFE type 132 11 The unsigned value of the integer is stored in the rest 5 _______________ bits of the byte and 7 additional bytes -OTFE type 164

Table 2

1.1.4. How to get any type serialized in byte array? Integers of any type are serialized in the same way as 1NT64, OTFEPrice -a Double number, indicating the Price; it can be represented as a decimal floating point number or as a fractional number.

The OTFEPrice can be of type Decimal or Fractional, If OTFEPrice is Decimal, then the number is represented as a Double value, followed by the minimum and the maximum length of the numbers after the floating point. For example, if we have a Decimal number 12.1, with minLength = 2, the number will be stored as 12.10; if we have a Decimal number 4.235431, with maxLength = 4, the number will be stored as 4.2354.

If OTFEPrice is Fractional, then the number is represented as an integer, containing the integer part of the number and a fraction with a denominator, stated in advance. So if we have a number 12.125 and a Denominator = 8, it will be represented as 12 1/8.

from INT64 to OTFE. The least significant (little endian) or the most significant (big endian) two bits of the first byte are used to determine the OTFE-type of the encoding. And then the value is represented in the same way as shown in Table 1 and Table 2.

Floating-point numbers (float and double) are represented in the message as standard floating-point 4 bytes for float and 8 bytes for double.

The value is written in the buffer unchanged.

OTFEString is represented as an array of U8: the string is converted into array of bytes that contains its byte-representation fin UTF-8 encoding; the length of the array is written in the byte buffer in OTFE-format; the values are written as a byte array in the buffer.

Enums are written in the message as integers, so they are represented as 1NT32 in OTFE-format.

DateTime is written in the byte array as a simple integer of type Long (64 bits).

When serializing a record from type OTFEPrice, the first thing to be stored in the buffer is the Double value of the number. Then a composition of the two short numbers minLength and maxLength is stored as an OTFESizeSigned number (132) -maxLength is shifted 16 bits to the left and binary added to minLength. If this number (minMax) is equal to -1, the OTFEPrice should be represented as a fractional number, so the next thing stored in the buffer is the denominator.

1.1.5. Arrays and Simple Types Some records in the tables (including the free-formatted template) may be represented as arrays of record-values. To distinguish them from the simple type records (scalar values), we use an index greater than the number of records in the current database. This index is formed by adding the index of the record in the database to the number of records it contains.

When we transform the message from the client-application into OTFEMessage, we compare the index of the record in the message to the number of records in the current database and in case it's greater, we obtain the original index of the record, by subtracting the number of records in the database from the index of the record in the message and than put it in the OTFEMessage.

1.1.6. What is Database and what do we use it for? Databases are used to describe the possible data in the OTFEMessages. They hold all the records that can be sent and also the exact templates of the messages (called tables). Records are listed in the Records section. It is not obligatory for a database to define tables. But when there is a ft table, it consists of the IDs and Names of the records, which are allowed for the messages it defines.

OTFEMessages are formed, according to the given templates (tables).

However, it is possible that a message doesn't contain all the records, listed in its template. This means that if some record-values remain unaltered, they don't need to be sent. The recordCount stored in the message, shows the exact number of the needed values, so we send them in continuous order.

OTFE defines a bitmask template (table), which enables the user to iterate over the records and apply updates only for the values different from null.

Not all possible messages can be defined with tables. A free-formatted table is always predefined, consisting only of pairs of record-indices and record-values, to form the structure of all other possible messages. It can hold unlimited number of records (all from the same database).

1.1.6.1. Lexical analysis Comments are allowed anywhere within the file; comments follow the normal C/C++/Java convention: enclosed within /* */ blocks or from "/1' to the end of the line.

Newlines are significant in the database file. They can terminate "/1'-style comments and they also terminate each entry in a particular section of the file, e.g. each record definition ends in a newline.

Each "token" in the file consists of a single word. A word is either a continuous sequence of non-whitespace characters or it is all of the characters between two balanced quotes (single or double).

The backslash " character is used to escape the following character in the file. A backslash at the end of a line "cancels" the effect of the newline, acting as a continuation character in the same way as for C source files.

The following escape sequences must be used when required in a definition: \", \\, \, \#. The following standard C escape sequences are supported: \f, \n, \t, \@ and it is always possible to specify the precise hexadecimal value with the escape sequence: \x17, etc. 1.1.6.2. Major structures The database file is split into the following sections: config enums records

tables

that appear in this order within the file. The enums and tables sections are optional but, when present, they must occur in the places indicated above.

The names of the sections are not case-sensitive, e.g. the "config" section may be called "config", "Config" or "CONFIG" etc. The contents of each section must be enclosed within curly braces. For

example:

config 1.1.6.3. Con fig section This section contains the ID for the database. The ID serves to identity this database so that it can be shared between different logical connections and to be sent once only to each destination so that it can be cached locally.

The GUID is a free-format word. It can be any string that makes sense to the user; it does not need to be a classic GUID. This definition uses the term GUID to distinguish this string from the databaseld which is a number that is used in OTFE messages. The GUID is specified on its own line within the config section with the word "guid" followed by the guid as a single word.

config guid 4D36E96A-E325-1 1 CE-BFC1 -08002BE1 0362 or: config guid "My Safe Place 1.1.7.6" The "config" keyword may also be followed by an optional name for the database that can be displayed in the debug output instead of the ID, where the ID is not very readable, e.g. config "My Safe Place" guid 4D36E96A-E325-1 1 CE-BFC1 -08002BE1 0362 The section may also include a flag to state whether the record names are case-sensitive within the database: config recordNamesCase 1 II Record names are case-sensitive by default aliasNamesCase 1 II Record alii are case-sensitive by default enumDisplaysCase 1 II enum record display strings are case-sensitive the word "recordNamesCase" is not, itself case-sensitive and it may be followed by a number that evaluates to a non-zero value to indicate that the record names are case-sensitive. The value 0 indicates that the record names are not case-sensitive in the database. The default values when no flag is present are listed in the example above.

The section may also include a flag to state whether duplicate entries in the "enums" section are allowed: (0 config enumDisplaystjuplicatjon 1 I/allowed duplicate entries in the enums section By default duplicate entries are allowed as shown above. A 0 value indicates that Strings in the "enums" section must be unique. This means the declaration: tickDirection 1 up 2 up 3 down is not allowed.

1.1.6.4. Enums section This section contains the legal values for all of the enumerated records along with their display strings. Each enum table has a name and this name is used to link the record with its enum table in the "records" section. The enum table is a bracketed sequence of value-string pairs of words, each on a separate line and nested within curly braces. When we want to access a value we refer to it with its ID (e.g. we want AFA => we refer to it with 4). enums

tickEnums 1 \xDE II 1 should be displayed as the character \xDE' 2 \XDF II 2 should be displayed as the character \xDF' ] currencies 4AFA 8 ALL 12 DZD

ADP 24A0N

1.1.6.5. Records section This section defines all the records that can be used in messages, encoded using this database or any of the tables on it. The definition of each record is on a separate line and words that contain embedded spaces are enclosed in quotes. The definition includes some required data and, optionally, extra non-OTFE specific data. The required data includes everything that is needed for the OTFE implementation to encode and decode a message using this database. The required data comes first on each line, in the order: recordid, recordName, recordAlias and recordType. If the recordType is enumerated, then there is an extra field which is the name of the enum table that controls its values. More than one record may use the

same enum table.

1 Server.Build "BUILD VERSION" S 3 Bid.Price.1 "BID 1" D 2 Bid.Price BID.PRICE D ripple=Bid.Price. 1,precision=4 4 Bid.Price.2 "BID 2" D Currency "CURRENCY" E 16 Currency_i "CURRENCY 1" E 88 Tick.Direction "TICK UP/DOWN" E 202 File.Length LENGTH 132 Notes on attributes of the record definitions: 1. The recordld, name and alias must all be unique as they can each be used to identify an individual record.

2. The records can be listed in any order but that order must match the order of the same records at the other end of the socket connection (this will happen automatically if the database is transmitted over the wire).

3. The alias for recordld 2, is BID.PRICE and this is not enclosed in quotes because it contains no spaces. It would be harmless to enclose the value in quotes, if desired.

4. The recordName and recordAlias namespaces are not distinct. The same string cannot be used for the alias of one record if it is also the name of another record in the same database. The alias and name can, however, be the same for a single record.

1.1.6.6. Record Types The following codes are used to identify the various OTFERecordlypes: OTFERecordTypel8 18 OTFERecordTypel 16 116 OTFERecordTypeI32 132 OTFERecordTypel64 164 OTFERecordTypeFloat F OTFERecordTypeDouble D OTFERecordTypeString S OTFERecordTypeEnum E OTFERecordTypeU8 U8 OTFERecordTypeUl6 U16 OTFERecordTypeU32 U32 OTFERecordTypeU64 [U64 OlFERecordTypePrice I D OTFERecordTypeDateTime J T 1.1.6.7. Tables section This section lists the tables and all records in them, in order. All tables have names and these names are not case-sensitive within the database.

Each table must have a unique name.

As for the records, the tables may be listed in any order but that order must match the order of the same records at the other end of the socket connection (this will happen automatically if the database is transmitted over the network).

Tables

uploadFile ) sendFile Each table definition lists the records, identifying them by recordld or Name. The order of records within the table is significant to the optimal performance of OTFE. The most heavily-used records should be listed first so that the protocol can optimise rarely used records from most messages: updateAsk { 9[] If the record identifier is followed by "[}", this indicates that the record value is an array and not a scalar.

1.1.6.8. Nested files Any line may start with "#include" whereupon the definition will read in the contents of a single nested file just as if that file were included directly within its parent. The name of the file follows the #include" keyword, e.g.: #include OTFE/config.OTFE #include "OTFE\\file name has spaces.OTFE/* backslashes must be escaped (unlike C preprocessor) */ There are no syntactical limits to the location of the "#include" directive, as long as it is the first non-whitespace token on the line. For example, the sequence: config #include "OTFE/config.OTFE" is allowed, as long as the first non-commented line in the tile "OTFE/config.OTFE" starts with an open-curly brace ({), being the beginning of the config section for the database. Or this is also legal (though rather contrived): -config id 4D36E96A-E325-1 1 CE-BFC1 -08002BE1 0362] #include "OTFE/closeSection.QTFE" as long as the file "OTFE/closeSection.QTFE" starts with a close curly-brace (},) A nested file can, itself, contain "#include" commands that recurse. It is important to note that a relative pathname always relates to the directory of the current file, not the base directory file. This matches the behaviour of standard C/C++ header directives. As an example of this, assume the following is stored in a base directory file called "/OTFE/config/database.OTFE": #include "/etc/OTFE/otherContig.OTFE" and the file "/etc/OTFE/otherConfig.OTFE" also contains an #include directive like this: #include "./sys/sysConfig.OTFE" then the "sysConfig.OTFE" is located in the directory "/etc/OTFE/sys", not in the directory "/OTFE/config/sys".

1.1.7. Messages and OTFE The Messages in OTFE are formed as a single byte buffer but they can be logically divided into two parts: Header area and Records area. In the Header area we store information about the records we will send. In the Records area we store the exact values of these records.

1.1.7.1. Header structure The header consists of packageLength, databaseld, sequence number (if the message is from the low-priority queue), streamld, tableld and recordCount, stored in the message in the same order.

The packageLength is stored in the beginning of the Header, but is calculated and put in the message, after all other parts of the Header and Records area. The packageLength holds the whole length of the transmitted message (this means the length of the record-values and the length of the information, stored in the Header). It is represented in 137-format.

The databaseld is the index of the database, where the records, transmitted in the Records area, are stored on the source computer. It is written in the Byte buffer as a signed integer in 137-format, If the message is from the low-priority queue, the databaseld is transmitted as a negative number-we add ito the ID and then multiply it with (-1); this will guarantee that even when an ID = 0, the ID in the message will be a negative number.

When decoding the message, we multiply with (-1) and subtract 1, to obtain the real databaseld. For a message from the high-priority queue, the databaseld is transmitted, without being changed.

Both low-and high-priority messages have a sequence number, but only the one of the low-priority messages is transmitted over the network (on the side of the recipient the sequence number is incremented automatically for a high-priority message). The sequence number helps the Client determine in what sequence the received messages should be placed in its cache, It is represented as an unsigned integer in 137-format.

The streamld allows us to have multiple channels in a single stream of messages. It is used to identify to which group of messages (a channel in the stream), the current message belongs. (For example: if we have a couple of messages for the stocks of IBM, they will have the same streamld.) The stream Id in the message is stored as an unsigned integer in OTFE-format.

The tableld is the index of the table in the database, provided by the databaseld, which contains the records, transmitted in the Records area.

tableld = -1 indicates that we have a free-formatted table. tableld < -1 indicates that a bitmask template is used. This is the predefined template, where records are stored in a sequence of record-index and record-value.

The tableld is stored in the message in OTFESizeSigned-format. tablelds >= O indicate an ordinary template (table), tableld = -1 indicates the free-formatted template and tablelds < -1 indicate the bitmask templates. The bitmask tablelds are formed by multiplying the real tableld of the template with -1 and subtracting 2 from the received value (tableld*(1)2). When deserializing the message, we add 2 and then multiply with -ito obtain the real index of the bitmask table.

The recordCount holds the number of records that have to be stored in the OTFEMessage. It is represented in 137-format.

If the tableld shows that a bitmask template is used to represent the message, the length of the bitmask, followed by the bitmask itself is being sent after the header. The bitmask has value 1, where the record is present, and 0, for the records with null value.

1.1.7.2. Records structure The Records structure holds all the record values that should be written in the message. If the tableld is not equal to -1 (it is not a free-formatted table), only the record-values are stored in the message, otherwise the record-index (in 137-format) is sent before the value.

Record-values may be scalars (including strings) or arrays of scalars (including arrays of strings). When we store scalars, we put only the record-value in the message. For arrays of values, we store the length of the array first (as a signed OTFE-number). If the length of the array is a positive number, then it is followed by the record-values.

A negative value of the length, is used to indicate what kind of operation with the array should be done and from which position on. The exact length of the array is obtained by multiplying the negative number, stored in the message with -1 and then subtracting 1. In this case, the length is followed by a number in OTFESigned-format, indicating the operation and the position of the element(s) to be altered in the array. If the value of this number is negative, it indicates that a new element should be inserted on position ((_1*number)_1). And the record-values are written next in the message, followed by the value of the new element.

A positive number, stored after the length indicates that more than one element should be altered in the array. The value is the position from which on all elements should be changed with the values supplied by another array.

For example: if we have an array with three numbers (1, 23, 53) and 5 elements (5, 67, 34, 85, 97) that should be placed from the l position on, the new array will look like this (1, 5, 67, 34, 85, 97). The old values till position 1 are placed in the array unaltered and the new elements replace the old ones.

A table from the database represents a template of the OTFEMessage, but not all records in this template have to be put in the message -some of the records remain unaltered and it's pointless to send them over the network.

When we send a sequence of records, we should send them in continuous order in the same way they are structured in the database's records section (e.g. if we need the 2 and the 8th record, we need to send all the records between them). This is not valid for the free-formatted template, because we send the indices and the values together, so no misplacing of the records is possible.

Note: For all types, except for Enum, the exact value of the record is written in the message. When we have a recordType Enum -we send the integer that represents its value and not the index of the value in the current enum. For example, if we have: enums currencies 4AFA 8 ALL 12 DZD

ADP

24 AON and we want to refer to "ADP", we don't use 3 (which is its index in the "currencies" enumeration), but with 20 -and we write 20 in the OTFEMessage.

1.1.7.3. General When constructing an OTFEMessage, we define an initial size for the Buffer first, and then begin to fit in the information, It is possible for the information-size to become greater than the capacity of the Buffer. In this case, we redefine its size, by adding some additional bytes, until all the information can be fitted.

1.1.8. How is Handshake working? When a Client connects to the Server, they both exchange information (using system messages) on how data is to be transferred (encrypted, compressed, etc.). On agreement, they make a Handshake -a connection is established and messages can be sent in both directions. If one of the sides of the connection doesn't support some of the operations on the other side (or doesn't fulfill its requirements), the connection is closed.

1.1.8.1. System Messages To establish a connection, Client and Server exchange system messages. They are processed with the highest priority. The databaselndex in a system message is always a negative number (this means we add 1 to the real databaseindex and multiply it with -1) and their sequence number is set to 0. That's how we distinguish them from normal messages, when decoding. 12.

1.1.8.2. Client-Server Handshake (First byte) 1.1.8.2.1. Server Handshake: When the Server accepts a connection from a new Client, it responds with a one-byte bitmask and optional associated data. The bitmask tells the client what to expect and how to communicate with the server: Bit When set (1) When not set (0) 0 little endian big endian 1 RSA key present server is not encrypted 2 low-priority zlib-low-priority uncompressed _______ compressed I If the bitmask indicates that an RSA key is present, the Client is entitled to send a random generated 512-bit blowfish key, encrypted with the known Server public key. The blowfish key will be used for the further communication.

If the bitmask indicates that messages are compressed, then compression is applied to the messages before their encryption.

1.1.8.2.2. Client Handshake When the Client receives the initial Server configuration, it sends its own configuration as a one-byte bitmask: Bit When set (1) When not set (0) 0 client is a source -client is a destination -1 blowfish key included no encryption 2 username included no login 3 password included no login This tells the Server whether the Client is using anonymous login or is supplying a user-name(host name)/password and whether the Client wishes to use blowfish to encrypt the connection.

According to the value of the bitmask, it is then followed by the blowfish key, username and password (in the same order, if all present) represented as strings. It is legal, though unusual, for the response to include a password without a username. This supports anonymous login, such as for FTP where the convention is to send the email address as password. The following information is encrypred with the server public key already known from the client, if in the server handshake byte the RSA bit is set.

Server uses its private RSA key, if supplied, to decode the Client's response. If the Client supplied a username and/or password, the server tries to login the user using this information and accepts or rejects the login accordingly. If the Client did not supply a username and password, the Server either accepts the Client (if anonymous login is in order) or rejects it (if not). If the Client is rejected, Server closes the connection immediately. Otherwise a connection is initialized and the Database-Handshake takes place.

1.1.8.3. Database-Handshake After the initial Handshake is made, Server and Client synchronize their database-compatibility. This means that both Client and Server should have exactly the same database in application-cache before the message-transfer begins, If the Client acts as a Source, it is possible that its database isn't present in the Server's cache. If the Client acts as a Destination the situation may be reverse.

There are four system messages that help the synchronization of thedatabases: * Verification whether the Database is present in the list of databases of the receiver (e.g. Client) * Answer -the Database is present in the list of databases of the receiver * Answer -the Database isn't present in the list of databases of the receiver * Sending the Database in "Wire-format" 1.1.8.3.1. Verification of the presence of the Database If the Client acts as a Source, it sends a system message to verify if the Database is present in the Server's list of databases. If the Client is a Destination, than the Servers makes this verification. The message consists of size, databaselndex (as a negative number in OTFE-format), sequence number, code of the operation (1 in this case), and the guid (as OTFEString).

1.1.8.3.2. Answer -the Database is present in the list of databases The receiving part sends a system message to acknowledge, that it has the needed Database in its list of databases. The message consists of size, databaselndex of the asking part (as a negative number in OTFE-format), sequence number, code of the operation (2 in this case), and databaseindex of the answering part.

1.1.8.3.3. Answer -the Database is not present in the list of databases. Request for sending the Database The receiving part (Server or Client-Destination) sends a system message to the asking part (Server or Client-Source), requesting the forwarding of the Database. The message consists of size, databaseindex (as a negative number in OTFE-format), sequence number, code of the operation (3 in this case).

1.1.8.3.4. Sending the Database in "Wire format" The Database is transmitted over the wire using a single (though large) OTFE system message, containing the Database in Wire format with the new databaseindex that the receiving part (Client-Destination or Server, if the Client is a Source) did not recognize during the Database-Handshake. The message consists of size, databaseindex (as a negative number in OTFE-format), sequence number, code of the operation (4 in this case), and the Database in "Wire format".

After receiving the Database, the Client-Destination or Server (if the Client is a Source) sends a system message of type 2 (1.1.8.3.2) to acknowledge that the Database is currently placed in its cache. After that the Server or Client-Source can send messages, using the databaselndex.

1.1.8.4. Heartbeat Message When the connection between Server and Client with the Handshake and Database-Handshake is established, it is possible that no messages are sent between them. A heart-beat system message is then sent in certain intervals to keep the connection alive. This message contains no information -it consists only of size, a negative number (-1), 0 sequence number (in order to distinguish it from the normal messages as a system message) and code of operation set to 0.

1.1.8.5. Database "Wire Format" The Wire format of a database is made by writing all its components one by one in a byte array.

First the elements in the config-section are put into the array: recordNamesCase, aliasNamesCase, enumDisplaysCase, displayName and guid of the database.

Enum-section with all the enum-tables and the values and displayNames, hold in them is then placed in the array.

The maximum index of record in the database and the number of all records in it. The first value is used by the tables to operate with records with indices> maxRecordlndex (this indicates an array of records). The list of all records is then put in the array with their recordld, recordName, recordAlias, recordType (followed by index of enum-table, if recordType is enum) and recordOptions -value and index (if present) or 0 if not.

The last thing to be written in the Wire format is the tables. First the number of all the tables in the database is put, and then the table's name, record number and all recordlndexes are placed in the array.

1.1.9. Transport Layer After establishing the connection and exchanging the Database information, the Client-Destination (or Server if Client is a Source) can send its requests for information to the Server (or Client-Source).

1.1.9.1. High-and low-priority messages The transport layer supports both low-and high-priority messages and it is the developer's responsibility to choose which message should be high-and which should be low-priority.

There are two message queues: low-and high-priority, so when a message can be transferred, the server chooses one from either of the queues and processes it to its destination. Usually if both queues are not empty, on a certain number of high-priority messages, only one from the low-priority queue is processed. The maximum ratio of high-to low- priority messages (maxHiPriority) is set in the beginning of the connection -the default value is 10. In the worst case (ii high-priority queue is full), on 10 high-priority message only one low-priority will be sent. When the high-priority queue is not full, its maximum size is divided in maxHiPriority intervals and the ratio -high-to low-priority messages is defined, according to the place of current queue-size in one of the intervals. E.g. if we have max queue size 100 and maxHiPriority = 10, if the current queue-size is 8 (interval [0-10) -first interval), on 1 high-priority message, 1 low-priority is sent; if the current queue-size is 78 (interval [70 -80) -eighth interval), on 8 high-priority messages, 1 low-priority is sent,etc.

System messages are processed first, with the highest priority.

1.1.9.2. Tokens When the application, using OTFE, is ready to send a message over the network, it sends a token to the network layer. When the network layer can accept the message, it sends a token in turn to request it. Then the message, corresponding to the initial token is passed to the network layer and is transferred to its destination.

In the simplest case the token can contain itself the message, but more often it is an indication whether and when a message can be sent. Its main purpose is to avoid overflowing and jamming the network.

Claims (4)

1. THE EMBODIMENTS OF THE INVENTION IN WHICH AN EXCLUSIVE

   PROPERTY OR PRIVILEGE IS CLAIMED ARE DEFINED AS FOLLOWS: I. A server, the server comprising: -A network interface -A connection acceptor -A protocol described in claim 3 encoder -A protocol described in cailm3 decoder -A message described in claim 4 receiver -A message described in claim 4 sender
2. 2. A client -A network interface -A module able to establish connection to server described in claim I -A protocol described in claim
3. 3 encoder -A protocol described in cailm3 decoder -A message described in claim 4 receiver -A message described in claim 4 sender 3. A protocol, the protocol comprising: -A way of storing and compressing messages described in claim 4 in array of bytes -A way of decompressing and restoring messages described in claim 4 from array of bytes -A protocol properties defining the way that the data in stored -A handshaking process, specific way to negotiate the protocol properties.
4. 4. A message, the message comprising: -A memory -A data -A functionality to read data in the memory -A functionality to write data in the memory