RU2780979C1 - Dynamically allocated database with topological information binding - Google Patents

Abstract

FIELD: computing technology.

SUBSTANCE: method for collecting, storing, updating, and processing information about an n-dimensional space pre-divided into topological, including geometric, topographic areas; including collecting information by the collection tools located on devices with coordinates in the described space, capable of sharing information, jointly forming a connected graph; placing the information related to the topological area on the information storage media located on said devices connected to a community, currently having coordinates within this and the neighbouring areas; exchanging information with devices with coordinates in this or other, primarily adjacent, areas; deleting the information related to the topological area when the coordinates of the device are changed to ones remote from the area.

EFFECT: reduced demand for computing resources and bandwidth of the network channel with an increase in the database size.

17 cl, 1 dwg

Description

FIELD OF KNOWLEDGE

The invention relates to ordered arrays of information logically organized into databases stored on physical media, such as magnetic disks, and to methods for managing databases implemented on the basis of software that processes information using computers.

The database (hereinafter referred to as the DB) is characterized by its logical model and consists of separate records - data containing distinctive characteristics. Data groups are grouped into subsets.

The structure of information includes the actual data that describes the objects of the subject area, and metadata that systematizes the objects of the subject area. The physical structure of the data provides for the storage and processing of information by software.

The database management method is based on performing operations of writing, reading, deleting, modifying, multi-user access to information and restoring internal consistency of information after an abnormal shutdown of one or more database users. Operations with information are performed in the form of transactions - groups of sequential actions, which are logical units of work with data, and are recorded in the form of a log (transaction log, https://en.wikipedia.org/wiki/Transaction log), fixing the sequence of transactions and their specific content.

The architecture of a distributed database is claimed, functioning as a "swarm" - a collection of independent devices. Each device works with a limited data fragment and periodically (as often as possible) synchronizes the state of the database fragment visible to it with other (mainly "neighboring") devices. Thanks to the topological (for example, geometric or topographic) N-dimensional segmentation of the database, the transaction log, as well as an effective synchronization procedure between agents, the proposed database, implemented on the basis of a "swarm" of independent devices, can effectively work with an almost unlimited amount of data and operations on them. change. It is possible, for example, to use in computer games, transport systems, etc. Devices exchange database updates between themselves directly via P2P connections, mediated by each other or by the server.

BACKGROUND AND PROTOTYPE

Known (US patent No. 8276084) method of delivering content to the area of virtual space (VP), including: detection (detection) of multiple interactions between participants (devices); tracking multiple identified interactions; evaluating the load level of the server supporting the VP region to determine if the load has exceeded a predetermined threshold; forming a set of subgroups of devices in the area, if a predetermined load threshold is exceeded, each of the set of subgroups containing one or more devices for which detection and tracking is performed; and responsible for exceeding the prev. assigned load threshold, if the amount of data transferred to or from the server exceeds the pre-assigned threshold of the amount of (broadcast) traffic, transferring at least a part of the amount of IP data between devices to one specific subgroup from the list of subgroups using P2P data exchange (broadcast), where all devices in one specific subgroup of a set of subgroups transmit at least a portion of the VP data to or from the server, where, if the amount of VP data is below a pre-assigned level (i.e., decreases) of traffic volume, stops transmitting part of the amount of VP data between devices of a particular subgroup from a set of subgroups.

The disadvantages of the method are as follows. The method frees the server network channel from distributing non-priority, static content, while the exchange of new information (changes) between clients is carried out strictly through the mediation of the server, which, on the one hand, leaves a significant load on the server and, on the other hand, makes clients dependent on server response time. Thus, this method does not benefit from an increase in customer activity. Also, the document does not describe the mechanisms for reading, writing and synchronizing the database for devices.

Known (US patent No. 10825024) method, devices and systems for data management platforms based on DLT. Several embodiments of distributed accounting (accounting) systems (i.e. distributed systems using financial accounting technologies in a decentralized computer network) are described that can be used to operate and manage the distribution and access to data so as to retain the advantages of a distributed accounting (financial ) technology allowing also private data exchange. In particular, the described distributed accounting systems include a data management platform based on distributed accounting technology (DLT), which can be applied at the nodes of the data management network.

The technology requires that the full log be stored and updated on each client, that is, the method does not have scaling mechanisms, since the requirements for the computing resources of clients and the bandwidth grow indefinitely and synchronously with the growth of the number of clients and/or transactions.

Known (US patent No. 9571571 dated 09/04/2012). The document reveals a way to create communities in the P2P network.

In accordance with the P2P protocol, the P2P network includes many clubs (communities) and many devices (ordinary participants) that receive content distributed by means of the network. Each of the many ordinary members is part of at least 2 communities. The uniting participant identifies the community (club) of the P2P network to join (which one he wants to join) and receives identification information for the member of the specified community (club). The joining member randomly chooses among the members of the selected community (club) at least the first member and the second member with whom to connect. The connection is proportionally weighted (weighted average) of the number of connections of the first participant with other participants and the number of connections of the second participant with other participants. The joining party establishes a first connection with the first party, and a second connection with the second party.

The disadvantage of this method is that it is dedicated to the particular task of joining the community and does not solve the problem of providing device access to a consistent database, as well as optimizing and/or reducing the need for computing resources with an increase in the size of the database.

Known (US patent No. 9210085 dated 05.10. 2007). Procedure Disclosed A P2P protocol, such as BitTorrent, is used to assist in streaming data. Participants download streaming content from the P2P network while simultaneously playing the downloaded content. While the stream is playing, the end system downloads the skipped parts directly from the server or other infrastructure node. This method is equivalent to a significant increase (squared) in server capacity and can be fine-tuned to achieve an average of 0 (1)th server regardless of the number of competing users. In this way, BitTorrent aids the streaming process better than traditional server-client and infrastructure-only solutions, each of which requires multiple infrastructure nodes that increase linearly to the number of users.

The disadvantage of this method is that it is dedicated to the particular problem of one-way transmission of streaming content, and solves the problem of optimizing and/or reducing the need for computing resources and network bandwidth with an increase in the number of clients receiving the data stream, but does not provide clients with the ability to interactively read data from DB and records in the DB.

Known (U.S. Patent No. 9177328 dated June 21, 2012) A method for adding a member to a network community is disclosed. A nearby member sends joining member information (contact information) about those members to the joining member, and the joining member establishes a connection with them. Community members place announcements for chat rooms to join a chat room, the participant finds out which participant is closest to the chat room by the chat room ID and notifies them that they are a member of the chat room.

The main similarity is the way the community is organized and the interaction between the nodes, incl. directly, supporting a connected graph of all network participants, creating and breaking P2P connections on the initiative of the participants in order to form direct connections of the desired configuration. The disadvantage of this method is that it is dedicated to the particular task of joining a community and rebuilding network connections to form communities, but does not solve the problem of providing access for all connected devices to a single consistent database for all participants.

US patent No. 8276084 was chosen as a prototype.

The technical result achieved by the present invention is to provide devices with access to a logically consistent database, as well as reducing the need for computing resources and bandwidth of the network channel of clients (and the server, if any) with an increase in the volume of the database, the number of users and recording intensity.

Known methods and devices do not allow to achieve the claimed technical result.

SHORT DESCRIPTION

The claimed technical result is achieved by the proposed method for collecting, storing, updating and processing dynamically allocated information with topological reference of data storage locations.

The claimed method for collecting, storing, updating and processing information about an n-dimensional space, pre-divided into topological (for example, geometric) areas, includes the collection of information by means of collection located on devices that have coordinates in the described space and are capable of exchanging information with each other, all together forming a community, where each device can exchange information with each directly or through the mediation of other devices (connected graph), placing information related to the topological area on information storage facilities located on the specified (connected to the community) devices that have at that moment coordinates within this area and neighboring (adjacent) areas, exchange of information with devices having coordinates in this or other, mostly adjacent, areas, deletion of information relating to the topological area, when the device coordinates change to others remote from this area.

A dynamic system for collecting, processing and placing information about an n-dimensional space is claimed, including

- n-dimensional space, previously divided into segments, forming a tiling (parquet) with full space coverage.

- one or more devices assigned coordinates in n-dimensional space, equipped with one or more means of collecting information, means of storing information, means of communication (reception and transmission) with other devices, with the ability to move both within one topological area and on any of the adjacent topological regions.

The information storage means may further include an information status storage means, a fragment of a shared distributed transaction log (DLT).

The transaction log (DLT) can store individual transactions. In this case, each operation changes the data only on a pre-fixed set of the specified selected areas of the n-dimensional space.

A complete log of all operations can be stored as a set of samples from this complete log, where each specified sample consists of a complete list of operations related to a particular topological area.

The logical address of each piece of information (each record in the database) may contain a unique identifier that uniquely identifies the area to which the record belongs.

Information about each area of space may include a list of all devices connected to the community (graph) that have coordinates in this area.

Information (transaction), including about the current area (affecting the current area), can be written to the local memory (storage medium) by the initiating device (source device) initially in the "inconsistent" status, then after confirmation - in the " agreed."

The local memory (storage medium) of each device can store information about the last operation affecting its own and any of the adjacent areas, before which the device knows all the operations performed by all devices for its own and adjacent areas.

New information (transaction) about the current area can be written to the local memory (storage medium) of the initiator device (source device) initially in the "accepted" status.

It is possible to additionally have one or more devices with server functions for primary authorization of devices, proxying information and organizing P2P interaction between agents (for example, for organizing UDP / IP punch hole), which does not belong to the community and does not have coordinates in the specified n-dimensional space.

A software product is claimed placed on a computer-readable storage device for collecting, storing, exchanging, moving content in a geometric region of n-dimensional space, a computer-readable storage device containing a program code that allows a computer to perform the following actions: collect information information by means of collection located on devices having coordinates in the described space, placing information related to the geometric area on the information storage means located on the indicated devices having coordinates within this area at that moment, exchanging information with devices having coordinates in this or other, mostly adjacent, areas .

The claimed software product may additionally allow the computer of a device having coordinates within a geometric region to transmit information to other devices having coordinates within the same region and adjacent regions during the recording process.

The claimed software product may additionally allow the device computer to perform data writing procedures within the area in the following order:

• each device to store information about the last operation affecting its own and any of the adjacent areas, before which the device knows all the operations performed by all devices in the community for its own and adjacent areas;

• each device to write to the local memory (means of storage) of the device in the "accepted" status new information (transaction) about its area;

• for each device to keep track of all changes (transactions) made by other devices that have coordinates in the same area where the device is located and in adjacent areas, and also periodically send information to all neighboring devices about which transaction was recorded last by this device or information that the recording was not carried out;

• each device should keep track of the completeness of its log of operations and request missed transactions if it learns about them, or update information about the operation for itself, before which all operations are known to this device.

The claimed software product may additionally allow the device computer to perform data writing procedures within the area in the following order

- primary recording in the local memory (storage medium) of one device,

- assignment of the primary record of the status "not agreed",

- distribution (direct or by forwarding) of data to other devices in the same area (neighbors) by area in adjacent areas,

- receiving a response from all neighboring devices within the area and adjacent areas that data has been received,

- assignment of the primary record of the status "approved".

Declared hosted on a computer system for the exchange, storage, delivery of information in the field of n-dimensional space, including information bus attached to inf. bus processor, information collection device, memory device (information storage), containing instructions: for collecting information, exchanging data with other devices located in the n-dimensional space.

The claimed system hosted on a computer may additionally contain instructions: for devices having coordinates within a geometric region, to transfer information to other devices having coordinates within the same region and adjacent regions during the recording process.

The claimed computer-hosted system may additionally contain instructions that each procedure for writing data to the area consists of

- primary recording in the local memory (storage medium) of one device,

- assignment of the primary record of the status "inconsistent",

- distribution (direct or by forwarding) of data to other devices in the same area (neighbors) by area in adjacent areas,

- receiving a response from all neighboring devices within the area and adjacent areas that the data has been received,

- assignment of the primary record of the status "agreed".

The claimed system hosted on a computer may additionally contain instructions: assign server functions to one of the devices to account for device interaction or information exchange in the specified n-dimensional space.

DESCRIPTION OF GRAPHIC MATERIAL

On FIG. 1 shows the layout of objects on the map and their addressing in the database (example).

DETAILED DESCRIPTION

The claimed technical result is achieved by the proposed method for collecting, storing, updating and processing dynamically allocated information with topological reference of data storage locations.

The claimed method for collecting, storing, updating and processing information about an n-dimensional space, previously divided into geometric regions, includes the following steps.

Collection of information by means of collection located on devices having coordinates in the described space and capable of exchanging information with each other.

Means of collecting information can be, for example, means of photo and video recording, audio reading and recording devices, radiation control sensors, temperature sensors, wind force and direction sensors, manual data entry devices (for example, a keyboard), etc.

All devices together form a community where each device can exchange information with each directly or through other devices - a connected graph.

The placement of information related to the geometric area occurs on information storage media (memory devices) located on the specified (connected to the community) devices that at that moment have coordinates within this and neighboring (adjacent) areas.

The exchange of information occurs mainly with devices that have coordinates in the same area or in adjacent areas. The exchange of information can also take place with devices from other areas, but is carried out at the explicit request of the receiving device with a direct indication of what kind of data it needs to transfer.

Deletion of information relating to the geometric area occurs when the device coordinates are changed to others, remote from the given area.

Recording and changing information can be carried out through a distributed log of operations (Distributed ledger, DLT) https://en.wikipedia.org/wiki/Distributed ledger, where each operation changes data only about a pre-fixed set of specified selected areas of the n-dimensional space.

A complete log of all operations can be stored as a set of samples from this complete log, where each sample consists of a complete list of operations related to a particular geometric area.

Each device can store information about each region of space and may include a list of all devices connected to the community (graph) that have coordinates in the same (own) given region of space.

Devices having coordinates within a geometric region can transmit information to other devices having coordinates within the same region and/or adjacent regions during the recording process.

The initiating device (source device) can write information (transaction) affecting the current area to local memory (storage medium) initially in the "inconsistent" status, then the initiating device (source device) can send information directly or by sending it to all "neighbors" " - other devices having coordinates in the same area as the initiating device (source device) and in adjacent areas, and the specified information, the initiating device (source device) can assign the status "negotiated" after all the others devices from the same and related areas have come to a "consensus" (see https://en.wikipedia.org/wiki/Consensus (computer science)) that the record (transaction) has been successfully registered, i.e. received by all destination devices.

When exchanging messages between devices, they can perform asymmetric message encryption (see, for example, https://docs.cntd.ru/document/1200004855).

When messages are exchanged between devices, the messages may be signed to ensure the authenticity of the messages.

Each device can store information about the last operation affecting its own and any of the adjacent areas, before which the device knows all the operations performed by all devices regarding its own and adjacent areas.

It is possible to write new information (transaction) about its area to the local memory (storage medium) of the device immediately in the "accepted" status.

At the same time, each device can track all changes (transactions) made by other devices that have coordinates in the same area where the device is located and in adjacent areas, and can also periodically send information to all neighboring devices about which transaction was recorded by this device. last or information that the recording was not carried out.

Each device can keep track of the completeness of its log of operations and requests missed transactions if this device becomes aware of them indirectly, or updates information about the operation for itself, before which all operations are known to this device.

Each procedure for writing data to an area may include the following steps. Writing to the local memory (storage medium) of a single device. The assignment of this device to the entry status "inconsistent". Sending data (directly or by forwarding) by a recording device to other devices in the same area and adjacent areas. Receiving a response from all devices within the changed area and adjacent areas that the data is accepted (negotiated). The purpose of the status record is "approved".

Information (transaction) about the current area can be written to the local memory (storage medium) of the device immediately in the "accepted" status. And then notify other devices in the same area and in adjacent areas about it

There may be one or more devices with server functions for primary authorization of devices, proxying information and organizing P2P interaction between agents (for example, for organizing a UDP / IP punch hole), which does not belong to the community and does not have coordinates in the specified n-dimensional space.

Any of the devices, when connected to a community (swarm), can go through the authorization procedure.

Any of the devices can be assigned additionally broader or truncated data access rights, extended or reduced responsibilities, and other additional. responsibilities or restrictions in relation to other devices in the community.

A dynamic system for collecting, processing and placing information about an n-dimensional space is claimed.

The system includes:

- n-dimensional space, previously divided into geometric regions,

- one or more devices assigned coordinates in n-dimensional space, equipped with one or more means of collecting information, means of storing information, means of communication (reception and transmission) with other devices, with the ability to move both within one geometric area and on any of the adjacent geometric regions, and the specified information contains, among other things, the logical address of each such device.

The information storage means may further include an information status storage means, a partial log of the general distributed transaction log.

In the partial log of operations (DLT https://en.wikipedia.org/wiki/Distributed ledger), each operation can change data only about a pre-fixed set of specified selected areas of the n-dimensional space.

A complete log of all operations can be stored as a set of samples from this complete log, and each specified sample consists of a complete list of operations related to a particular geometric region.

The logical address of each piece of information (each record in the database) may contain a unique identifier that uniquely identifies the area to which the record belongs.

Information about each area of space may include a list of all devices connected to the community (graph) that have coordinates in this area.

New information (transaction) about the current area can be written to the local memory (storage medium) by the initiating device (source device) initially in the "not agreed" status, then after confirmation - in the "agreed" status.

The local memory (storage medium) of each device may store information about the last operation affecting its own and any of the adjacent areas, before which the device knows all the operations performed by all devices for its own and adjacent areas.

Information (transaction) about the current area can be written to the local memory (storage medium) of the initiator device (source device) initially in the "accepted" status.

It is possible to additionally have one or more devices with server functions for primary authorization of devices, proxying of information and organization of P2P interaction between agents (for example, for organizing a UDP/IP punch hole) that does not belong to the community and does not have coordinates in the specified n-dimensional space.

A software product placed on a computer-readable storage device for collecting, storing, exchanging, moving content in a topological region of n-dimensional space is claimed, a computer-readable storage device containing a program code that allows (forces) a computer to perform the following actions: collect information information by means of collection located on devices having coordinates in the described space, placing information related to the topological area on information storage facilities located on the indicated devices that have coordinates within this area at that moment, information exchange with devices having coordinates in this or other , mostly adjacent areas.

Additionally, said software product may allow a computer device having coordinates within the topological region to transmit information to other devices having coordinates within the same region and adjacent regions during recording.

Additionally, said software product may allow the computing device to perform procedures for writing data within an area in the following order:

- primary recording in the local memory (storage medium) of one device,

- assignment of the primary record of the status "not agreed",

- distribution (direct or by forwarding) of data to other devices in the same area (neighbors) in the area and in adjacent areas,

- receiving a response from all neighboring (within the area and adjacent areas) devices within the area and adjacent areas that the data has been received,

- assignment of the primary record of the status "approved".

Additionally, said software product may allow the computing device to perform procedures for writing data within an area in the following order:

• each device to store information about the last operation affecting its own and any of the adjacent areas, before which the device knows all the operations performed by all devices in the community for its own and adjacent areas;

• each device to write to the local memory (storage medium) of the device in the "accepted" status new information about its area;

• each device to keep track of all changes made by other devices that have coordinates in the same area where the device is located and in adjacent areas? and also periodically send to all neighboring devices information about which transaction was recorded by this device last or information that the recording was not carried out;

• each device monitors the completeness of its log of operations and request missed transactions if it learns about them, or update information about the operation for itself, before which all operations are known to this device;

• each device should keep track of the completeness of its log of operations and request missed transactions if it learns about them, or update information about the operation for itself, before which all operations are known to this device.

A computer-based system for exchanging, storing, and delivering information in an n-dimensional space is also claimed, including an information bus connected to inf. bus processor, information collection device, memory device (information storage), containing instructions for collecting information, exchanging data with other devices located in the n-dimensional space.

Also placed on a computer system for the exchange, storage, delivery of information in the area of n-dimensional space, including an information bus attached to inf. On the bus, a processor, an information collection device, a memory (information storage) device, may additionally contain instructions for devices having coordinates within the topological area to transfer information to other devices having coordinates inside the same area and adjacent areas during the recording process.

Also placed on a computer system for the exchange, storage, delivery of information in the area of n-dimensional space, including an information bus attached to inf. bus processor, information collection device, memory device (storage of information) may additionally contain instructions to initially assign the status "inconsistent" ("in agreement"), the information (transaction) that the agent wrote to its local memory (storage medium) of the area specified information subsequently assign the status "approved" (including the agent-author) only after all the inhabitants of the area and adjacent areas have accepted and confirmed to the agent-initiator its receipt.

Also placed on a computer system for the exchange, storage, delivery of information in the area of n-dimensional space, including an information bus attached to inf. bus processor, information collection device, memory (information storage) device may additionally contain instructions that each procedure for writing data to the area consists of:

- primary recording in the local memory (storage medium) of one device,

- assignment of the primary record of the status "inconsistent",

- distribution (direct or by forwarding) of data to other devices in the same area (neighbors) by area in adjacent areas,

- receiving a response from all neighboring (within the area and adjacent areas) devices within the area and adjacent areas that the data has been received,

- assignment of the primary record of the status "agreed".

Also placed on a computer system for the exchange, storage, delivery of information in the area of n-dimensional space, including an information bus attached to inf. bus processor, information collection device, memory (storage) device • each device to store information about the last operation affecting its own and any of the adjacent areas, before which the device knows all operations performed by all devices in the community for its own and adjacent areas;

• each device to store information about the last operation affecting its own and any of the adjacent areas, before which the device knows all the operations performed by all devices in the community for its own and adjacent areas;

• each device to write to the local memory (storage medium) of the device in the "accepted" status new information about its area;

• each device to keep track of all changes made by other devices that have coordinates in the same area where the device is located and in adjacent areas? and also periodically send to all neighboring devices information about which transaction was recorded by this device last or information that the recording was not carried out;

• each device should keep track of the completeness of its log of operations and request missed transactions if it learns about them, or update information about the operation for itself, before which all operations are known to this device.

ADDITIONAL EXPLANATION ON THE TOPIC.

The type of database used in the claimed method may be different, but the best result is obtained with a "Key-Value" type database (https://en.wikipedia.org/wiki/Key%E2%80%93value database) with support for hierarchical addressing or without. In databases of this type, "Key" is a sequence of bytes of arbitrary length, "address", "Value" - stored data of arbitrary type, a sequence of bytes of arbitrary length.

Each "Key" of the database contains (uniquely calculated or explicitly stored as a sequence of N integers) the address of the area in the N-dimensional space. In this case, many different keys can belong to one area.

Any database/programming language offers many data operations. Of this set of operations for the claimed database, only those that meet the criteria below are applicable, which somewhat narrows the standard meaning of the expression "data operation".

Also, the proposed database has limitations on the characteristics of the tasks performed. This means that the proposed database does not provide an advantage in solving problems where there is no viewing area, binding to an area (in n-dimensional space) and the client can read / write any data with equal probability. Unlimited database scaling without a linear increase in the load on the network and computing resources will manifest itself only on tasks whose business logic meets the above criteria.

An N-dimensional space is described, all data are assigned coordinates in this space, then the space is divided into segments (topological tiling of an N-dimensional surface). Further, the data of each segment (each tile) is processed separately. The topological terms "tilings", "mosaics", "parquets" describe the procedure for the applied space segmentation.

Areas (segments) of the database form a discrete N-dimensional space, and have the standard topological properties of "neighborhood", i.e. each one area is adjacent to one or more others, and "remoteness", i.e. the comparative distance between any regions in N-dimensional space can be calculated.

Information about each area is stored in the computer memory and updated independently of other areas.

To change a record in the database, special byte sequences are used - "operations". An operation is a set of data describing the changes made to the database (operation type, data address, new value, etc.). The closest analogue is a SQL query, but not all operations available in SQL are suitable for the claimed database.

Only transactions that meet the following criteria are eligible:

• operation describes changes in a fixed set of areas,

• the list of areas affected by the operation does not depend on the state of the database,

• the operation must describe deterministic changes and, when executed on an identical database, always give an identical result,

• the operation must be "reversible", that is, for each operation there must be a "reverse" operation ("cancel" operation, inverse function), the use of which after the operation will return the database to the state "before the start of the operation",

• the operation is atomic, i.e. can either be completely successfully applied to the database, or completely "fail", a partial version is not allowed. A successful operation can write to the database, a failed operation leaves the database unchanged.

Operations are combined into a "transaction". A transaction, as understood in this document, as well as in common practice, is an ordered list of one or more transactions. A transaction inherits all the properties of a single operation, including atomicity. A transaction, as a set of data, in addition to the list of included operations, contains service information including, but not limited to, the time of creation, the creator of the transaction, and the electronic signature. A transaction, like an operation, can be "applied" to a database.

Transactions applied to the database are placed in the "transaction log": a list of transactions uniquely ordered according to fixed rules. It is convenient to sort transactions in the log in descending order according to the unique combination of the creation time of the transaction and the ID of the creator.

Based on the unique sorting of transactions, database users put new transactions in the log in an arbitrary order. Due to the fact that the order of transactions in the log depends only on the description of the transactions, and not on the order in which they were added, an identical log will always be obtained from an identical list of transactions at the output.

TASKS THAT THE DATABASE SOLVES

The problem is solved when in a certain space, previously divided into regions, many independent agents - devices, simultaneously and inconsistently read and write information to a single database (hereinafter referred to as the "DB"). Agents can perform, for example, the following tasks:

• Cars (agents) when driving on the road using sensors analyze the traffic situation and put into the database information about the obstacles encountered, the quality of the roadway, parked cars, their own location and other information useful for other cars and/or for traffic control systems. Also, each car reads from the database information recorded by other cars to search for free parking lots, select a driving mode, etc.

• Combat vehicles and fighters on the battlefield, having connected to the base with information about the territory, monitor each other's positions and identified hostile targets, upload information about their own position, observed objects to the database, place marks for artillery strikes or transmit instructions to unmanned vehicles.

• Participants of a computer game, having connected to a common game base, perform competitive actions in the game universe - move in space (update the record "position in space of the spaceship No. 1"), inflict damage on other players (update the record "Ship No. players) and game objects, create and/or destroy buildings, exchange items with other players, etc.

The situations described above are characterized by the following key features:

1. The database describes the objects located in the n-dimensional space, i.e. the database stores descriptions of objects. The dimension of such a space can be any, the main thing is that the space is uniquely divided into, preferably, non-intersecting topological (geometric, topographic) regions (or simply "regions"), and for each such region of space one could find a finite number of neighboring regions. In the simplest case, this is a fragment of a plane (an area on the ground), divided into squares (for example, 1 km × 1 km). As examples of information collected in the area:

• Placement and properties of buildings, vegetation and roads in the area (a map of the area can be viewed as a two-dimensional space)

• In a computer game, the placement of players and in-game objects on the playing field (2D or 3D)

• Position of stars and star systems in the universe (three-dimensional space).

2. The agent accessing the database (be it writing or reading) has a relatively stable and clearly defined area of interest - a list of topological areas (often in contact with each other, but not necessarily). The zone of interest can be of arbitrary shape and size (most often an n-dimensional sphere can be expected). A relatively stable area of interest means that the vast majority of read/write operations are performed without changing the area of interest, and changes to the area of interest are rarely performed. The fewer operations of changing the area of interest per one read/write operation, the greater the gain in performance and used computer resources gives the proposed architecture. Examples that illustrate the area of interest include the following:

• A car looking for a parking space needs information about free parking lots within a radius of ~500 m,

• It is useful for the BMP (infantry fighting vehicle) gunner to know about the positions of the MANPADS (anti-aircraft missile system) within a radius of several (for example, 4) kilometers. What happens at a distance further than 4 km (for example) is no longer so important, but useful. Outside a radius of 10 km, information is not needed.

• A player in a computer game can only trade and trade with players and NPCs in their immediate vicinity, usually within their line of sight. On a sufficiently large game map, the zone "visible" to the player at any given time can be a fraction of a percent of the total game area.

• The zone of interest is determined in each case individually, for example, a player who has connected to a network game from a full-fledged computer can have a larger visibility zone than players who have connected from mobile phones, and a car that has not found a parking lot within a radius of 200 meters can expand the field of view of parking lots up to 400 or 600 meters.

• An agent whose zone of interest covers the entire database can also work with the database. For example, a division headquarters, having a powerful computer and communication networks, can collect information from a vast territory, while a fighter may have access to a small fragment with a radius of one and a half kilometers.

3. Each read or write operation can be guaranteed and unambiguously correlated with a certain area of the space described in the database. Therefore, each series of operations will also have a finite list of affected areas. Global operations (applying to all areas simultaneously) are either not used at all or are an exception (they are performed using other tools not described here).

• If the car reports a occupied parking space, it means that the exact location of the parking space is known, and which "map area" needs to be updated, and which areas (all other areas) will remain unchanged.

• If a player needs a list of all the avatars of other players (participants) within a certain radius from his avatar, then it is possible to accurately determine the final list of areas "on the map", that is, the sections of the database that are necessary and sufficient to examine in order to obtain an exhaustive list of neighboring avatars.

4. Data and/or changes occurring outside the data "zone of interest" are not needed by the agent (device). It also means that changes made to the database by an agent are not required by other agents whose area of interest does not cover the area being changed. In fact, at every moment of time, each agent works with a "selected fragment" of the base, while the rest of the base exists completely outside its visibility zone.

• The car's built-in computer or a mobile phone with an application will not be able to process the traffic situation in the whole city, but it is possible to download over the mobile network and process the state of parking lots within a radius of 1 kilometer without having significant computing and network resources.

• Players that are actively generating events (avatar moved there, avatar moved here) collectively generate a huge amount of traffic. It is impossible to process in real time all the events of the game universe, where several hundreds or thousands of players are playing at the same time, on a conventional computer. Neither the bandwidth of a standard Internet connection will be enough, nor the computing resources. But if the player's computer receives information only about events in the neighborhood (and the player does not need others), then it will receive such data over the network and process it with significantly fewer resources.

5. Agents have the ability (on their own or at the behest of other authorized agents) to change the "zone of interest". The implementation is extremely important, in which the area of interest is stored directly in the database, updated there by agents (possibly appointed by authorized agents) and public, that is, the agents will be informed about each other's area of interest, which opens up wide optimization opportunities.

• As the car moves through the city, the need to view and collect/process information from one area disappears, but it becomes necessary to view (collect/process information) another area.

• The player (agent) can move around the game world: at the moment he needs to see the northern side of the continent and the opponents on it, in five minutes - the eastern one, and then he will fly into space. Or the player could, for example, change the weapon from "knife" to "sniper rifle", which would require granting him write access to a significantly expanded area of "space".

6. It is possible to provide, not necessarily direct, communication between agents whose "zones of interest" overlap and/or touch

• Players, through the mediation of the server and / or directly, through P2P connections, can exchange information with players who are close to them in the game universe.

• Vehicles can establish radio contact with nearby vehicles either directly or via cellular repeaters, for example.

7. The database must have built-in access control and change verification tools: primary user authorization (for example, by login/password and/or public key), defining different access rights to different layers/areas of the database for different users, and other standard access control functions to the DB.

• A city road network database can grant a city administration the right to position parking spaces on a map (write) and view the positions of parked cars (read). The DB will allow the built-in computers of cars to move the label of their car (and only their own) and read the location of parking spaces. The car does not have the right to set the location of parking lots.

• The player must be able to change the speed and direction of movement of his avatar, but not the avatars of other players. The game administration will see all areas of the database, including those that are completely hidden from the players (for example, the contents of secret chests, which can be found out only by coming close to the chest).

8. The database needs (required from the database) (should) fully function and maintain logical integrity, including under conditions of intensive, simultaneous and inconsistent data updates in the database by many participating devices (agents) (See also https:// en.wikipedia.org/wiki/Database_translation). At the same time, it is necessary (required from the database) to remain reliable, protected from unauthorized access (secure), complete and accessible to all agents, that is, to provide a standard level of reliability and responsiveness for a server-based database (SQL or NoSQL).

• Even in a relatively small parking lot, cars can arrive and depart every minute. The situation in several parking lots in a metropolis can change every second, and five-minute-old data is no longer of value. On a citywide scale, thousands of cars will update data on their position, occupied parking lots, etc. every second.

• On the battlefield, the situation changes every second. The set of visible objects, positions and actions of the participants in the collision can change extremely rapidly. Fighters and drones move, new targets are identified and suppressed, information about changes in the situation can come from sensors, drones, be entered manually, come from headquarters, etc.

Additional explanations are as follows. Integrity implies that (conditionally) if a car can be either on the road or in the parking lot, then there will never be a situation where the same car will be both on the road and in the parking lot. That is, at each moment of time the car exists (the integrity is not broken - nothing fell out) and is in a consistent, adequate state (either there or there). It is achieved through replication and the use of not individual operations (remove a car, add a car), but through transactions (either remove + add to the database, or nothing will arrive, no wreckage and partial updates).

Thus information is not lost unless it is intentionally done.

If it is necessary to make the database area working and storing data even when there are no agents left in the area, the custodian of the "area" is forcibly appointed, for example, a dedicated "agent" controlled by the server is placed in the area, and the agent remains in the area until it is required to save data from this area.

9. The database must fully function and maintain integrity in case of violations of the connectivity of individual agents and groups of agents, with arbitrary connection and disconnection of agents, including abnormal and / or forced ones. For example, a swarm of "content givers" in BitTorrent (https://en.wikipedia.org/wiki/Glossary_of_BitTorrent_terms#Peer or https://patents.justia.com/assignee/bittorrent-inc) can expand or contract, individual nodes can disconnect, but the swarm as a whole still provides content.

• A player connected to the game via a mobile network may have an unstable connection: spontaneous changes in speed, packet loss, temporary disconnections, complete disconnection. However, other players should not receive delays and / or errors.

• On the battlefield, any device can be disabled at any time, and communication may be unavailable due to interference and damage to wire lines. At the same time, the surviving devices should continue to work, exchange (if possible) data, adequately display the situation according to the data that they have and try to restore synchronization.

10. The base may consist of several separate "layers", some of which are visible to some agents and invisible to others. For layers, you can, for example, use "Type", as in the example above.

• Trucks may ignore car-only parking spaces.

• The UAV (unmanned aerial vehicle) can only receive data related to targeting for it, while targeting for mortars will be transmitted in another "layer".

Tree database and topological (topographic) addressing

It is possible to represent the proposed database of objects in the form of a tree structure, where the address of each node consists of a sequence of tokens, for example, numbers. The first number can indicate the type of data, such as "1 - buildings, 2 - trees". Then, in the address, a sequence of N numbers (for two-dimensional space - two numbers) indicates belonging to a certain area in space. For a two-dimensional space, we get [/Type/CoordinateX/CoordinateY]. For example, at [/1/3/2], we might have a list of buildings located on a square kilometer 3 kilometers east and two kilometers south of the 0:0 point. At the address [2/3/2], trees will be stored on the same square kilometer 3:2. And at the address [/1/2/2] in the database, buildings on the next square kilometer with coordinates 2:2 will be available (2:2 is adjacent to the 3:2 square on the left).

An example of placing objects on the map and addressing them in the database (Fig. 1).

View Addressing

[/Type/CoordinateX/CoordinateaY/OrdinalNumber]

Addressing such "regions of space" can be done in different ways: [/Type/CoordinateX/CoordinateY], [/CoordinateY/CoordinateX/Type] or generally [/CoordinateY/CoordinateX], etc. Any address that meets the criteria will do:

1. The address of any database entry must contain the address of the topological area (for example, an N-dimensional vector) to which the object belongs

2. Each object stored in the database is guaranteed to be associated with its topological area (preferably, unambiguously)

3. For each region, you can select the addresses of all neighboring (adjacent) regions

Within each region of space, you can address objects in the same way, numerically (which gives a simple end-to-end numerical addressing of the form /#/#/#/#/… for the entire database) or in some other way (up to using SQL), which slightly complicates the implementation, but in some cases it can be useful. The branching depth of the resulting "tree" (and, accordingly, the length of the addresses) can be any.

Examples of possible addresses for fields from such a database:

• [/Type/KilometerX/KilometerU/MillimeterX/MillimeterU/OrdinalNumber]

• [/KilometerX/KilometerT/Type/Unique1B]

• [/KilometerX/KilometerU/GroupID/SubgroupID/UniqueID]

Thus, it is possible to place in a single tree-like database a description of any "territory", arbitrarily large, having (preferably uniform) coordinates, and address the objects of this database, knowing which region of the N-dimensional space they belong to and, knowing the address of the object, set the area in which the object is stored.

Recording on the example of a tree-based addressing database, transaction log

Let's assume that the entire database is available to some agent for reading and writing. An agent, having an address, can read the objects stored in the database (or make sure that there is no object at the specified address in the database). The agent can also create new objects in the database, delete them and modify them based on the address.

Write operations to such a database will be performed by the agent through three hierarchically related objects:

1. "Operations"

2. Consisting of operations "transactions"

3. Consisting of transactions "transaction log". OPERATION

An "operation" describes a minimal change to objects in the database and can be applied to (performed on) the database. The operation includes the type, address of the target object(s), and the data needed to complete the operation. An example of an operation would be:

• Creation of an object. In this case, the data is the contents of the object to be created, for example, "create an integer 8 at the address [/5/8/12/8]".

• Deleting an object. The data in this case is an empty field, for example "delete an object at [/5/8/12/8]"

• Transferring an object from address A to address B. In this case, the data is address B, to which you need to transfer data from address A. For example, "move an object from address [/5/8/12/8] to address [/5/7 /12/8]".

• Comparing an object for equality. In this case, the data is the value with which the object must be compared. For example "check that the address [/5/8/12/8] contains the integer 8"

There can be many types of such operations, from the simplest (creation / deletion, arithmetic operations, more / less operations, operations of changing the type from "integer" to "floating point number", etc.) to more complex ones, if the object is a complex structure (rebuild the 3D model of the star system at [/98585/9982157/6654781] taking into account the current gravitational fields).

Operations can be different, but they must meet the following criteria:

1. Affect a finite, rigidly determined number of database topological areas. An operation can read or make changes to one specific object or to a group of objects, but the list of affected areas must be uniquely determined and not depend on the state of the database. Ideally, the operation affects either one object in the database (changing an object) or two (moving an object), but there may be more complex cases.

For example, an operation like "move the object referred to by the database object at the address ..." is not, in general, valid. The fact is that in this case, depending on the state of the database at the time of the operation, the affected areas may differ.

2. Be deterministic, that is, when performed on a database containing the same data, the operation must always give the same result. If the "write random number" operation will write a newly generated (that is, usually new) random number each time it is executed on the same database (on the same agent or on different ones), then this is an incorrect operation.

3. The operation may or may not succeed ("fail").

An example of unsuccessful (failed) operations:

a. The operation "arithmetically add to the object "string" try'" the object "integer 5" is impossible - it is impossible to perform arithmetic operations with the string.

b. The operation of moving a non-existent object. If there is no object [/1/8/9/9/8] in the database, then the operation "move the object [/1/8/9/9/8] to a new address [/1/8/9/9/5]" will not be executed because it is not possible.

c. The comparison operation if the comparison returned "false". For example, if we have an object "integer 5" and we apply the operation "check equality to integer 8" to it, such an operation will not be successful.

4. The operation must be "rollback" back. If we replaced the value "integer 5" with "integer 8", we can remember the previous value, and then we can return the database to the "before operation" state. So with any other operation, it should be possible to "roll back" back, returning the database to the state "before the operation was executed" (comparison operations, since they do not make changes to the database, do not need a special rollback procedure).

TRANSACTION

To make changes to the database, individual operations can be combined into "transactions". A transaction can consist of one or more transactions. Transactions are logically executed on the database sequentially, not in parallel, that is, while the state of the database is changed by one transaction, other transactions of the database do not change (this requirement is "logical", that is, the actual implementation that tracks the areas affected by transactions can de facto execute non-overlapping transactions to improve performance). Operations in a transaction are performed in a given sequence, which, thanks to the properties of operations, allows both to execute a transaction on the database (in this case, all operations are performed in direct order, until the first unsuccessful operation), and to "roll back" (in this case, completed operations are rolled back in reverse order). and thus the database is guaranteed to return to the state before the transaction).

Relying on the ability to roll back individual operations, it is possible to make all transactions atomic. That is, the operations of a transaction are either all executed or not all executed. Failure (failure) of any of the operations included in the transaction leads to the failure of the entire transaction, while all operations already performed are rolled back in reverse order, and the database remains unchanged as a result of the unsuccessful execution of the transaction.

Atomicity, in addition to eliminating inconsistent and unpredictable database changes, is useful for implementing the basic "if ... then" logic built into the transaction, which is extremely useful for working in conditions of uncertainty of the current state of the database on each individual agent. For example, when combining compare operations with write operations, the transaction might look like this:

1. Check that the address [/1/2/2/8/7] contains the integer 12

2. Set the integer at address [/1/2/2/8/7] to 6

3. Set the integer at [/1/2/2/8/9] to 3

4. Check that the sum of [/1/2/2/8/7] and [/1/2/2/8/1] is 8

Thus, you can be sure that if any check is not successfully passed (the first one, placed in the transaction before the assignment operations, or the final one, checking the already changed values), then none of the operations from this transaction will be executed.

Due to the fact that for each operation the topographic areas affected in it are precisely known, it is also possible for each transaction (just by examining operations) to establish a list of database areas that will be affected by the transaction.

TRANSACTION LOG

Changes in the database are proposed to be made by sequential execution of transactions. In the simplest case, the database will initially be completely empty, and on this database, transactions with the "create object" operations are first executed, then any other transactions.

A general, unconditionally ordered list of such sequentially executed transactions, we will define as a "transaction log".

Transactions (on each agent and at the level of the global database log) will be placed in the "transaction log" in a fixed order. The order can, for example, be set as follows:

1. a unique ID will be generated for each transaction,

2. The ID will include the current time and the ID of the agent that created the transaction,

3. it is also possible to add a transaction counter of this agent to the transaction ID, which will allow other actors to detect "skipped" transactions,

4. since each next transaction will be created later than the previous one, each transaction will have a unique ID,

5. if all transactions are sorted based on such an ID (transactions), first in ascending order of time, and then in ascending order of the agent ID, and then the counter (if it is used, which reduces the requirements for the accuracy of the time counter), then we get a simple and unambiguous way put all transactions in order.

TRANSACTION LOG EXECUTION

The resulting log (ordered list) of transactions can be executed (perform all transactions in direct order, from oldest to newest) and "undone" (roll back all transactions in reverse order, from newest to oldest). The log can be executed (and canceled) in whole or in part.

When adding a new transaction to the log, it can get not only to the end of the list, but also to an arbitrary place. For example, if agent A created a transaction five minutes ago, but, due to network connectivity problems, was able to transfer it to be logged to agent B just now (i.e. with a delay), and other agents had already logged by this time your transactions. In this case, if the transactions that were in the list after ("before") the new transaction have already been executed on the database, they should be "rolled back" (back), then execute the "new" (and in fact the old) transaction, and then again execute all subsequent transactions that have just been undone. Functionally, this is equivalent to returning the database to the starting state and sequentially applying all transactions from the final log, but it is incomparably faster, especially if the log is large and includes many transactions.

Since the transaction log is built so that after the completion of the data exchange it will be identical for all agents and all operations are deterministic, the final state of the database on all agents will also be identical. If some operations added by agents are not compatible with the final state of the database, they will not be successful at the same time and will not be executed for all agents, without generating any inconsistency.

By tracking the affected branches of the DB tree addressing considered in the example, you can even roll back and re-execute not all transactions, but only those whose results the newly added transaction could have affected (if any), which will give even greater performance.

Due to the ability to put transactions in the log in an arbitrary order, multiple agents can asynchronously and inconsistently add new transactions to the log. In this case, it is always possible:

1. uniquely determine the sequence of transactions in the log,

2. deterministically apply the transaction log available to the agent to the database,

3. receive new transactions, insert them into the transaction log (including "backdating") and correctly apply the final transaction log, bringing the database to the state of sequential execution of the entire final log

4. based on points 1-3 above, an unlimited number of agents can participate, each working with their own local copy of the database, while receiving identical database states, provided that they successfully exchange their transactions and each agent eventually receives a complete list of transactions, moreover, the exchange of transactions between agents can take place in any order.

5. Based on the atomicity of transactions, it is possible to expect that even in the process of the ongoing collection and execution of the transaction log, each agent will possibly have a unique database state, but this state will be consistent and, in general, will be suitable for reading / writing at any given time, regardless of the speed and regularity of the exchange of transactions with other agents.

6. A side negative consequence of this method is the inability to guarantee the successful completion of the transaction at the time it is placed in the log by the agent. It is possible to determine exactly whether a transaction was executed or failed only at the moment of "clarity", see below.

It should be noted that the use of the "DB + Log Transactions" (DLT, Distrit ledger) technology for distributed work with data is well known and widely used in many storage systems, for example, in P2P cryptocurrencies such as Bitcoin. The novelty of the proposed solution (described in detail below) lies in the fact that the general transaction log is divided into segments due to and in connection with the "topological (topographic) segmentation of the database" and agents, working only with selected segments at any time, can quickly integrate changes and as a result, interact with a single database of arbitrary size with extremely limited resources.

DISTRIBUTED DB STORAGE

LOCAL COPY OF THE DB FRAGMENT

It is proposed to store the database (in whole or in part) on the agent's computer device as a set of independent objects, for example, in the form of files on a disk and / or objects in the computer's memory. There must be one isolated storage object per topological area. Each such storage object may, in turn, itself consist of several objects, depending on the implementation, but in this case it is important that a local object belonging to one area can be changed, deleted or copied to the agent from the outside, completely independent of the objects related to other areas. An object that stores data in a separate area does not affect neighboring objects, and neighboring objects do not affect it.

Additionally, you can divide objects by "layers" (for example, types), so that the agent can download one layer for itself for the area, but not download data for other layers.

In its simplest form (convenient for explanation), the database can be stored on the agent's hard disk as a direct indication of the path to the file and the address of the object. In this case, the object at the address [/1/2/1/1] will (for example) be stored in the file "/storage/1/2/1/1.dat", and the object at the address [/1/3/2/3 /15] in the file "/storage/1/3/2/3/15.dat". Reading/writing from such a database is trivial, and data belonging to one layer of one topological area is represented by a folder on disk. Such a folder can be easily deleted, its contents changed, or copied from another agent, and this will not affect the folders that store the data of neighboring areas.

A more realistic implementation could use berkleydb (or a similar key-value storage technology, e.g. https://ru.wikipedia.org/wiki/%D0%91%D0%B0%D0%B7%D0%B0%D0%B4% D0%B0%D0%BD%D0%BD%D1%8B%D1%85%C2%AB%D0%BA%D0%BB%D1%8E%D1%87-%D0%B7%D0%BD%D0 %B0%D1%87%D0%B5%D0%BD%D0%B8%D0%B5%C2%BB) as the realm data store. If the amount of data is large and the write rate is high, multiple berkleydb storages can be used, one for each "layer". If the amount of data is small and partial updates are rare, you can use standard serialization / deserialization tools, for example https://habr.com/en/post/4 7 9462/. For session applications that do not require long-term data storage, such as "session games" where data is not needed at the end of the game, you can limit yourself to storing data for each area in the application's memory, in the form of dynamically created OOP objects or other storage-oriented data in the programming language used, structures (for example, struct in C ++ https://en.wikipedia.org/wiki/Struct (C programming language))

ZONE OF VISIBILITY

Each agent can have an arbitrary visibility zone (specified by the agent himself or set to him from the outside). For example, a car computer looking for a parking space needs information about free parking lots within a radius of 500 m, but it does not need information about parking spaces on the other side of the city. The visibility zone can change over time - move, change coverage, etc.

In most cases, information about the agent's visibility zone should be stored in the database.

Since the agent does not need to read or write data outside the visibility zone, each agent can store and update only information about the topological areas included in its "visibility zone".

As the "visibility zone" changes, the agent will delete (or archive) objects that are not related to its new "visibility zone" and receive from the outside (from neighboring agents or from the server) data on areas that have appeared in the visibility zone.

As a result, each agent at each moment of time will store and update a fragment of the database, which is several areas from the common "mosaic".

SEGMENTED TRANSACTION LOG

In the same way that, based on topological areas, the entire database is divided into independently stored and updated fragments, it is possible to deal with the transaction log.

As already noted in the description of "transaction" objects, for each transaction the list of topological areas affected by this transaction is always known exactly. Thus, it is possible to store (ideologically "single" for the entire database) transaction log as a set of "local" transaction logs related to specific topological areas. In this case, if a transaction affects (changes) several areas, it is possible to store several copies of this transaction, one for each affected area. Duplicate transactions can be detected because each transaction is assigned a unique ID, which allows you to get rid of repetitions when combining the transaction logs of several areas into a single transaction log.

In this case, each agent stores the transaction log as a set of independent logs related to separate areas. For example, for layer 1 and area 2:1, the log can be stored in the file "/storage/1/2/1/1.log".

Since each log fragment contains all transactions related to this area, it is independent and self-sufficient. For each area, you can store either such a log from the creation of the database, or, better, store a "snapshot" of the intermediate state of the database (a snapshot can only be taken in the state of "clarity" (fully defined), see below) of the area, plus information about which the transaction was last executed in this snapshot, plus the transaction log for this area, containing only transactions not included in the snapshot, placed in it after the last snapshot transaction (snapshot). From such a set (a snapshot with the exact creation time, plus a log of newer transactions), it is possible to get the state of the area in the same way as if you had access to all transactions in the global log and the entire database.

The only exception in this case may be the presence of operations such as copying/transferring objects from areas that are not included in the agent's visibility zone to an area that is in the visibility zone. This problem can be solved by selectively storing the deployed version of such transactions in the log. For example, in the target area [/2/3/4] instead of "copy number from address [/1/1/1/4] to address [/2/3/4/5]" there will be the operation "create number 5 at address [ /2/3/4/5]", and instead of "transfer" "transfer or, if the source is not available, create"). It is also possible to create a data exchange protocol between agents so that the agent can request the data fragment it needs from its neighbors and copy inaccessible objects from the received fragment. Both solutions assume that the donor area is in a state of "clarity" (complete certainty) (see below for more details), that is, there will be no "backdated" transactions and the agent knows exactly the final state of the copied object at the time of copying.

A SINGLE DB AS A SUM OF LOCAL COPIES

If there are many independent agents using the same database and exchanging data with each other (for example, via the Internet, see details below), such a "mosaic" database may not be fully accessible to any agent and may not be stored entirely anywhere. In this case, each agent will work with its own fragment and will be able, if necessary, to receive the necessary fragments from other agents.

For individual applications (or for individual layers of a more complex database), such a "mosaic database" of fragments scattered over agents is already fully functional. Such a database does not require a server or a dedicated agent that stores the entire database. It as a whole does not physically exist on any separate computer, each user agent sees and stores only its own fragment. If the fragment does not have any "keeper" - then it's just an empty fragment.

For example, if the database contains only information about the position and actions of player avatars in the game universe, information about topological areas where there are no player avatars does not exist and is not available. But, since there are no player avatars there, this lack of data is a) formally correct (an empty set) and b) this is of no interest or problem for any agent, because if an empty place is not in the visibility zone of any player, then the data from this areas are not needed. The opposite is also true, in each area where the player's avatar is represented, the computer of this player stores and maintains the corresponding fragment of the database. And if another avatar falls into his field of vision, they will successfully exchange information about each other, since the agent of the other player also maintains the visibility area of his avatar and has an up-to-date database of the corresponding areas. Thus, such a database will provide all the necessary and sufficient information to each agent.

A similar implementation can work for a database in which the data is rapidly becoming outdated - data from sensors that are updated every second, etc. As soon as the sensor stops serving its area (fails down, loses connection), the topological area corresponding to it will become "inaccessible", but this quite accurately reflects the state of affairs, so we can assume that the database has done its job.

For more complex applications, where data must exist in the absence of an "observer", it will be necessary to use (dedicated) agents that purposefully archive data from a set of areas and provide "archived" data to agents whose corresponding area is in view. Such agents may or may not cover the entire "territory". You can consider such dedicated agents as "servers", although this does not quite correspond to the standard client-server ideology and, in many implementations, the "keeper" function can be performed by agents that are ordinary in all other respects (for example, always-on sensors with sufficient computing resources) .

COMMUNICATION BETWEEN AGENTS

In order for the described database to function successfully, each agent must be able to:

1. Identify (preferably all) agents whose visibility zone intersects with the visibility zone of this agent (hereinafter referred to as "neighbors")

2. Communicate and/or exchange data with some (or all) neighbors

3. The connection between neighbors should ensure the connection of each with each, indirectly (my neighbor's neighbor is connected with me) or directly. In other words, all neighbors with overlapping visibility zones must form (form, form) a connected graph.

For example, in the case of cars, communication and identification of neighbors can be carried out directly, via WiFi connections between agents, or via a standard cellular network. When using a standard cellular network for data transmission, agents will not be able to identify neighbors on their own, information about neighbors will have to be provided either by a cellular network or some other provider.

In the case of a military unit, the list of nodes connected to the network can be predefined and/or external, since arbitrary nodes do not need access to such a network. Connectivity between the nodes here will probably be quite heterogeneous and can be provided taking into account the characteristics of the available equipment and environment - some channels will be wired, some radio, somewhere you can use optical transmission. But the logic of operation of the described database is not sensitive to the nature and heterogeneity of channels insofar as agents can systematically exchange messages containing transactions, etc. (see below). Communication "each with each" is not required, it is enough that the graph of agents is connected.

In a multiplayer game, the connections will also often be heterogeneous: a dedicated server may be required at the stage of connecting the agent to the community, later, depending on the game’s demands for security and responsiveness, clients may either not use the server’s services at all, or resort in extreme situations ( for example, to create P2P channels between agents connected via NAT (Network Address Translation), or (optimally) to transfer the maximum possible amount of data through P2P channels, and what cannot be processed through direct connections (due to security requirements and / or from - due to the inability to create a P2P connection) to process with the involvement of the server.

EXCHANGE OF INFORMATION BETWEEN AGENTS

Communication between agents will at least consist of the following steps:

• the agent will send to all neighbors (directly or in transit through directly connected neighbors) information about transactions placed by the agent in the database log. Information, in addition to the list of transactions and service information related to the transaction, may include an electronic signature of the transaction with a secret key, which will verify the authorship of transactions,

• the agent can send to all neighbors a "pulse" (synchronizing messages) containing the current time and the ID of the last transaction issued by the agent (see the "clarity" section below),

• the agent can certify transactions received from neighbors (for example, certify the time and/or feasibility of the transaction with a secret key) and send a separate certification and/or an already certified "A issued, B approved" transaction,

• the agent can forward received transactions (and confirmations of transactions) further, to other neighbors (functionally acting as a "transit gateway"). In this case, the agent can simply forward transactions further to all neighbors that did not send a transaction with this ID, or use optimization algorithms based on knowledge of the state of the "network" of neighbors,

• the agent can, upon request, send to its neighbor a "snapshot" of the DB area known to it (see below) and/or a fragment of the transaction log related to the DB area known to it; the snapshot can be signed with an electronic signature to certify the integrity and authorship of the "database snapshot" and "transaction log snapshot",

• An agent can, upon neighbor's request, send a transaction with a specific ID to it (for example, it is useful if the transaction was not delivered due to network problems, and the neighbor knows from the heartbeat that it exists).

AREA STATUS - EXCHANGE BETWEEN AGENTS

In the case when the agent's area of interest (which the agent does in a situation where his area of interest) includes a topological area about which the agent does not have information, he performs the following actions:

1. the agent finds a neighbor who already has the desired area in his area of interest (and is authorized (possessing the rights) to give a "snapshot" of area information on request),

2. the agent sends a "area snapshot required" request to this neighbor,

3. a neighbor, if he does not have information on the area of interest, can forward such a request further or return a refusal; if he has the information, the neighbor satisfies the request; for this:

a. the neighbor finds out the last known transaction, before which there will definitely not be new transactions (i.e., the “clear moment”) (see the “CLARITY” section below)

b. rolls back (in the simplest case) generally all transactions executed after the “clear moment”, or (slightly more complicated) transactions related to the desired area and placed in the log after the “clarity moment”, plus the transactions on which these transactions depend,

c. takes a snapshot of the area (in a simple example with files, this can be a zip archive with the contents of the area folder),

d. transfers the received impression to the requester,

e. passes to the requester the ID of the last transaction executed on the snapshot,

f. sends to the requester a list of neighboring agents with the state of their counters (in the case of calculating "clarity" based on transaction counters),

g. transfers to the requester all known transactions related to the requested area and available to the giver in the log after the "moment of clarity",

h. all together or separately can be signed with a secret key before transmission,

4. Having received and placed this data set, the agent receives a full-fledged working section of the database responsible for the area, and can continue to work normally (including providing a "cast" to other neighbors if he is authorized for such operations),

5. if the first neighbor did not return the requested snapshot, the agent requests a snapshot from another neighbor, then from the next one, and so on,

a. If no neighbor authorized to give "snapshots" (including neighbors acting as dedicated archiving agents, server agents, etc., if any) has returned the requested snapshot, the agent may act as if its connection to the community was physically torn apart.

The agent regularly receives subsequent transactions from mailings (or explicitly requests if it becomes clear from the pulse that some transactions added by other agents to the log after the "snapshot" moment are missing).

ROLES OF AGENTS

Depending on the requirements of a particular application, the agents (participant devices) that make up the community can be completely identical and equal, or they can have an explicit separation of functions. The roles of agents can vary from an "all equal" structure to a multi-level interlaced scheme, where different agents have different rights to different database layers, different types of transactions, etc.

The division, for example, might look like this:

• an agent can act as a "non-trusted actor" (untrusted member of the community): has the right to create transactions that other agents must verify and approve or reject,

• an agent can act as an "arbitrator" who does not have the right to create transactions, but has the right to certify transactions; such an "arbiter" can, for example, be forced to assign to each agent-actor a server,

• Some of the agents controlled by the administration can act as a "server" with exclusive rights to sign and certify their own transactions, with the right to write to the database layer "user registry".

• agent roles can be assigned by a dedicated agent-server,

for example, entering data on the status of each agent in the database in a section that is writable only by the server, and all other agents will be guided by this information.

Agents - arbitrators

It should also be borne in mind that several agents in different roles (or even in different roles on different databases) can work on the same computer (with the same database) at the same time. For example, by connecting to the authorization server of a computer network game, the program, in addition to connecting to the database to control the player's avatar as an "untrusted actor", can get the role of "arbiter" in the same (or another) database. In this case, the agent will check and certify the actions of another player randomly selected by the server in order to exclude fraudulent actions prohibited by the game.

Such validation "on the side of a random player" is not absolutely reliable protection against unauthorized changes to the database, validation on the server side allows you to achieve greater reliability. But full validation on the server side is extremely resource intensive and therefore rarely used. In addition, the resource intensity of server-side validation increases linearly with the number of players, which often makes it completely inapplicable. Validation by one player of another player scales infinitely, in parallel with the growth in the number of players. By assigning each player two random "arbitrators" from among the players, the probability of "collusion" between the arbitrators and the player is 0.01% in a game universe where there are only 100 players. As the number of players increases, the attractiveness of hacking grows linearly, but at the same time, the probability of "collusion" between the player and arbitrators also decreases non-linearly (inversely proportional to the square of the number of players), which gives an excellent balance of security and ease of implementation.

CLARITY

In a database operating in the described mode due to unpredictable delivery delays, connectivity interruptions, and network restructuring, the agent may face the question: where new transactions can still be added to the existing transaction log, and where the log is already fixed and there will be no new transactions guaranteed ?

This is important, for example, when transferring a "snapshot" of a database area, since the snapshot should consist of the actual data snapshot in the database and the transaction log that were added after the snapshot was taken. If we take a snapshot as of some point in time T and transfer such a snapshot (imprint) to a neighbor, and then it turns out that a new transaction has appeared dated by time T-1, then the neighbor, not having a full-fledged log for the period preceding T, will not be able to apply such a transaction to your database.

In the case of sorting the transaction log by ID, described above, consisting of time and actor ID, the question is: "what is the minimum transaction ID that we can still get from other neighbors?"

The following methods can be used to establish "clarity".

CONSENSUS

Each transaction is authenticated by each agent with write access to the area, by a majority or otherwise selected set of agents, or even just by the server. It is possible, for example, to use the BLS algorithm (https://ru.wikipedia.org/wiki/BLS), which allows you to compactly store a series of signatures, or use another method. The consensus approach is not efficient, plus it potentially allows "cancellation" of transactions, when some of the agents who did not have time to "vote" are forced to cancel the transaction. But in some database applications, consensus/certification/signature is required in any case, for example, when working with cryptocurrencies, to make sure that the transaction meets all the criteria for security, validity, etc. For example, if it is a money transfer and we want to be sure that both the "payer" and the "recipient" agree with the transaction and we have a sufficient number of "witnesses", etc. At the same time, the consensus on transaction B can be released only after the consensus on transaction A has been reached (for example, the signature of each subsequent transaction must include the signature of the previous one, the standard blockchain), which will automatically give both the confirmation of the transaction and the tracking of "clarity".

PULSE + COUNTERS (How "clarity" is achieved: how to detect transactions missed in the local log)

1. The transaction ID includes the incremental transaction counter of each agent. The agent increases the value of the counter by 1 in each next created transaction (the counter can be rotated, for example, increased to 255 and then reset to 0),

2. each agent has a public (visible in the database to other agents) "active area", that is, a list of areas in which the agent can write, i.e. place new transactions,

3. Agents send out a "pulse" at a specified interval containing information about the current time and the last transaction ID that this agent placed in the log (and this ID contains the counter described above),

4. all agents keep track of the full list of neighbors that have the right to write to each area in their visibility zone and listen to their "pulse", they also track the latest transactions of these actors,

5. if the heartbeat counter is greater than the counter of the neighbor's last transaction, or if the neighbor's last received transaction increased the counter by more than 1, this means that the neighbor created a transaction that the agent does not have and he requests it to be repeated from the neighbors,

6. when the heartbeat and/or fresh transactions from all writeable neighbors are received, the agent will know exactly where the "clear point" is (less than what ID all logged transactions are already known to the agent).

The method is simple, but a potential problem is the unforeseen "falling out" of individual agents, for example due to a power outage, which will create a delay in establishing a "point of clarity" due to the lack of a heartbeat and, therefore, information about the last transaction of the "disconnected" agents. But the "point of clarity" is needed mainly to optimize the exchange of snapshots between agents and the system remains fully functional even if an emergency event occurs and the point of clarity "gets stuck" after (does not move further) the first transaction in the log. In case of an abnormal disconnection of agents, as an option, the server or other authorized agent, having detected the "agent" that has lost connection, can issue a "retroactive" transaction that removes the dropped agent from the list of "active agents" at the moment when the connection was lost, after which the establishment procedure clarity will continue to work normally.

Also, the "bottleneck" of such a system is the appearance of neighbors. If, for some reason, the agent does not know that he has a new neighbor, the agent will not track his pulse and will not know that he missed transactions. This problem can be solved by forcing "core change" type transactions, for example, through a dedicated agent-server that guarantees the delivery of all "neighbor arrival" type transactions. It is possible to force nothing to be sent, but to consider all far areas "unclear" and determine the clarity for them not by listening to the pulse, but by listening to the clarity confirmation, including the "last transaction ID" from agents with write access to the area. In this case, it is possible to additionally force transactions to be sent for "empty areas" (when an agent enters an area where there is no one and when the "last" agent leaves the area). This method (a combination of "clearness defined by agents that have write access to the area and forced transactions for empty areas) works well if each agent has a visibility zone (the area from which the agent reads data) with a margin covering the "active zone", for example if an agent can only write to one area of a 2D database, then it can read to this area and 8 adjacent areas.

DB DISTRIBUTION METHOD

The proposed database can exist both as a library and as a separate application available in the form of a "server" to simple NOSQL clients, for example, via the memcached protocol (https://github.com/memcached/memcached/blob/master/doc/ protocol.txt).

When setting up, the database administrator describes (in the configuration file or in the program code) the dimension and structure of the database, types and powers of agents, methods (or arbitrary program procedures) of transaction validation (certification) and other parameters, after which he either integrates the library into his software, where working with the database is part of another functionality, or runs the database on your computer.

Claims (45)

1. A method for collecting, storing, updating and processing information about an n-dimensional space, previously divided into topological, including geometric, topographic, areas, including the collection of information by means of collection located on devices that have coordinates in the described space and are capable of exchanging between information that all together form a connected graph, placement of information related to the topological area, on information storage facilities located on the specified devices connected to the community, which at that moment have coordinates within this area and neighboring areas, exchange of information with devices having coordinates in this or other, mostly adjacent, areas, deletion of information relating to the topological area, when changing the coordinates of the device to others remote from this area, moreover, the recording and modification of information is carried out through a distributed log of operations (DLT), where each operation changes data only about a pre-fixed set of specified selected areas of the n-dimensional space, moreover, the complete log of all operations is stored as a set of samples from this complete log, where each sample consists of a complete list of operations related to a particular topological area, moreover, each device stores a list of all devices connected to the community that have coordinates in its own and adjacent areas of space, moreover, devices having coordinates inside the topological area transmit information to other devices having coordinates inside the same area and / or in adjacent areas during the recording process, moreover, the initiating device writes information about data operations to the local memory initially in the "inconsistent" status, then the initiating device sends information directly or by sending it to all other devices that have coordinates in the same area as the initiating device and in adjacent areas, and the specified information, the initiating device assigns the status "consistent" after all other devices from the same and adjacent areas have come to a "consensus" that the transaction has been successfully registered, i.e. received by all destination devices. 2. The method for collecting, storing, updating and processing information about the n-dimensional space according to claim 1, in which the "consensus" is fixed after confirmation of receipt of information from all devices in the same and adjacent areas. 3. The method of collecting, storing, updating and processing information about the n-dimensional space according to claim 1, in which, when exchanging messages between devices, asymmetric encryption of messages is carried out. 4. The method of collecting, storing, updating and processing information about the n-dimensional space according to claim 1, in which, when exchanging messages between devices, messages are provided with an electronic signature, which makes it possible to guarantee the authenticity of messages. 5. The method of collecting, storing, updating and processing information about the n-dimensional space according to claim 1 or 2, in which - each device stores information about the last operation affecting its own and any of the adjacent areas, before which the device knows all the operations performed by all devices in the community for its own and adjacent areas; - a new transaction about its area is written to the local memory of each device in the status "accepted"; - at the same time, each device monitors all changes made by other devices that have coordinates in the same area where the device is located and in adjacent areas, and also periodically sends information to all neighboring devices about which transaction was recorded last by this device, or information that the recording was not carried out; - each device monitors the completeness of its operation log and requests missed transactions if it learns about them, or updates information about the operation for itself, before which all operations are known to this device. 6. The method of collecting, storing, updating and processing information about the n-dimensional space according to claim 1, in which there is one or more device with server functions that performs at least the primary authorization of devices, proxying information and organizing P2P interaction between agents. 7. The method of collecting, storing, updating and processing information about the n-dimensional space according to claim 6, in which the specified server does not belong to the community and does not have coordinates in the specified n-dimensional space. 8. The method for collecting, storing, updating and processing information about the n-dimensional space according to claim 1, in which any of the devices, upon entering the specified n-dimensional area, passes the authorization procedure. 9. The method of collecting, storing, updating and processing information about the n-dimensional space according to claim 1, in which any of the devices is assigned additionally broader or truncated data access rights, extended or reduced responsibilities, other additional. responsibilities or restrictions compared to other devices in the community. 10. Dynamic system for collecting, processing, exchanging and placing information about n-dimensional space, including - n-dimensional space, previously divided into topological areas; - one or more devices assigned coordinates in n-dimensional space, equipped with one or more means of collecting information, a means of storing information, a means of communication with other devices, with the ability to move both within one topological area and to any of the adjacent topological areas, moreover, the specified information contains, among other things, a logical address, moreover, the information storage means additionally includes an information status storage means, a partial log - a selection from the general distributed transaction log, moreover, in the partial and full log of operations, each operation changes data only on a pre-fixed set of specified selected areas of the n-dimensional space, moreover, the complete log of all operations is stored as a set of samples from this complete log, and each specified sample consists of a complete list of operations related to a particular topological area, moreover, information about the current area is recorded in the local memory of the initiating device, initially in the "not agreed" status, then after confirmation - in the "agreed" status. 11. The dynamic system for collecting, storing, updating and processing information about the n-dimensional space according to claim 10, in which the logical address of each piece of information necessarily contains a unique identifier that uniquely identifies the area to which this record belongs. 12. A dynamic system for collecting, storing, updating and processing information about n-dimensional space according to claim 10, in which information about each area of space includes a list of all devices connected to the community that have coordinates in this area. 13. The dynamic system for collecting, storing, updating and processing information about the n-dimensional space according to claim 10, in which the transaction about the current area is recorded in the local memory of the initiating device initially in the "accepted" status. 14. A dynamic system for collecting, storing, updating and processing information about the n-dimensional space according to claim 10, in which there is one or more devices with server functions for primary authorization of devices, proxying information and organizing P2P interaction between agents that does not belong to the community and having no coordinates in the specified n-dimensional space. 15. A computer-based system for collecting, processing, exchanging and placing information in an n-dimensional space, including an information bus, a processor attached to the information bus, an information collection device, a memory device, - containing instructions for collecting information, exchanging data with other devices located in the n-dimensional space, - additionally containing instructions for devices having coordinates inside the topological area to transfer information to other devices having coordinates inside the same area and adjacent areas during the recording process, - additionally containing instructions that each procedure for writing data to the area consists of • primary recording in the local memory of one device, • assignment of the primary record of the status "inconsistent", • broadcasting (direct or by forwarding) data to other devices in the same area, neighbors in adjacent areas, • receiving a response from all neighboring devices within the area and adjacent areas that data has been received, • assignment of the primary record of the status "agreed". 16. The system hosted on a computer according to claim 15, additionally containing instructions: assign server functions to one of the devices for primary authorization of devices, proxying information and organizing P2P interaction between agents that does not belong to the community and does not have coordinates in the specified n-dimensional space. 17. A computer-hosted system according to claim 15, additionally containing instructions: • each device to store information about the last operation affecting its own and any of the adjacent areas, before which the device knows all the operations performed by all devices in the community for its own and adjacent areas; • each device to write to the local memory (storage medium) of the device in the "accepted" status new information about its area; • each device keep track of all changes made by other devices that have coordinates in the same area where the device is located and in adjacent areas, as well as periodically send information to all neighboring devices about which transaction was last written by this device, or information about that no recording has been made, • each device should keep track of the completeness of its operation log and request missed transactions if it learns about them, or update information about the operation for itself, before which all operations are known to this device.

Images