RU2621628C1 - Way of the linked data storage arrangement - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: way of the linked data storage arrangement comprises stages on which the records are united in at least one group according to the similarity; the identification code is assigned to every mentioned above group and to every record that is in this group; the link between the group ID and its location address is formed on the memory storage in such a way that the translocation of groups on the memory storage as well as among memory storages do not change the group ID value; the link between the record identifier and its location address within the group is formed on the memory storage in such a way that the record relocation on the memory storage within the group does not change the record identifier; the records of one group are saved on the memory storage in consecutive areas.

EFFECT: data base management system productivity improvement.

9 cl, 20 dwg

Description

FIELD OF TECHNOLOGY

[0001] This technical solution relates to the field of computer engineering, in particular to a method for storing data, and can be used in a database management system (DBMS).

BACKGROUND

[0002] The patent for the invention US 8838593 B2 "Method and system for storing, organizing and processing data in a relational database" is known in the prior art, which proposes to group records by attribute values in order to increase compression of stored data and accelerate aggregate operations, however, the method does not oblige to place records of one group in physically adjacent areas of the storage device (memory), but introduces additional logical structures with a total size of up to 1% of all stored data.

[0003] It is known that specialized cluster objects are implemented in Oracle Database, which are used to jointly store several tables that are often joined together in SQL queries. Clustered tables in Oracle Database slow down the operations of adding, updating, and deleting rows, and require the creation of a cluster index.

[0004] The closest is the patent US 20090187591 A1 "Retrieving database records for aggregation without redundant database read operations", in which, in order to reduce I / O, and thereby improve the performance of the database management system, it is proposed to store records with the identical value of one of attributes in physically adjacent areas of the storage device. The described solution is inefficient for storing data in tasks involving a high rate of change of records, since modifying the value of the attribute by which the grouping is performed will lead to the obligatory rebuilding of the groups and the physical transfer of the modified record to the corresponding memory area. In addition, the method proposed in US 20090187591 A1 does not imply the presence in the same database of groups formed according to different criteria.

SUMMARY OF THE INVENTION

[0005] This technical solution is aimed at eliminating the disadvantages inherent in existing solutions for storing related data.

[0006] A method for organizing the storage of related data is claimed, in which records are combined into at least one group according to a similar characteristic; assign an identifier to each of the above groups and each record in this group; forming a link of the group identifier with the address of its location on the storage device in such a way that moving the group on the storage device, as well as between the storage devices, does not change the value of the group identifier; forming a connection of the record identifier with the address of its location within the group on the storage device in such a way that moving the record on the storage device within the group does not change the record identifier; store records of one group on a storage device in adjacent areas.

[0007] The technical result is to increase the performance of a database management system.

[0008] The technical result is achieved by reducing the input / output operations due to the storage in one group of records related by the domain logic and having similar characteristics, as well as the relative addressing of the record within the group, which reduces the frequency of changes of the referenced data structure.

[0009] In some embodiments, when combining records into groups, a similar sign is that the records are in a relational relation.

[00010] In some embodiments, when combining records into groups, a similar feature is the storage of versions of the same data.

[00011] In some embodiments, when combining records into groups, a similar feature is that they belong to the same horizontal section.

[00012] In some embodiments, when combining records into groups, a similar feature is the order in which the records are added.

[00013] In some embodiments, when combining records into groups, a similar feature is the identity of one or more attributes.

[00014] In some embodiments, the storage device is RAM, and / or ROM, and / or magnetic memory, and / or flash memory, and / or a magnetic disk, and / or an optical disk.

[00015] In some embodiments, the record identifier and the group are not a relationship attribute.

BRIEF DESCRIPTION OF THE DRAWINGS

[00016] The features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings, in which:

[00017] In FIG. Figure 1 shows an example of a relationship (120) with attributes (121) and several records (123-125) that are stored in memory (software).

[00018] In FIG. 2 shows a scheme for splitting records into groups. Records (222-226) are divided into two groups, each of which is stored in a separate block (212, 214) of a storage device (memory) (210). Records 222, 224-226 are grouped together because they have the identical attribute (230). Record 226 illustrates the storage of unfinished transaction data in the memory (210).

[00019] In FIG. Figure 3 shows the addressing scheme of records and groups from the referencing data structure, where 300 is the addressing structure, 310 is the address converter, 330 is the slot converter, 320 is the memory unit in which the groups are stored, and 324 and 322 are the records themselves.

[00020] FIG. 4 shows the results of measurements of the performance of random reading of pages.

[00021] In FIG. 5 shows the results of measurements of sequential page read performance.

[00022] FIG. Figure 6 shows an example implementation of the MATCHES table.

[00023] In FIG. 7 shows a layout for storing records in pages.

[00024] FIG. 8 shows the table “EMPLOYEES”.

[00025] In FIG. 9 shows the breakdown of records into groups “501” and “502” and their location on the pages of the memory with the approximate name “slow”.

[00026] In FIG. 10 shows the placement of the “501” group on the “slow” memory page, and the “502” group on the productive memory with the approximate name “fast”.

[00027] In FIG. 11 shows the breakdown of records into the following groups: “2016-02” and “2016-01”.

[00028] In FIG. 12 shows the contents of the data pages in the initial state (before transferring the entries of the “502” group to another memory).

[00029] In FIG. 13 shows the contents of the data pages after transferring the “502” group to the “fast” memory.

[00030] FIG. 14 shows an exemplary embodiment of group c1.

[00031] In FIG. 15 shows an embodiment of addressing records without an address converter (before transferring records);

[00032] In FIG. 16 shows an embodiment of addressing records without an address converter (after transferring records);

[00033] FIG. 17 shows an embodiment of addressing records using an address converter (before transferring records);

[00034] FIG. 18 shows an embodiment of addressing records using an address converter (after transferring records);

[00035] In FIG. 19 shows an embodiment in which, as a result of updating the data, record No. 2 has become longer;

[00036] In FIG. Figure 20 shows the actual movement of records within pages without changing the addressing slot number.

DETAILED DESCRIPTION OF THE INVENTION

[00037] Below will be described the concepts and definitions necessary for the detailed disclosure of the invention.

[00038] A record is a set of values of various types intended for storage in a memory.

[00039] A group is a set of records that is considered as an independent unit in read / write operations, involving physically integrated storage in a memory.

[00040] Physically adjacent areas are areas of memory that involve minimizing auxiliary positioning and addressing operations in one-time read / write operations from these areas. Depends on the memory implementation, for example: neighboring sectors on disk media, adjacent virtual / physical memory addresses.

[00041] A group identifier is a data set (in a particular case, a serial number) that uniquely identifies a group of records among the entire set of groups.

[00042] A record identifier is a data set (in a particular case, a serial number) that uniquely identifies a record within a group (but not necessarily among all records of all groups).

[00043] Address converter is a device that calculates the address of a block in a memory in which a group of records is stored by the group identifier.

[00044] A slot converter is a device that calculates a recording start address in a data block by a recording identifier.

[00045] Addressing structure - a structure that uniquely determines the current physical location of a record in memory, for example: LBA address in disk drives or a pointer to a virtual memory area.

[00046] According to the technical solution, a method for organizing the storage of related data includes the following steps:

[00047] grouping records into at least one group on a similar basis;

The sign by which the grouping of records occurs is not restrictive, since the criteria can be different and are largely determined directly by the data and subject area. For example, this criterion may be the location of records in a relational relation or the storage of versions of the same data or belonging to the same horizontal section. Consider several embodiments of the technical solution, which are not limiting.

Example 1

Let there be a database describing the holding of the hockey championship and containing the “MATCHI” relational relation (Fig. 6) with the attributes ID, TEAM HOST, TEAM GUESTS, DATE AND CARRY-OUT:

When creating this relationship, the characteristic of the controlled grouping was determined: “month and year of the match”, which predetermined the location of records corresponding to matches in February 2016 (ID = 1,2) to one group, and records corresponding to matches in January of the same year ( ID = 3,4) - to another. Each group is allocated a range of pages in the data file, let pages 0 to 1 be allocated for the group “2016-01”, and pages 2 to 3 for the group “2016-02”, then the layout for storing records in pages can be represented in this way as shown in FIG. 7.

Pages 1 and 3 are not used, but reserved to add new entries to the group to which they are assigned.

Example 2

Let there be a database of employees of the enterprise containing contact information of the personnel in relation to “EMPLOYEES” (Fig. 8) with the following attributes: ID, SURNAME, CABINET NUMBER.

When creating this relationship, the characteristic of the controlled grouping was determined: “Cabinet number”, in addition, the reading operations of the groups are calculated, the results of which determine the location of the group in a particular memory. Thus, the actual location of the group with the records is adaptive and is formed automatically as the method is used.

Let the following entries be added to the relation:

At the time of adding the record, the statistics of reading groups is not known. The formed groups with the names “502” and “501” are initially placed in pages 0-1 of the “slow” storage (Fig. 9).

Suppose that after some time of operation of the database it was possible to collect the following statistics of accesses to records:

Since the group is an independent unit in the read / write operations, the above statistics mean that there were 58 calls to the group with the conditional name “502” (the largest number of indicators by records ID = 2, ID = 3), and to the group with the conditional name “501” - 3 hits. The DBMS moves the “502” group to the more efficient “fast” memory (Fig. 10).

[00048] assign an identifier to each of the above groups and each record in this group;

Each group has a unique (within the framework of the relationship) identifier, which is necessary for addressing the group as a whole, and each record is a unique identifier within the group for addressing the record itself. Identifiers are not an attribute of a relationship.

Thus, for an addressing structure (for example, a B-tree), for a unique addressing of each record, a pair of values is required: the group identifier and the record identifier. Moreover, a pair of values does not require changes in the addressing structure in the following cases:

- the group is transferred entirely to another memory;

- the group is transferred entirely to other memory blocks;

- the records change the order and position inside the memory unit allocated to the corresponding group.

The correspondence between the actual position of the memory unit and the group identifier is determined by the address converter, and the recording position inside the memory unit is determined by the slot converter. The concepts of address converter are introduced to simplify the description of this technical solution. One skilled in the art will appreciate that the functions of an address and slot converter can be performed by an application or graphics processor, not limited to.

Example 1

Since, according to the conditions of the example, the group was created at the stage of creating / filling the relationship, the above records are initially divided into the following groups: “2016-02” and “2016-01” (Fig. 11). Let the group “2016-01” be assigned the identifier c1, and the group “2016-01” - c2. Each record will receive an identifier in the order of filling the pages: s1 - for the first record added, s2 - for the second, etc.

Example 2

Let the order of assigning identifiers be based on the order of adding records and groups to the database, then in the initial state (before transferring the group) the contents of the data pages can be arbitrarily presented in the manner shown in FIG. 12, and after transfer, as shown in FIG. 13.

The above illustration (Fig. 13) shows an acceptable behavior scenario: records from groups c1 and c2 have end-to-end numbering s1-s4, since this does not violate the condition of uniqueness of identifiers within each group.

[00049] form a link of the group identifier with the address of its location on the storage device so that moving the group on the storage device, as well as between storage devices does not change the value of the group identifier;

In an exemplary embodiment, the processor performs the functions of the address converter.

Example 1

The address converter receives a group identifier at the input and returns the data page number (let's say that the memory in this example is common for all groups):

Example 2

The address converter receives a group identifier at the input and returns the storage and page numbers with data.

It should be noted that a pair of record identifiers (for example c2: s2 - group “502”, record No. 2) does not change for the addressing structure (for example, B-tree) when transferring the whole group, as mentioned above.

[00050] linking the record identifier with the address of its location within the group on the storage device so that moving the record on the storage device within the group does not change the record identifier;

In an exemplary embodiment, the functions of the slot converter may be performed using a processor.

Example 1

Consider the group c1 (Fig. 14).

Example 2

Consider the group c1 (Fig. 12).

After moving the group (Fig. 13) for group c1:

It follows that a pair of record identifiers (for example c1: s4 - group “501”, record No. 4) does not change for the addressing structure (for example, B-tree) when transferring the group, as mentioned above.

[00051] Records may be subject to change during the operation of the database, the reason for this may be:

- modification of the record, after which it no longer meets the previous criteria for grouping and should be transferred to another group.

- expansion / truncation of group size. The group may run out of free space to add new records or the free space has been allocated too much and the storage is used inefficiently. In such cases, to optimize the group (and therefore the entries in it) can be moved inside the repository.

[00052] Groups may be subject to change during the operation of the database, the reason for this may be:

- change of grouping criteria, as a result of which all groups should be rebuilt.

- Transfer groups from one store to another. Such a need may arise if part of the relationship is transferred to the archive or if some of the records are placed on a more efficient medium, in cases of sharding, etc.

[00053] store records of one group on a storage device in adjacent areas.

Adjacent areas are areas of memory that involve minimizing auxiliary positioning and addressing operations in one-time read / write operations from these areas. In both examples considered, a HDD is assumed as a storage device, in this case sectors with adjacent addresses are assumed under adjacent areas, which allows full use of the forward reading of sectors implemented in modern drives. Measurements of the performance of the prototypes (Fig. 3-4) illustrate this.

[00054] The storage device may be implemented in the form of any type of non-volatile memory, non-volatile memory, or a combination thereof, for example, static random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory ( EPROM), programmable read-only memory (ROM), read-only memory (ROM), magnetic memory, fl br memory, a magnetic disk or optical disk.

[00055] The specified technical result is achieved as follows. When estimating the number of operations generated by moving a group of records as a whole, for cases with an address converter (or processor) and without it it turns out: let a group of three records move from page N to page M. Let the table index refer to all three records, and the record # 2 is also a structure in RAM (for example, a transaction is waiting to unlock). Without using an address converter, all addressing structures refer to a record using a pair of values - page number, slot number on the page. Similar solutions are used in some modern relational DBMSs: Oracle, Firebierd. In the example considered in FIG. 15 and FIG. 16, even in the case of a simple transfer of a page with three entries, it is necessary to update three sheets in the index B-tree and one structure in RAM (the changes are highlighted in red in Figure 16). In the case of using an address converter, as shown in FIG. 17 and FIG. 18, only the service data of the address converter itself is updated, i.e. once in one place (changes are highlighted in red in Figure 18). Consider the slot converter in FIG. 19. Let the record No. 2 become longer as a result of data updating, then it moves to the tail of the page. As seen in FIG. 20, the actual movement of the records inside the pages did not change the addressing slot number, the record changed its position on the page, but did not change its identifier.

[00056] Thus, the joint use of the address and slot converters allows, without changing the addressing structures, to move records within the repository. The address converter is entirely responsible for the freedom of movement of the group, and the slot converter is responsible for the freedom of movement of single entries in the group. The effect of sharing address and slot converters is used to store records of one group “next to each other” in physically adjacent areas.

[00057] Consider how the claimed technical solution differs from hash functions. According to D. Knut (The Art of Programming, Volume 3, § 6.4), hashing or the method of open memory suggests the existence of a function h (K) such that

[00058] In addition, a good hash function according to R. Floyd must satisfy the requirement of minimizing collisions. Those. in general

Address converter a () and slot converter s () do not satisfy condition (1). Let there be two identical entries

According to the proposed solution, they belong to one group (C-identifier of the group):

Within the group, the identifier of slot I must be unique, i.e.

Since a pair of values {C _R , I _R } is used to uniquely identify the record, then, as we see, (7) violates requirement (1). Those. a () and s () do not have the necessary properties of hash functions.

Moreover, (7) will always be true only if the whole group is located in one memory block, otherwise, in some situations, the same records are located in the same group, but in different blocks, and then the first equality in (7 ) will not be for such cases.

In addition, as indicated in the description of the groups, they can be moved inside the repository during the operation of the DBMS (for example, to expand if the group is full), i.e. at different times t _n and t _m expression (5) may be incorrect even for the same record:

In turn, it is possible to free slots in a group (for example, after deleting an entry) and then the slot can be used again later for another group entry, i.e.:

In conclusion, it should be noted that the appointment of the pair {C, I} for new records is not a function in its classical sense, but is carried out by a mechanism that implements a decision-making algorithm that takes into account:

- sign of grouping;

- the presence of conflicting signs of a grouping (a record falls under more than one condition) and the rules for resolving them (for example, use group priorities in case of conflict to determine the target group);

- fullness of the blocks in which the groups are located (not to allow the complete filling of blocks for the possibility of expanding records);

- fullness of slots inside the block (reuse reused identifiers);

- DBMS state (rebuild groups in the background during idle time, for example).

[00059] The present detailed description is made up of various non-restrictive and exhaustive embodiments. At the same time, it will be apparent to those skilled in the art that various replacements, modifications, or combinations of any of the embodiments disclosed herein (including in part) can be reproduced within the scope of this technical solution. Thus, it is understood and understood that the present description includes additional embodiments, the essence of which is not set forth here in an explicit form. Such embodiments may be obtained, for example, by combining, modifying, or transforming any actions, components, elements, properties, aspects, characteristics, limitations, etc., related to the embodiments presented herein and not being restrictive.

EXAMPLES OF IMPLEMENTATION

[00060] In the framework of the implementation of the proposed technical solution, an embodiment of the method was implemented and measurements were taken of the speed of reading pages in groups as well as without them. The operation of the DBMS LINTER v5.9-6.0 mechanisms without the essential features described in this technical solution was compared with the DBMS LINTER AT prototype, where the organization of storing related data was implemented.

[00061] The following measurement procedure was used:

1. The following values received by the hdparm utility are accepted as reference values of the read / write speed on the test hardware:

1. reading from the OS page cache: 13225 Mb / s;

2. disk reading (sequential): 147 Mb / s.

2. The tests were carried out in series of 5 starts with the same settings, the extreme values of the monitored parameters were discarded, and the arithmetic mean was taken from the remaining three.

3. Each test series was carried out on two types of format: Linter v5.9-6.0 (without groupings of related data) and Linter AT (with groups of related data) for the following values of parameter n (number of generated records): 50 thousand, 100 thousand , 200 thousand, 250 thousand, 500 thousand, 1 million, 5 million, 10 million, 15 million

4. The value of the parameter m (sample size) in all tests was taken equal to 50% of the value of n.

5. Tracked values: file generation time, file size, speed of reading records in sequential and random modes.

6. In each of the tests, the body size of the generated record in bytes was a random number from the range [8 ... 256], which was rounded to the smallest even. Taking into account the headers, the average value of the total recording size in all measurements lay in the range [133 ... 135] bytes.

7. The testing tools did not use or emulate the DBMS cache.

8. In all cases, the entries were placed on the pages as densely as possible - no place was reserved for expanding entries or for sharing new ones.

9. In all cases, a sign of grouping was the procedure for adding entries.

[00062] The results of comparing key measurement indicators for different formats are shown in FIG. 4 and in FIG. 5.

[00063] The implementation of the method is carried out by changing the addressing method of the record from the addressing structure (see Fig. 3) and the introduction of address and slot converters.

[00064] In the particular case, the functions of the address and slot converter can be performed using an application or graphics processor.

[00065] The addressing structure (300) contains the group identifier C _R (307) and the record identifier I _R (305) within the group, which (identifiers) are logical and do not correlate with the actual location of the record in the memory. A pair of values (C _R , I _R ) uniquely identifies a single entry R. To obtain the address P _{R of the} actual location of the R record in the memory, an address converter (310) is used to calculate the address P _{c of} block (320) in the memory in which (block) the group C _{R is located} . Directly in the block (320) itself, the correspondence (326, 327) of the record identifiers and the location of the record body (322) relative to the beginning of the block are stored. The calculation of this offset produces a slot converter (330), which calculates the address of the beginning of the recording relative to the beginning of the block containing it. With this addressing, moving and modifying a record within one group, as well as moving the entire group to a new memory unit, will not lead to a change in the values of the pair (C _R , I _R ).

[00066] Consider an example implementation of a slot converter in Linter-AT.

[00067] The data necessary to determine the start address of the record in the data block by the record identifier is stored directly in the data page, here is its format.

[00068] The data page contains packed records in accordance with the NSM (N-agu Storage Model) storage model. In addition to the header at the end of the page is a directory of slots (slot directory) of the page, which represents an array of slots. The size of each entry in the slot is 4 bytes, while:

[00069] 8-19 bits - SLOT_LENGTH size (actual) of the record;

[00070] 20-31 bits - OFFSET offset to write relative to the beginning of the page;

[00071] 7 bits - COLLECTION_FLAG sign that the slot contains a collection of records;

[00072] 3-6 bits - reserved;

[00073] The types of slots may be as follows:

[00074] SLOT_FREE (0 × 0) slot is free, OFFSET and SLOT LENGTH can be other than 0 and indicate a free area;

[00075] SLOT_ADDRESS - the slot contains only the address of the record, but not the record itself;

[00076] SLOT_NORMAL - the record corresponds to the group and is completely located in the slot;

[00077] SLOT_BIG - the slot describes a "large" record;

[00078] SLOT_RELOCATED - the record contains the transferred;

[00079] SLOT_FRAGMENT - the slot contains a fragment of a "long" record without titles;

[00080] SLOT_DELETED - the slot is marked for deletion.

[00081] Entries are placed sequentially from the heading to the end of the page, slots from the end of the page to the heading. The space between the end of the last record and the first slot is available for placing new records. When deleting a record, the value of the corresponding slot is set to 0 × 00. When moving the recording body within one page, the slot number does not change. The order of entries on the page may not coincide with the order of referring slots.

[00082] Consider an example implementation of an address converter in Linter-AT.

[00083] The data necessary for calculating the address of the block in the memory in which the group of records is stored by the group identifier is stored in the data file in the data pages of the address converter. The pages of the address converter contain ordered records that determine the location of groups in the database file pages and the actual occupancy of the slots in the corresponding group. The recording format is independent of the isolation model.

[00084] FREE_SPACE_FACTOR contains the estimated value of free space to accommodate a new record, where 0 × 0 - free, 0 × FF - fully occupied;

[00085] PAGENUM - page number in the data file in which the group is located;

[00086] FNUM - file number;

[00087] BITMAP is a bitmask of slots in a group (necessary to know in advance about slot occupancy).

[00088] Consider the difference between the group identifiers and the entries of the claimed technical solution from the physical ROWID in Oracle DBMS.

[00089] The physical ROWIDs (https://docs.oracle.com/cd/B28359_01/server.111/b28318/datatype.htm#i6732) describe the current position of the row. for example

[00095] Any movement of a record to another block will result in a change in the physical ROWID.

[00096] In the claimed technical solution, the identifiers are logical and the group can "move" to another memory altogether without changing the identifier.

[00097] Consider the difference between group and record identifiers from logical ROWID in Oracle DBMS.

[00098] First, the logical ROWIDs are obtained based on the value of the indexed attribute, and changing it will cause the logical ROWID to change. In the claimed technical solution there is no such binding of the identifier to the value of the record or its attribute.

[00099] Secondly, the logical ROWID contains an unidentifiable component, the so-called physical guess, which actually indicates the last location of the record, but it may become outdated when moving the record and the read operation may fail, in which case the record is searched.

[000100] Third, the ROWID does not display the “proximity” of the location of the records in the memory in any way. In the claimed solution, the identifiers of two records of the same group will have the same value C _R , on the basis of which it can immediately be determined that these records are guaranteed to be stored “next to” on the memory.

Claims (14)

1. A method of organizing the storage of related data, in which: - combine records in at least one group on a similar basis; - Assign an identifier to each of the above groups and each record in this group; - form a relationship between the group identifier and the address of its location on the storage device in such a way that moving the group on the storage device, as well as between the storage devices, does not change the value of the group identifier; - linking the record identifier with the address of its location within the group on the storage device so that moving the record on the storage device within the group does not change the record identifier; - save the records of one group on the storage device in adjacent areas. 2. The method according to p. 1, in which when combining records in groups, a similar sign is the finding of records in a relational relation. 3. The method according to claim 1, in which when combining records into groups, a similar feature is the storage of versions of the same data. 4. The method according to p. 1, in which when combining records in groups, a similar sign is the belonging to one horizontal section. 5. The method according to claim 1, in which when combining records into groups, a similar feature is the order of adding records. 6. The method according to claim 1, in which when combining records into groups, a similar feature is the identity of one or more attributes. 7. The method of claim 1, wherein the storage device is RAM, and / or ROM, and / or magnetic memory, and / or flash memory, and / or a magnetic disk, and / or an optical disk. 8. The method according to claim 1, in which the identifier of the record and group are not an attribute of the relationship. 9. The method according to claim 1, in which the records are moved in the case of a modification of the records or changes in the grouping criteria.

Images