CN111046034A

CN111046034A - Method and system for managing memory data and maintaining data in memory

Info

Publication number: CN111046034A
Application number: CN201910972980.4A
Authority: CN
Inventors: 邓龙; 王太泽; 黄亚建; 范晓亮; 刘永超
Original assignee: 4Paradigm Beijing Technology Co Ltd
Current assignee: 4Paradigm Beijing Technology Co Ltd
Priority date: 2018-10-12
Filing date: 2018-10-12
Publication date: 2020-04-21
Anticipated expiration: 2038-10-12
Also published as: WO2020073854A1; CN111046034B; CN109299100A; CN109299100B

Abstract

Methods and systems for managing memory data and maintaining data in memory are provided. The method for managing the memory data comprises the following steps: setting a plurality of fragment groups, wherein each fragment group comprises at least one fragment, and all the fragments in each fragment group correspond to a uniform index field and a uniform sequencing field; and respectively constructing a corresponding first-level jump table and a corresponding second-level jump table for each fragment, wherein the first-level jump table corresponding to each fragment is set as a node which is used for storing the key as the value of the data record related to the index field and the value as a pointer or an object indicating the second-level jump table, and the second-level jump table corresponding to each fragment is set as a node which is used for storing the key as the value of the data record related to the sorting field and the value as a pointer of a storage space indicating the value of at least one attribute field for storing the data record. According to the method and the system, the efficiency of data writing and data query can be improved.

Description

Method and system for managing memory data and maintaining data in memory

The application is a divisional application of patent applications with the application date of 2018, 10 and 12 months and the application number of 201811187043.X, and the title of the patent applications is 'method and system for managing memory data and maintaining data in memory'.

Technical Field

The present disclosure relates generally to the field of memory data management and maintenance, and more particularly, to a method and system for managing memory data and a method and system for maintaining data in memory.

Background

Existing databases include relational and non-relational databases. Relational databases such as MySQL and SQL Server are mainly used to manage and maintain relational data. Non-relational databases such as Redis and MongoDB are primarily used to manage and maintain non-relational data. Relational data means data based on a Relational Model (RM for short). Non-relational data means data that is not based on a relational model.

In order to process time series data, a time series Database (TSDB for short) such as infiluxdb is proposed. In order to process data in the memory, a memory database such as VoltDB is proposed.

However, in a specific scenario where it is necessary to rapidly process data and simultaneously perform a large number of data writing tasks and/or data query tasks, the conventional database including the above-listed databases has a problem that it takes a long time to read/write data, and also has a problem that the number of data writing tasks and/or data query tasks that can be simultaneously performed per unit time is small. In summary, the prior art has the defects of low data writing efficiency and low data query efficiency.

Disclosure of Invention

Exemplary embodiments of the present disclosure aim to overcome the disadvantages of the prior art that data writing efficiency is low and data query efficiency is low.

According to an exemplary embodiment of the present disclosure, there is provided a method of managing memory data, including: setting a plurality of fragment groups, wherein each fragment group comprises at least one fragment, all fragments in each fragment group correspond to a uniform index field and a uniform sequencing field, the index fields of different fragment groups are different, and the sequencing fields of different fragment groups are the same or different; and respectively constructing a corresponding first-level jump table and a corresponding second-level jump table for each fragment, wherein the first-level jump table corresponding to each fragment is set as a node for storing a value corresponding to a key by using a value of an index field of each fragment in a data record and a pointer or an object indicating the second-level jump table, and the second-level jump table corresponding to each fragment is set as a node for storing a value corresponding to the key by using a value of a sorting field of each fragment in the data record as the key and a pointer indicating a storage space for storing a value of at least one attribute field of the data record.

Optionally, the storage space is shared by the second-level skip tables corresponding to all the slice groups and stores values of all the attribute fields of the data record, or the same storage space is shared by the second-level skip tables corresponding to the same slice group and stores values of all the attribute fields of the data record except the index field and the sorting field corresponding to the same slice group.

According to another exemplary embodiment of the present disclosure, there is provided a method of maintaining data in a memory, including: for each of a plurality of slice groups, determining a corresponding slice according to a value of an index field of each slice group of a data record to be inserted, wherein each slice group comprises at least one slice, all slices in each slice group correspond to a uniform index field and a uniform sorting field, the index fields of different slice groups are different, the sorting fields of different slice groups are the same or different, each slice corresponds to a first-level skip list, and the first-level skip list is used for storing a node which takes the value of the index field of the corresponding slice in the data record as a key and takes a pointer or an object indicating a second-level skip list as a value corresponding to the key; searching a node which takes the value of the index field of each slice group of the data record to be inserted as a key word from a first-level skip table corresponding to the determined slice; and adding a node which takes the value of the sorting field of the data record to be inserted, which is related to each slice group, as a key and takes a pointer of a storage space for storing the value of at least one attribute field of the data record to be inserted as a value corresponding to the key in a second-level skip table indicated by a pointer or an object in the searched node under the condition that the node which takes the value of the index field of the data record to be inserted, which is related to each slice group, as a key is searched.

Optionally, the step of determining the corresponding segment includes: calculating a hash value corresponding to the value of the index field of each slice group of the data record to be inserted; obtaining a remainder obtained by dividing the calculated hash value by the total number of the fragments of each fragment group; and determining the segment corresponding to the obtained remainder as the corresponding segment.

Optionally, each slice stores a pointer or an object indicating the corresponding first-level skip list.

Optionally, the method for maintaining data in the memory further includes: and under the condition that a node which takes the value of the index field of each slice group of the data record to be inserted as a key cannot be found, creating a second-level skip list, creating a node which takes the value of the index field of each slice group of the data record to be inserted as the key and takes a pointer or an object indicating the created second-level skip list as a value corresponding to the key in the first-level skip list, and adding a node which takes the value of the sorting field of each slice group of the data record to be inserted as the key and takes a pointer indicating a storage space for storing the value of at least one attribute field of the data record to be inserted as a value corresponding to the key in the created second-level skip list.

Optionally, a storage space for storing a value of at least one attribute field of the data record to be inserted stores a character string obtained by one of the following ways: merging the values of the at least one attribute field according to a preset character string merging rule, serializing the values of the at least one attribute field according to a preset JSON format, serializing the values of the at least one attribute field according to a preset protocol buffer format, and serializing the values of the at least one attribute field according to a preset Schema format.

Optionally, the method for maintaining data in the memory further includes: determining the sharded groups with the same index fields as the index fields of the data records to be inquired in the plurality of sharded groups; in the determined fragment group, determining a corresponding fragment according to the value of the index field of the data record to be queried; searching a node which takes the value of the index field of the data record to be inquired as a key word from a first-stage skip list corresponding to the determined fragment; searching pointers of keywords in the nodes in the value range of the sequencing fields of the data records to be searched from the pointers in the searched nodes or the second-level skip list indicated by the objects; and taking out the value of at least one attribute field of the data record to be queried from the storage space indicated by the queried pointer.

Optionally, the step of retrieving, from the storage space indicated by the queried pointer, a value of at least one attribute field of the data record to be queried includes: taking out the value of the at least one attribute field of the data record to be queried by one of the following methods: splitting the value of the at least one attribute field according to a preset character string splitting rule, deserializing the value of the at least one attribute field according to a preset JSON format, deserializing the value of the at least one attribute field according to a preset protocol buffer format, and deserializing the value of the at least one attribute field according to a predefined Schema format.

Optionally, the step of determining the corresponding segment includes: calculating a hash value corresponding to the value of the index field of the data record to be queried; obtaining a remainder obtained by dividing the calculated hash value by the determined total number of the fragments of the fragment group; and determining the segment corresponding to the obtained remainder as the corresponding segment.

Optionally, the data record to be inserted and/or the data record to be queried is a time-series data record, and the sorting fields are both timestamps.

Optionally, the value range specifies a start value and an end value of the timestamp or specifies an end value of the timestamp.

Optionally, the step of adding a node in the second-level skip list includes: and adding the nodes according to the time indicated by the value of the timestamp, so that the nodes in the second-stage skip list are arranged according to the sequence of the time from near to far.

Optionally, the method for maintaining data in the memory further includes: setting a node quantity threshold corresponding to a second-level skip list, wherein the step of querying the pointer of the keyword in the node in the value range of the sequencing field of the data record to be queried from the second-level skip list indicated by the pointer or the object in the searched node comprises the following steps: and taking out the pointers stored in each node of the preset number of nodes of the keywords in the value range from near to far from the second-level jump table indicated by the pointers or objects in the searched nodes, wherein the preset number does not exceed the threshold value of the number of nodes.

Optionally, the method for maintaining data in the memory further includes: setting a node quantity threshold corresponding to the second-stage skip list; respectively traversing a first-stage skip list and a second-stage skip list corresponding to each slice group in a preset period; when the number of traversed nodes in the second-level hop table corresponding to each slice group exceeds a node number threshold, deleting all nodes in the second-level hop table, which are arranged in the order after the node corresponding to the node number threshold, according to the arrangement order of the nodes in the second-level hop table, and deleting the storage space indicated by the pointer stored in the deleted nodes, wherein the arrangement order is the arrangement order of time from near to far.

Optionally, the method for maintaining data in the memory further includes: setting the length of an expiration period; traversing the first-stage skip list and the second-stage skip list corresponding to each fragment in a preset period, and integrally deleting nodes behind the nodes in the second-stage skip list according to the arrangement sequence of the nodes in the second-stage skip list from near to far by positioning the nodes with the time stamps in the second-stage skip list reaching the expiration period length.

According to another exemplary embodiment of the present disclosure, there is provided a system for managing memory data, including: the system comprises a fragment management device and a fragment processing device, wherein the fragment management device is used for setting a plurality of fragment groups, each fragment group comprises at least one fragment, all the fragments in each fragment group correspond to a uniform index field and a uniform sequencing field, the index fields of different fragment groups are different, and the sequencing fields of different fragment groups are the same or different; the skip list management device is used for respectively constructing a corresponding first-level skip list and a corresponding second-level skip list for each fragment, wherein the first-level skip list corresponding to each fragment is set to be used for storing nodes which take the value of the index field of each fragment in the data record as a key and take a pointer or an object indicating the second-level skip list as a value corresponding to the key, and the second-level skip list corresponding to each fragment is set to be used for storing nodes which take the value of the sorting field of each fragment in the data record as a key and take a pointer of a storage space for storing the value of at least one attribute field of the data record as a value corresponding to the key.

According to another exemplary embodiment of the present disclosure, there is provided a system for maintaining data in a memory, including: the system comprises a fragment determining device and a fragment determining device, wherein the fragment determining device is used for determining corresponding fragments according to values of index fields of each fragment group of a data record to be inserted, each fragment group comprises at least one fragment, all the fragments in each fragment group correspond to uniform index fields and sequencing fields, the index fields of different fragment groups are different, the sequencing fields of different fragment groups are the same or different, each fragment corresponds to a first-level skip list, and the first-level skip list is used for storing nodes which take the values of the index fields of the corresponding fragments in the data record as keywords and take pointers or objects indicating the second-level skip list as values corresponding to the keywords; a node searching device, configured to search, from a first-level skip table corresponding to the determined slice, a node using a value of the index field of each slice group of the data record to be inserted as a key; and a node management device, configured to, when a node is found that uses the value of the index field of each slice group of the to-be-inserted data record as a key, add, to a second-level skip list indicated by a pointer or an object in the found node, a node that uses the value of the sorting field of the to-be-inserted data record that is related to each slice group as a key and uses a pointer indicating a storage space for storing the value of at least one attribute field of the to-be-inserted data record as a value corresponding to the key.

Optionally, the fragmentation determination device calculates a hash value corresponding to a value of an index field of each of the groups of fragments of the data record to be inserted, obtains a remainder obtained by dividing the calculated hash value by a total number of fragments of each of the groups of fragments, and determines a fragment corresponding to the obtained remainder as a corresponding fragment.

Optionally, each slice stores therein a pointer or an object indicating the corresponding first-level skip table.

Optionally, the system for maintaining data in the memory further includes: the node management device creates a second-level skip list when a node with the value of the index field of each slice group of the data record to be inserted as a key cannot be found, creates a node with the value of the index field of each slice group of the data record to be inserted as a key and a pointer or an object indicating the created second-level skip list as a value corresponding to the key in the first-level skip list, and adds a node with the value of the sorting field of each slice group of the data record to be inserted as a key and a pointer of a storage space for storing the value of at least one attribute field of the data record to be inserted as a value corresponding to the key in the created second-level skip list.

Optionally, the system for maintaining data in the memory further includes: slice group determining means for determining a slice group of the plurality of slice groups having an index field identical to an index field of a data record to be queried, and data obtaining means for obtaining the data record from the plurality of slice groups, determining a corresponding fragment according to the value of the index field of the data record to be queried, searching a node which takes the value of the index field of the data record to be queried as a keyword from a first-level jump table corresponding to the determined fragment by a node searching device, querying a pointer of the keyword in a node in a value range of a sorting field of the data record to be queried from a pointer in the searched node or a second-level jump table indicated by an object by a data acquisition device, and taking out the value of at least one attribute field of the data record to be queried from the storage space indicated by the queried pointer.

Optionally, the data obtaining device takes out, according to the queried pointer, a value of the at least one attribute field of the data record to be queried in one of the following manners: splitting the value of the at least one attribute field according to a preset character string splitting rule, deserializing the value of the at least one attribute field according to a preset JSON format, deserializing the value of the at least one attribute field according to a preset protocol buffer format, and deserializing the value of the at least one attribute field according to a predefined Schema format.

Optionally, the fragmentation determination device calculates a hash value corresponding to a value of an index field of the data record to be queried, obtains a remainder obtained by dividing the calculated hash value by the total number of the fragments of the determined fragmentation group, and determines a fragment corresponding to the obtained remainder as a corresponding fragment.

Optionally, the node management apparatus adds the nodes according to the time indicated by the value of the timestamp, so that the nodes in the second-level skip list are arranged in the order of time from near to far.

Optionally, the system for maintaining data in the memory further includes: and the node number threshold setting device is used for setting a node number threshold corresponding to the second-level skip list, wherein the data acquisition device takes out the pointers stored in each node of the preset number of nodes in the value range of the keywords from the pointer in the searched node or the second-level skip list indicated by the object according to the sequence from near to far, and the preset number does not exceed the node number threshold.

Optionally, the system for maintaining data in the memory further includes: the device comprises a node quantity threshold setting device, a skip list traversing device and a data deleting device, wherein the node quantity threshold setting device sets a node quantity threshold corresponding to a second-level skip list, the skip list traversing device respectively traverses a first-level skip list and a second-level skip list corresponding to each slice group in a preset period, when the number of traversed nodes in the second-level skip list corresponding to each slice group exceeds the node quantity threshold, the node deleting device deletes all nodes in the second-level skip list, which are arranged in sequence after the nodes corresponding to the node quantity threshold, according to the arrangement sequence of the nodes in the second-level skip list, and deletes a storage space indicated by pointers stored in the deleted nodes, wherein the arrangement sequence is an arrangement sequence from near to far in time.

Optionally, the system for maintaining data in the memory further includes: the data deleting device integrally deletes nodes of an arrangement sequence after a node of which the value of a timestamp in the second-stage skip list reaches the expiration deadline length, wherein the arrangement sequence is an arrangement sequence of time from near to far.

According to another exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein the instructions, when executed by at least one computing device, cause the at least one computing device to perform the method as described above.

According to another exemplary embodiment of the present disclosure, a system is provided comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the method as described above.

In the method and system according to the exemplary embodiments of the present disclosure, a plurality of slice groups may be set, where all slices in each slice group correspond to a unified index field and a unified sorting field, and a corresponding first-level skip table and a second-level skip table may be respectively constructed for each slice, such that the first-level skip table corresponding to each slice stores a node having a value regarding the index field in a data record as a key and a pointer or an object indicating the second-level skip table as a value corresponding to the key, and the second-level skip table corresponding to each slice stores a node having a value regarding the sorting field in the data record as a key and a pointer indicating a storage space of a value of at least one attribute field of the data record as a value corresponding to the key.

In this way, when data writing is performed, the multiple fragment groups can be processed in parallel, the corresponding fragments and the corresponding first-level skip lists can be quickly located based on the index fields, and the values of the attribute fields can be stored by using the shared storage space. In this case, the processing time can be reduced and the processing efficiency can be improved. When the query is carried out, the corresponding fragment and the first-level skip list and the second-level skip list corresponding to the fragment can be quickly positioned in the corresponding fragment group based on the value of the index field of the data record to be queried, and the storage space for storing the value of the attribute field can be quickly positioned based on the sorting field, so that the processing time can be reduced, and the processing efficiency can be improved. When data writing is executed, a plurality of fragment groups can be processed in parallel, and data writing can also be performed on fragments in the same fragment group in parallel; when data queries are performed, data queries may be performed in parallel on different slices from the same slice group or different slice groups. This increases the number of data write tasks and/or data search tasks that can be executed simultaneously per unit time. In addition, any attribute field of the data record can be set as an index field, so that the data record can be flexibly and conveniently written and inquired according to the required attribute field.

Additional aspects and/or advantages of the present general inventive concept will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the general inventive concept.

Drawings

The above and other objects and features of the exemplary embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings which illustrate exemplary embodiments, wherein:

FIG. 1 illustrates a flow chart of a method of managing memory data according to an exemplary embodiment of the present disclosure;

FIG. 2 shows a schematic view of a tile according to an exemplary embodiment of the present disclosure;

FIG. 3 illustrates a flow chart of a method of maintaining data in memory according to an exemplary embodiment of the present disclosure;

FIG. 4 illustrates a schematic diagram of operations to insert a data record, according to an example embodiment of the present disclosure;

FIG. 5 illustrates a flowchart of an operation for querying data in memory according to an exemplary embodiment of the present disclosure;

FIG. 6 illustrates a block diagram of a system for managing memory data in accordance with an exemplary embodiment of the present disclosure; and

FIG. 7 illustrates a block diagram of a system for maintaining data in memory in accordance with an exemplary embodiment of the present disclosure.

Detailed Description

Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described below in order to explain the present disclosure by referring to the figures.

Fig. 1 illustrates a flowchart of a method of managing memory data according to an exemplary embodiment of the present disclosure. As shown in fig. 1, a method of managing memory data according to an exemplary embodiment of the present disclosure may include step S110 and step S120. In step S110, a plurality of slice groups are set, where each slice group includes at least one slice, all slices in each slice group correspond to a uniform index field and an ordering field, the index fields of different slice groups are different, and the ordering fields of different slice groups are the same or different. In step S120, a corresponding first-level skip table and a second-level skip table are respectively constructed for each fragment, where the first-level skip table corresponding to each fragment is set to be used for storing a node that takes a value of an index field of each fragment in a data record as a key and a pointer or an object indicating the second-level skip table as a value corresponding to the key, and the second-level skip table corresponding to each fragment is set to be used for storing a node that takes a value of a sorting field of each fragment in the data record as a key and a pointer indicating a storage space for storing a value of at least one attribute field of the data record as a value corresponding to the key. As an example, the index field and the ordering field of each slice group are different.

As an example, each slice has stored therein a pointer or object indicating the corresponding first level skip table.

As an example, the second-level skip tables corresponding to all slice groups share the storage space and the storage space stores values of all attribute fields of the data record, which may correspond to a case where the ordering fields of different slice groups are the same. As an example, the second-level skip tables corresponding to the same slice group share the same storage space and the same storage space stores values of all attribute fields of the data record except for the index field and the ordering field corresponding to the same slice group, which may correspond to a case where the ordering fields of different slice groups are different.

In an exemplary embodiment of the present disclosure, a group of slices may be a collection of slices, and one group of slices may include one or more slices; the index field of the slice group refers to the index field corresponding to the slice group, and the sequencing field of the slice group refers to the sequencing field corresponding to the slice group; the data record may have one or more attribute fields, the index field may be an attribute field of the data record, and the sort field may also be an attribute field of the data record. For example, the index field is a card number or a Merchant Category Code (MCC), and the sorting field is a timestamp or an age.

In an exemplary embodiment of the present disclosure, the first level jump table and the second level jump table are skiplists, also referred to as jump tables. The shards may be associated with the first level skip list through pointers or objects. Accordingly, the first level skip list corresponding to a tile may be located by a pointer or object stored in the tile. The first level jump table may be associated with the second level jump table by a pointer or object. Accordingly, the second level jump table corresponding to the first level jump table may be located by a pointer or object stored in the first level jump table. The objects herein are similar to those defined in Object Oriented programming (OO) technology.

As an example, when creating a data table, an index field and/or an ordering field of a slice group may be defined, e.g. which field is defined as index field and which field is defined as ordering field. Alternatively, the order of the index field and/or the sort field may be defined. For example, a first index field is defined as a card number and a second index field is defined as a merchant category code. As another example, a first ordering field is defined as a timestamp and a second index field is defined as an age.

As an example, when creating a data table, it may be specified whether the ordering fields of different slice groups are the same or different. For example, the ordering fields of different slice groups may be designated as timestamps.

In an exemplary embodiment of the present disclosure, the data table may include: the system comprises a plurality of slice groups, a first-stage skip list and a second-stage skip list, wherein the first-stage skip list and the second-stage skip list correspond to each slice in the plurality of slice groups respectively. The data table may or may not also include storage space for storing values of attribute fields of the data records. Optionally, the storage space for storing the values of the attribute fields of the data records is independent of the data table.

As an example, the data table may be created through an interface create (table _ name, ttl, ttl-type, key _ 1: type _ 1: index, key _ 2: type _ 2: index, … …, key _ N: type _ N: index, value: type). The user can create a data table in the memory through the interface create. Accordingly, when a request for a user to create a data table is received through the interface create, step S110 and step S120 may be performed. In other words, information required to perform steps S110 and S120 may be received through the interface create, where "table _ name" represents a name of the data table; ttl (time to live) represents a survival threshold of data stored in the data table, which has different value ranges according to different survival expiration types (the type is indicated by a ttl-type field), for example, when the ttl-type field indicates that expired data is deleted according to a quantity threshold limit, the value of ttl may be the maximum reserved quantity of the data record, for example, when tll is 100, the data table may store 100 pieces of data (for example, the latest 100 pieces of data), and the data beyond the part may be deleted; for another example, when the ttl-type field indicates that expired data is deleted according to the time threshold limit, the value of ttl may be the maximum retention period of the data record, for example, when tll is 3 days, it indicates that the data table may store the data of the last three days, and the data before this is deleted. "key _ i: type _ i: index' represents the name of an index field (which can be called as the ith index field) corresponding to the ith slice group and the data type (1 ≦ i ≦ N) of the index field, N is a natural number, where key _ i represents the name of the ith index field, type _ i represents the data type of the ith index field, and "index" represents that key _ i is an index field, which is only an index field identifier and can be replaced by any character or character combination representing that key _ i is an index field; "value: type "represents a name of the attribute field of the at least one non-index field as a whole (wherein a value of the at least one attribute field may be encoded as a whole) and a corresponding data type. In the present example, the sort field is a preset attribute field, for example, a time stamp, and thus, the sort field may not be set in the interface. Exemplary of the present disclosure are data types in examples that may include: string type, floating point type, etc.

As an example, the data table may be created through an interface create (table _ name, ttl, ttl-type, key _ 1: type _ 1: index, key _ 2: type _ 2: index, … …, key _ N: type _ N: index, value _ 1: value _ type _1, … …, value _ M: value _ type _ M). The parenthesis after the interface create is the parameters of the interface. Except for "value _ 1: value _ type _1, … …, value _ M: the setting of parameters other than value _ type _ M "is similar to that of the corresponding parameters in the above example of interface create, and is not described here again. For "value _ 1: value _ type _1, … …, value _ M: value _ type _ M ", value _ j: value _ type _ j represents the name and type of the attribute field of the jth non-index field, wherein j is more than or equal to 1 and less than or equal to M, and M is a natural number.

As an example, in any of the above examples of interface create, a parameter "field _ 1: field _ type _ 1: order, field _ 2: field _ type _ 2: order, … …, field _ N: field _ type _ N: order ", where field _ i represents the name (1 ≦ i ≦ N) of the sorting field (which may be referred to as the ith sorting field) corresponding to the ith slice group, N is a natural number, field _ type _ i represents the data type of the ith sorting field, and" order "represents that field _ i is a sorting field, which is only a sorting field identifier, and may be replaced with any character or character combination that represents that field _ i is a sorting field.

The above example of the interface create is only used to describe the concept of the present disclosure, and is not used to limit the scope of the present disclosure, and other interfaces for creating the data table are also possible, for example, one or more parameters in the above interfaces may be omitted, and one or more parameters may be added to the above interfaces.

Data records in exemplary embodiments of the present disclosure may be understood with reference to table 1. Table 1 shows data records relating to banking transactions according to an exemplary embodiment of the present disclosure. The data records shown in Table 1 may include the following attribute fields: card number, time stamp, transaction amount, transaction location, POS (Point of Sale, POS for short) number. Table 1 includes 3 data records. The card number may be used as an index field and the timestamp may be used as an ordering field.

TABLE 1

Card number	Time stamp	Amount of transaction	Transaction location	POS number
					6222XXXX01	2018052814520505	100	Kyoto road	10xxx
6222XXXX01	2018052815520505	50	Beijing-xi-two-flag xx shop	20xxx
					6222XXXX02	2018052811520505	1000	South Beijing Drum building area xxx	30xxx

Fig. 2 shows a schematic view of a tile according to an exemplary embodiment of the present disclosure. As shown in fig. 2, slices 0 to n are n +1 slices, where n is a natural number greater than 2. Each of the slices corresponds to a first level skip list. The first-level skip list corresponding to the fragment 0 includes nodes 11 to 1m, where m is a natural number. Each slice may have stored therein a pointer or object indicating the corresponding first level jump table to facilitate location to the first level jump table corresponding to the slice. Each node in the first level hop may correspond to a second level hop table. The second-level hop table corresponding to the node 11 includes nodes 41 to 4k, the second-level hop table corresponding to the node 12 includes nodes 31 to 3j, and the second-level hop table corresponding to the node 1m includes nodes 21 to 2i, where i, j, and k are natural numbers. Each node of the first-level hop table may have stored therein a pointer or object indicating the corresponding second-level hop table to facilitate location to the second-level hop table corresponding to the node of the first-level hop table.

A key-value pair may be set in a node of the skip list. Specifically, for a node of the first-level skip list, a value of an index field of the data record may be set to a key (key), and a pointer or an object indicating the second-level skip list may be set to a value (value) corresponding to the key. For a node in the second-level skip list, a value of the sorting field of the data record may be set as a key, and a value corresponding to the key is a pointer indicating a storage space in which a value of at least one attribute field of the data record is stored.

As an example, the value of the at least one attribute field includes a value of an index field and/or a value of a sorting field of the data record, or the value of the at least one attribute field includes neither a value of an index field of the data record nor a value of a sorting field of the data record.

In addition, a pointer or object may be stored in a node of the first level hop table that indicates another node in the first level hop table. A pointer or object may also be stored in a node of the second level hop table that indicates another node in the second level hop table. For the first-level hop table or the second-level hop table, when a node already exists in the hop table, an object or a pointer indicating a node belonging to the same hop table as the node is stored in each node except for the tail node, so that the hop table forms a chain structure. When a third node is inserted between the first node and the second node, it is necessary to change a pointer or an object indicating the second node in the first node to indicate the third node, and indicate the second node with the pointer or the object in the third node.

Fig. 3 illustrates a flow chart of a method of maintaining data in a memory according to an exemplary embodiment of the present disclosure. As shown in fig. 3, a method of maintaining data in a memory according to an exemplary embodiment of the present disclosure may include step S210, step S220, and step S230.

In step S210, for each slice group of the multiple slice groups, a corresponding slice is determined according to a value of an index field of each slice group of a data record to be inserted, where each slice group includes at least one slice, all slices in each slice group correspond to a uniform index field and a sorting field, the index fields of different slice groups are different, and the sorting fields of different slice groups are the same or different, each slice corresponds to a first-level skip table, and the first-level skip table is used to store a node that takes the value of the index field of the corresponding slice in the data record as a key and takes a pointer or an object indicating the second-level skip table as a value corresponding to the key. As an example, the index field and the ordering field of each slice group are different.

As an example, the step of determining the corresponding slice comprises: calculating a hash value corresponding to the value of the index field of each slice group of the data record to be inserted; obtaining a remainder obtained by dividing the calculated hash value by the total number of the fragments of each fragment group; and determining the segment corresponding to the obtained remainder as the corresponding segment. The value of the index field may be computed using a hash function to obtain a hash value. For example, the hash function used may be the hash function murmurmurhash proposed by austinnappleby. Of course, the present disclosure is not limited to the hash function used, and other hash functions may be used for the calculation of the hash value. In the case that one slice group includes slices 0 to n, for example, each slice shown in fig. 2 belongs to the same slice group, if the remainder is 0, then slice 0 in the one slice group corresponds to the data record to be inserted; if the remainder is h (0< h ≦ n), the slice h in the one slice group corresponds to the data record to be inserted.

In step S220, a node using the value of the index field of each slice group of the data record to be inserted as a key is searched from the first-level skip table corresponding to the determined slice.

In step S230, in a case that a node is found which uses the value of the index field of each slice group of the to-be-inserted data record as a key, a node which uses the value of the sorting field of the to-be-inserted data record as a key and uses a pointer indicating a storage space for storing the value of at least one attribute field of the to-be-inserted data record as a value corresponding to the key is added to a second-level skip list indicated by a pointer or an object in the found node.

As an example, the method of maintaining data in a memory according to an exemplary embodiment of the present disclosure further includes: and under the condition that a node which takes the value of the index field of each slice group of the data record to be inserted as a key cannot be found, creating a second-level skip list, creating a node which takes the value of the index field of each slice group of the data record to be inserted as the key and takes a pointer or an object indicating the created second-level skip list as a value corresponding to the key in the first-level skip list, and adding a node which takes the value of the sorting field of each slice group of the data record to be inserted as the key and takes a pointer indicating a storage space for storing the value of at least one attribute field of the data record to be inserted as a value corresponding to the key in the created second-level skip list.

As an example, when the insertion of the data record is performed, each slice group is processed, and the processing of the slice group includes processing of a slice corresponding to the slice group and a first-level skip table and a second-level skip table corresponding to the slice. Specifically, when the insertion of the data record is performed, a value regarding each index field and a value of at least one attribute field of the data record may be received as the data record, and a value of an ordering field of the data record may also be received. The above-mentioned values may be received from a user input, and when the user input is received, the processing may be performed according to the above-described steps S210, S220, and S230.

As an example, the data record may be inserted through the interface put (table _ name, ts, key _1, key _2, … …, key _ N, value), which may correspond to a case where the ordering fields of different slice groups are the same. Before inserting the data record, it may be pre-specified that the ordering fields of different slice groups are the same, e.g. the ordering fields of different slice groups are the same timestamps. Here, "table _ name" represents the name of the data table into which the data record is to be inserted, "ts" represents the value of the sorting field, key _1, key _2, … …, and key _ N represents the value of each index field of the data record to be inserted, and "value" represents the value of at least one attribute field of the data record to be inserted. As an example, the value may be a value obtained by encoding (e.g., merging or serializing) the at least one attribute value according to a specific rule.

As an example, the order of the values of the index fields of the data record to be inserted is the same as the order of the index fields of the multiple slice groups. In other words, the arrangement order of the values (key _1, key _2, … …, key _ N) of each index field in the interface put is the same as the arrangement order of each index field of the slice group of the data table. In this case, the index field corresponding to each value can be determined according to the arrangement sequence of key _1, key _2, … …, and key _ N in the interface put, so that the data record can be inserted conveniently and efficiently.

As an example, the data record may be inserted through an interface put (table _ name, key _1, key _2, … …, key _ N, field _1, field _2, … …, field _ N, value), which may correspond to a case where the sorting fields of different slice groups are different, where field _1, field _2, … …, field _ N represents the value of each sorting field corresponding to key _1, key _2, … …, and key _ N, respectively. Other parameters in the interface can be understood with reference to the description of the interface put (table _ name, ts, key _1, key _2, … …, key _ N, value).

The above examples of the interface put are only used for describing the concept of the present disclosure, and are not used for limiting the scope of the present disclosure, and other interfaces for inserting data records are also possible, for example, one or more parameters in the above interfaces may be omitted, and one or more parameters may be added to the above interfaces.

As an example, the arrangement order of the values of the respective sorting fields of the data record to be inserted is the same as the arrangement order of the respective sorting fields of the plurality of slice groups. In other words, the arrangement order of the values (field _1, field _2, … …, field _ N) of the respective sorting fields in the interface input is the same as the arrangement order of the respective sorting fields of the slice group of the data table. In this case, the index field corresponding to each value in field _1, field _2, … … can be determined according to the arrangement sequence of field _1, field _2, … … and field _ N in the interface put, so that the insertion operation of data record can be conveniently and efficiently performed.

According to an exemplary embodiment of the present disclosure, the data records to be inserted into the data table may be data records as shown in table 1. The data records in table 1 can be inserted one by means of the above-mentioned interface put. The card number may be used as an index field and the timestamp as an ordering field. The value of the card number, the value of the timestamp, the value of the transaction amount, the value of the transaction location, and the value of the POS number may be encoded (e.g., serialized or merged) to be stored in a memory space shared with all second-level skip lists.

As an example, a storage space for storing a value of at least one attribute field of the data record to be inserted stores a character string obtained by one of the following ways: merging the values of the at least one attribute field according to a preset character string merging rule, serializing the values of the at least one attribute field according to a preset JSON format, serializing the values of the at least one attribute field according to a preset protocol buffer format, and serializing the values of the at least one attribute field according to a preset Schema format.

As an example, the predetermined string merging rule includes merging by a specific symbol (e.g., "|"). For example, the value of the transaction amount, the value of the transaction location, and the value of the POS number may be combined into a character string "100 | xx road |10xxx on beijing, based on a preset symbol, such as" | ", and the character string may be stored in the storage space. Besides combining the values of the at least one attribute field according to a predetermined string combination rule to obtain a string, the above-mentioned obtaining of the string using the JSON format, the protocol buffer format, and the Schema format is also feasible. Of course, the above description is by way of example only and should not be taken as limiting.

As an example, the data records to be inserted are time-series data records, and the sorting fields are all time stamps. As an example, the step of adding a node in the second level skip list comprises: and adding the nodes according to the time indicated by the value of the timestamp, so that the nodes in the second-stage skip list are arranged according to the sequence of the time from near to far. The time can be determined by comparing the values of the timestamps, and the time corresponding to the value of the larger timestamp is shorter than the time corresponding to the value of the smaller timestamp. Therefore, in the second-stage skip list, the node with the larger value of the timestamp can be arranged before the node with the smaller value of the timestamp.

The following describes a method for maintaining data in the memory by taking the data record to be inserted as the 1 st data record in table 1 as an example. The data table is assumed to include two slice groups, an index field of the first slice group is a card number, an index field of the second slice group is a POS number, sequencing fields of the two slice groups are timestamps, and second-level skip tables of the two slice groups share a storage space SP. In step S210, for the first slice group, determining a corresponding slice (for example, slice 0) according to the value "6222 XXXX 01" of the card number; for the second slice group, the corresponding slice (for example, slice 3) is determined according to the value "10 xxx" of the POS number. In step S210, for the first slice group, a node with "6222 XXXX 01" as a key is looked up from the first-level skip list corresponding to slice 0 (for example, the node 11 is looked up); for the second slice group, the node keyed by "10 xxx" is looked up from the first level hop table corresponding to slice 3 (e.g., to node 333). In step S230, for the first slice group, a node having the value "2018052814520505" of the timestamp as a key and a pointer pointing to the storage space SP as a value corresponding to the key is inserted into the second-level skip list corresponding to the node 11; in the second slice group, a node having a value "2018052814520505" of the timestamp as a key and a pointer to the storage space SP as a value corresponding to the key is inserted into the second-stage skip list corresponding to the node 333, and "6222 XXXX 01", "2018052814520505", "100", "xx pass in beijing," and "10 xxx" are encoded and stored in the storage space S.

In addition, in the case where the data records to be inserted are the 1 st to 3 rd data records in table 1. For the same slice group, the 1 st data record and the 2 nd data record have the same card number, and therefore, the two data records correspond to the same second level skip list. In the second-level skip list, after the 1 st data record is added, since the value of the timestamp of the 2 nd data record is greater than that of the timestamp of the 1 st data record, the node corresponding to the 2 nd data record is added before the node corresponding to the 1 st data record.

FIG. 4 illustrates a schematic diagram of operations to insert a data record, according to an example embodiment of the present disclosure. As shown in fig. 4, a data record may be inserted into a data table in memory through a put interface (e.g., the put interface described above). The present exemplary embodiment corresponds to a case where the sorting fields of different slice groups are the same. The values of the index fields input through the put interface are key1, key2 and key3, key1, key2 and key3 respectively correspond to dimension a, dimension b and dimension c, the value of the sorting field input through the put interface is ts, and the value of at least one attribute field input through the put interface is value. The data table may include a slice group a, a slice group b, and a slice group c. The index fields of slice group a, slice group b, and slice group c correspond to key1, key2, and key3, respectively. The ordering fields of slice group a, slice group b, and slice group c correspond to ts. The arrangement order of the index field of the slice group a, the index field of the slice group b and the index field of the slice group c is the same as the arrangement order of the key1, the key2 and the key 3. When the sequence of any one of the keys 1, 2 and 3 is known, the index field value can be determined.

Fig. 5 illustrates a flowchart of an operation of querying data in a memory according to an exemplary embodiment of the present disclosure. As shown in fig. 5, the operation of querying data in the memory according to the exemplary embodiment of the present disclosure includes steps S310 to S360.

In step S310, an index field of a data record to be queried, a value of the index field, and a value range of the sorting field are received. Here, the index field of the data record to be queried refers to the name of the index field of the data record to be queried, e.g., a card number.

As an example, the data records to be queried are time-series data records, and the sorting fields are all time stamps.

As an example, the value range specifies a start value and an end value of the timestamp or specifies an end value of the timestamp.

In step S320, a shard group of the plurality of shard groups is determined, where the index field is the same as the index field of the data record to be queried.

In step S330, in the determined fragment group, a corresponding fragment is determined according to a value of the index field of the data record to be queried.

As an example, the shard corresponding to the data record to be queried may be determined by: calculating a hash value corresponding to the value of the index field of the data record to be queried; obtaining a remainder obtained by dividing the calculated hash value by the determined total number of the fragments of the fragment group; and determining the segment corresponding to the obtained remainder as the corresponding segment. The value of the index field may be computed using a hash function to obtain a hash value. For example, the hash function used may be the hash function murmurmurhash proposed by austinnappleby. Of course, the present disclosure is not limited to the hash function used, and other hash functions may be used for the calculation of the hash value.

In step S340, a node using the value of the index field of the data record to be queried as a key is searched from the first-level skip table corresponding to the determined fragment.

In step S350, the pointers in the nodes whose keywords are within the value range of the sorting field of the data record to be queried are queried from the pointers in the searched nodes or the second-level skip list indicated by the object.

In step S360, a value of at least one attribute field of the data record to be queried is taken out from the storage space indicated by the queried pointer.

As an example, the data query may be performed through an interface scan (table _ name, key _ name, key _ value, start _ time, end _ time), where table _ name is used to define the name of a data table from which data is queried, key _ name is used to define an index field of a data record to be queried, key _ value is used to define the value of the index field of the data record to be queried, and start _ time and end _ time are used to define the value range of the data record to be queried, such as a start time and an end time. This example may correspond to the case where the ordering fields corresponding to all slice groups of the data table are the same.

As an example, data query may be performed through an interface get (table _ name, key _ name, key _ value, ts), where table _ name is used to define a name of a data table from which data is queried, key _ name is used to define an index field of a data record to be queried, key _ value is used to define a value of the index field of the data record to be queried, and ts is used to define a value range of the data record to be queried, for example, ts is used to define a value of a timestamp of the data record to be queried, in which case, data whose timestamp is ts is actually desired to be queried; for another example, ts is used to define the termination time of the data record to be queried, in which case, it is actually desired to query for data from the time the data is queried to the specified ts. This example may correspond to the case where the ordering fields corresponding to all slice groups of the data table are the same.

The above examples of the interface scan and the interface get are only used for describing the concept of the present disclosure, and are not used for limiting the scope of the present disclosure, and other interfaces for data query are also possible, for example, one or more parameters in the above interfaces may be omitted, and one or more parameters may be added to the above interfaces.

Taking the data records in table 1 as an example, data with a card number value of "6222 XXXX 01" and a timestamp value of "2018052815520505" and an end value of "2018052814520505" needs to be queried. Assuming that the fragment corresponding to the data record to be queried in the fragment group with the card number as the index field is determined to be fragment 0 according to the '6222 XXXX 01', the node with the '6222 XXXX 01' as the key is found from the node 11 node 1m of the first-level hop table corresponding to the fragment 0 to be the node 11, and the second-level hop table corresponding to the node 11 is determined to comprise the nodes 41 to 4 k. The node having the start value of "2018052815520505" and the end value of "2018052814520505" of the key is searched for from the node 41 to the node 4 k. And taking out the data from the storage space pointed by the pointer stored in the node of the searched second-stage skip list. For another example, when the value range only specifies the end value (e.g., "2018052814520505") of the timestamp, a node in the second-level skip list whose value of the keyword is greater than or equal to "2018052814520505" may be searched, and data is fetched from the storage space pointed by the pointer stored in the searched node in the second-level skip list.

As an example, the step of retrieving the value of at least one attribute field of the data record to be queried from the storage space indicated by the queried pointer includes: taking out the value of the at least one attribute field of the data record to be queried by one of the following methods: splitting the value of the at least one attribute field according to a preset character string splitting rule, deserializing the value of the at least one attribute field according to a preset JSON format, deserializing the value of the at least one attribute field according to a preset protocol buffer format, and deserializing the value of the at least one attribute field according to a predefined Schema format.

For example, values corresponding to the keywords are found from the found nodes as "100 | road |10xxx xx in beijing prefecture" and "50 |20xxx | in xx shop | in beijing west prefecture", respectively. The method can split 100. the xx road |10xxx "in Beijing according to a preset symbol such as" | ", and obtain a value 100 of a first split character string as a transaction amount according to the meaning of the preset split character string, a value xx road in Beijing in Shang and a value 10 xxx" of a POS number as a value of a transaction place of a second split character string. Similarly, the value "50" for the transaction amount, the value "xxxx shop" for the transaction location, and the value "20 xxx" for the POS number may be obtained from "20 xxx | xxxx shop | beijing west xi xx shop".

As an example, to ensure that excessive data is not stored in a memory (e.g., a data table of the memory), a threshold number of nodes corresponding to the second-level skip table may be set. On this basis, the step of querying the pointer of the keyword in the node in the value range of the sorting field of the data record to be queried from the pointer in the searched node or the second-level skip list indicated by the object includes: and taking out the pointers stored in each node of the preset number of nodes of the keywords in the value range from near to far from the second-level jump table indicated by the pointers or objects in the searched nodes, wherein the preset number does not exceed the threshold value of the number of nodes.

As an example, a node number threshold corresponding to the second-level skip list may be set, and the periodic deletion may be performed according to the set node number threshold, that is: respectively traversing a first-stage skip list and a second-stage skip list corresponding to each slice group in a preset period; when the number of traversed nodes in the second-level hop table corresponding to each slice group exceeds a node number threshold, according to the arrangement order of the nodes in the second-level hop table, deleting all nodes in the second-level hop table, which are arranged sequentially behind the node corresponding to the node number threshold (for example, when the node number threshold is 10, according to the arrangement order of the nodes in the second-level hop table, the node corresponding to the node number threshold is the 10 th node), and deleting the storage space indicated by the pointer stored in the deleted node, wherein the arrangement order is the arrangement order of time from near to far.

As an example, to improve processing efficiency, the following stale data deletion operations may be performed: setting the length of an expiration period; traversing a first-stage skip list and a second-stage skip list corresponding to each fragment in a preset period (for example, 3 months), and integrally deleting nodes behind the nodes in the second-stage skip list according to the arrangement sequence of the nodes in the second-stage skip list from near to far by positioning the nodes of which the values of the timestamps in the second-stage skip list reach the expiration period length. For example, if the value of the timestamp corresponding to the set expiration date length is 2018060000000000, all of the 3 nodes corresponding to the 3 pieces of data in table 1 added to the second-stage skip list in the above example can be deleted by the above-described expiration data deletion operation.

FIG. 6 illustrates a block diagram of a system for managing memory data according to an exemplary embodiment of the present disclosure. As shown in fig. 6, a system for managing memory data according to an exemplary embodiment of the present disclosure includes: a slice management device 410 and a skip list management device 420.

The slice management device 410 is configured to set multiple slice groups, where each slice group includes at least one slice, all slices in each slice group correspond to a uniform index field and an ordering field, index fields of different slice groups are different, and ordering fields of different slice groups are the same or different.

The skip list management device 420 is configured to respectively construct a corresponding first-level skip list and a corresponding second-level skip list for each fragment, where the first-level skip list corresponding to each fragment is configured to store a node that takes a value of an index field of each fragment in a data record as a key and a pointer or an object indicating the second-level skip list as a value corresponding to the key, and the second-level skip list corresponding to each fragment is configured to store a node that takes a value of an ordering field of each fragment in the data record as a key and a pointer indicating a storage space for storing a value of at least one attribute field of the data record as a value corresponding to the key.

As an example, the storage space is shared by the second-level skip tables corresponding to all slice groups and stores values of all attribute fields of the data record, or the same storage space is shared by the second-level skip tables corresponding to the same slice group and stores values of all attribute fields of the data record except for the index field and the sorting field corresponding to the same slice group.

FIG. 7 illustrates a block diagram of a system for maintaining data in memory according to an exemplary embodiment of the present disclosure. As shown in fig. 7, a system for maintaining data in a memory according to an exemplary embodiment of the present disclosure includes: a fragmentation determination means 510, a node lookup means 520 and a node management means 530.

The fragment determining device 510 is configured to determine, for each of multiple fragment groups, a corresponding fragment according to a value of an index field of each fragment group of a data record to be inserted, where each fragment group includes at least one fragment, all fragments in each fragment group correspond to a unified index field and a unified sorting field, the index fields of different fragment groups are different, and the sorting fields of different fragment groups are the same or different, each fragment corresponds to a first-level skip table, and the first-level skip table is used to store a node that takes the value of the index field of the corresponding fragment in the data record as a key and takes a pointer or an object indicating the second-level skip table as a value corresponding to the key.

The node searching means 520 is configured to search, from the first-level skip table corresponding to the determined slice, a node using a value of the index field of each slice group of the data record to be inserted as a key.

The node management device 530 is configured to, when a node is found which uses the value of the index field of each slice group of the to-be-inserted data record as a key, add, to a second-level skip list indicated by a pointer or an object in the found node, a node which uses the value of the sorting field of the to-be-inserted data record related to each slice group as a key and uses a pointer of a storage space for storing the value of at least one attribute field of the to-be-inserted data record as a value corresponding to the key.

As an example, the fragmentation determining means 510 calculates a hash value corresponding to a value of the index field of the data record to be inserted with respect to each of the fragmentation groups, obtains a remainder obtained by dividing the calculated hash value by the total number of the fragmentation of each of the fragmentation groups, and determines a fragmentation corresponding to the obtained remainder as a corresponding fragmentation.

As an example, the system for maintaining data in a memory according to an exemplary embodiment of the present disclosure further includes: a skip list management device, wherein, when a node using the value of the index field of each slice group of the to-be-inserted data record as a key cannot be found, the skip list management device creates a second-level skip list, the node management device 530 creates a node using the value of the index field of each slice group of the to-be-inserted data record as a key and using a pointer or an object indicating the created second-level skip list as a value corresponding to the key in the first-level skip list, and adds a node using the value of the sorting field of the to-be-inserted data record as a key and using a pointer indicating a storage space for storing the value of at least one attribute field of the to-be-inserted data record as a value corresponding to the key in the created second-level skip list.

As an example, the system for maintaining data in a memory according to an exemplary embodiment of the present disclosure further includes: a sharded group determining means (not shown) and a data acquiring means (not shown), wherein the sharded group determining means determines the sharded group of the plurality of sharded groups whose index field is the same as the index field of the data record to be queried, the sharded determining means 510 determines, among the determined sharded groups, determining a corresponding fragment according to the value of the index field of the data record to be queried, searching a node using the value of the index field of the data record to be queried as a keyword from a first-level skip list corresponding to the determined fragment by the node searching device 530, querying a pointer of the keyword in a node within the value range of the sorting field of the data record to be queried from a pointer in the searched node or a second-level skip list indicated by an object by the data obtaining device, and taking out the value of at least one attribute field of the data record to be queried from the storage space indicated by the queried pointer.

As an example, the data obtaining apparatus, according to the queried pointer, takes out a value of the at least one attribute field of the data record to be queried by one of the following manners: splitting the value of the at least one attribute field according to a preset character string splitting rule, deserializing the value of the at least one attribute field according to a preset JSON format, deserializing the value of the at least one attribute field according to a preset protocol buffer format, and deserializing the value of the at least one attribute field according to a predefined Schema format.

As an example, the fragmentation determining means 510 calculates a hash value corresponding to a value of an index field of the data record to be queried, obtains a remainder obtained by dividing the calculated hash value by the total number of fragments of the determined fragmentation group, and determines a fragment corresponding to the obtained remainder as a corresponding fragment.

As an example, the data record to be inserted and/or the data record to be queried is a time-series data record, and the sorting fields are both time stamps.

As an example, the node management apparatus 530 adds the nodes according to the time indicated by the value of the timestamp, so that the nodes in the second-level skip list are arranged in the order of time from the near to the far.

As an example, the system for maintaining data in a memory according to an exemplary embodiment of the present disclosure further includes: and a node number threshold setting device (not shown) configured to set a node number threshold corresponding to the second-level skip list, where the data obtaining device takes out, from the second-level skip list indicated by the pointer or the object in the searched node, pointers stored in each node of a predetermined number of nodes within the value range by using the keyword according to a sequence from near to far, where the predetermined number does not exceed the node number threshold.

As an example, the system for maintaining data in a memory according to an exemplary embodiment of the present disclosure further includes: the device comprises a node number threshold setting device (not shown), a skip list traversing device (not shown) and a data deleting device (not shown), wherein the node number threshold setting device sets a node number threshold corresponding to a second-level skip list, the skip list traversing device traverses a first-level skip list and a second-level skip list corresponding to each slice group respectively in a preset period, when the number of traversed nodes in the second-level skip list corresponding to each slice group exceeds the node number threshold, the node deleting device deletes all nodes in the second-level skip list, the nodes are arranged in the order behind the nodes corresponding to the node number threshold, and a storage space indicated by pointers stored in the deleted nodes is deleted, wherein the arrangement order is an arrangement order from near to far in time.

As an example, the system for maintaining data in a memory according to an exemplary embodiment of the present disclosure further includes: the data deleting device integrally deletes nodes of which the arrangement sequence is after the node of which the value of the timestamp in the second-level skip list reaches the expiration period length, wherein the arrangement sequence is the arrangement sequence of time from near to far.

It should be understood that specific implementations of the system for managing memory data and the system for maintaining data in a memory according to the exemplary embodiment of the present disclosure may be implemented with reference to the related specific implementations described in conjunction with fig. 1 to 5 and table 1, and are not described herein again.

The apparatus shown in fig. 6 and 7 may be configured as software, hardware, firmware, or any combination thereof, respectively, that performs certain functions. These means may correspond, for example, to an application-specific integrated circuit, to pure software code, or to a combination of software and hardware elements or modules. Further, one or more functions implemented by these apparatuses may also be collectively performed by components in a physical entity device (e.g., a processor, a client, a server, or the like).

The method and system for managing memory data and maintaining data in memory according to exemplary embodiments of the present disclosure is described above with reference to fig. 1 through 5. It is to be understood that the above-described method may be implemented by a program recorded on a computer-readable medium, for example, according to an exemplary embodiment of the present disclosure, there may be provided a computer-readable storage medium storing instructions that, when executed by at least one computing device, cause the at least one computing device to perform: setting a plurality of fragment groups, wherein each fragment group comprises at least one fragment, all fragments in each fragment group correspond to a uniform index field and a uniform sequencing field, the index fields of different fragment groups are different, and the sequencing fields of different fragment groups are the same or different; and respectively constructing a corresponding first-level jump table and a corresponding second-level jump table for each fragment, wherein the first-level jump table corresponding to each fragment is set as a node for storing a value corresponding to a key by using a value of an index field of each fragment in a data record and a pointer or an object indicating the second-level jump table, and the second-level jump table corresponding to each fragment is set as a node for storing a value corresponding to the key by using a value of a sorting field of each fragment in the data record as the key and a pointer indicating a storage space for storing a value of at least one attribute field of the data record.

For another example, according to an example embodiment of the present disclosure, there may be provided a computer-readable storage medium storing instructions that, when executed by at least one computing device, cause the at least one computing device to perform: for each of a plurality of slice groups, determining a corresponding slice according to a value of an index field of each slice group of a data record to be inserted, wherein each slice group comprises at least one slice, all slices in each slice group correspond to a uniform index field and a uniform sorting field, the index fields of different slice groups are different, the sorting fields of different slice groups are the same or different, each slice corresponds to a first-level skip list, and the first-level skip list is used for storing a node which takes the value of the index field of the corresponding slice in the data record as a key and takes a pointer or an object indicating a second-level skip list as a value corresponding to the key; searching a node which takes the value of the index field of each slice group of the data record to be inserted as a key word from a first-level skip table corresponding to the determined slice; and adding a node which takes the value of the sorting field of the data record to be inserted, which is related to each slice group, as a key and takes a pointer of a storage space for storing the value of at least one attribute field of the data record to be inserted as a value corresponding to the key in a second-level skip table indicated by a pointer or an object in the searched node under the condition that the node which takes the value of the index field of the data record to be inserted, which is related to each slice group, as a key is searched.

The computer program in the computer-readable storage medium described above may be run in an environment deployed in a computer apparatus, such as a processor, a client, a host, a proxy device, a server, etc., for example, by at least one computing device in a stand-alone environment or a distributed cluster environment, as examples of which may be computers, processors, computing units (or modules), clients, hosts, proxy devices, servers, etc. It should be noted that the computer program may also be used to perform additional steps in addition to the above steps or perform more specific processing when performing the above steps, and the contents of these additional steps and further processing have already been described with reference to fig. 1 to 5, and will not be described again here in order to avoid repetition.

It should be noted that the system for managing memory data and the system for maintaining data in memory according to the exemplary embodiments of the present disclosure may completely depend on the operation of the computer program to realize the corresponding functions, that is, each device corresponds to each step in the functional architecture of the computer program, so that the whole system is called by a special software package (e.g., lib library) to realize the corresponding functions.

Alternatively, the various means shown in fig. 6 and 7 may be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the corresponding operations may be stored in a computer-readable medium such as a storage medium, so that a processor may perform the corresponding operations by reading and executing the corresponding program code or code segments.

For example, according to an exemplary embodiment of the present disclosure, a system may be provided comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the following steps of managing memory data: setting a plurality of fragment groups, wherein each fragment group comprises at least one fragment, all fragments in each fragment group correspond to a uniform index field and a uniform sequencing field, the index fields of different fragment groups are different, and the sequencing fields of different fragment groups are the same or different; and respectively constructing a corresponding first-level jump table and a corresponding second-level jump table for each fragment, wherein the first-level jump table corresponding to each fragment is set as a node for storing a value corresponding to a key by using a value of an index field of each fragment in a data record and a pointer or an object indicating the second-level jump table, and the second-level jump table corresponding to each fragment is set as a node for storing a value corresponding to the key by using a value of a sorting field of each fragment in the data record as the key and a pointer indicating a storage space for storing a value of at least one attribute field of the data record.

For another example, according to an exemplary embodiment of the present disclosure, a system may be provided that includes at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the following steps of maintaining data in memory: for each of a plurality of slice groups, determining a corresponding slice according to a value of an index field of each slice group of a data record to be inserted, wherein each slice group comprises at least one slice, all slices in each slice group correspond to a uniform index field and a uniform sorting field, the index fields of different slice groups are different, the sorting fields of different slice groups are the same or different, each slice corresponds to a first-level skip list, and the first-level skip list is used for storing a node which takes the value of the index field of the corresponding slice in the data record as a key and takes a pointer or an object indicating a second-level skip list as a value corresponding to the key; searching a node which takes the value of the index field of each slice group of the data record to be inserted as a key word from a first-level skip table corresponding to the determined slice; and adding a node which takes the value of the sorting field of the data record to be inserted, which is related to each slice group, as a key and takes a pointer of a storage space for storing the value of at least one attribute field of the data record to be inserted as a value corresponding to the key in a second-level skip table indicated by a pointer or an object in the searched node under the condition that the node which takes the value of the index field of the data record to be inserted, which is related to each slice group, as a key is searched.

The system may constitute a stand-alone computing environment or a distributed computing environment including at least one computing device and at least one storage device, where the computing device may be, for example, a general purpose or special purpose computer, a processor, etc., a unit that performs processing solely using software, or a combination of software and hardware. That is, the computing device may be implemented as a computer, processor, computing unit (or module), client, host, proxy device, server, or the like. Further, the storage devices may be physical storage devices or logically partitioned storage units that may be operatively coupled to the computing device or may communicate with each other, such as through I/O ports, network connections, and the like.

Furthermore, for example, exemplary embodiments of the present disclosure may also be implemented as a computing device comprising a storage component having stored therein a set of computer-executable instructions that, when executed by the processor, perform a method of managing memory data and/or a method of maintaining data in memory.

In particular, the computing devices may be deployed in servers or clients, as well as on node devices in a distributed network environment. Further, the computing device may be a PC computer, tablet device, personal digital assistant, smart phone, web application, or other device capable of executing the set of instructions described above.

The computing device need not be a single computing device, but can be any device or collection of circuits capable of executing the instructions (or sets of instructions) described above, individually or in combination. The computing device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces with local or remote (e.g., via wireless transmission).

In the computing device, the processor may include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a programmable logic device, a special purpose processor system, a microcontroller, or a microprocessor. By way of example, and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.

Certain operations described in the method of managing memory data and/or the method of maintaining data in memory according to the exemplary embodiments of the present disclosure may be implemented by software, certain operations may be implemented by hardware, and further, these operations may be implemented by a combination of hardware and software.

The processor may execute instructions or code stored in one of the memory components, which may also store data. Instructions and data may also be transmitted and received over a network via a network interface device, which may employ any known transmission protocol.

The memory component may be integral to the processor, e.g., having RAM or flash memory disposed within an integrated circuit microprocessor or the like. Further, the storage component may comprise a stand-alone device, such as an external disk drive, storage array, or any other storage device usable by a database system. The storage component and the processor may be operatively coupled or may communicate with each other, such as through an I/O port, a network connection, etc., so that the processor can read files stored in the storage component.

Further, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the computing device may be connected to each other via a bus and/or a network.

Operations involved in a method of managing memory data and/or a method of maintaining data in memory according to exemplary embodiments of the present disclosure may be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams may be equally integrated into a single logic device or operated on by non-exact boundaries.

While various exemplary embodiments of the present disclosure have been described above, it should be understood that the above description is exemplary only, and not exhaustive, and that the present disclosure is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. Therefore, the protection scope of the present disclosure should be subject to the scope of the claims.

Claims

1. A method of managing memory data, comprising:

setting a plurality of fragment groups, wherein each fragment group comprises at least one fragment, all fragments in each fragment group correspond to a uniform index field and a uniform sequencing field, the index fields of different fragment groups are different, and the sequencing fields of different fragment groups are the same or different;

and respectively constructing a corresponding first-level jump table and a corresponding second-level jump table for each fragment, wherein the first-level jump table corresponding to each fragment is set as a node for storing a value corresponding to a key by using a value of an index field of each fragment in a data record and a pointer or an object indicating the second-level jump table, and the second-level jump table corresponding to each fragment is set as a node for storing a value corresponding to the key by using a value of a sorting field of each fragment in the data record as the key and a pointer indicating a storage space for storing a value of at least one attribute field of the data record.

2. A method of maintaining data in a memory, comprising:

for each of a plurality of slice groups, determining a corresponding slice according to a value of an index field of each slice group of a data record to be inserted, wherein each slice group comprises at least one slice, all slices in each slice group correspond to a uniform index field and a uniform sorting field, the index fields of different slice groups are different, the sorting fields of different slice groups are the same or different, each slice corresponds to a first-level skip list, and the first-level skip list is used for storing a node which takes the value of the index field of the corresponding slice in the data record as a key and takes a pointer or an object indicating a second-level skip list as a value corresponding to the key;

searching a node which takes the value of the index field of each slice group of the data record to be inserted as a key word from a first-level skip table corresponding to the determined slice;

and adding a node which takes the value of the sorting field of the data record to be inserted, which is related to each slice group, as a key and takes a pointer of a storage space for storing the value of at least one attribute field of the data record to be inserted as a value corresponding to the key in a second-level skip table indicated by a pointer or an object in the searched node under the condition that the node which takes the value of the index field of the data record to be inserted, which is related to each slice group, as a key is searched.

3. The method of claim 2, wherein the second-level skip tables corresponding to all slice groups share the storage space and the storage space stores values of all attribute fields of the data record, or the second-level skip tables corresponding to a same slice group share a same storage space and the same storage space stores values of all attribute fields of the data record except for an index field and an ordering field corresponding to the same slice group.

4. The method of claim 2, wherein the step of determining the corresponding slice comprises:

calculating a hash value corresponding to the value of the index field of each slice group of the data record to be inserted;

obtaining a remainder obtained by dividing the calculated hash value by the total number of the fragments of each fragment group;

and determining the segment corresponding to the obtained remainder as the corresponding segment.

5. The method of claim 2, wherein each slice has stored therein a pointer or object indicating the corresponding first level skip list.

6. The method of claim 2, further comprising:

and under the condition that a node which takes the value of the index field of each slice group of the data record to be inserted as a key cannot be found, creating a second-level skip list, creating a node which takes the value of the index field of each slice group of the data record to be inserted as the key and takes a pointer or an object indicating the created second-level skip list as a value corresponding to the key in the first-level skip list, and adding a node which takes the value of the sorting field of each slice group of the data record to be inserted as the key and takes a pointer indicating a storage space for storing the value of at least one attribute field of the data record to be inserted as a value corresponding to the key in the created second-level skip list.

7. A system for managing memory data, comprising:

the system comprises a fragment management device and a fragment processing device, wherein the fragment management device is used for setting a plurality of fragment groups, each fragment group comprises at least one fragment, all the fragments in each fragment group correspond to a uniform index field and a uniform sequencing field, the index fields of different fragment groups are different, and the sequencing fields of different fragment groups are the same or different;

the skip list management device is used for respectively constructing a corresponding first-level skip list and a corresponding second-level skip list for each fragment, wherein the first-level skip list corresponding to each fragment is set to be used for storing nodes which take the value of the index field of each fragment in the data record as a key and take a pointer or an object indicating the second-level skip list as a value corresponding to the key, and the second-level skip list corresponding to each fragment is set to be used for storing nodes which take the value of the sorting field of each fragment in the data record as a key and take a pointer of a storage space for storing the value of at least one attribute field of the data record as a value corresponding to the key.

8. A system for maintaining data in a memory, comprising:

the system comprises a fragment determining device and a fragment determining device, wherein the fragment determining device is used for determining corresponding fragments according to values of index fields of each fragment group of a data record to be inserted, each fragment group comprises at least one fragment, all the fragments in each fragment group correspond to uniform index fields and sequencing fields, the index fields of different fragment groups are different, the sequencing fields of different fragment groups are the same or different, each fragment corresponds to a first-level skip list, and the first-level skip list is used for storing nodes which take the values of the index fields of the corresponding fragments in the data record as keywords and take pointers or objects indicating the second-level skip list as values corresponding to the keywords;

a node searching device, configured to search, from a first-level skip table corresponding to the determined slice, a node using a value of the index field of each slice group of the data record to be inserted as a key;

and a node management device, configured to, when a node is found that uses the value of the index field of each slice group of the to-be-inserted data record as a key, add, to a second-level skip list indicated by a pointer or an object in the found node, a node that uses the value of the sorting field of the to-be-inserted data record that is related to each slice group as a key and uses a pointer indicating a storage space for storing the value of at least one attribute field of the to-be-inserted data record as a value corresponding to the key.

9. A computer-readable storage medium storing instructions that, when executed by at least one computing device, cause the at least one computing device to perform the method of any of claims 1 to 6.

10. A system comprising at least one computing device and at least one storage device storing instructions that, when executed by the at least one computing device, cause the at least one computing device to perform the method of any of claims 1 to 6.