CN111966690A

CN111966690A - Method and device for loading table full data, computer equipment and storage medium

Info

Publication number: CN111966690A
Application number: CN202010847191.0A
Authority: CN
Inventors: 胥亮; 贾淋淋; 何毅朗
Original assignee: Xi'an Huanyu Satellite Tt & C And Data Application Co ltd
Current assignee: Xi'an Huanyu Satellite Tt & C And Data Application Co ltd
Priority date: 2020-08-21
Filing date: 2020-08-21
Publication date: 2020-11-20
Anticipated expiration: 2040-08-21
Also published as: CN111966690B

Abstract

The invention discloses a method, a device, computer equipment and a storage medium for loading table full data, wherein the method comprises the steps of searching key values in a REDIS database and acquiring all keys associated with table examples; storing all keys associated with the table instance to a container, acquiring a field name of a first key value element in the container, and setting the field name as a table column name; sequentially traversing each key value element of the container, and creating a row object by using the key value elements; and adding the row object into a table, acquiring the field names and values of the key value elements corresponding to the row object, and filling the field names and values into the cells of the row object. The method encapsulates the interaction with the REDIS in the table data full-load process, including key value retrieval, table column name creation and row data storage, and reduces the difficulty of access to the REDIS; the cells are assigned by the cell names, so that the problem of field disorder in the hash type caused by the change of the internal data structure of the REDIS is solved.

Description

Method and device for loading table full data, computer equipment and storage medium

Technical Field

The invention relates to the technical field of memory database caching, in particular to a method and a device for loading table full data, computer equipment and a storage medium.

Background

With the rapid development of the internet technology and the continuous improvement of a distributed architecture system, users put higher requirements on the response speed of software and the overall performance of the system. The caching mechanism is adopted to solve the problems of high concurrent requests and hotspot data storage, and the caching mechanism becomes a popular preferred scheme. The cache mechanism stores data in the memory by utilizing the high-speed read-write characteristic of the memory so as to improve the data access speed and reduce the performance loss of the system at the data read-write part.

The Redis (remote Dictionary Server) is a non-relational memory database which takes Key-Value as a data model, is free from source and supports high-performance concurrent reading and writing, can support a plurality of data storage structures such as character strings, lists, sets, ordered sets and hash tables, provides a communication mode of issuing and subscribing messages, and has strong data management capability widely applied to scenes such as cache, data issuing and subscribing. The REDIS hash type is a mapping table of fields and value fields of a string type, and the (key-field-value) structure of the REDIS hash type is similar to the (identification-attribute name-attribute value) structure of an object, so that the REDIS hash type can be used for storing the object which is simple in structure and is not required to be queried in a complex relationship.

Although REDIS has many advantages of key value distributed storage, no coupling among data, strong expandability, fast search speed and the like, the stored data has no structure and no coupling, and only binary type also becomes a very obvious defect. When a developer uses a strong type-related language such as C/C + +, conversion between integer, floating point, Boolean and other programming language-related data types and REDIS data types needs to be processed, and the correlation and cohesion of business data are not easy to understand. Meanwhile, Redis provides a plurality of client commands for executing operations such as adding, deleting, modifying, checking and the like on a Redis server side, and also provides client APIs of a plurality of programming languages so as to complete interaction with a Redis database in the software development process. The Hiredis is a lightweight C language client function library for accessing a Redis database, and basically realizes the minimum support to the Redis protocol. The Hiredis comprises a synchronous API and an asynchronous API, and the use method of the native interface is flexible. The asynchronous API needs to work with a third party asynchronous event library. For example, Libevent is an open-source, lightweight, cross-platform, high-performance event notification library written in C language and supporting event-driven and multiple I/O multiplexing technologies, and is widely used as an underlying network library.

In summary, before using the REDIS, software developers need to spend a lot of time and energy to learn the REDIS command and the application method of the client API, the time and labor costs are high, the overall progress of software development is influenced to a certain extent, and the complexity of later maintenance of the software is increased.

Based on the above reasons, the applicant proposes an automatic table data synchronization method, but in the table data synchronization method, the difficulty of accessing the REDIS is high in the process of interaction with the REDIS in the process of loading the whole amount of table data, which is not beneficial to a user, and in some cases, field disorder in the hash type caused by the change of the internal data structure of the REDIS can occur.

The information disclosed in this background section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.

For the reasons, the applicant provides a method, a device, a computer device and a storage medium for loading table full-scale data.

Disclosure of Invention

In order to meet the above-mentioned requirements, a first object of the present invention is to provide a method for loading table full-scale data.

The second purpose of the invention is to provide a device for loading table full data.

It is a third object of the invention to provide a computer apparatus.

It is a fourth object of the invention to provide a non-transitory computer-readable storage medium.

In order to achieve the purpose, the invention adopts the following technical scheme:

on one hand, the embodiment of the application provides a method for loading table full data, which comprises the following steps:

searching key values in a REDIS database, and acquiring all keys associated with table examples;

storing all keys associated with the table instance to a container, acquiring a field name of a first key value element in the container, and setting the field name as a table column name;

sequentially traversing each key value element of the container, and creating a row object by using the key value elements;

and adding the row object into a table, acquiring the field names and values of the key value elements corresponding to the row object, and filling the field names and values into the cells of the row object.

In one possible embodiment, the step of searching for key values in the REDIS database includes implementing the search for key values in a progressive traversal manner.

In a possible embodiment, the step of implementing the lookup of the key value in a progressive traversal manner includes obtaining a table name and a cursor initial value, and inputting the table name and the cursor initial value into a cursor-based iterator.

In a possible embodiment, the step of creating a row object by using key-value elements includes obtaining all fields and values in the key-value elements, analyzing a row number, and creating a row object.

In one possible embodiment, the step of parsing out the row number and creating the row object includes making the intra-row cell name and the table column cell name consistent.

In a possible implementation manner, the step of obtaining the field name and the value of the key-value element corresponding to the row object and filling the field name and the value into the cell of the row object further includes returning the successfully loaded row number.

In a possible embodiment, the step of obtaining the field names and values of the key-value elements corresponding to the row object and filling the field names and values into the cells of the row object further includes returning the number of columns that are successfully loaded.

In another aspect, an embodiment of the present application further provides an apparatus for loading table full data, including the following units:

the key value searching unit is used for searching key values in the REDIS database and acquiring all keys related to the table examples;

the container filling unit is used for storing all keys associated with the table instance into a container, acquiring the field name of the first key value element in the container, and setting the field name as the table column name;

the element traversing unit is used for sequentially traversing each key value element of the container and creating a row object by using the key value elements;

and the data filling unit is used for adding the row object into a table, acquiring the field names and values of the key value elements corresponding to the row object and filling the field names and values into the cells of the row object.

In a third aspect, an embodiment of the present application further provides a computer device, including a memory, a processor, and a program loaded with table full amount data, which is stored in the memory and is executable on the processor, where the program loaded with table full amount data, when executed by the processor, implements the method for loading table full amount data as described in any one of the above.

In a fourth aspect, embodiments of the present application propose a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of loading tabular full size data as set forth in any one of the above.

Compared with the prior art, the invention has the beneficial effects that: the method for loading the table full data provided by the embodiment of the application encapsulates the interaction with the REDIS in the table data full loading process, and comprises the steps of retrieving key values, creating table column names and storing row data, so that the difficulty of accessing the REDIS is reduced; the cells are assigned by the cell names, so that the problem of field disorder in the hash type caused by the change of the internal data structure of the REDIS is solved.

The invention is further described below with reference to the accompanying drawings and specific embodiments.

Drawings

Fig. 1 is a schematic flowchart of an embodiment of a method for loading table full data according to an embodiment of the present disclosure;

fig. 2 is a structural block diagram of a two-dimensional table of a specific application scenario of a method for loading table full data according to an embodiment of the present application;

FIG. 3 is a schematic diagram of the attributes and methods of the cell class definition of the table of FIG. 2;

FIG. 4 is a schematic diagram of the attributes and methods of the cell set class definition of the table of FIG. 2;

FIG. 5 is a schematic diagram of the attributes and methods of the data row class definition of the table of FIG. 2;

FIG. 6 is a schematic diagram of the attributes and methods of the data row set class definition of the table of FIG. 2;

FIG. 7 is a schematic diagram of the attributes and methods of the class definition of the table of FIG. 2;

FIG. 8 is a schematic block diagram of an apparatus for loading table full data provided by an embodiment of the present application;

fig. 9 is a schematic block diagram of a computer device provided in an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

Referring to fig. 1, as shown in fig. 1, a flowchart of an embodiment of a method for loading table full data provided by the embodiment of the present application includes the following steps:

s101, searching key values in a REDIS database, and acquiring all keys associated with table instances;

specifically, in the present embodiment, the key value is searched in a progressive traversal manner.

For example, the SCAN command of REDIS is executed by the REDIS client side object of the table class to acquire all keys related to the table instance. The SCAN command is a cursor-based iterator, which means that each time the command is called, the cursor returned by the last call is used as the cursor parameter of the call, so as to continue the previous iteration process. In this example, to adapt the SCAN grammar, the table name, the initial cursor value, and the input to the cursor-based iterator are obtained.

In some embodiments, assuming that the table name is TestTbl, the TestTbl table data in the REDIS memory database is stored in a hash table with several keys TestTbl: N, where N represents a unique row number in the data table, and has the same meaning as the primary key of the data table, so that a row of data can be uniquely determined. Therefore, the format of the SCAN command to be executed is SCAN cursor MATCH table name, the initial value of the cursor is 0, and the command will return the cursor searched next time and several results satisfying the matching pattern searched this time every time the command is executed. When the return cursor is 0, the SCAN command is shown to complete one complete traversal.

Step S102, storing all keys associated with the table instance in a container, acquiring a field name of a first key value element in the container, and setting the field name as a table column name;

in the above steps, the key value traversed by the SCAN command has been acquired, assuming that the container name is TBL _ KEYS.

In this embodiment, if the container is not empty, the first element is fetched, assuming TestTbl:1, and the HKEYS command to REDIS retrieves all the field names in the TestTbl:1 key, i.e., the column names of the Table TestTbl.

Step S103, traversing each key value element of the container in sequence, and creating a row object by using the key value elements;

specifically, a REDIS key to row object conversion operation firstly needs to execute a HGETALL command of REDIS to acquire all fields and values in the key, analyzes a row number RID from a key value in a table name-row number format, creates a new row object with a row number RID and consistent row unit cell names and table column unit cells, and fills all fields and values in the key into the unit cells in the new row object.

Wherein the Redis Hgetall command is used to return all fields and values in the hash table. In the return value, each field name is immediately followed by a value (value) of the field, so the length of the return value is twice the size of the hash table. By adopting the command in the embodiment, the analysis of the line number can be accurately realized.

And step S104, adding the row object into a table, acquiring the field names and values of the key value elements corresponding to the row object, and filling the field names and values into cells of the row object.

Specifically, since the data is already loaded from the REDIS database, there is no need to synchronize the REDIS database for the bulk add operation, and the data loading method returns the number of rows and columns successfully loaded to the caller.

Interaction with the REDIS in the table data full-load process is encapsulated through the steps, including key value retrieval, table column name creation and row data storage, so that the difficulty of access to the REDIS is reduced; the cells are assigned by the cell names, so that the problem of field disorder in the hash type caused by the change of the internal data structure of the REDIS is solved.

The following describes a specific implementation process of the embodiment of the present invention with a specific example:

referring to fig. 2 to 7, fig. 2 is a block diagram of a structure of a two-dimensional table in a specific application scenario according to an embodiment of the present application, fig. 3 is an attribute and a method defined by a cell class of the table of fig. 2 in this example, fig. 4 is an attribute and a method defined by a cell set class of the table of fig. 2 in this example, fig. 5 is an attribute and a method defined by a data row class of the table of fig. 2 in this example, fig. 6 is an attribute and a method defined by a data row set class of the table of fig. 2 in this example, and fig. 7 is an attribute and a method defined by a two-dimensional table class in this.

The purpose of this embodiment is to implement a thread-safe distributed virtual two-dimensional table based on REDIS, that is, a specific application scenario of the table data automatic synchronization method provided in the embodiment of the present application, and aims to enable a developer to access the REDIS database only through row and column operations of the table without learning the REDIS related technology, without concerning data storage details and underlying implementation, and without defining field names and formats for data to be stored in advance when the REDIS database is operated through the table, thereby solving the problems of high REDIS learning and use costs, no data coupling, and difficulty in understanding.

In the embodiment of the application, the problems that in the table synchronization method, the REDIS is difficult to access in the process of interaction with the REDIS in the process of loading the full amount of table data, the use by a user is not facilitated, and fields in a hash type are out of order due to the change of an internal data structure of the REDIS caused in some cases are solved.

The cells and rows provided in this example describe a two-dimensional table model, organizing data in rows and columns that is logically more understandable to the user. The table is internally provided with a plurality of classes to realize the overall function in a coordinated mode, and a plurality of interfaces are provided for users to use. The functions provided by the form comprise form creation, form field and index design, batch increase, modification, deletion, row data query, form row and column number acquisition, form clearing, change notification callback registration and the like.

In order to implement the table function, the technical solution adopted in this example includes three modules, namely, a table definition module, an asynchronous subscription module and a REDIS operation module. The table definition module comprises a cell class, a cell set class, a data row set class and a two-dimensional table class definition. The asynchronous subscription module uses Hiredis and libervent open source software to connect with a RELIS server in an asynchronous mode, executes SUBSCRIBE command to SUBSCRIBE channel data named by a table name, starts an event thread to asynchronously receive subscription response data, and timely notifies an outer-layer user. The REDIS operation module comprises a REDIS command class, a synchronous client class and a response analysis class definition, wherein the command class stores all parameters of a REDIS command; the response analysis class is responsible for completing the conversion from the REDIS server side to the custom response structure; the synchronization client class uses Hiredis to connect with the REDIS server in a synchronization mode, executes a plurality of commands in a pipeline mode, and returns the REDIS response parameters.

The implementation scheme of the data automatic synchronization table based on the REDIS is provided in the example, the scheme designs related classes in C + + language and encapsulates a series of interfaces available for external use.

The scheme also comprises the design and encapsulation of a REDIS command class, a REDIS synchronous client class, a REDIS response analysis class, a REDIS asynchronous subscription class, a cell collection class, a data line class, a line collection class, a table class and the like.

The used REDIS server is deployed into a master-slave environment of a single machine mode, and the REDIS synchronous client class encapsulates an interface for interaction between a synchronous mode and the REDIS server. The REDIS asynchronous subscription class realizes the asynchronous mode to connect with the REDIS server, takes the form name as the channel name, executes the SUBSCRIBE command of the REDIS to SUBSCRIBE the change notification of the form, and notifies the form to update data after receiving the change message.

Each row record of the created table may map to a key-value pair of a value type hash table in the REDIS database. The mapping is explained as follows: assuming that there is a table example DT _ Person storing personal information, whose table Name is Person, three columns of Name, Age, City are defined in the table, and information of two persons, a and B, are stored, which are { a, 10, west }, { B, 18, guangzhou }, respectively, the two information will be stored in the hash table of the REDIS database through the HMSET command of REDIS. The command corresponding to the user A is as follows: HMSET Person:1Name A Age 10 City Saisan. Wherein the number 1 in the key Person:1 represents the row number recorded in the Person table. The mapping mode of the user B is the same as that of the user A.

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

The REDIS command class is named RedisCommand, which stores all elements of a REDIS command in a dynamic string array, including the command name, key name, and its parameters. Redis Command uses template function technology to realize the function of converting non-character string type such as numerical value, Boolean and the like into character string type, defines the interface for dynamically adding parameters into the Redis command through overloading < < and () operator, and returns a new Redis Command instance, through which the user can dynamically splice the parameters of the Redis command in an additional mode.

The REDIS synchronization client class is named as ReDISHA, the synchronization mode is connected with the REDIS server, a series of REDIS commands are executed on the REDIS server in a pipeline mode, and an execution result returned by the REDIS server is analyzed.

The REDIS response parsing class is responsible for uniformly converting response messages (redisReply) returned by the Hiredis open source software into a self-defined response structure. The self-defined response structure is named Redisceply and comprises a data type, an integer value, a character string value and a dynamic array of the Redisceply type. The data type corresponds to 6 return types of character strings, arrays, integers, null, states and failures defined by the REDIS protocol. When the response data type is integer, the value of the response data is stored in the integer value attribute of the RedisRely class; when the response data type is a character string, a state or a failure, the value of the response data is stored in the character string value attribute of the RedisRely class; when the response data type is an array, the value of the response data type is stored in the dynamic array attribute of the Redisceply type, and the value of the Redisceply type of the array element is stored according to the rule.

The REDIS asynchronous subscription class is named Redis Subscriber, and defined core data members comprise a libevent event object, a Hiredis asynchronous object, an initialization notification callback and a message notification callback. The subscription class initialization process needs to complete the following processing: firstly, connecting a REDIS server in an asynchronous mode to obtain a Hiredis asynchronous object, setting connection and disconnection callbacks, creating a libevent object instance, and binding the libIS asynchronous object instance to the asynchronous object to enable the REDIS asynchronous callbacks to be associated with events; secondly, the user needs to register an initialization notification callback and a message notification callback to the subscription class; thirdly, starting an independent thread to start event distribution processing; and fourthly, with the form name as a channel name, executing the SUBSCRIBE command of the REDIS to SUBSCRIBE the change notice of the form, and registering a command feedback callback. And the command feedback callback discriminates and processes the SUBSCRIBE command execution result and the subscription message. And after receiving the subscription message, calling the change callback registered by the outer-layer user to notify the outer layer of the changed data.

The cell class is named DataCell, abbreviated DC within this document. A cell is a basic data unit in a two-dimensional table, and has attributes such as a cell name (a character string type), a storage value (a character string type), a change flag (boolean type), and a table where the cell is (a waak _ ptr type).

The interfaces provided by the cell classes are shown in table 1.

TABLE 1 cell class interface Table

DCI _3 is a DC class assignment operator overloading interface, and the interface resets the internal data of the current object according to the participating object. For the only pointer data member within the DC class, i.e. the table to which it belongs, the assignment operation is simply to reset its value.

DCI _4 is a DC-like copy constructor that calls the DCI _3 interface directly to implement its function.

DCI _8 is an interface for obtaining a cell value by a DC class, and the interface converts a character string value stored in a cell into a plurality of basic data types by using a function template of a C + + language and a overloading technology, so that a user can conveniently obtain a value according to an actually required type.

DCI _9 is an interface for setting a cell value in a DC class, and in order to support a user to transmit a plurality of C + + language basic data types, the interface uniformly converts a plurality of different input types into a character string type for storage by using a function template, a heavy load and a character stream technology.

DCI _14 is a comparison function interface for cell class encapsulation, which defines two parameters, where parameter 1 is a string type for storing relational operators, such as: ">", "<", "|! "," ═ like "," between "," in "; the form 2 is a string array type for storing relational operator parameters. The method calls a relation operator interface of the internal overload of the DC class and compares the relation between the stored values in the two cells.

The cell collection class is named DataCellCollection, abbreviated DCC within this document. A set of cells comprising a plurality of cells, the set being logically equivalent to a row or a column; the DCC class stores a plurality of cell addresses (shared _ ptr < DataCell >) using sequential and associative containers. The intelligent pointer is used to conveniently manage the dynamic memory and avoid memory leakage. Storage in sequential containers may facilitate indexing of cells with subscripts; the mapping relation can be established for the column names and the cells by storing the mapping relation in the association container, so that the cells can be conveniently searched according to the column names.

The interface provided by the cell set class is shown in table 2.

TABLE 2 cell set class interface Table

The DCCI _3 is a DCC class assignment operator overloading interface, the DCCI _4 is a DCC class copy construction function interface, the two interfaces define a DC class copy control mechanism, and the class object copying process is required to construct a completely independent copy, namely a cell object pointer stored in a class must adopt a deep copy mode.

DCCI _7 is an interface defined by the DCC class that obtains its location index from the cell name. The position index of a cell within a set is defined as the subscript of the cell object in the sequential container.

The DCCI _8 interface detects whether a cell of a specified name exists within the set of cells.

DCCI _9 is the first kind of operation adding interface defined by DCC class, and the interface mainly realizes the function of inserting specified cell in the specified position of the current object.

In this embodiment, the default parameter identification is incremented only on the current row. Assuming that the user calls the DCCI _9 interface through the DCC class object C, and transmits a DC object address A and an insertion position P, the DCCI _9 interface processes the following in detail:

step 1, judging whether the table to which the cell object A to be inserted belongs is the same as the table to which the set C to be inserted belongs. If the result is the same, it is determined that A is already in C, and the process is ended.

And 2, judging whether the inserting position P is out of range. If the boundary is out of range, the process is terminated.

And 3, assigning the table address of the table associated with the C object to the A object. And acquiring a shared _ ptr intelligent pointer of the C object. The DCCI _10 interface is called.

And 4, judging whether the object C is associated with the data table. If not, the process is terminated.

And 5, judging whether the C object associated data table has data or not. If not, the process is terminated.

And 6, inserting the reference cell object A into the specified positions of all the rows of the C object association table.

DCCI _10 is a second kind of add-on operation interface defined by DCC class, which mainly implements the function of inserting a specified cell at a specified position of a specified cell set object. And the default parameter of the participating cells, the insertion position and the parameter of the participating cell set of the interface are legal, and the insert method of the sequence container and the associated container is directly called to add the participating cell objects to the participating cell set.

DCCI _11 is a third type of add operation interface defined by the DCC class that appends a cell to the collection, i.e., inserts at the end position.

DCCI _12 is a fourth kind of increasing operation interface defined by DCC class, adds a specified name cell to the set, and the interface firstly constructs a cell object according to the name of the entering cell and then calls DCCI _11 interface;

DCCI _13 is the first delete operation interface defined by the DCC class that deletes the cells in the collection that specify the name. For example: a user calls a DCCI _13 interface through the DCC class object C, a cell name N is introduced, the DCCI _13 interface firstly obtains a cell index from a container according to the cell name N, and then calls a DCCI _14 interface;

DCCI _14 is a second type of delete operation interface defined by the DCC class that deletes the cells in the set that specify the index. For example: a user calls a DCCI _14 interface through the DCC class object C, a cell position index P is transmitted, the DCCI _14 interface judges whether the entry reference index P is out of range, and if the entry reference index P is out of range, the processing is terminated; the cell object pointer ObjPtr is acquired from the sequential container according to the index P. And searching the cell object name pointed by the ObjPtr in the associated container, and if the corresponding element is not found in the associated container, only deleting the designated index element from the sequential container. If the associated container is found, the associated container is deleted. If the C object has associated a table, then each row of the associated table calls the DCCI _15 interface.

DCCI _15 is an internal delete operation interface defined by the DCC class that deletes the cells in the specified set that specify the index.

DCCI _16 is a third type of delete operation interface defined by the DCC class that deletes the cell object in the current set that is named as the reference. For example: a user calls a DCCI _16 interface through the DCC class object C and transmits a cell object Obj, the DCCI _16 interface obtains a cell name N through the cell object Obj and then calls a DCCI _13 interface;

DCCI _17 is a copy interface defined by the DCC class that enables cells beginning at the current set designation index to be copied into the reference designation set. A user calls a DCCI _17 interface through the DCC class object C, a starting position S and a cell set M are transmitted, and the DCCI _17 interface firstly checks whether the starting position S is legal or not in the object C; if not, the process is terminated. And the object M circularly calls a DCCI _9 interface to sequentially add the cell object copies in the C set at the tail end.

DCCI _18 is an interface provided by the DCC class to obtain a copy of the cell object from the cell name.

DCCI _19 is an interface provided by the DCC class to obtain a copy of the cell object from the cell index.

The data line class is named DataRow, abbreviated DR within this document. The data line class describes the attributes and methods that a single line should possess. A single row should have at least one set of cells and two attributes of the table in which it is located.

The row class provides the interface as shown in table 3.

Table 3 data row interface table

DRI _2 is a DR class assignment operator overloading interface, DRI _3 is a DR class copy constructor interface, the two interfaces define a DR class copy control mechanism, and the class object copying process is required to construct a completely independent copy, namely, a cell object pointer stored in a class must adopt a deep copy mode.

DRI _5 is the first type of retrieve cell interface provided by the DR class, which retrieves from within-line according to the cell name, and calls the DCCI _18 interface of the cell collection class directly internally.

DRI _6 is a second type of retrieve cell interface provided by the DR class that retrieves from within-line according to cell location, calling the DCCI _19 interface of the cell collection class directly inside the interface.

DRI _7 is an interface provided by the DR class to get the in-line cell collection object.

DRI _8 is an interface for obtaining a row number by a DR class, and each row object in the table has a unique row number in the table, namely the row number is equal to a table primary key.

DRI _9 is an interface for DR class setting line object identification.

DRI _10 is an interface for DR class to obtain row primary key values, and the primary key of the row object is "table name: row number".

DRI _11 is the interface provided by the DR class to get all field names in the row.

DRI _12 is the interface provided by the DR class to get all the field names and values in the row, and the dynamic array returned stores F1-V1-F2-V2 … in turn.

The data line set class is named DataRowCollection, abbreviated DRC within this document. The DRC class describes the attributes and methods that multiple rows should possess. The line set class is provided with a plurality of line element sets and attributes of a table where the line element sets are located; a plurality of row object pointers (shared _ ptr < DR >) are managed through a doubly linked list container, a row set index table is managed through a secondary hash table, the primary hash table takes column names as keys, the value range of the primary hash table is also a hash table, the secondary hash table takes the stored values of the column cell objects as keys, and the value range is a row object pointer.

The interfaces provided by the rowset class are shown in table 4.

Table 4 data row set interface table

The row set class needs to support setting a plurality of index fields at the same time, and in order to realize the function, the index table adopts a two-stage hash table structure, wherein the column name of the first-stage hash table is used as a key, and the value of the second-stage hash table is used as a value. The secondary hash table is keyed by the column value and takes the row class instance to which the column value belongs as the value. DRCI _7 is an interface for DRC class to set index, when the row set class sets index, if the row number of the table to which the row set class belongs is 0, the mapping relation between the column name and the empty second-level hash table is only needed to be added into the first-level hash table. If the table row is not 0, a secondary hash table is also constructed according to each row of data in the table.

DRCI _9 is an interface that adds a single line provided by the DRC class. When a new row is added to the row set, if the index is set in the table, the index table needs to be updated synchronously. The specific treatment process comprises the following steps: traversing the index table, and acquiring a cell object corresponding to the index column name from the row object to be added; obtaining a stored value from the cell object, and searching whether the stored value exists in an index list value corresponding to the current list name; if so, i.e., an index field value conflict, adding the new row fails. And if the column name does not exist, inserting the corresponding relation between the new column value and the new row into the index table corresponding to the current column name. And finally inserting a new row into the row set container.

DRCI _10 is the first single line of deletion interface provided by the DRC class. When a row object is deleted from the row set, if the table has an index set, the index table also needs to be updated. The specific treatment process comprises the following steps: checking whether an index column exists in the row object to be deleted; if yes, deleting the corresponding column value and row object mapping relation from the index table; if not, searching the specified line object from the line set container and deleting the specified line object;

DRCI _11 is a delete row interface provided by the row collection class based on the index field name and value.

DRCI _12 is a line set class that provides a method to retrieve lines based on location. The detailed treatment process comprises the following steps: checking whether the incoming position index is out of range, if so, terminating the processing; otherwise, the incoming position index is corresponding to the relative position of the line object in the line set list, and the line object at the specified position is returned.

DRCI _13 is an interface provided by the row set class that fetches row objects based on row identification.

DRCI _14 is an interface provided by the line set class that fetches line objects from index field names and values.

DRCI _15 is an interface for updating row data provided by the row set class.

DRCI _16 is an interface provided by the rowset class to find rowsets that satisfy user specified conditions.

DRCI _17 is a line-by-line traversal interface provided by the line set class.

DRCI _18 is a judge line presence interface provided by the line set class based on the line identification.

The table class describes a two-dimensional table model containing rows and columns, each defined as a set of cells, for temporarily storing a virtual table data in memory. A two-dimensional table is made up of rows and columns. The rows are represented by a row set, the row set comprises a plurality of row instances, each row instance comprises a cell set instance, one cell set instance comprises a plurality of cell instances, and each cell instance stores valid names and values. The multiple columns are represented by a cell set, the column cell set comprises multiple cell instances, only the name in each cell instance is valid, and no meaningful value is stored. The table also has attributes such as name, number, subscribe sub-object, and REDIS client sub-object.

The table class, named DataTable, abbreviated DT in this document, can be created and used independently. The table class provides the interface as shown in table 5.

TABLE 5 Table interface Table

DTI _3 is a static method provided by the DT class to create a table instance, and the internal process flow of this method is shown on the right side of FIG. 7. The user transmits parameters such as REDIS server IP, port, password, table name, database index and subscription mark to the interface. The REDIS service end IP is an IP address for deploying the REDIS service, the port is a REDIS service monitoring port, and the password is an authentication password when the REDIS client is connected with the REDIS service end. The subscription flag indicates whether to turn on the subscription form data change function, and the subscription form data change function is turned on when the value is true and turned off when the value is false. If the caller does not transmit the database index and the subscription mark parameter to the interface, the interface uses a default value, the database index default value is 0, namely the default table data is stored in a number 0 database of the REDIS server; the subscription flag default value is true, i.e. the table instance turns on the subscription data change function by default.

The static method firstly constructs a table instance dynamically on the heap, and stores the instance address in the intelligent pointer. After the table is instantiated, internal data including a row set, a column set, a REDIS client instance and a subscription instance are initialized. The table initially created has no schema, i.e. no column names are set, nor does it store any row of data. Therefore, initializing a row set requires dynamically creating an empty DRC instance, and initializing a column name set requires dynamically creating an empty DCC instance. In order that the table instance can be normally used as a bridge between a user and a REDIS database, a REDIS client terminal object needs to be initialized when the table is created for the first time, and the connection establishment between the REDIS client object and a REDIS server is ensured to be normal. In order to sense the change of the form data in time, the subscription sub-object is initialized after the form instance is created, the condition that the asynchronous connection between the subscription sub-object and the REDIS server is normally established is ensured, meanwhile, the SUBSCRIBE command of the REDIS is successfully executed, the asynchronous event processing thread is normally started, and all messages on the channel named by the form name are subscribed. After the form object is created successfully and all initializations are completed, the static method returns the form instance address to the user. The user can obtain the address of the table instance, i.e., a shared _ ptr < DT > type variable. An example of a static method call is as follows:

shared_ptr<DT>PTR＝DT::CreateDataTable(“192.168.1.1”,6973,“123456”,“TestTbl”,true,1)；

DTI _5 is an interface for the DT class setup table column field names. After obtaining the table instance address, if a brand-new data table needs to be designed, the user can call the column name of the data stored in the interface setting table through the table instance address according to the application scene. For example, setting 3 column names F1, F2, and F3, respectively, a call is performed: vector < string > headers { "F1", "F2", "F3" }; PTR- > sethead (heads); the internal part of the interface traverses the transmitted array of column names, each element is taken out in sequence and used as a table column cell set object to call the entry parameter of the DCC type DCCI _12 interface, and the cells with the specified names are added in a circulating mode.

DTI _6 is an interface for DT class to set the table index field, and the index is an optional setting item, allowing the table to have no index. The user can create indexes for the table examples at any time according to the service requirement, allow the indexes to be created on a plurality of columns, allow the indexes to be set for a plurality of times, and select two modes of resetting or adding during the setting. The reset mode can clear the previous index information and construct the index table again according to the table data; the adding mode can keep the original index information and supplement the index table data according to the newly set index column. Although one table is allowed to have a plurality of index columns, only one column is allowed to be searched, and multi-column combination search is not supported. The interface of the table class setting index directly calls the line set class setting index interface.

DTI _8 is a DT class setting table name interface.

DTI _9 is a DT class get table name interface. The table name is case-specific and the default value is null.

DTI _10 is a DT class get table number interface. The table number is obtained through the static variable of the self-increment table class, is unique in a single process, and can be the same among multiple processes.

DTI _11 is a DT-type interface for loading table full data, i.e. another specific embodiment of the method for loading table full data proposed in this embodiment. In this example, if the user only needs to load the existing table data from the REDIS database without setting a new table, the DTI _11 interface of the table class is called by the table instance address, and the example is called: auto ret ═ PTR- > LoadData (); the method for loading table data first needs to execute the SCAN command of REDIS through the REDIS client terminal object of the table class to obtain all keys related to the table instance. Assuming that the table name is TestTbl, the TestTbl table data in the REDIS memory database is stored in a hash table with several keys of TestTbl: N, where N represents a unique row number in the data table, and has a meaning equal to that of a primary key of the data table, so that a row of data can be uniquely determined. Therefore, the format of the SCAN command to be executed is SCAN cursor MATCH table name, the initial value of the cursor is 0, and the command will return the cursor searched next time and several results satisfying the matching pattern searched this time every time the command is executed. When the return cursor is 0, the SCAN command is shown to complete one complete traversal. All KEYS retrieved during the SCAN traversal are saved to the container, assumed to be TBL _ KEYS. If the container is not empty, the first element is fetched, assuming TestTbl:1, and the HKEYS command to REDIS gets all the field names in the TestTbl:1 key, i.e., the column names of the Table TestTbl. Calling the DTI _8 interface sets the table column name. And circularly traversing each key element in the TBL _ KEYS, completing the conversion operation from key values to row objects, and finally calling a table type batch increasing data row interface to add all rows created according to the TBL _ KEYS into the table. Since the data is loaded from the REDIS database, there is no need to synchronize the REDIS library for bulk add operations. The data loading method returns the number of rows and columns successfully loaded to the caller.

The conversion operation from REDIS key to row object firstly needs to execute HGETALL command of REDIS to obtain all fields and values in the key, analyzes row number RID from the key value in the format of table name and row number, creates a new row object with row number RID and consistent row cell name and table column cell, and fills all fields and values in the key into the cell in the new row object.

The technical effect obtained by the method for loading the table full data is as follows: firstly, the method encapsulates the interaction with the REDIS in the table data full-load process, including key value retrieval, table column name creation and row data storage, and reduces the difficulty of access to the REDIS; secondly, the method assigns the value to the cell through the cell name, and avoids the problem of field disorder in the hash type caused by the change of the internal data structure of the REDIS.

DTI _12 is a DT type acquisition table row and column number interface.

DTI _13 is a DT-type clear table row data interface.

DTI _15 is a DT type acquisition table new line number interface. Each table specifies a globally unique line number identifier name. For example, a table instance is named TableNameA, and the row number identifier name of the table can be defined as TableNameA _ SEQ. The method for generating a new line number requires first entering the line number identifier name, and then executing the INCR command of REDIS to add 1 to the value stored in TableNAME _ SEQ for KEY. The REDIS mechanism ensures that if the designated KEY does not exist in the REDIS database, the value of the designated KEY is initialized to 0, then INCR operation is executed, finally the KEY is stored in the REDIS database, and a result after the INCR execution is returned to the client. The single-thread serial processing mechanism of the REDIS ensures that the problem of data competition can not occur when the new line numbers of the same table are generated concurrently in a multi-thread environment.

DTI 16 is the DT class create table new row object interface. The method for creating the new row object comprises the steps of dynamically creating an empty new row object on a heap, copying a list cell set of a table object into the new row object, setting a data table associated with the new row object as a current table object, and returning the address of the new row object to a user so that the user can set row data.

DTI-17 is a DT-type batch new line interface.

DTI _18 is a DT-type bulk delete line interface.

DTI _19 is a DT-type bulk modify line interface. Batch operations are performed in a transactional manner, with success being all success and failure all failure. When the rows are added, deleted or modified in batches, the table data needs to be locked first, and then data backup is carried out, wherein the backup is to rollback the table data when partial row operation fails. After the backup is finished, the single-row interface is called to add, delete or modify the definition of the row set class to circularly process the plurality of rows which are transmitted. And after all the rows are successfully processed, unlocking the table data, and selecting whether to synchronously update the REDIS database according to business needs.

The method for updating the REDIS database needs to transmit batch row data to be updated and operation identification, and the operation identification can be set to be added, deleted or modified. The method comprises the steps of firstly analyzing each row of data into a subscription and release data structure, and then converting the subscription and release structure into a REDIS command according to an operation identifier. If the operation identifier is added or modified, the line data needs to be converted into the HMSET KEY F1V 1[ F2V 2 … ] command of REDIS. If the operation identifier is deleted, the row data needs to be converted into HDEL KEY F1[ F2 … ] command of REDIS. KEY in the command is "table name: row number", Fn and Vn correspond to the name and value of each cell in the row. All the rows to be operated are converted into a REDIS command according to rules, then the whole publishing and subscribing structure is serialized, and a PUBLISH CHANNEL MESSAGE command is added into a set of REDIS commands to be executed, wherein CHANNEL is a table name, and MESSAGE is serialized publishing and subscribing data. And executing all REDIS commands in a channel mode, analyzing an execution result, and returning success or failure to the caller according to the command execution result.

The publish-subscribe data structure is defined as a composite type and comprises an operation identifier, a database index, a table name, a table number and change line data. The row data comprises a row number and a row attribute set; the row attribute data includes an attribute name and an attribute value. The table instance for publishing change notifications serializes the publish-subscribe composite structure into a JSON character string, and the table instance for subscribing change notifications deserializes the received JSON character string into a composite structure.

DTI-20 is a DT-type row data query interface. The line query method allows a user to enter a self-defined query function object, the function object defines a self-defined line query condition, and a query function object interface is defined as pool (DR &). The row query method circularly traverses all rows in the table, takes each row as the input parameter of a query function object, executes the query function, and judges whether the input row meets the query condition or not according to the return value. If yes, putting the copy of the incoming row into the query result set. And finally, returning the query result set to the caller.

DTI _21 is a DT-type batch new column interface.

DTI _22 is a DT-type bulk delete column interface. The method for adding or deleting columns in batches calls an interface for adding or deleting cells of a cell set circularly, a table address is stored in the cell set of the column list of the table, and when the cells are added or deleted, if data of the rows exist in the table, the cells are added or deleted in the cell set corresponding to each row.

DTI _23 is the progressive traversal interface provided by the DT class.

DTI _24 is a DT class registration line add notification callback interface.

DTI _25 is a DT class registration line delete notification callback interface.

DTI _26 is a DT class registration line modification notification callback interface. In order to realize the automatic synchronization function of the table data, the table needs to realize a data change callback method. When the asynchronous event processing thread which subscribes the sub-object in the form receives the change message, the data change callback registered by the form class is called. The change callback of the table type can synchronously update the memory data of the table type according to the operation identifier in the subscription message body. The user can selectively register the form data to add, modify or delete the change callback interface to the form class, when the form updates the object memory data due to receiving the subscription change message, the callback interface registered by the user is called, and the user can customize the processing logic of the change data in the callback interface.

DTI _27 is a DT class registration subscription notification callback interface. The form class is an upper layer user of the subscription class, and a subscription notification callback must be registered in the subscription class so as to receive a form data change notification message in time. And after the form class receives the data change notification, deserializing the received JSON character string into a composite structure, and then adding, deleting or modifying according to the operation identifier. When the number of the line in the changed data is increased, the existence of the line number in the table instance memory is checked, and if the line number does not exist, the changed data is increased. Otherwise, the non-processing is ignored. And during modification and deletion, if the line number exists in the table instance memory, updating or deleting the corresponding line data, otherwise, ignoring and not processing the line data. The change message is passively received, the triggered table data change does not need to be synchronized to the REDIS database, and the REDIS database data change is completed by the table instance actively issuing the change.

DTI _28 is a DT class check table presence interface. The user can check the presence of any table data in the REDIS database by the table instance. For example, detecting whether a TestCheck table already exists by a TestTbl table instance, PTR- > TableExist ("TestCheck") may be performed; if the reference table name is not specified, the table instance itself is checked for presence by default, i.e., PTR- > TableExist () is equivalent to PTR- > TableExist ("TestTbl").

DTI _29 is the DT class interface to the search line based on the index field name value pair. When the index table is used for searching rows, only one column is allowed to be searched, and multi-column combined search is not supported.

Fig. 8 is a schematic block diagram of an apparatus for loading table full data according to an embodiment of the present invention. As shown in fig. 8, the present invention also provides a device for loading table full data, corresponding to the above method for loading table full data. The device for loading table full data comprises a unit for executing the method for loading table full data, and can be configured in a desktop computer, a tablet computer, a portable computer, and the like. Specifically, referring to fig. 8, the apparatus 200 for loading table full data includes:

a key value searching unit 201, configured to search key values in the REDIS database, and obtain all keys associated with the table instance;

a container filling unit 202, configured to store all keys associated with the table instance in a container, obtain a field name of a first key-value element in the container, and set the field name as a table column name;

an element traversing unit 203, configured to sequentially traverse each key-value element of the container, and create a row object by using the key-value elements;

and the data filling unit 204 is configured to add the row object to the table, obtain a field name and a value of a key value element corresponding to the row object, and fill the field name and the value into a cell of the row object.

It should be noted that, as can be clearly understood by those skilled in the art, the specific implementation process of each subunit of the apparatus 200 for loading table full data may refer to the corresponding description in the foregoing method embodiment, and for convenience and brevity of description, no further description is provided herein.

The apparatus 200 for loading table full data can be implemented in the form of a computer program, which can be run on a computer device as shown in fig. 9.

Referring to fig. 9, the computer device 300 includes a processor 302, memory, and a network interface 305 connected by a system bus 301, where the memory may include a non-volatile storage medium 303 and an internal memory 304.

The nonvolatile storage medium 303 may store an operating system 3031 and a computer program 3032. The computer program 3032 includes program instructions that, when executed, cause the processor 302 to perform a method of loading table full data.

The processor 302 is used to provide computing and control capabilities to support the operation of the overall computer device 300.

The internal memory 304 provides an environment for the execution of the computer program 3032 in the non-volatile storage medium 303, and the computer program 3032, when executed by the processor 302, causes the processor 302 to perform a method of loading table full data.

The network interface 305 is used for network communication with other devices. Those skilled in the art will appreciate that the configuration shown in fig. 9 is a block diagram of only a portion of the configuration associated with the present application and does not constitute a limitation of the computer apparatus 300 to which the present application is applied, and that a particular computer apparatus 300 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

The storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, which can store various computer readable storage media.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.

The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be merged, divided and deleted according to actual needs. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention.

While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for loading table full data, comprising the steps of:

2. The method of loading table full data according to claim 1, wherein the step of searching for key values in the REDIS database comprises implementing the search for key values in a progressive traversal.

3. The method of claim 2, wherein the step of performing a lookup of key values in a progressive traversal comprises obtaining a table name and a cursor initial value, and inputting the table name and the cursor initial value into a cursor-based iterator.

4. The method of claim 1, wherein the step of creating a row object using key-value elements comprises obtaining all fields and values in a key-value element, parsing out a row number and creating a row object.

5. The method of claim 4, wherein the step of parsing out the row number and creating the row object comprises matching the intra-row cell name with the tabular column cell name.

6. The method of claim 1, wherein the step of obtaining field names and values of key-value elements corresponding to the row objects to fill in cells of the row objects further comprises returning the number of rows successfully loaded.

7. The method of claim 1, wherein the step of obtaining field names and values of key-value elements corresponding to the row object to fill in cells of the row object further comprises returning the number of columns successfully loaded.

8. An apparatus for loading table full data, comprising the following elements:

9. A computer device comprising a memory, a processor, and a program loaded with tabular full size data stored in the memory and executable on the processor, wherein the program loaded with tabular full size data when executed by the processor implements a method of loading tabular full size data as claimed in any one of claims 1-7.

10. A non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of loading tabular full size data as claimed in any one of claims 1 to 7.