CN110888939A

CN110888939A - Data management method and device

Info

Publication number: CN110888939A
Application number: CN201811036754.7A
Authority: CN
Inventors: 罗金
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2018-09-06
Filing date: 2018-09-06
Publication date: 2020-03-17

Abstract

The invention discloses a data management method and device, and relates to the technical field of computers. One embodiment of the method comprises: performing full-field analysis on a target data table in a database, and determining a header field of the target data table; creating an entity object in the memory according to the header field of the table, and performing full data query in the target data table to obtain full data; and storing the full data into the entity object so as to realize the extraction of the data in the database into the memory. After the data in the database are extracted to the memory, the utilization rate of the memory can be improved; in addition, because the data in the memory is transparent, the data in the memory can be acquired without additionally processing the database query request, and only one query is needed, which is approximately closed loop application, so that the query efficiency can be further improved.

Description

Data management method and device

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a data management method and apparatus.

Background

There are many current reading schemes for data in relational data tables, such as: 1. direct query; 2. and after query, performing cache and multi-level cache and the like.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:

1. direct query: a large amount of storage relational data are not used by a memory, and the utilization rate of the memory is low; when the data volume to be inquired is large, the database needs to be inquired for many times, and the inquiry efficiency is low;

2. and caching and multi-level caching after query: the data query from the cache can improve the query efficiency to a certain extent, but the query of the relational data in the cache is still not high because the relational data need to be processed according to different business requirements and then is completed by customized encoded query statements.

Disclosure of Invention

In view of this, embodiments of the present invention provide a data management method and apparatus, so that after data in a database is extracted into a memory, the utilization rate of the memory can be increased; in addition, because the data in the memory is transparent, the data in the memory can be acquired without additionally processing the database query request, and only one query is needed, which is approximately closed loop application, so that the query efficiency can be further improved.

To achieve one or more of the above objects, according to an aspect of an embodiment of the present invention, there is provided a data management method.

The data management method according to the embodiment of the invention comprises the following steps:

performing full-field analysis on a target data table in a database, and determining a header field of the target data table;

creating an entity object in the memory according to the header field of the table, and performing full data query in the target data table to obtain full data;

and storing the full data into the entity object so as to realize the extraction of the data in the database into the memory.

Optionally, user configuration information is acquired when the application program is initialized, and the target data table is determined according to the user configuration information; wherein the user configuration information includes: database domain name or IP address, database name, table name.

Optionally, the data management method of the embodiment of the present invention further includes: inquiring data in the memory according to the following steps:

intercepting a database query request for querying a target data table in a database;

analyzing a query condition from the database query request, and converting the database query request into a memory query request of a corresponding memory according to the query condition;

and performing data query in the memory according to the memory query request.

Optionally, the data management method of the embodiment of the present invention further includes: presetting key fields of each target data table; when the key field is contained in the query request, the query request is judged to be a database query request aiming at the target table.

According to still another aspect of an embodiment of the present invention, there is provided a data management apparatus.

The data management apparatus according to an embodiment of the present invention includes:

the analysis module is used for carrying out full-field analysis on a target data table in the database and determining a header field of the target data table; performing full data query in the target data table according to the header field of the table to obtain full data;

the creating module creates an entity object in the memory according to the header field;

and the storage module stores the full data into the entity object so as to extract the data in the database into the memory.

Optionally, the parsing module is further configured to: acquiring user configuration information during initialization of an application program, and determining the target data table according to the user configuration information; wherein the user configuration information includes: database domain name or IP address, database name, table name.

Optionally, the data management apparatus according to the embodiment of the present invention further includes:

the intercepting module intercepts a database query request for querying a target data table in a database;

the conversion module is used for analyzing the query condition from the database query request and converting the database query request into a memory query request of a corresponding memory according to the query condition;

and the query module is used for performing data query in the memory according to the memory query request.

Optionally, the intercepting module is further configured to: presetting key fields of each target data table; when the key field is contained in the query request, the query request is judged to be a database query request aiming at the target table.

According to another aspect of an embodiment of the present invention, there is provided a data management electronic device.

The data management electronic device according to the embodiment of the present invention includes:

one or more processors;

a storage device for storing one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors implement the data management method provided in the first aspect of the embodiments of the present invention.

According to another aspect of an embodiment of the present invention, a computer-readable medium is provided.

According to the computer readable medium of the embodiment of the present invention, a computer program is stored thereon, and when the program is executed by a processor, the data management method provided by the first aspect of the embodiment of the present invention is realized.

One embodiment of the above invention has the following advantages or benefits: an entity object is created in the memory according to the header field of the target data table in the database, and then the full data of the target data table is stored in the entity object, so that the data in the database can be extracted into the memory, and the utilization rate of the memory is improved. Because the memory stores the full data in the target data table, the data does not need to be inquired from the database, and the inquiry efficiency can be greatly improved. Intercepting a database query request for querying a target data table in a database, converting the database query request into a memory query request corresponding to a memory according to the analyzed query condition, so that the data in the memory is transparent, the data in the memory can be obtained without additionally processing the database query request, and the query efficiency can be further improved by nearly applying a closed loop through only one-time query.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of a main flow of a data management method according to an embodiment of the present invention;

FIG. 2 is a schematic illustration of a main flow of a data management method according to some embodiments of the invention;

FIG. 3 is a schematic illustration of a main flow of a data management method according to further embodiments of the invention;

FIG. 4 is a schematic illustration of a main flow of a data management method according to further embodiments of the invention;

FIG. 5 is a schematic diagram of the main modules of a data management apparatus according to an embodiment of the present invention;

FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 7 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Some of the terms mentioned in the present invention are explained below:

relational data: the method is a database established on the basis of a relational model, and data in the database is processed by means of mathematical concepts and methods such as set algebra and the like.

Reading only: only query operations, and no update or delete operations.

Memory: memory is one of the important components in a computer, and it is the bridge to communicate with the CPU.

An interceptor: a mechanism is provided that allows a developer to define code that executes before and after an operation (action) is performed, and to prevent execution of an action before it is performed, as well as to provide a way in which reusable portions of an action can be extracted. In AOP (Aspect-organized Programming) an interceptor is used to intercept and then add certain operations before or after a certain method or field is accessed.

According to an aspect of an embodiment of the present invention, there is provided a data management method.

Fig. 1 is a schematic diagram of a main flow of a data management method according to an embodiment of the present invention, and as shown in fig. 1, the data management method according to the embodiment of the present invention includes: step S101, step S102, and step S103.

Step S101, carrying out full-field analysis on a target data table in a database, and determining a header field of the target data table.

In the practical application process, the tool adopting the data management method of the embodiment of the invention can provide the user-defined configuration service for the user accessing the tool, and the user can complete the designation of each target data table through the service. An application program accessed to the tool can trigger the full-field analysis service of the tool when the application program is initialized, and the target data tables needing to be analyzed are confirmed by reading data in user-defined configuration.

Suppose the following read-only relational quotation data table is provided, and the domain name of the database is as follows: com, database name: quote _ db, table name: storage _ quote. Specific information of a specified table is needed in user-defined configuration, and the configuration information is as follows: mysql, storage, com/quote _ db? table _ name ═ storage _ quote. This designates the data table storage _ quote as the target data table.

TABLE 1 read-only relational quotation data sheet

id	quote_code	quote_name	service_code	service_name
					1	BOOK001	Book quotation	storage01	Storage service class
2	FRESH001	Fresh quoted price	storage01	Storage service class
					3	GLOBAL001	Global purchase offer	storage01	Storage service class

Performing full-field analysis on the target data table in the table 1, and analyzing the structure of the target data table, namely reading the header information of the target data table to obtain the following fields: id, quote _ code, quote _ name, service _ code, service _ name.

And step S102, creating an entity object in the memory according to the header field of the table, and performing full data query in the target data table to obtain full data.

Taking the embodiment corresponding to table 1 as an example, the target data table name storage _ quote may be used as the name of the entity object, and the entity is created in the memory: storage _ quote. Second, a full data query is performed. The full data refers to all data in the target data table.

Step S103, storing the full amount of data into the entity object to extract the data in the database into the memory.

Taking the embodiment corresponding to table 1 as an example, the specific information stored in the memory is as follows:

in the prior art, when relational data are needed, a coding query relational database is needed so as to obtain the relational data, the main scheme for obtaining the relational data is obtained in the relational database in real time, a large amount of relational data are not stored by using a memory, and the utilization rate of the memory is low.

According to the invention, the entity object is created in the memory according to the header field of the target data table in the database, and then the full data of the target data table is stored in the entity object, so that the data in the database can be extracted into the memory, and the utilization rate of the memory is improved. Because the memory stores the full data in the target data table, the data does not need to be inquired from the database, and the inquiry efficiency can be greatly improved. In addition, since the target data table is read in full, there is no back source (i.e. if no data is obtained in the memory, the query and extraction in the database is continued) theoretically.

It should be noted that the method of the embodiment of the present invention is applicable to various types of data, such as relational data or phenanthrene relational data, and further such as read-only data or other non-read-only data.

FIG. 2 is a schematic diagram of the main flow of a data management method according to some embodiments of the invention. As shown in fig. 2, the main flow of the data management method includes:

firstly, providing a user-defined configuration service for a user accessing the tool, and enabling the user to complete the specification of a target data table through the service;

when an application system accessed to the tool is initialized, reading data in the custom configuration service to confirm which target data tables need to be analyzed;

if the configuration does not exist in the custom configuration service, indicating that no target data table needing to be analyzed exists, and ending the process;

otherwise, initializing a relational data full-field analysis service; analyzing the target table according to user-defined configuration, generating a memory data structure, and inquiring total data in the target data table; triggering a memory reading/storing service, and storing the full data into a memory; and the initialization of the application system is completed, and the process is finished.

Optionally, the data management method of the embodiment of the present invention further includes: inquiring data in the memory according to the following steps: intercepting a database query request for querying a target data table in a database; analyzing a query condition from the database query request, and converting the database query request into a memory query request of a corresponding memory according to the query condition; and performing data query in the memory according to the memory query request.

In the prior art, after the relational data are stored in the memory, the relational data need to be processed according to different business requirements, and then customized encoding is performed to complete query of the relational data in the memory, so that query efficiency is low.

The invention intercepts the database query request for querying a target data table in the database, converts the database query request into the memory query request corresponding to the memory according to the analyzed query condition, so that the data in the memory is transparent, the data in the memory can be obtained without additionally processing the database query request, and the query efficiency can be further improved by nearly applying closed loop only by once query.

Taking the embodiment shown in table 1 as an example, when the keyword select & & storage _ quote (select & & storage _ quote is a preset keyword indicating a query operation for the target data table) is intercepted, that is, the query request contains the keyword select & & storage _ quote, the database query request for the target data table at the time of the current query request is determined. When the command conversion operation for converting the database query request into the memory query request corresponding to the memory is executed according to the query condition, the specific conversion operation is as follows:

example 1: user desires to execute a command (i.e., database query request): select from mstorage _ quotewhere id 1;

then the conversion is: storage _ quote _ set.foreach (if (storage _ quote:: getId () ═ 1) return storage _ quote;))); i.e. data with id equal to 1 is matched in the warehousing quote set.

Example 2: the user desires to execute the command: select from storage _ quote where _ quote _ code ═ globl 001 'and service _ code ═ storage 01';

then the conversion is: store _ quote _ set _ foreach (store _ quote:: getQuote _ code (),. equals ("gloobl 001") & & getService _ code. equals ("store 01")) { return store _ quote; }); that is, data in the warehousing quote set is matched with quote _ code equal to "gloobl 001" and service _ code equal to "storage 01".

FIG. 3 is a schematic illustration of the main flow of a data management method according to further embodiments of the present invention. As shown in fig. 3, the main flow of the data management method includes:

firstly, extracting the total data in the target data table in the database into a memory by adopting the method of the embodiment of the invention, wherein the specific extraction method refers to the embodiment shown in fig. 2 or the step S101 to the step S103, which is not described again;

receiving a query command (i.e., a query request);

intercepting a query command interceptor (the interceptor is used for intercepting a database query request for querying a target data table in a database);

reading data in the user-defined configuration service;

if the current query command does not need to be intercepted, namely the current query command is not queried aiming at the target data table in the database, the process is ended;

otherwise, calling a query command conversion service to convert the database query request into a memory query request of a corresponding memory; the query command conversion service transmits the conversion result (i.e. the memory query request) to the memory data reading/storing service; the memory data read/store service executes the memory data read command (i.e., the memory query request) and returns the result, so far, the process ends.

FIG. 4 is a schematic illustration of the main flow of a data management method according to further embodiments of the present invention. As shown in fig. 4, the main flow of the data management method includes:

when the application system is initialized, a full-field parsing service of the relational data table is called to extract the full data in the target data table into the memory, and specific steps can refer to the flow of the embodiment shown in fig. 2, which is not described herein again;

when a query command (namely a query request) is received, calling a command interceptor (the interceptor is used for intercepting a database query request which is queried aiming at a target data table in a database) to intercept the query command; if the current query command needs to be intercepted, namely the current query command is queried aiming at a target data table in a database, a query command conversion service is called to convert the database query request into a memory query request of a corresponding memory; then calling the memory data reading/storing service to execute the memory data reading command (namely the memory query request) and returning the result, and ending the process.

Fig. 5 is a schematic diagram of main blocks of a data management apparatus according to an embodiment of the present invention, and as shown in fig. 5, a data management apparatus 500 according to an embodiment of the present invention includes:

the analysis module 501 performs full-field analysis on a target data table in the database to determine a header field of the target data table; performing full data query in the target data table according to the header field of the table to obtain full data;

a creating module 502, which creates an entity object in the memory according to the header field;

the storage module 503 stores the full amount of data into the entity object, so as to extract the data in the database into the memory.

an interception module (not shown in the figure) for intercepting a database query request for querying a target data table in a database;

a conversion module (not shown in the figure), which analyzes the query condition from the database query request, and converts the database query request into a memory query request corresponding to the memory according to the query condition;

and a query module (not shown) for performing data query in the memory according to the memory query request.

one or more processors;

a storage device for storing one or more programs,

Fig. 6 shows an exemplary system architecture 600 to which the data management method or data management apparatus of an embodiment of the present invention may be applied.

As shown in fig. 6, the system architecture 600 may include

terminal devices

601, 602, 603, a network 604, and a server 605. The network 604 serves to provide a medium for communication links between the

terminal devices

601, 602, 603 and the server 605. Network 604 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.

A user may use the

terminal devices

601, 602, 603 to interact with the server 605 via the network 604 to receive or send messages or the like. The

terminal devices

601, 602, 603 may have installed thereon various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 605 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the

terminal devices

601, 602, 603. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.

It should be noted that the data management method provided by the embodiment of the present invention is generally executed by the server 605, and accordingly, the data management apparatus is generally disposed in the server 605.

It should be understood that the number of terminal devices, networks, and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 701.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor comprising: the analysis module is used for carrying out full-field analysis on a target data table in the database and determining a header field of the target data table; performing full data query in the target data table according to the header field of the table to obtain full data; the creating module creates an entity object in the memory according to the header field; and the storage module stores the full data into the entity object so as to extract the data in the database into the memory. The names of these modules do not in some cases form a limitation on the module itself, for example, a parsing module may also be described as a "module that creates an entity object in memory from the header field".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: performing full-field analysis on a target data table in a database, and determining a header field of the target data table; creating an entity object in the memory according to the header field of the table, and performing full data query in the target data table to obtain full data; and storing the full data into the entity object so as to realize the extraction of the data in the database into the memory.

According to the technical scheme of the embodiment of the invention, the entity object is created in the memory according to the header field of the target data table in the database, and then the full data of the target data table is stored in the entity object, so that the data in the database can be extracted into the memory, and the utilization rate of the memory is improved. Because the memory stores the full data in the target data table, the data does not need to be inquired from the database, and the inquiry efficiency can be greatly improved. Intercepting a database query request for querying a target data table in a database, converting the database query request into a memory query request corresponding to a memory according to the analyzed query condition, so that the data in the memory is transparent, the data in the memory can be obtained without additionally processing the database query request, and the query efficiency can be further improved by nearly applying a closed loop through only one-time query.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for managing data, comprising:

creating an entity object in a memory according to the header field, and performing full data query in the target data table to obtain full data;

and storing the full data into the entity object so as to extract the data in the database into the memory.

2. The method of claim 1, wherein user configuration information is obtained at application initialization, the target data table being determined based on the user configuration information; wherein the user configuration information comprises: database domain name or IP address, database name, table name.

3. The method of claim 1, further comprising: inquiring data in the memory according to the following steps:

and performing data query in the memory according to the memory query request.

4. The method of claim 3, further comprising: presetting a key field of each target data table; when the key field is contained in the query request, judging that the query request is a database query request aiming at the target table.

5. A data management apparatus, comprising:

the analysis module is used for carrying out full-field analysis on a target data table in a database and determining a header field of the target data table; performing full data query in the target data table according to the table header field to obtain full data;

and the storage module is used for storing the full data into the entity object so as to extract the data in the database into the memory.

6. The apparatus of claim 5, wherein the parsing module is further to: acquiring user configuration information during initialization of an application program, and determining the target data table according to the user configuration information; wherein the user configuration information comprises: database domain name or IP address, database name, table name.

7. The apparatus of claim 5, further comprising:

the conversion module is used for analyzing the query conditions from the database query request and converting the database query request into a memory query request of a corresponding memory according to the query conditions;

8. The apparatus of claim 7, wherein the interception module is further to: presetting a key field of each target data table; when the key field is contained in the query request, judging that the query request is a database query request aiming at the target table.

9. An electronic device for data management, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-4.

10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-4.