CN107943846B - Data processing method and device and electronic equipment - Google Patents
Data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN107943846B CN107943846B CN201711058820.6A CN201711058820A CN107943846B CN 107943846 B CN107943846 B CN 107943846B CN 201711058820 A CN201711058820 A CN 201711058820A CN 107943846 B CN107943846 B CN 107943846B
- Authority
- CN
- China
- Prior art keywords
- data
- data table
- processed
- attribute information
- record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data processing method and device and electronic equipment. The method comprises the following steps: receiving an sql request sent by an agent interface, and determining a data table to be processed and data attribute information corresponding to the data table to be processed according to the sql request; searching whether a loading record of the data table to be processed exists in a pre-established data table loading record library; if yes, determining whether the data attribute information exists in a loading record of the data table to be processed; and if so, processing the data table to be processed according to the sql request, and returning a processing result. The method adopts a plurality of distributed memory databases to process data, records the loaded data through the data table loading record library, so that the same data does not need to be loaded for the second time in the memory databases, the loaded data volume is reduced, the data processing efficiency is improved, the data processing speed is improved, and the load pressure loaded by the server is reduced.
Description
Technical Field
The present invention relates to the field of computer communications, and in particular, to a data processing method and apparatus, and an electronic device.
Background
With the continuous development of the power industry and the continuous increase of the business volume, a large amount of data is generated and needs to be calculated and analyzed by related personnel of an enterprise. However, in the process of implementing the present invention, the inventor finds that at least the following problems exist in the related art, because the data volume related to the power assets inside an enterprise is huge and the calculation formula is complex, when the existing database is used for data processing, a series of steps of accessing the database, extracting a large amount of related data and generating a report are required, and these steps all depend on the hard disk to access the data, the data processing speed is greatly reduced due to data transmission among the hard disk, the memory and the CPU, the time consumed in the whole process is long, the enterprise demand cannot be responded in time, and the data processing efficiency is low.
Disclosure of Invention
The data processing method, the data processing device and the electronic equipment provided by the embodiment of the invention are used for at least solving the problems in the prior art.
A first aspect of an embodiment of the present invention provides a data processing method, including:
receiving an sql request sent by an agent interface, and determining a data table to be processed and data attribute information corresponding to the data table to be processed according to the sql request, wherein the agent interface stores the corresponding relation between each data table and a memory database; searching whether a loading record of the data table to be processed exists in a pre-established data table loading record library; if the loading record of the data table to be processed exists, determining whether the data attribute information exists in the loading record of the data table to be processed; and if the data attribute information exists in the loading record of the data table to be processed, processing the data table to be processed according to the sql request, and returning a processing result.
Optionally, the method further comprises: if the loading record of the data table to be processed does not exist, determining data to be loaded in the data table to be processed according to the data attribute information; loading the data to be loaded in the data table to be processed from a disk database; and recording the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table recording record library.
Optionally, the method further comprises: if at least one piece of data attribute information does not exist in the loading record of the data table to be processed, determining the data to be loaded according to the at least one piece of data attribute information which does not exist in the loading record of the data table to be processed; loading the data to be loaded in the data table to be processed from a disk database; and updating the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table recording record library.
Optionally, the method further comprises: and updating the attribute information of the data table to be processed in the data table record database.
Optionally, the method further comprises: determining a data table stored in the memory database and data stored in the data table; monitoring whether the data in the data table of the disk database is updated or not; and if the data in the data table is updated, synchronizing the update to the data table stored in the memory database.
A second aspect of an embodiment of the present invention provides a data processing apparatus, including:
the system comprises a receiving module, a processing module and a memory database, wherein the receiving module is used for receiving an sql request sent by an agent interface and determining a data table to be processed and data attribute information corresponding to the data table to be processed according to the sql request, and the agent interface stores the corresponding relation between each data table and the memory database; the searching module is used for searching whether the loading record of the data table to be processed exists in a pre-established data table loading record library; a first determining module, configured to determine whether the data attribute information exists in a load record of the to-be-processed data table if the load record of the to-be-processed data table exists; and the processing module is used for processing the data table to be processed according to the sql request and returning a processing result if the data attribute information exists in the loading record of the data table to be processed.
Optionally, the apparatus further comprises: the second determining module is used for determining the data to be loaded in the data table to be processed according to the data attribute information if the loading record of the data table to be processed does not exist; the first loading module is used for loading the data to be loaded in the data table to be processed from a disk database; and the recording module is used for recording the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table recording record library.
Optionally, the apparatus further comprises: a third determining module, configured to determine, if at least one piece of data attribute information does not exist in a load record of the to-be-processed data table, the to-be-loaded data according to the at least one piece of data attribute information that does not exist in the load record of the to-be-processed data table; the second loading module is used for loading the data to be loaded in the data table to be processed from the disk database; and the first updating module is used for updating the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table record database.
Optionally, the apparatus further comprises: and the second updating module is used for updating the attribute information of the data table to be processed in the data table record database.
Optionally, the apparatus further comprises: a fourth determining module, configured to determine a data table stored in the memory database and data stored in the data table; the monitoring module is used for monitoring whether the data in the data table of the disk database is updated or not; and the third updating module is used for synchronizing the update to the data table stored in the memory database if the data in the data table is updated.
A third aspect of an embodiment of the present invention provides an electronic device, including: at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to cause the at least one processor to perform a data processing method according to the first aspect of the embodiments of the present invention.
According to the technical solutions above, the data processing method, the data processing apparatus and the electronic device provided by the embodiments of the present invention,
drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present invention, and it is also possible for a person skilled in the art to obtain other drawings based on the drawings.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flowchart of a data processing method according to a second embodiment of the present invention;
FIG. 3 is a block diagram of a data processing apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic diagram of a hardware structure of an electronic device executing a data processing method according to a first embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments of the present invention shall fall within the scope of the protection of the embodiments of the present invention.
Conventional database management systems manage all data on disk and are therefore referred to as disk databases. The disk database needs to frequently access the disk to perform data operations, and since the operations of reading and writing data on the disk are performed by mechanical movement of the magnetic head on one hand and affected by system call time on the other hand, when the data volume is large and the operations are frequent and complicated, many problems are exposed. The principle of the memory database is that the data is put in the memory to be directly operated, so that the CPU period and the memory can be more effectively used to improve the processing speed. Compared with a magnetic disk, the data read-write speed of the memory is higher by several orders of magnitude, and the application performance can be greatly improved by storing data in the memory compared with accessing from the magnetic disk.
For a disk database, the actual average execution time of transactions is greatly different from the estimated worst-case execution time due to disk access, data transfer of internal and external memories, buffer management, queue waiting, lock delay and the like, and if the whole database or the main 'working' part of the database is put into a memory, each transaction has no I/O (input/output) in the execution process, the system can accurately estimate and arrange the operation time of the transactions, so that the transactions have better dynamic predictability, and meanwhile, the system lays a foundation for realizing the timing limitation of the transactions.
The technical solution of the embodiment of the present invention is described in detail below.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention. As shown in fig. 1, a data processing method provided in an embodiment of the present invention is applied to a memory database, and specifically includes:
s101, receiving an sql request sent by an agent interface, and determining a data table to be processed and data attribute information corresponding to the data table to be processed according to the sql request.
The agent interface stores the corresponding relation between each data table and the memory database.
In the embodiment of the invention, the distributed storage of the memory database is realized by introducing the proxy interface so as to reduce the load pressure of the server. Generally, data is stored in a disk database in a form of a data table, a proxy interface stores in advance a corresponding relationship between a plurality of data tables in the disk database and a memory database, and after receiving an sql request from a client, the proxy interface distributes the sql request to the corresponding memory database according to the data table related to the sql request. sql (structured query language) request is a database query and programming language for querying and analyzing data, and the like.
After the memory database receives the sql request sent by the proxy interface, the corresponding to-be-processed data table is determined by analyzing the sql request, optionally, the data table identifier in the sql request can be obtained by analyzing the sql request, and the to-be-processed data table (which may be the name of the to-be-processed data table) is determined according to the data table identifier. Generally, the data in the same column in the data table generally corresponds to the same information category, each data in the same column has a respective row identifier, which information category in the data table to be processed belongs to the data to be processed and which part of data under the information category belongs to the data to be processed are determined by analyzing the processing instruction to the data table to be processed in the sql request, the information category and the row identifier are used as data attribute information of the data table to be processed, and which data in the data table to be processed is processed by the sql request, that is, the data which needs to be loaded into the memory database, is determined according to the data attribute information.
S102, searching whether the loading record of the data table to be processed exists in a pre-established data table loading record library.
Specifically, in the actual data processing process, the same data in the same data table may be referred to multiple times, so that the same data is not loaded when used next time, the loaded data amount is reduced, and the data processing efficiency is improved. Therefore, a data table loading record library is established in the memory database in advance, and the data table loading record library comprises the loading records of each data table loaded into the memory database. Optionally, the load record includes, but is not limited to, attribute information of the data table and data attribute information corresponding to the data table, where the attribute information includes, but is not limited to, a name, a load time, and a load frequency of the data table, and the data attribute information includes a column information category and a row identifier, which are used to determine data loaded in the in-memory database in the data table. The record of each data table in the data table loading record base is generally named by the name of the data table, so that when the data table is loaded, whether the loading record of the data table exists or not is searched in the data table loading record base.
Optionally, the loading records in the record library may be recorded by a data table, and the data table stored in the memory database and the data stored in each data table are determined; monitoring whether the data in the data table in the disk database is updated or not; and if the data in the data table is updated, synchronizing the update to the data table stored in the memory database. Specifically, the monitoring may be real-time monitoring or periodic monitoring, and may depend on the update frequency of each data table, which is not limited herein. Generally, data in a data table of a disk database is updated, such as modified, added, deleted, and the like, and if the in-memory database is not monitored, the stored data is expired data, so that the processing result of the data is inaccurate.
In this step, through the to-be-processed data table determined in step S101, the data table loading database may be searched according to the name of the to-be-processed data table, and it is determined whether a loading record of the to-be-processed data table exists.
S103, if the loading record of the data table to be processed exists, determining whether the data attribute information exists in the loading record of the data table to be processed.
And S104, if the data attribute information exists in the loading record of the data table to be processed, processing the data table to be processed according to the sql request, and returning a processing result.
In this embodiment, if there is a load record of the to-be-processed data table, the data attribute information in the load record is obtained, and the data attribute information in the load record can determine the data in the to-be-processed data table that is previously loaded to the memory database, that is, the data in the to-be-processed data table that is currently stored in the memory database. Specifically, the information category and the row identifier of the data in the to-be-processed data table stored in the memory database can be determined through the data attribute information of the loading record, and accordingly, whether the information category and the row identifier completely include the information category and the row identifier in the data attribute information determined in step S101 is determined, if the information category and the row identifier completely include the information category and the row identifier, it is indicated that the data in the to-be-processed data table corresponding to the current sql request is already stored in the memory database, reloading is not required, the to-be-processed data table can be directly processed according to the processing instruction in the sql request, and the processing result is transmitted through the proxy interface and returned to the client in real time for displaying. In addition, the attribute information of the data table to be processed is updated in a data table record library, and the attribute information comprises loading time, loading times and the like.
As another optional implementation manner of this embodiment, if there is no load record of the to-be-processed data table, it is described that any data in the to-be-processed data table has not been loaded into the memory database before, that is, the first load of the data currently belonging to the to-be-processed data table. Therefore, it is necessary to determine the row identifier and the column information category of the data according to the data attribute information corresponding to the to-be-processed data table determined in step S101, so as to determine the to-be-loaded data in the to-be-processed data table. And loading the data to be loaded in the data table to be processed from the disk database to the memory database, finishing the subsequent processing of the data to be loaded in the memory database based on the data to be loaded, and transmitting the processing result to the client in real time through the proxy interface for showing. In addition, a loading record of the to-be-processed data table needs to be created in the data table recording record library, and the attribute information of the to-be-processed data table and the data attribute information corresponding to the to-be-processed data table are recorded.
As another optional implementation manner of this embodiment, if a load record of the to-be-processed data table exists, but at least one piece of the data attribute information does not exist in the load record of the to-be-processed data table, specifically, as described above, if the information type and the row identifier of the data in the to-be-processed data table stored in the memory database do not completely include the information type and the row identifier in the data attribute information determined in step S101, that is, at least one information type or row identifier in the data attribute information determined in step S101 is not stored in the memory database, that is, the data in the to-be-processed data table corresponding to the current sql request is not completely stored in the memory database. And determining to-be-loaded data to be loaded into a data database according to at least one piece of data attribute information which does not exist in the loading record of the to-be-processed data table, loading the to-be-loaded data in the to-be-processed data table from a disk database to a memory database, wherein the data of the to-be-processed data table stored in the memory database completely comprises data corresponding to the current sql request, processing the to-be-processed data table directly according to a processing instruction in the sql request, and transmitting a processing result to a client through a proxy interface in real time for showing. In addition, it is also necessary to update the attribute information of the to-be-processed data table and the data attribute information corresponding to the to-be-processed data table in the data table recording record library, specifically, record the information category and the row identifier corresponding to the to-be-loaded data to the attribute information of the to-be-processed data table, and update the loading time and the loading frequency.
The embodiment of the invention provides a data processing method, a data processing device and electronic equipment, wherein a plurality of distributed memory databases are adopted for processing data, and a data table loading record base is used for recording the loaded data, so that the same data does not need to be loaded secondarily in the memory databases, the loaded data volume is reduced, the data processing efficiency is improved, the data processing speed is improved, and the load pressure loaded by a server is reduced.
Fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention. As shown in fig. 2, the data processing method provided in the second embodiment of the present invention specifically includes:
s201, receiving an sql request sent by an agent interface, and determining a data table to be processed and data attribute information corresponding to the data table to be processed according to the sql request.
The agent interface stores the corresponding relation between each data table and the memory database.
S202, searching whether the loading record of the data table to be processed exists in a pre-established data table loading record library.
If the loading record of the data table to be processed exists in the pre-established data table loading record library, executing step S203; and if the pre-established data table loading record base does not have the loading record of the data table to be processed, executing the step S204-the step S206.
S203, determining whether the data attribute information exists in the loading record of the data table to be processed.
If the data attribute information exists in the loading record of the data table to be processed, executing step S207-step S208; if at least one piece of the data attribute information does not exist in the load record of the data table to be processed, step S209-step S211 are executed.
And S204, determining the data to be loaded in the data table to be processed according to the data attribute information.
S205, loading the data to be loaded in the data table to be processed from a disk database;
and S206, recording the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table recording record library.
And S207, processing the data table to be processed according to the sql request, and returning a processing result.
And S208, updating the attribute information of the data table to be processed in the data table record library.
S209, determining the data to be loaded according to at least one piece of data attribute information which does not exist in the loading record of the data table to be processed.
S210, loading the data to be loaded in the data table to be processed from a disk database.
And S211, updating the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table recording record library.
The embodiment of the invention provides a data processing method, a data processing device and electronic equipment, wherein a plurality of distributed memory databases are adopted for processing data, and a data table loading record base is used for recording the loaded data, so that the same data does not need to be loaded secondarily in the memory databases, the loaded data volume is reduced, the data processing efficiency is improved, the data processing speed is improved, and the load pressure loaded by a server is reduced.
An application scenario of an embodiment of the present invention is described below.
The main disadvantage of the program for calculating the depreciation of the electric fixed assets by adopting the conventional ORACLE database is that the program directly accesses the ORACLE database to process a large amount of data and generate related reports, the performance of the application program is not high, the processing time of the waiting program is about 30 minutes, and the processing time of the waiting program is likely to be further increased along with the increase of the asset data of the enterprise. Therefore, in the embodiment of the present invention, the SAP system database is used to manage and process data, it is determined whether data in the disk database needs to be loaded to the in-memory database based on the method provided above, if so, a required data table is loaded to the in-memory database through an slt (SAP LT replication server) trigger (a platform responsible for data loading and constructed based on an ORACLE native trigger mechanism), and the in-memory database uses a cluster scheme to reduce the load pressure of a server. An hsql program (a memory database programming language) is compiled based on a memory database to analyze an sql request to determine a processing instruction, the data are processed according to the processing instruction, and a calculation result is returned in real time and is provided to a front end fixed asset depreciation report application program for displaying through interface transmission, so that the time for processing the data of the fixed assets and the accumulated depreciation report is reduced, and the data processing efficiency is improved.
Fig. 3 is a structural diagram of a data processing apparatus according to a third embodiment of the present invention. As shown in fig. 3, the apparatus specifically includes: a receiving module 310, a searching module 320, a first determining module 330 and a processing module 340.
The receiving module 310 is configured to receive an sql request sent by an agent interface, and determine a to-be-processed data table and data attribute information corresponding to the to-be-processed data table according to the sql request, where a corresponding relationship between each data table and a memory database is stored in the agent interface; the searching module 320 is configured to search whether a loading record of the to-be-processed data table exists in a pre-established data table loading record library; the first determining module 330 is configured to determine whether the data attribute information exists in the load record of the to-be-processed data table if the load record of the to-be-processed data table exists; the processing module 340 is configured to, if the data attribute information exists in the load record of the to-be-processed data table, process the to-be-processed data table according to the sql request, and return a processing result.
Optionally, the apparatus further comprises: the second determining module is used for determining the data to be loaded in the data table to be processed according to the data attribute information if the loading record of the data table to be processed does not exist; the first loading module is used for loading the data to be loaded in the data table to be processed from a disk database; and the recording module is used for recording the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table recording record library.
Optionally, the apparatus further comprises: a third determining module, configured to determine, if at least one piece of data attribute information does not exist in a load record of the to-be-processed data table, the to-be-loaded data according to the at least one piece of data attribute information that does not exist in the load record of the to-be-processed data table; the second loading module is used for loading the data to be loaded in the data table to be processed from the disk database; and the first updating module is used for updating the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table record database.
Optionally, the apparatus further comprises: and the second updating module is used for updating the attribute information of the data table to be processed in the data table record database.
Optionally, the apparatus further comprises: a fourth determining module, configured to determine a data table stored in the memory database and data stored in the data table; the monitoring module is used for monitoring whether the data in the data table of the disk database is updated or not; and the third updating module is used for synchronizing the update to the data table stored in the memory database if the data in the data table is updated.
The data processing apparatus provided in the third embodiment of the present invention is specifically configured to execute the method provided in the first embodiment and/or the second embodiment, and the implementation principle, the method, and the functional use thereof are similar to those in the first embodiment and/or the second embodiment, and are not described herein again.
Fig. 4 is a schematic diagram of a hardware structure of some electronic devices that execute a data processing method according to an embodiment of the present invention. As shown in fig. 4, the electronic device includes:
one or more processors 410 and a memory 420, with one processor 410 being an example in fig. 4.
The apparatus for performing the data processing method may further include: an input device 430 and an output device 430.
The processor 410, the memory 420, the input device 430, and the output device 440 may be connected by a bus or other means, such as the bus connection in fig. 4.
The memory 420, which is a non-volatile computer-readable storage medium, may be used for storing non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the data processing methods in the embodiments of the present application. The processor 410 performs various functional applications of the server and data processing, i.e., implements the data processing method, by executing nonvolatile software programs, instructions, and modules stored in the memory 420.
The memory 420 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by use of the data processing apparatus provided according to the third embodiment of the present invention, and the like. Further, the memory 420 may include high speed random access memory 420, and may also include non-volatile memory 420, such as at least one piece of disk memory 420, flash memory device, or other non-volatile solid state memory 420. In some embodiments, memory 420 may optionally include memory 420 located remotely from processor 410, and such remote memory 420 may be connected to the data processing apparatus via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 430 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the data processing device. The input device 430 may include a pressing module or the like.
The one or more modules are stored in the memory 420 and, when executed by the one or more processors 410, perform the data processing method.
The product can execute the method provided by the corresponding embodiment of the application, and has the corresponding functional module and the beneficial effect of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the methods provided in the corresponding embodiments of the present application.
The electronic device of the embodiments of the present application exists in various forms, including but not limited to:
(1) mobile communication devices, which are characterized by mobile communication capabilities and are primarily targeted at providing voice and data communications. Such terminals include smart phones (e.g., iphones), multimedia phones, functional phones, and low-end phones, among others.
(2) The ultra-mobile personal computer equipment belongs to the category of personal computers, has calculation and processing functions and generally has the characteristic of mobile internet access. Such terminals include PDA, MID, and UMPC devices, such as ipads.
(3) Portable entertainment devices such devices may display and play multimedia content. Such devices include audio, panoramic video players (e.g., ipods), handheld game consoles, electronic books, as well as smart toys and portable car navigation devices.
(4) The server is similar to a general computer architecture, but has higher requirements on processing capability, stability, reliability, safety, expandability, manageability and the like because of the need of providing highly reliable services.
(5) And other electronic devices with data interaction functions.
The embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer-executable instructions, wherein when the computer-executable instructions are executed by an electronic device, the electronic device is caused to execute the data processing method in any method embodiment described above.
Embodiments of the present invention provide a computer program product, wherein the computer program product comprises a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, wherein the program instructions, when executed by an electronic device, cause the electronic device to perform the data processing method in any of the above method embodiments.
The above-described embodiments of the apparatus are merely illustrative, wherein the modules described as separate parts may or may not be physically separate, and the parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions and/or portions thereof that contribute to the prior art may be embodied in the form of a software product that can be stored on a computer-readable storage medium including any mechanism for storing or transmitting information in a form readable by a computer (e.g., a computer). For example, a machine-readable medium includes Read Only Memory (ROM), Random Access Memory (RAM), magnetic disk storage media, optical storage media, flash memory storage media, electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others, and the computer software product includes instructions for causing a computing device (which may be a personal computer, server, or network device, etc.) to perform the methods described in the various embodiments or portions of the embodiments.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the embodiments of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (4)
1. A data processing method is applied to a memory database, and is characterized by comprising the following steps: receiving an sql request sent by an agent interface, and determining a data table to be processed and data attribute information corresponding to the data table to be processed according to the sql request, wherein the agent interface stores the corresponding relation between each data table and a memory database, and the memory database is a distributed memory database;
searching whether a loading record of the data table to be processed exists in a pre-established data table loading record library;
if the loading record of the data table to be processed exists, determining whether the data attribute information exists in the loading record of the data table to be processed;
if the data attribute information exists in the loading record of the data table to be processed, processing the data table to be processed according to the sql request, and returning a processing result;
if the loading record of the data table to be processed does not exist, determining data to be loaded in the data table to be processed according to the data attribute information;
loading the data to be loaded in the data table to be processed from a disk database;
recording attribute information of the data table to be processed and data attribute information corresponding to the data table to be processed in the data table recording record library;
if at least one piece of data attribute information does not exist in the loading record of the data table to be processed, determining the data to be loaded according to the at least one piece of data attribute information which does not exist in the loading record of the data table to be processed;
loading the data to be loaded in the data table to be processed from a disk database;
updating the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table recording record library;
the data attribute information comprises column information categories and row identifiers and is used for determining data loaded into the memory database in the data table;
further comprising: determining a data table stored in the memory database and data stored in the data table;
monitoring whether the data in the data table of the disk database is updated or not;
and if the data in the data table is updated, synchronizing the update to the data table stored in the memory database.
2. The method of claim 1, further comprising:
and updating the attribute information of the data table to be processed in the data table record database.
3. A data processing device applied to an in-memory database is characterized by comprising:
the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving an sql request sent by an agent interface and determining a data table to be processed and data attribute information corresponding to the data table to be processed according to the sql request, the agent interface stores the corresponding relation between each data table and a memory database, and the memory database is a distributed memory database;
the searching module is used for searching whether the loading record of the data table to be processed exists in a pre-established data table loading record library;
a first determining module, configured to determine whether the data attribute information exists in a load record of the to-be-processed data table if the load record of the to-be-processed data table exists;
the processing module is used for processing the data table to be processed according to the sql request and returning a processing result if the data attribute information exists in the loading record of the data table to be processed;
the second determining module is used for determining the data to be loaded in the data table to be processed according to the data attribute information if the loading record of the data table to be processed does not exist;
the first loading module is used for loading the data to be loaded in the data table to be processed from a disk database;
the recording module is used for recording the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table recording record library;
a third determining module, configured to determine, if at least one piece of data attribute information does not exist in a load record of the to-be-processed data table, the to-be-loaded data according to the at least one piece of data attribute information that does not exist in the load record of the to-be-processed data table;
the second loading module is used for loading the data to be loaded in the data table to be processed from the disk database;
the first updating module is used for updating the attribute information of the data table to be processed and the data attribute information corresponding to the data table to be processed in the data table recording record library;
the data attribute information comprises column information categories and row identifiers and is used for determining data loaded into the memory database in the data table;
the device further comprises:
a fourth determining module, configured to determine a data table stored in the memory database and data stored in the data table;
the monitoring module is used for monitoring whether the data in the data table of the disk database is updated or not;
and the third updating module is used for synchronizing the update to the data table stored in the memory database if the data in the data table is updated.
4. The apparatus of claim 3, further comprising:
and the second updating module is used for updating the attribute information of the data table to be processed in the data table record database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711058820.6A CN107943846B (en) | 2017-11-01 | 2017-11-01 | Data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711058820.6A CN107943846B (en) | 2017-11-01 | 2017-11-01 | Data processing method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107943846A CN107943846A (en) | 2018-04-20 |
| CN107943846B true CN107943846B (en) | 2021-05-11 |
Family
ID=61933200
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711058820.6A Active CN107943846B (en) | 2017-11-01 | 2017-11-01 | Data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107943846B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109344142B (en) * | 2018-08-22 | 2023-05-30 | 中国平安人寿保险股份有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN109766362B (en) * | 2018-12-28 | 2021-07-20 | 北京字节跳动网络技术有限公司 | Data processing method and device |
| CN111797063A (en) * | 2020-06-28 | 2020-10-20 | 中孚信息股份有限公司 | Streaming data processing method and system |
| CN111813783B (en) * | 2020-07-27 | 2024-03-26 | 南方电网数字电网研究院有限公司 | Data processing method, device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102937957A (en) * | 2012-06-29 | 2013-02-20 | 华为技术有限公司 | Processing method and processing device for data access and server |
| CN103218416A (en) * | 2013-03-27 | 2013-07-24 | 华为技术有限公司 | Method, device and system for loading database |
| CN103853727A (en) * | 2012-11-29 | 2014-06-11 | 深圳中兴力维技术有限公司 | Method and system for improving large data volume query performance |
| CN105989049A (en) * | 2015-02-05 | 2016-10-05 | 天脉聚源（北京）科技有限公司 | Data middle layer realizing method and system |
-
2017
- 2017-11-01 CN CN201711058820.6A patent/CN107943846B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102937957A (en) * | 2012-06-29 | 2013-02-20 | 华为技术有限公司 | Processing method and processing device for data access and server |
| CN103853727A (en) * | 2012-11-29 | 2014-06-11 | 深圳中兴力维技术有限公司 | Method and system for improving large data volume query performance |
| CN103218416A (en) * | 2013-03-27 | 2013-07-24 | 华为技术有限公司 | Method, device and system for loading database |
| CN105989049A (en) * | 2015-02-05 | 2016-10-05 | 天脉聚源（北京）科技有限公司 | Data middle layer realizing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107943846A (en) | 2018-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107943846B (en) | Data processing method and device and electronic equipment | |
| US20170364697A1 (en) | Data interworking method and data interworking device | |
| CN111352902A (en) | Log processing method and device, terminal equipment and storage medium | |
| US20160147778A1 (en) | Applying a database transaction log record directly to a database table container | |
| US20140149527A1 (en) | Slave Side Transaction ID Buffering for Efficient Distributed Transaction Management | |
| CN109471851B (en) | Data processing method, device, server and storage medium | |
| CN113535721B (en) | Data writing method and device | |
| US9292341B2 (en) | RPC acceleration based on previously memorized flows | |
| CN110704194A (en) | Method and system for managing memory data and maintaining data in memory | |
| WO2021068113A1 (en) | Method and apparatus for compiling duration statistics, electronic device, and computer-readable medium | |
| CN109947729A (en) | A kind of real-time data analysis method and device | |
| CN110990365A (en) | Data synchronization method, device, server and storage medium | |
| US9229969B2 (en) | Management of searches in a database system | |
| CN114840565A (en) | Sampling query method, device, electronic equipment and computer readable storage medium | |
| US20170169044A1 (en) | Property retrieval apparatus, method and system | |
| CN111488386A (en) | Data query method and device | |
| CN107016296B (en) | Data index construction method, data index reading method, data index construction device, data reading device and electronic equipment | |
| CN112631517B (en) | Data storage method and device, electronic equipment and storage medium | |
| CN114417112A (en) | Data processing method, data processing apparatus, electronic device, and storage medium | |
| CN111221817B (en) | Service information data storage method, device, computer equipment and storage medium | |
| CN109543079B (en) | Data query method and device, computing equipment and storage medium | |
| CA3048876C (en) | Retroreflective join graph generation for relational database queries | |
| CN113268483A (en) | Request processing method and device, electronic equipment and storage medium | |
| CN112667949A (en) | Data processing method and device for front-end gateway | |
| CN111161047A (en) | Bank business data processing and inquiring method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |