CN112579678A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN112579678A CN112579678A CN202011438493.9A CN202011438493A CN112579678A CN 112579678 A CN112579678 A CN 112579678A CN 202011438493 A CN202011438493 A CN 202011438493A CN 112579678 A CN112579678 A CN 112579678A
- Authority
- CN
- China
- Prior art keywords
- database
- memory
- operation request
- data
- external memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 230000007246 mechanism Effects 0.000 claims abstract description 30
- 238000000034 method Methods 0.000 claims description 26
- 238000011084 recovery Methods 0.000 claims description 13
- 230000001360 synchronised effect Effects 0.000 claims description 13
- 230000000903 blocking effect Effects 0.000 claims description 12
- 239000008186 active pharmaceutical agent Substances 0.000 claims description 10
- 238000001514 detection method Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method and a data processing device, wherein a database is respectively established in a memory and an external memory of a terminal device, the data is the same in an initial state, and under the condition of receiving a database operation request, an SQLite engine is called to execute the operation corresponding to the database operation request on the database in the memory, so that the reading and writing speed of the database is improved. Meanwhile, the data of the database in the internal memory and the data of the database in the external memory are consistent through an asynchronous operation preset synchronization mechanism, and even if the data in the database in the internal memory is lost due to reasons such as power failure, the data of the database in the external memory cannot be lost and is not influenced, so that the safety and the stability of the operation of the database are ensured.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
The SQLite engine is widely used in terminal devices due to its characteristics of light weight, rapidness, high reliability and complete functions.
Currently, a database file is generally established in an external memory in a terminal device, so that an SQLite engine is used to perform read and write operations on the database in the external memory. However, the read-write operation of the database in the external memory is limited by the read-write speed of the external memory, which is slow and poor in user experience.
Disclosure of Invention
In view of this, the present invention provides a data processing method, apparatus and system, which can improve the database read/write speed and ensure the security and stability of the database operation.
In order to achieve the above purpose, the invention provides the following specific technical scheme:
a data processing method is applied to a processor in terminal equipment, wherein the data of a memory and a database in an external memory of the terminal equipment are the same in an initial state, and the method comprises the following steps:
under the condition of receiving a database operation request, calling an SQLite engine to execute an operation corresponding to the database operation request on a database in a memory;
and asynchronously operating a preset synchronous mechanism to make the data of the memory consistent with the data of the database in the external memory.
Optionally, the asynchronously operating preset synchronization mechanism includes:
judging whether the database operation request is a query operation;
if so, not executing the operation corresponding to the database operation request on the database in the external memory;
if not, judging whether the operation corresponding to the database operation request is correctly executed in the database in the memory;
if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into an SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the database in the external memory.
Optionally, in a case of a database execution error in the memory of an operation corresponding to the database operation request, the method further includes:
detecting whether a database in a memory or an external memory is damaged;
if the database in the memory is damaged, recovering the database in the memory according to the data of the database in the external memory;
if the database in the external memory is damaged, the database in the external memory is recovered according to the data of the database in the memory.
Optionally, the terminal device receives the database operation request by providing a set of interfaces to the outside, where the interfaces may be APIs and/or services.
Optionally, the restoring the database in the memory according to the data of the database in the external storage includes:
blocking the interface call;
discarding transactions that cause database corruption in memory;
under the condition that all the database operation requests in the SQL queue are executed, loading the data of the database in the external memory to the database in the internal memory;
and releasing the interface call.
Optionally, the recovering the database in the external storage according to the data of the database in the memory includes:
blocking the interface call;
emptying the SQL queue;
newly building a database in an external memory;
loading the data of the database in the memory to a newly built database in an external memory;
deleting the damaged database in the external memory;
and releasing the interface call.
Optionally, in a case that a storage space of the memory is insufficient, the method further includes:
the database in the memory only stores the core data;
and asynchronously operating a preset synchronization mechanism to make the core data of the database in the memory and the external memory consistent.
Optionally, the asynchronously operating preset synchronization mechanism includes:
under the condition of receiving the database operation request, judging whether the database operation request is a query operation;
if the operation is the query operation, judging whether the data corresponding to the database operation request is loaded to a database in the memory;
if the operation request is not loaded to the database in the memory, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory;
if the operation request is loaded to the database in the memory, under the condition that the database operation request in the SQL queue is completely executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the memory;
if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory;
judging whether the operation corresponding to the database operation request is executed correctly by the database in the external memory;
if the operation is correctly executed, judging whether the data corresponding to the database operation request is core data;
if the data is core data, under the condition that the transaction corresponding to the database operation request is completed, adding the database operation request into the SQL queue, and asynchronously calling the SQLite engine to execute the operation corresponding to the database operation request in the SQL queue on the database in the memory.
Optionally, in a case that a storage space of the memory is insufficient, the method further includes:
deleting the database in the memory;
newly building a backup database in an external memory, and loading data in an original database in the external memory into the backup database;
and asynchronously operating a preset synchronous mechanism to make the data of the original database consistent with the data of the backup database.
Optionally, the asynchronously operating preset synchronization mechanism includes:
under the condition of receiving the database operation request, judging whether the database operation request is a query operation;
if the operation is the query operation, under the condition that all the database operation requests in the SQL queue are executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the original database;
if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the backup database;
judging whether the operation corresponding to the database operation request is correctly executed in the backup database;
if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into the SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the original database.
A data processing device is applied to a processor in a terminal device, wherein the memory of the terminal device is the same as the data of a database in an external memory in an initial state, and the device comprises:
the memory database operation unit is used for calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the memory under the condition of receiving the database operation request;
and the synchronous processing unit is used for asynchronously operating the preset synchronous mechanism to ensure that the data of the database in the memory and the external memory are consistent.
Optionally, the synchronization processing unit is specifically configured to:
judging whether the database operation request is a query operation;
if so, not executing the operation corresponding to the database operation request on the database in the external memory;
if not, judging whether the operation corresponding to the database operation request is correctly executed in the database in the memory;
if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into an SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the database in the external memory.
Optionally, the apparatus further comprises:
a failure detection unit, configured to detect whether a database in the memory or the external storage is damaged in a case where a database execution error in the memory of an operation corresponding to the database operation request occurs;
the memory database recovery unit is used for recovering the database in the memory according to the data of the database in the external memory;
and the external storage database recovery unit is used for recovering the database in the external storage according to the data of the database in the memory.
Optionally, the terminal device receives the database operation request by providing a set of interfaces to the outside, where the interfaces may be APIs and/or services, and the memory database recovery unit is specifically configured to:
blocking the interface call;
discarding transactions that cause database corruption in memory;
under the condition that all the database operation requests in the SQL queue are executed, loading the data of the database in the external memory to the database in the internal memory;
and releasing the interface call.
Optionally, the terminal device receives the database operation request by providing a set of interfaces to the outside, where the interfaces may be APIs and/or services, and the external storage database recovery unit is specifically configured to:
blocking the interface call;
emptying the SQL queue;
newly building a database in an external memory;
loading the data of the database in the memory to a newly built database in an external memory;
deleting the damaged database in the external memory;
and releasing the interface call.
Alternatively to this, the first and second parts may,
the memory database operation unit is also used for storing only core data in the database in the memory under the condition that the storage space of the memory is insufficient;
the synchronization processing unit is further configured to:
under the condition of receiving the database operation request, judging whether the database operation request is a query operation;
if the operation is the query operation, judging whether the data corresponding to the database operation request is loaded to a database in the memory;
if the operation request is not loaded to the database in the memory, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory;
if the operation request is loaded to the database in the memory, under the condition that the database operation request in the SQL queue is completely executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the memory;
if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory;
judging whether the operation corresponding to the database operation request is executed correctly by the database in the external memory;
if the operation is correctly executed, judging whether the data corresponding to the database operation request is core data;
if the data is core data, under the condition that the transaction corresponding to the database operation request is completed, adding the database operation request into the SQL queue, and asynchronously calling the SQLite engine to execute the operation corresponding to the database operation request in the SQL queue on the database in the memory.
Alternatively to this, the first and second parts may,
the memory database operation unit is also used for deleting the database in the memory under the condition that the storage space of the memory is insufficient;
the synchronization processing unit is further configured to:
newly building a backup database in an external memory, and loading data in an original database in the external memory into the backup database;
under the condition of receiving the database operation request, judging whether the database operation request is a query operation;
if the operation is the query operation, under the condition that all the database operation requests in the SQL queue are executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the original database;
if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the backup database;
judging whether the operation corresponding to the database operation request is correctly executed in the backup database;
if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into the SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the original database.
Compared with the prior art, the invention has the following beneficial effects:
the invention discloses a data processing method, which is characterized in that a database is respectively established in a memory and an external memory of terminal equipment, the data is the same in an initial state, and under the condition of receiving a database operation request, an SQLite engine is called to execute the operation corresponding to the database operation request on the database in the memory, so that the reading and writing speed of the database is improved. Meanwhile, the data of the database in the internal memory and the data of the database in the external memory are consistent through an asynchronous operation preset synchronization mechanism, and even if the data in the database in the internal memory is lost due to reasons such as power failure, the data of the database in the external memory cannot be lost and is not influenced, so that the safety and the stability of the operation of the database are ensured.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic flow chart illustrating a data processing method according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a synchronization mechanism according to an embodiment of the present invention;
FIG. 3 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating another exemplary synchronization mechanism disclosed in the present invention;
FIG. 5 is a diagram illustrating another exemplary synchronization mechanism disclosed in the present invention;
fig. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention utilizes the characteristic that the SQLite engine has high speed of reading and writing the database in the memory to establish a database in the memory of the terminal equipment, all database operation requests are executed in the database in the memory, the reading and writing speed of the database is improved, and in order to solve the problem that the data in the database in the memory is lost due to power failure and the like, a database is also established in the external memory of the terminal equipment, such as embedded equipment, PC and the like, the data of the memory and the database in the external memory are the same in the initial state of the terminal equipment, the data of the database in the memory and the external memory are consistent through an asynchronous operation preset synchronization mechanism, the data recovery when the data are damaged is realized, and the safety and the stability of the database operation are ensured.
Specifically, the data processing method disclosed in this embodiment is applied to a processor in a terminal device, and the data of a database in a memory and the data of a database in an external memory of the terminal device are the same in an initial state, please refer to fig. 1, and the data processing method disclosed in this embodiment includes the following steps:
s101: under the condition of receiving a database operation request, calling an SQLite engine to execute an operation corresponding to the database operation request on a database in a memory;
the data operation request can be a query type operation or a non-query type operation.
S102: and asynchronously operating a preset synchronous mechanism to make the data of the memory consistent with the data of the database in the external memory.
The preset synchronization mechanism may be: and when the operation corresponding to the database operation request is correctly executed in the database in the internal memory and the transaction corresponding to the database operation request is completed, asynchronously executing the same database operation in the database in the external memory to ensure that the data of the internal memory and the data of the database in the external memory are consistent.
Since the query operation does not change the data in the database, in order to reduce the resource occupation and improve the data processing efficiency, please refer to fig. 2, the preset synchronization mechanism may also be: if the database operation request is a query operation, not executing the operation corresponding to the database operation request on the database in the external memory; if the database operation request is a non-query operation, the database operation of the transaction is asynchronously executed in the database in the external memory under the condition that the operation corresponding to the database operation request is correctly executed in the database in the internal memory and the transaction corresponding to the database operation request is completed, so that the data of the database in the internal memory and the data of the database in the external memory are consistent.
Specifically, referring to fig. 3, based on the second synchronization mechanism, the data processing method disclosed in this embodiment is as follows:
s201: under the condition of receiving a database operation request, calling an SQLite engine to execute an operation corresponding to the database operation request on a database in a memory;
s202: judging whether the database operation request is a query operation;
if yes, S203: not performing an operation corresponding to the database operation request on the database in the external memory;
if not, S204: judging whether the operation corresponding to the database operation request is correctly executed in the database in the memory;
if the execution is correct, S205: under the condition that the transaction corresponding to the database operation request is completed, adding the database operation request into an SQL queue, and asynchronously calling an SQLite engine to execute the operation corresponding to the database operation request in the SQL queue on the database in the external memory;
it should be noted that, a transaction may include multiple database operation requests, and in a case that the database in the internal memory of the transaction is executed correctly, multiple non-query-class database operation requests corresponding to the transaction are sequentially added to the SQL queue and are executed asynchronously in the database in the external memory in a first-in first-out order.
If the execution is wrong, S206: detecting whether a database in a memory or an external memory is damaged;
for example, whether the database in the memory or the external storage is damaged can be detected by whether the system table such as the sqlite _ master or the specific service data table can be correctly queried.
S207: if the database in the memory is damaged, recovering the database in the memory according to the data of the database in the external memory;
s208: if the database in the external memory is damaged, the database in the external memory is recovered according to the data of the database in the memory.
The terminal equipment receives the database operation request by providing a group of interfaces to the outside, the external interfaces can be APIs and/or services, and the application is easy to use by providing a group of APIs which are the same as or similar to the SQLite engine to the outside; by providing a set of services for receiving a request from a network or a local socket, a storage service can be provided for the outside like a multi-user database such as MySQL.
Based on this, the above S207: restoring the database in the memory according to the data of the database in the external memory, comprising:
blocking interface calls;
discarding transactions that cause database corruption in memory;
under the condition that all database operation requests in the SQL queue are executed, loading the data of the database in the external memory to the database in the internal memory;
the interface call is released.
The above S208: restoring the database in the external memory according to the data of the database in the memory, comprising:
blocking interface calls;
emptying the SQL queue;
newly building a database in an external memory;
loading the data of the database in the memory to a newly built database in an external memory;
deleting the damaged database in the external memory;
the interface call is released.
Further, under the condition that the storage space of the memory is insufficient, only the core data can be stored in the database in the memory, and the core data of the database in the memory and the core data of the database in the external memory are consistent through the asynchronous operation preset synchronization mechanism.
Specifically, referring to fig. 4, in the case of receiving the database operation request, the preset synchronization mechanism determines whether the database operation request is a query operation; if the operation is the query operation, judging whether the data corresponding to the database operation request is loaded to a database in the memory; if the operation request is not loaded to the database in the memory, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory; if the operation request is loaded to the database in the memory, under the condition that the database operation request in the SQL queue is completely executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the memory; if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory; judging whether the operation corresponding to the database operation request is executed correctly by the database in the external memory; if the operation is correctly executed, judging whether the data corresponding to the database operation request is core data; if the data is core data, under the condition that the transaction corresponding to the database operation request is completed, adding the database operation request into the SQL queue, and asynchronously calling the SQLite engine to execute the operation corresponding to the database operation request in the SQL queue on the database in the memory.
Further, under the condition that the storage space of the internal memory is insufficient, the database in the internal memory can be deleted, a backup database is newly built in the external memory, the data in the original database in the external memory is loaded into the backup database, and a preset synchronous mechanism is asynchronously operated, so that the data of the original database is consistent with the data of the backup database.
Specifically, referring to fig. 5, in the case of receiving the database operation request, the preset synchronization mechanism determines whether the database operation request is a query operation; if the operation is the query operation, under the condition that all the database operation requests in the SQL queue are executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the original database; if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the backup database; judging whether the operation corresponding to the database operation request is correctly executed in the backup database; if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into the SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the original database.
Based on the data processing method disclosed in the foregoing embodiment, this embodiment correspondingly discloses a data processing apparatus, which is applied to a processor in a terminal device, where data of a memory of the terminal device in an initial state is the same as data of a database in an external memory, please refer to fig. 6, and the apparatus includes:
the memory database operation unit 100 is configured to, in a case that a database operation request is received, invoke an SQLite engine to perform an operation corresponding to the database operation request on a database in a memory;
the synchronous processing unit 200 is configured to asynchronously run a preset synchronous mechanism, so that the data in the memory and the data in the database in the external memory are consistent.
Optionally, the synchronous processing unit 200 is specifically configured to:
judging whether the database operation request is a query operation;
if so, not executing the operation corresponding to the database operation request on the database in the external memory;
if not, judging whether the operation corresponding to the database operation request is correctly executed in the database in the memory;
if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into an SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the database in the external memory.
Optionally, the apparatus further comprises:
a failure detection unit, configured to detect whether a database in the memory or the external storage is damaged in a case where a database execution error in the memory of an operation corresponding to the database operation request occurs;
the memory database recovery unit is used for recovering the database in the memory according to the data of the database in the external memory;
and the external storage database recovery unit is used for recovering the database in the external storage according to the data of the database in the memory.
Optionally, the terminal device receives the database operation request by providing a set of interfaces to the outside, where the interfaces may be APIs and/or services, and the memory database recovery unit is specifically configured to:
blocking the interface call;
discarding transactions that cause database corruption in memory;
under the condition that all the database operation requests in the SQL queue are executed, loading the data of the database in the external memory to the database in the internal memory;
and releasing the interface call.
Optionally, the terminal device receives the database operation request by providing a set of interfaces to the outside, where the interfaces may be APIs and/or services, and the external storage database recovery unit is specifically configured to:
blocking the interface call;
emptying the SQL queue;
newly building a database in an external memory;
loading the data of the database in the memory to a newly built database in an external memory;
deleting the damaged database in the external memory;
and releasing the interface call.
Alternatively to this, the first and second parts may,
the memory database operation unit 100 is further configured to, in a case that a storage space of the memory is insufficient, store only the core data in the database in the memory;
the synchronization processing unit 200 is further configured to:
under the condition of receiving the database operation request, judging whether the database operation request is a query operation;
if the operation is the query operation, judging whether the data corresponding to the database operation request is loaded to a database in the memory;
if the operation request is not loaded to the database in the memory, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory;
if the operation request is loaded to the database in the memory, under the condition that the database operation request in the SQL queue is completely executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the memory;
if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory;
judging whether the operation corresponding to the database operation request is executed correctly by the database in the external memory;
if the operation is correctly executed, judging whether the data corresponding to the database operation request is core data;
if the data is core data, under the condition that the transaction corresponding to the database operation request is completed, adding the database operation request into the SQL queue, and asynchronously calling the SQLite engine to execute the operation corresponding to the database operation request in the SQL queue on the database in the memory.
Alternatively to this, the first and second parts may,
the memory database operation unit 100 is further configured to delete a database in a memory when a storage space of the memory is insufficient;
the synchronization processing unit 200 is further configured to:
newly building a backup database in an external memory, and loading data in an original database in the external memory into the backup database;
under the condition of receiving the database operation request, judging whether the database operation request is a query operation;
if the operation is the query operation, under the condition that all the database operation requests in the SQL queue are executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the original database;
if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the backup database;
judging whether the operation corresponding to the database operation request is correctly executed in the backup database;
if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into the SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the original database.
In the data processing apparatus disclosed in this embodiment, a database is created in the internal memory and the external memory of the terminal device, and the data is the same in the initial state, and when a database operation request is received, the SQLite engine is called to execute an operation corresponding to the database operation request on the database in the internal memory, so as to improve the read-write speed of the database. Meanwhile, the data of the database in the internal memory and the data of the database in the external memory are consistent through an asynchronous operation preset synchronization mechanism, and even if the data in the database in the internal memory is lost due to reasons such as power failure, the data of the database in the external memory cannot be lost and is not influenced, so that the safety and the stability of the operation of the database are ensured.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable external memory, CD-ROM, or any other form of storage medium known in the art.
The above embodiments can be combined arbitrarily, and the features described in the embodiments in the present specification can be replaced or combined with each other in the above description of the disclosed embodiments, so that those skilled in the art can implement or use the present application.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (17)
1. A data processing method applied to a processor in a terminal device, wherein a memory of the terminal device is the same as data of a database in an external memory in an initial state, the method comprising:
under the condition of receiving a database operation request, calling an SQLite engine to execute an operation corresponding to the database operation request on a database in a memory;
and asynchronously operating a preset synchronous mechanism to make the data of the memory consistent with the data of the database in the external memory.
2. The method of claim 1, wherein the asynchronously operating a presupposed synchronization mechanism comprises:
judging whether the database operation request is a query operation;
if so, not executing the operation corresponding to the database operation request on the database in the external memory;
if not, judging whether the operation corresponding to the database operation request is correctly executed in the database in the memory;
if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into an SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the database in the external memory.
3. The method of claim 2, wherein in the event of a database execution error in the in-memory of the operation corresponding to the database operation request, the method further comprises:
detecting whether a database in a memory or an external memory is damaged;
if the database in the memory is damaged, recovering the database in the memory according to the data of the database in the external memory;
if the database in the external memory is damaged, the database in the external memory is recovered according to the data of the database in the memory.
4. The method according to claim 1, wherein the terminal device receives the database operation request by externally providing a set of interfaces, which may be APIs and/or services.
5. The method of claims 3 and 4, wherein the restoring the database in the memory according to the data of the database in the external storage comprises:
blocking the interface call;
discarding transactions that cause database corruption in memory;
under the condition that all the database operation requests in the SQL queue are executed, loading the data of the database in the external memory to the database in the internal memory;
and releasing the interface call.
6. The method according to claims 3 and 4, wherein the restoring the database in the external storage according to the data of the in-memory database comprises:
blocking the interface call;
emptying the SQL queue;
newly building a database in an external memory;
loading the data of the database in the memory to a newly built database in an external memory;
deleting the damaged database in the external memory;
and releasing the interface call.
7. The method of claim 1, wherein in case of insufficient memory space in the memory, the method further comprises:
the database in the memory only stores the core data;
and asynchronously operating a preset synchronization mechanism to make the core data of the database in the memory and the external memory consistent.
8. The method of claim 7, wherein the asynchronously operating a presupposed synchronization mechanism comprises:
under the condition of receiving the database operation request, judging whether the database operation request is a query operation;
if the operation is the query operation, judging whether the data corresponding to the database operation request is loaded to a database in the memory;
if the operation request is not loaded to the database in the memory, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory;
if the operation request is loaded to the database in the memory, under the condition that the database operation request in the SQL queue is completely executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the memory;
if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory;
judging whether the operation corresponding to the database operation request is executed correctly by the database in the external memory;
if the operation is correctly executed, judging whether the data corresponding to the database operation request is core data;
if the data is core data, under the condition that the transaction corresponding to the database operation request is completed, adding the database operation request into the SQL queue, and asynchronously calling the SQLite engine to execute the operation corresponding to the database operation request in the SQL queue on the database in the memory.
9. The method of claim 1, wherein in case of insufficient memory space in the memory, the method further comprises:
deleting the database in the memory;
newly building a backup database in an external memory, and loading data in an original database in the external memory into the backup database;
and asynchronously operating a preset synchronous mechanism to make the data of the original database consistent with the data of the backup database.
10. The method of claim 9, wherein the asynchronously operating a presupposed synchronization mechanism comprises:
under the condition of receiving the database operation request, judging whether the database operation request is a query operation;
if the operation is the query operation, under the condition that all the database operation requests in the SQL queue are executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the original database;
if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the backup database;
judging whether the operation corresponding to the database operation request is correctly executed in the backup database;
if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into the SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the original database.
11. A data processing apparatus, applied to a processor in a terminal device, wherein a memory of the terminal device is identical to data of a database in an external storage in an initial state, the apparatus comprising:
the memory database operation unit is used for calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the memory under the condition of receiving the database operation request;
and the synchronous processing unit is used for asynchronously operating the preset synchronous mechanism to ensure that the data of the database in the memory and the external memory are consistent.
12. The apparatus according to claim 11, wherein the synchronization processing unit is specifically configured to:
judging whether the database operation request is a query operation;
if so, not executing the operation corresponding to the database operation request on the database in the external memory;
if not, judging whether the operation corresponding to the database operation request is correctly executed in the database in the memory;
if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into an SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the database in the external memory.
13. The apparatus of claim 12, further comprising:
a failure detection unit, configured to detect whether a database in the memory or the external storage is damaged in a case where a database execution error in the memory of an operation corresponding to the database operation request occurs;
the memory database recovery unit is used for recovering the database in the memory according to the data of the database in the external memory;
and the external storage database recovery unit is used for recovering the database in the external storage according to the data of the database in the memory.
14. The apparatus according to claim 13, wherein the terminal device receives the database operation request by providing a set of interfaces to the outside, where the interfaces may be APIs and/or services, and the in-memory database recovery unit is specifically configured to:
blocking the interface call;
discarding transactions that cause database corruption in memory;
under the condition that all the database operation requests in the SQL queue are executed, loading the data of the database in the external memory to the database in the internal memory;
and releasing the interface call.
15. The apparatus according to claim 13, wherein the terminal device receives the database operation request by providing a set of interfaces to the outside, where the interfaces may be APIs and/or services, and the external storage database recovery unit is specifically configured to:
blocking the interface call;
emptying the SQL queue;
newly building a database in an external memory;
loading the data of the database in the memory to a newly built database in an external memory;
deleting the damaged database in the external memory;
and releasing the interface call.
16. The apparatus of claim 11,
the memory database operation unit is also used for storing only core data in the database in the memory under the condition that the storage space of the memory is insufficient;
the synchronization processing unit is further configured to:
under the condition of receiving the database operation request, judging whether the database operation request is a query operation;
if the operation is the query operation, judging whether the data corresponding to the database operation request is loaded to a database in the memory;
if the operation request is not loaded to the database in the memory, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory;
if the operation request is loaded to the database in the memory, under the condition that the database operation request in the SQL queue is completely executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the memory;
if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the database in the external memory;
judging whether the operation corresponding to the database operation request is executed correctly by the database in the external memory;
if the operation is correctly executed, judging whether the data corresponding to the database operation request is core data;
if the data is core data, under the condition that the transaction corresponding to the database operation request is completed, adding the database operation request into the SQL queue, and asynchronously calling the SQLite engine to execute the operation corresponding to the database operation request in the SQL queue on the database in the memory.
17. The apparatus of claim 11,
the memory database operation unit is also used for deleting the database in the memory under the condition that the storage space of the memory is insufficient;
the synchronization processing unit is further configured to:
newly building a backup database in an external memory, and loading data in an original database in the external memory into the backup database;
under the condition of receiving the database operation request, judging whether the database operation request is a query operation;
if the operation is the query operation, under the condition that all the database operation requests in the SQL queue are executed, calling the SQLite engine to execute the operation corresponding to the database operation request on the original database;
if the operation is not the query operation, calling the SQLite engine to execute the operation corresponding to the database operation request on the backup database;
judging whether the operation corresponding to the database operation request is correctly executed in the backup database;
if the operation is correctly executed, under the condition that the transaction corresponding to the database operation request is completed, the database operation request is added into the SQL queue, and the SQLite engine is asynchronously called to execute the operation corresponding to the database operation request in the SQL queue on the original database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011438493.9A CN112579678B (en) | 2020-12-07 | 2020-12-07 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011438493.9A CN112579678B (en) | 2020-12-07 | 2020-12-07 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112579678A true CN112579678A (en) | 2021-03-30 |
| CN112579678B CN112579678B (en) | 2024-04-12 |
Family
ID=75130488
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011438493.9A Active CN112579678B (en) | 2020-12-07 | 2020-12-07 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579678B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR19990058861A (en) * | 1997-12-30 | 1999-07-26 | 윤종용 | Apparatus and method for backing up real-time database system in multiprocess environment |
| GB201421743D0 (en) * | 2014-12-08 | 2015-01-21 | Ibm | Controlling a multi-database system |
| CN111125129A (en) * | 2019-12-10 | 2020-05-08 | 珠海格力电器股份有限公司 | Data processing method and device, storage medium and processor |
| CN111159106A (en) * | 2019-12-30 | 2020-05-15 | 亚信科技（中国）有限公司 | Data query method and device |
-
2020
- 2020-12-07 CN CN202011438493.9A patent/CN112579678B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR19990058861A (en) * | 1997-12-30 | 1999-07-26 | 윤종용 | Apparatus and method for backing up real-time database system in multiprocess environment |
| GB201421743D0 (en) * | 2014-12-08 | 2015-01-21 | Ibm | Controlling a multi-database system |
| CN111125129A (en) * | 2019-12-10 | 2020-05-08 | 珠海格力电器股份有限公司 | Data processing method and device, storage medium and processor |
| CN111159106A (en) * | 2019-12-30 | 2020-05-15 | 亚信科技（中国）有限公司 | Data query method and device |
Non-Patent Citations (2)
| Title |
|---|
| 朱国奇;: "基于内存数据库的战场态势生成方法", 数字技术与应用, no. 02 * |
| 潘明明;李丁丁;汤庸;刘海;: "一种基于中间件的异构数据库融合访问方法及系统", 计算机科学, no. 05 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112579678B (en) | 2024-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110865888B (en) | Resource loading method and device, server and storage medium | |
| CN111897498A (en) | Multi-level storage method and device for block chain data | |
| US9778998B2 (en) | Data restoration method and system | |
| CN108959407B (en) | Strong consistency writing method of data and terminal equipment | |
| CN107506266B (en) | Data recovery method and system | |
| CN111383031B (en) | Intelligent contract execution method and system in block chain and electronic equipment | |
| US10936411B2 (en) | Memory scrub system | |
| WO2018006587A1 (en) | File storage method, terminal, and storage medium | |
| CN113961153B (en) | Method and device for writing index data into disk and terminal equipment | |
| CN111414362A (en) | Data reading method, device, equipment and storage medium | |
| CN115599807A (en) | Data access method, device, application server and storage medium | |
| CN111324622A (en) | Resource first-aid-purchase processing method, device, equipment and storage medium | |
| US11249866B1 (en) | Snapshot-based data corruption detection | |
| CN108205559B (en) | Data management method and equipment thereof | |
| CN116126581B (en) | Memory fault processing method, device, system, equipment and storage medium | |
| CN112579678B (en) | Data processing method and device | |
| CN109240773B (en) | Method, device, terminal and readable storage medium for solving room string data | |
| CN115543918A (en) | File snapshot method, system, electronic equipment and storage medium | |
| CN112764973B (en) | Data disaster tolerance method and device, electronic equipment and readable storage medium | |
| CN113901010A (en) | Android application package compression method and device | |
| US20170364418A1 (en) | Integrity of transactional memory of card computing devices in case of card tear events | |
| CN108959405B (en) | Strong consistency reading method of data and terminal equipment | |
| CN112181590A (en) | Method, device and equipment for deleting resources in cloud platform and readable storage medium | |
| CN111625500A (en) | File snapshot method and device, electronic equipment and storage medium | |
| CN112231290A (en) | Method, device and equipment for processing local log and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |