CN111125129A

CN111125129A - Data processing method and device, storage medium and processor

Info

Publication number: CN111125129A
Application number: CN201911261795.0A
Authority: CN
Inventors: 杨丰玮; 宋德超; 王沅召; 张家琪; 赵慧子
Original assignee: Gree Electric Appliances Inc of Zhuhai; Zhuhai Lianyun Technology Co Ltd
Current assignee: Gree Electric Appliances Inc of Zhuhai; Zhuhai Lianyun Technology Co Ltd
Priority date: 2019-12-10
Filing date: 2019-12-10
Publication date: 2020-05-08

Abstract

The invention discloses a data processing method and device, a storage medium and a processor. Wherein, the method comprises the following steps: receiving an operation request for processing data stored in an embedded database, wherein the operation request at least comprises: keywords corresponding to data to be processed; processing the memory based on the operation request, wherein the memory is used for storing data to be processed; and after the memory processing is finished, processing a target data table in the embedded database based on the operation request, wherein the target data table is used for storing data to be processed. The invention solves the technical problem of low data processing efficiency caused by poor reading and writing speed of the lightweight database in the related technology.

Description

Data processing method and device, storage medium and processor

Technical Field

The present invention relates to the field of databases, and in particular, to a data processing method and apparatus, a storage medium, and a processor.

Background

The database is very important in system development, a Key-Value type database of a non-relational database (NoSQL) has the characteristics of high processing speed and large data volume, and common databases include Redis, Berkeley, levelDB and the like. Although the database is very powerful in performance, reliability and functionality, the database is large in size and is not suitable for systems that do not require relatively heavy-weight services.

For the system, a self-contained Key-Value type database (such as Map in Java) or other lightweight relational database (such as SQLite) in a development language can be adopted. However, the Key-Value object carried by the development language is only stored in the memory, but not in the hard disk, and persistent storage of data cannot be realized. And the lightweight relational database is used for realizing the storage of Key-Value, and due to the limitation of the read-write performance of the hard disk, the performance of the magnetic disk can limit the performance of the system when the concurrency of the system is large.

Aiming at the problem of low data processing efficiency caused by poor reading and writing speed of a lightweight database in the related technology, an effective solution is not provided at present.

Disclosure of Invention

The embodiment of the invention provides a data processing method and device, a storage medium and a processor, which are used for at least solving the technical problem of low data processing efficiency caused by poor reading and writing speed of a lightweight database in the related technology.

According to an aspect of an embodiment of the present invention, there is provided a data processing method including: receiving an operation request for processing data stored in an embedded database, wherein the operation request at least comprises: keywords corresponding to data to be processed; processing the memory based on the operation request, wherein the memory is used for storing data to be processed; and after the memory processing is finished, processing a target data table in the embedded database based on the operation request, wherein the target data table is used for storing data to be processed.

Further, the operation request further includes: an operation type, wherein processing the memory based on the operation request comprises: under the condition that the operation type is the query type, retrieving in the memory based on the keywords, and if the keywords are retrieved in the memory, outputting the data to be processed stored in the memory; under the condition that the operation type is the adding type, adding data to be processed and keywords in a memory, wherein the operation request further comprises the data to be processed; under the condition that the operation type is a modification type, retrieving in the memory based on the keywords, and if the keywords are retrieved in the memory, modifying the data to be processed stored in the memory based on the target data, wherein the operation request also comprises the target data; and deleting the data to be processed and the keywords stored in the memory under the condition that the operation type is a deletion type.

Further, before adding the data to be processed and the keywords in the memory, the method further comprises: retrieving in the memory based on the keywords; if the keywords are searched in the memory, deleting the original data and the keywords stored in the memory, and adding the data to be processed and the keywords in the memory; if the key words are not searched in the memory, the data to be processed and the key words are added in the memory.

Further, if the key is not retrieved in the memory, the target data table is processed based on the operation request.

Further, in a case that the operation type of the operation request is a query type, processing the target data table based on the operation request includes: retrieving in the operation queue and the target data table based on the keyword; if the keywords are retrieved from the operation queue, judging whether the operation statement corresponding to the keywords is a deletion statement; if the keywords are searched in the target data table and the operation statement is not a deletion statement, storing the data to be processed and the keywords in a memory and outputting the data to be processed; and if the keywords are not retrieved from the operation queue and the target data table or the operation statement is a deletion statement, outputting prompt information, wherein the prompt information is used for representing that the target data table does not store the data to be processed.

Further, the target data table also stores the expiration time of the data to be processed, wherein before the data to be processed and the keyword are stored in the memory, the method further includes: judging whether the data to be processed is overdue or not based on the expiration time; and if the data to be processed is judged not to be out of date, storing the data to be processed and the key words into the memory.

Further, the expiration time is determined by the effective time carried in the operation request and the time of receiving the operation request.

Further, in a case that the operation type of the operation request is the addition type, processing the target data table based on the operation request includes: retrieving in the target data table based on the keyword; if the keywords are retrieved from the target data table, deleting the original data and the keywords stored in the target data table, and storing the data to be processed and the keywords into the target data table; and if the key word is not retrieved in the target data table, storing the data to be processed and the key word in the target data table.

Further, before retrieving in the target data table based on the keyword, the method further comprises: retrieving in the operation queue based on the keyword; if the keywords are retrieved from the operation queue, judging whether the operation statement corresponding to the keywords is a deletion statement; if the operation statement is not a deletion statement, generating a deletion instruction based on the keyword, generating an insertion instruction based on the data to be processed and the keyword, and sequentially storing the deletion instruction and the insertion instruction to the tail of the operation queue; if the operation statement is a delete statement or no key is retrieved in the operation queue, a search is performed in the target data table based on the key.

Further, in the case that the operation type of the operation request is a modification type, processing the target data table based on the operation request includes: retrieving in the target data table based on the keyword; if the keywords are retrieved from the target data table, modifying the data to be processed stored in the target data table based on the target data; and if the keywords are not searched in the target data table, deleting the data to be processed and the keywords stored in the memory, and outputting prompt information.

Further, before retrieving in the target data table based on the keyword, the method further comprises: retrieving in the operation queue based on the keyword; if the keywords are retrieved from the operation queue, judging whether the operation statement corresponding to the keywords is a deletion statement; if the operation statement is not a deletion statement, generating a modification instruction based on the target data and the keyword, and storing the modification instruction to the tail of the operation queue; if the operation statement is a delete statement or no key is retrieved in the operation queue, a search is performed in the target data table based on the key.

Further, in the case where the key is not retrieved from the memory and the key is retrieved from the target data table, the target data and the key are stored to the memory.

Further, in a case that the operation type of the operation request is a delete type, processing the target data table based on the operation request includes: deleting an operation instruction corresponding to the keyword stored in the operation queue; generating a deletion instruction based on the keyword; and storing the deleting instruction to the tail part of the operation queue.

Further, the target data table is determined based on the identification information corresponding to the keyword.

Further, processing the target data table based on the operation request includes: calling a target operation method of the embedded database based on the operation type of the operation request; and processing the target data table based on the target operation method.

Further, acquiring a target data table and a time stamp of data stored in the memory according to a preset period, and deleting the data after determining the data expiration based on the time stamp; or under the condition that the operation type of the operation request is the query type, acquiring the time stamp of the data to be processed, and deleting the target data table and the data to be processed stored in the memory after determining the expiration of the data to be processed based on the time stamp.

According to another aspect of the embodiments of the present invention, there is also provided a data processing apparatus, including: a receiving module, configured to receive an operation request for processing data stored in an embedded database, where the operation request at least includes: keywords corresponding to data to be processed; the first processing module is used for processing the memory based on the operation request, wherein the memory is used for storing data to be processed; and the second processing module is used for processing a target data table in the embedded database based on the operation request after the memory processing is finished, wherein the target data table is used for storing data to be processed.

Further, the second processing module includes: the calling unit is used for calling a target operation method of the embedded database based on the operation type of the operation request; and the processing unit is used for processing the target data table based on the target operation method.

Further, the apparatus further comprises: the deleting module is used for acquiring the time stamps of the target data table and the data stored in the memory according to a preset period, and deleting the data after the data expiration is determined based on the time stamps; or under the condition that the operation type of the operation request is the query type, acquiring the time stamp of the data to be processed, and deleting the target data table and the data to be processed stored in the memory after determining the expiration of the data to be processed based on the time stamp.

According to another aspect of the embodiments of the present invention, there is also provided a storage medium including a stored program, wherein when the program runs, a device on which the storage medium is located is controlled to execute the data processing method described above.

According to another aspect of the embodiments of the present invention, there is also provided a processor, configured to execute a program, where the program executes the data processing method described above.

In the embodiment of the invention, after the operation request for processing the data stored in the embedded database is received, the memory can be processed based on the operation request, and after the memory is processed, the target data table in the embedded database is processed based on the operation request, so that the purpose of operating the embedded database is realized. Compared with the prior art, the embedded database stores data, an embedded zero configuration is realized, the complete database with light-weight development operation is provided, the data stored in the embedded database is quickly operated by directly processing the memory, the technical effect of improving the data processing efficiency is achieved, and the technical problem that the data processing efficiency is lower due to the fact that the light-weight database in the prior art is poor in reading and writing speed is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:

FIG. 1 is a flow chart of a method of data processing according to an embodiment of the invention;

FIG. 2 is a schematic diagram of an alternative database system architecture according to an embodiment of the present invention; and

fig. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

An embedded database: embedding the database into the application process eliminates the overhead associated with client server configuration, which is a lightweight database. Less memory is required at run-time, faster for embedded devices, and also provides a zero configuration mode of operation.

Example 1

In accordance with an embodiment of the present invention, there is provided a data processing method, it should be noted that the steps illustrated in the flowchart of the figure may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.

Fig. 1 is a flow chart of a data processing method according to an embodiment of the present invention, as shown in fig. 1, the method including the steps of:

step S102, receiving an operation request for processing data stored in an embedded database, wherein the operation request at least comprises: keywords corresponding to data to be processed;

in order to reduce the data volume of the database and achieve the purpose of database lightweight, an embedded database is adopted in the embodiment of the present invention, for example, SQLite may be adopted, the data volume of the database itself is less than 10M, and the data stored in the database may be in the form of Key-Value. To achieve persistent storage of data, the data in the database may be stored in a single disk directory on the hard disk.

When data stored in the database needs to be processed, a user can provide a Key Value (i.e., the Key word) corresponding to the data that needs to be processed, and the data to be processed is a Value corresponding to the Key Value. The user can perform operations such as query, addition, modification and deletion on the database, for the query, modification and deletion operations, the data required to be processed by the user is the data already stored in the database, and for the addition operation, the data required to be processed by the user is the data to be stored in the database, so the data to be processed in the steps can be the data to be stored in the database, and can also be the data already stored in the database.

It should be noted that the embodiment of the present application does not limit the specific form and content of the data at all.

Step S104, processing a memory based on the operation request, wherein the memory is used for storing data to be processed;

due to the limitation of the read-write performance of the hard disk, in order to improve the read-write speed of the database, data can be cached in the memory in advance, and when a user needs to process the data stored in the database, the memory can be directly operated. The memory operation method comprises the following steps: firstly, judging whether the Key value exists in the memory, and carrying out corresponding operation on the data in the memory based on the judgment result.

And step S106, after the memory processing is finished, processing a target data table in the embedded database based on the operation request, wherein the target data table is used for storing the data to be processed.

In the embodiment of the invention, the data is stored in the cache and the embedded database, but the data in the embedded database is always used as the standard. In an alternative embodiment, in order to ensure the accuracy of the data in the embedded database, after the memory operation is completed, the embedded database may be operated in an asynchronous operation mode.

For the data already stored in the embedded database, the target data table in the above step may be the data table storing the data; for the data to be stored in the embedded database, the target data table in the above step may be the data table to be stored. The target data table may be determined based on identification information corresponding to the keyword, where the identification information may be a table name set for a different data table.

In order to prevent all data from being stored in the same table and cause the data operation on the table to be slow when one table has excessive data, in the embodiment of the invention, the data table can be created and the created table name can be managed, and when the embedded database needs to be operated, the table name where the data should exist can be determined according to the Key value, so that the table data can be dispersed in different tables, and the speed of operating the embedded database is increased.

Further, the table name of the data table can be implemented in various ways, and in an alternative embodiment, the first character of underline + Key is used as the table name. For example, if the inserted Key is 123456, the record will be inserted into the "_ 1" table, and if the data in the "_ 1" table exceeds 10 ten thousand, then when the Key is inserted (123456), the first two digits of the underlined + Key will be used as the table name, and "_ 12" will be created and the data will be inserted. When the database needs to query, modify or delete the Key (123456), two table names of "_ 1" and "_ 12" can be acquired.

For the query request, the modification request or the deletion request, since the data already stored in the embedded database and the memory needs to be operated, the user can perform corresponding operation on the data only when the data which needs to be processed by the user exists in the embedded database and the memory. For the addition request, in order to avoid that the embedded database and the memory store the data, whether the data exists in the embedded database and the memory needs to be searched first, and then a mode for processing the data is determined according to a search result.

By adopting the embodiment of the invention, after the operation request for processing the data stored in the embedded database is received, the memory can be processed based on the operation request, and after the memory is processed, the target data table in the embedded database is processed based on the operation request, so that the purpose of operating the embedded database is realized. Compared with the prior art, the embedded database stores data, an embedded zero configuration is realized, the complete database with light-weight development operation is provided, the data stored in the embedded database is quickly operated by directly processing the memory, the technical effect of improving the data processing efficiency is achieved, and the technical problem that the data processing efficiency is lower due to the fact that the light-weight database in the prior art is poor in reading and writing speed is solved.

Optionally, in the foregoing embodiment of the present invention, the operation request further includes: an operation type, wherein processing the memory based on the operation request comprises: under the condition that the operation type is the query type, retrieving in the memory based on the keywords, and if the keywords are retrieved in the memory, outputting the data to be processed stored in the memory; under the condition that the operation type is the adding type, adding data to be processed and keywords in a memory, wherein the operation request further comprises the data to be processed; under the condition that the operation type is a modification type, retrieving in the memory based on the keywords, and if the keywords are retrieved in the memory, modifying the data to be processed stored in the memory based on the target data, wherein the operation request also comprises the target data; and deleting the data to be processed and the keywords stored in the memory under the condition that the operation type is a deletion type.

The operation types in the above steps may include a query type, a modification type, an addition type and a deletion type, and different operations may be performed on the embedded database for different types of operation requests. It should be noted that, for a modification type operation request, a user needs to modify data stored in the embedded database, and at this time, the operation request needs to carry the modified data (i.e., the target data).

For the operation request of the query type, it can be determined that the user needs to query the data stored in the embedded database, at this time, the user needs to search in the memory, determine whether the data is stored in the memory, and if the data is stored, the data can be output to the user, so as to achieve the purpose of data search.

For the operation request of the addition type, it can be determined that the user needs to add data in the embedded database, and the user needs to carry the data to be stored (i.e., the data to be processed) in the operation request, and then store the data carried in the operation request in the memory, thereby achieving the purpose of adding data.

For the operation request of the modification type, it can be determined that the user needs to modify the data stored in the embedded database, and the user needs to carry the target data in the operation request. At this time, it is necessary to search in the memory to determine whether the data is stored in the memory, and if so, modify the data based on the target data, so as to achieve the purpose of modifying the data.

For the deletion type operation request, it can be determined that the user needs to delete the data stored in the embedded database, at this time, it needs to search in the memory to determine whether the data is stored in the memory, and if so, the data can be directly deleted, so as to achieve the purpose of deleting the data.

It should be noted that, the same operation needs to be performed on the keyword while data is deleted or added.

Optionally, in the above embodiment of the present invention, before adding the data to be processed and the keyword into the memory, the method further includes: retrieving in the memory based on the keywords; if the keywords are searched in the memory, deleting the original data and the keywords stored in the memory, and adding the data to be processed and the keywords in the memory; if the key words are not searched in the memory, the data to be processed and the key words are added in the memory.

Since the user may have stored the data corresponding to the keyword (i.e., the original data), at this time, it may be determined whether the data corresponding to the keyword is stored in the memory first, and if so, the stored data is deleted, and the data carried in the operation request is stored in the memory; and if the data is not stored, directly storing the data carried in the operation request in the memory.

Optionally, in the above embodiment of the present invention, if the keyword is not retrieved in the memory, the target data table is processed based on the operation request.

In an optional embodiment, after determining that the operation of querying, modifying or deleting by the user does not exist in the memory, the target data table stored in the embedded database may be directly processed.

Optionally, in the foregoing embodiment of the present invention, when the operation type of the operation request is a query type, processing the target data table based on the operation request includes: retrieving in the operation queue and the target data table based on the keyword; if the keywords are retrieved from the operation queue, judging whether the operation statement corresponding to the keywords is a deletion statement; if the keywords are searched in the target data table or the operation statement is not a deletion statement, storing the data to be processed and the keywords in a memory, and outputting the data to be processed; and if the keywords are not retrieved from the operation queue and the target data table or the operation statement is a deletion statement, outputting prompt information, wherein the prompt information is used for representing that the target data table does not store the data to be processed.

For the operation request of the addition type, the modification type or the deletion type, because corresponding operation needs to be performed on the data already stored in the embedded database, if the data does not exist in the embedded database, the operation cannot be performed. The whole operation of the database is not perceptible to the user, so that the user can conveniently confirm whether the operation on the embedded database is successful, and in an alternative embodiment, after determining that the operation on the embedded database cannot be performed, prompt information without data can be returned to the user.

The embedded database operation is not blocked by using the search data of the embedded database, and the execution speed is high. In order to buffer the operation of the embedded database and accelerate the processing efficiency, an operation queue can be deployed in advance, and the asynchronous operation of the operation queue can realize the operation of the data stored in the embedded database. It should be noted that, the load of the embedded database often has different levels, and the asynchronous operation delays the completion of the data operation when the load of the embedded database is low.

On this basis, due to the asynchronous operation of the operation queue, the operation of adding data in the embedded database by the user may not be completed yet and still be in the queue. In addition, if the user has deleted the data, but the delete operation has not been completed, the data is still stored in the data table. In view of the above situation, in an optional embodiment, when it is determined that the user needs to query the data, the operations queue and the data table may be searched at the same time, determine whether there is related data, and determine whether the operation statement of the Key value in the operations queue is a delete statement. If the operation statement is not a deletion statement and the Key value exists in the data table, the SQL statement can be assembled to search from the embedded database, and after corresponding data is found, the SQL statement is directly returned to the user, so that the corresponding speed is improved; if the operation statement is a delete statement or the data does not exist in the operation queue and the data table, returning a prompt message without the data to inform the user that the query operation is unsuccessful.

For the embedded database, operations such as query, insertion, modification, deletion and the like can be realized through SQL sentences, corresponding SQL sentences can be generated for each operation of a user, and the SQL sentences are inserted into the tail part of the operation queue.

It should be noted that, it is possible to search whether the Key value exists from the tail of the operation queue.

Optionally, in the above embodiment of the present invention, the target data table further stores an expiration time of the data to be processed, where before storing the data to be processed and the keyword in the memory, the method further includes: judging whether the data to be processed is overdue or not based on the expiration time; and if the data to be processed is judged not to be out of date, storing the data to be processed and the key words into the memory.

In order to realize the life cycle management of data, a file not only stores a Key Value and a Value, but also stores corresponding expiration time, whether the data stored in an embedded database exceeds the period can be judged through the expiration time, if the data does not exceed the period, the data is valid and needs to be stored in a memory; if the period is over, the data is invalid, the embedded database is considered to have no corresponding data, and prompt information without data is directly returned to inform the user that the query operation is unsuccessful.

In the embodiment of the present invention, the expiration time may be determined by the valid time carried in the operation request and the time when the operation request is received.

In an alternative embodiment, when each data is stored in the embedded database, the user needs to give the valid time of each data, so that the expiration time of each data can be determined according to the sum of the current timestamp (i.e. the time when the operation request is received) of the system and the valid time. For example, assuming that the validity period of the data is 180 seconds and the system current timestamp is 1571033887, a corresponding expiration time of 1571034067 may be obtained.

Optionally, in the foregoing embodiment of the present invention, when the operation type of the operation request is an add type, processing the target data table based on the operation request includes: retrieving in the target data table based on the keyword; if the keywords are retrieved from the target data table, deleting the original data and the keywords stored in the target data table, and storing the data to be processed and the keywords into the target data table; and if the key word is not retrieved in the target data table, storing the data to be processed and the key word in the target data table.

For the operation request of the adding type, whether the data which needs to be added by the user already exists in the target data table is determined based on the judgment result, if yes, the related data in the target data table is deleted, and the data which needs to be added by the user is stored in the target data table; if not, the data which the user needs to add is directly stored in the target data table.

Optionally, in the above embodiment of the present invention, before performing the retrieval in the target data table based on the keyword, the method further includes: retrieving in the operation queue based on the keyword; if the keywords are retrieved from the operation queue, judging whether the operation statement corresponding to the keywords is a deletion statement; if the operation statement is not a deletion statement, generating a deletion instruction based on the keyword, generating an insertion instruction based on the data to be processed and the keyword, and sequentially storing the deletion instruction and the insertion instruction to the tail of the operation queue; if the operation statement is a delete statement or no key is retrieved in the operation queue, a search is performed in the target data table based on the key.

In an optional embodiment, when data is searched, table names possibly existing in the data can be obtained, the table names are subjected to SQL assembly, a search statement is executed on an embedded database to search whether the data exists, if yes, a deletion statement and an insertion statement are assembled, two SQL statements are sequentially inserted into an operation queue, and a result of successful operation is directly returned. When the operation queue turns to the two SQL sentences, the first deletion sentence is executed firstly, all the data of the Key stored in the embedded database are deleted, and then the second sentence is executed to perform insertion operation on the data.

Optionally, in the foregoing embodiment of the present invention, when the operation type of the operation request is a modification type, processing the target data table based on the operation request includes: retrieving in the target data table based on the keyword; if the keywords are retrieved from the target data table, modifying the data to be processed stored in the target data table based on the target data; and if the keywords are not searched in the target data table, deleting the data to be processed and the keywords stored in the memory, and outputting prompt information.

For the operation request of the modification type, the operation request can be firstly searched in the memory, if the data does not exist, the operation request is further searched in the target data table, and if the data exists, the operation request directly modifies the data. If not, a prompt message without data can be returned to inform the user that the query operation is unsuccessful. In addition, because the embedded database is asynchronous operation, if the data does not exist in the embedded database, the corresponding data in the memory can be deleted, and the synchronization of the two is ensured.

Optionally, in the above embodiment of the present invention, before performing the retrieval in the target data table based on the keyword, the method further includes: retrieving in the operation queue based on the keyword; if the keywords are retrieved from the operation queue, judging whether the operation statement corresponding to the keywords is a deletion statement; if the operation statement is not a deletion statement, generating a modification instruction based on the target data and the keyword, and storing the modification instruction to the tail of the operation queue; if the operation statement is a delete statement or no key is retrieved in the operation queue, a search is performed in the target data table based on the key.

For the operation request of the modification type, firstly searching in an operation queue, starting to search from the tail of the queue, if the operation statement exists and is not a deletion statement, directly assembling and modifying an SQL statement, and inserting the modification statement into the operation queue; if the operation queue does not exist, the SQL sentences are assembled to the embedded database for searching, if the result is found, the SQL sentences are kept in the memory, the data of the memory is updated and modified, meanwhile, the SQL sentences are assembled and modified, and the modified sentences are inserted into the database operation queue.

Optionally, in the above embodiment of the present invention, when the key is not retrieved from the memory and the key is retrieved from the target data table, the target data and the key are stored in the memory.

In order to ensure that the data stored in the embedded database and the memory are synchronized, the data stored in the embedded database may be synchronized into the memory.

Optionally, in the foregoing embodiment of the present invention, when the operation type of the operation request is a delete type, processing the target data table based on the operation request includes: deleting an operation instruction corresponding to the keyword stored in the operation queue; generating a deletion instruction based on the keyword; and storing the deleting instruction to the tail part of the operation queue.

For the operation request of the modification type, the table name of the target data table can be obtained, then the assembly of the SQL statement is deleted, the assembled statement is inserted into the operation queue, and the data is deleted.

Optionally, in the foregoing embodiment of the present invention, processing the target data table based on the operation request includes: calling a target operation method of the embedded database based on the operation type of the operation request; and processing the target data table based on the target operation method.

In an optional embodiment, various operation methods for operating the embedded database are encapsulated in the system architecture to form a database operation tool class, and when a user needs to process data stored in the embedded database, the corresponding operation methods in the embedded database operation tool class can be directly called, and corresponding parameters are introduced into the methods.

For example, for an operation request of query type, a query operation method may be called, and a Key value is passed in the method. After the method is invoked, a query operation may be performed on the embedded database.

For an add type operation request, an add operation method may be called, and a Key Value and a Value are passed in the method. After the method is invoked, an add operation may be performed on the embedded database.

For a modification type operation request, a modify operation method may be called, and a Key Value and a new Value are introduced in the method. After the method is invoked, a modify operation may be performed on the embedded database.

For a delete type operation request, a delete operation method may be called and a Key value is passed in the method. After the method is invoked, a delete operation may be performed on the embedded database.

Optionally, in the above embodiment of the present invention, the time stamps of the target data table and the data stored in the memory are obtained according to a preset period, and after determining that the data expires based on the time stamps, the data is deleted; or under the condition that the operation type of the operation request is the query type, acquiring the time stamp of the data to be processed, and deleting the target data table and the data to be processed stored in the memory after determining the expiration of the data to be processed based on the time stamp.

In the embodiment of the application, the lifecycle of the data can be managed in two ways, one is by a timing task, and the other is lazy deletion, that is, the management is realized when the system retrieves the Key value. The preset period in the above steps may be set according to actual use requirements, for example, the preset period may be daily, but is not limited thereto.

In an alternative embodiment, all data in the memory and the data file may be traversed according to a preset period, and the expired data may be deleted. In another alternative embodiment, when the system searches a Key, it checks whether the data has expired, if so, it returns that there is no data, and deletes the corresponding data stored in the memory and the file at the same time.

A preferred embodiment of the present invention will be described in detail below with reference to fig. 2, taking a Java short network address generation system as an example. The system is used for web site shortening services, and when a web site is registered on the system (e.g., https:// gmall. grid. com/staff/index. html), the system generates a short web site for the web site (e.g., http:// grid. cn/123456). Most of the requests of the system are to analyze and convert the short website into the original website again, so that a larger concurrency amount needs to be supported, the conversion speed is required to be higher, and a bottleneck cannot be caused to the processing speed of other business services. The mapping relation between the short website and the original website needs to be stored in a database, only one server is used for the service, and at the moment, the embedded database can be used, the Jar of the database is directly depended on, and the website mapping data is added, deleted, searched and modified by operating a database operation tool.

As shown in fig. 2, the system architecture is composed of a database interface layer, a cache operation layer, a database operation layer, a timing task, and a tool class, wherein the tool class includes: a memory management class and a database table management class. The cache stores database operations. The cache operation layer may be controlled by a memory management class, and the database operation layer may be controlled by a database table management class. The database table management class creates a table by converting the Key and returns an operation table name for the logic code. When the data amount stored in one data table reaches a certain amount, a new data table is created and the table name is returned.

In the interface layer of the database, the operation request input by the user can be obtained through the database operation tool class, and the specific type of the operation required by the user is determined, which is divided into query operation, adding operation, modifying operation and deleting operation.

For the search operation, firstly, judging whether the Key is in the cache or not in the cache operation layer, and if so, directly returning a result; if the Key does not exist, in the database operation layer, searching the Key in the database operation queue and the database table, judging whether the Key is found, if so, adding the Key and Value in the cache, and returning a result; if not, return no record found.

For the adding operation, firstly, judging whether the Key exists in a cache or not in a cache operation layer, if so, deleting the Key and the corresponding Value in the cache, and adding the Key and the Value in the cache; if not, directly adding the Key and Value in the cache. Then, searching a Key in a database operation queue and a database table in a database operation layer, judging whether the Key is found, splicing a deletion statement and an insertion statement if the Key is found, inserting the deletion statement and the insertion statement into the database operation queue in sequence, and directly returning a processing result; and if the Key is not retrieved, directly splicing the insertion statement, inserting the insertion statement into the database operation queue, and directly returning a processing result, wherein the deletion statement is used for deleting all records of the Key in the database, and the insertion statement is used for inserting data.

For modification operation, firstly, judging whether a Key exists in a cache or not in a cache operation layer, and if so, modifying data in the cache. If the Key does not exist in the memory or after the Key is modified in the memory, searching the Key in a database operation queue and a database table in a database operation layer, judging whether the Key is found, splicing an update statement if the Key is found, inserting the update statement into the database operation queue, directly returning a processing result, and if the Key does not exist in the memory, adding the modified record to a cache; if not, deleting the data stored in the cache and returning that the data does not exist, wherein the updating statement is used for updating the data of the Key.

For a delete operation, the Key and the corresponding Value are first deleted in the cache operation layer. And then deleting the operation of the Key in the database operation queue in the database operation layer, splicing the deleted statement, inserting the deleted statement into the database operation queue, and directly returning a processing result.

And the database operation in the cache is operated by a memory management class, and the embedded database is operated by reading the operation statements in the database operation queue.

The timing task can delete the expired data by traversing the data in the database table and caching the data.

Taking the example of adding a website mapping record in an embedded database, the system generates a short website (http:// grid.cn/123456) for a long website (https:// gram.grid.com/staff/index.html), the data needs to be inserted into the database at present, Key is the ID of the short website (such as 123456), and the effective time of the short website is 180 seconds, the system only needs to directly call the adding method of the tool class, and the Key, Value and the effective time are transmitted in the method. After the adding method is called, the database obtains the expiration time (1571034067) according to the current timestamp (such as 1571033887) + the validity period (180), forms a new character string (https:// gmall. grid.com/staff/index. html |1571034067) with the Value, searches whether the Key exists in the memory, deletes the data if the Key exists, adds the data (Key: 123456, Value: https:// gmall. grid.com/staff/index. html |1571034067), inserts the data into the memory, connects the database, obtains the table name which may exist in the data by using a database table management tool class, assembles the table name, and executes a lookup operation on the embedded database to check whether the data exists. If the data exists, assembling a deletion statement and an insertion statement, sequentially inserting the two SQL statements into a database operation queue, and directly returning the operation success to the database operation class. When the database operation queue turns to the two SQL sentences, the first deletion sentence is executed firstly, all the data of the Key in the embedded database are deleted, and then the second sentence is executed to perform insertion operation on the data.

Through the scheme, the database is developed by Java voice, and the data is stored by the SQLite embedded database, so that an embedded zero configuration is realized and a complete database stored under a single disk is realized. The database has light-weight development operation, reduces the waste of system resources, directly operates the memory, has higher performance than a common relational database, adds failure time to data, manages the life cycle of the data, and enhances the practicability of the database, so that the system uses a more appropriate database and is developed more quickly.

Example 2

According to an embodiment of the present invention, there is provided a data processing apparatus. The device may perform the method provided in embodiment 1, and the preferred embodiment or implementation in this embodiment is the same as that in embodiment 1, which is not described herein again.

Fig. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 3, the apparatus including:

a receiving module 32, configured to receive an operation request for processing data stored in the embedded database, where the operation request at least includes: keywords corresponding to data to be processed;

the first processing module 34 is configured to process a memory based on the operation request, where the memory is used to store data to be processed;

and a second processing module 36, configured to process a target data table in the embedded database based on the operation request after the memory processing is completed, where the target data table is used to store data to be processed.

By adopting the above embodiment of the present invention, after the operation request for processing the data stored in the embedded database is received by the receiving module, the memory can be processed by the first processing module based on the operation request, and after the memory is processed, the target data table in the embedded database is processed by the second processing module based on the operation request, so as to achieve the purpose of operating the embedded database. Compared with the prior art, the embedded database stores data, an embedded zero configuration is realized, the complete database with light-weight development operation is provided, the data stored in the embedded database is quickly operated by directly processing the memory, the technical effect of improving the data processing efficiency is achieved, and the technical problem that the data processing efficiency is lower due to the fact that the light-weight database in the prior art is poor in reading and writing speed is solved.

Optionally, in the foregoing embodiment of the present invention, the operation request further includes: the first processing module is further used for retrieving in the memory based on the keywords under the condition that the operation type is the query type, and outputting the data to be processed stored in the memory if the keywords are retrieved in the memory; under the condition that the operation type is the adding type, adding data to be processed and keywords in a memory, wherein the operation request further comprises the data to be processed; under the condition that the operation type is a modification type, retrieving in the memory based on the keywords, and if the keywords are retrieved in the memory, modifying the data to be processed stored in the memory based on the target data, wherein the operation request also comprises the target data; and deleting the data to be processed and the keywords stored in the memory under the condition that the operation type is a deletion type.

Optionally, in the above embodiment of the present invention, the first processing module is further configured to perform a search in a memory based on the keyword; if the keywords are searched in the memory, deleting the original data and the keywords stored in the memory, and adding the data to be processed and the keywords in the memory; if the key words are not searched in the memory, the data to be processed and the key words are added in the memory.

Optionally, in the foregoing embodiment of the present invention, the second processing module is further configured to process the target data table based on the operation request if the keyword is not retrieved from the memory.

Optionally, in the foregoing embodiment of the present invention, in a case that the operation type of the operation request is a query type, the second processing module is further configured to perform retrieval in the operation queue and the target data table based on the keyword; if the keywords are retrieved from the operation queue, judging whether the operation statement corresponding to the keywords is a deletion statement; if the keywords are searched in the target data table and the operation statement is not a deletion statement, storing the data to be processed and the keywords in a memory and outputting the data to be processed; and if the keywords are not retrieved from the operation queue and the target data table or the operation statement is a deletion statement, outputting prompt information, wherein the prompt information is used for representing that the target data table does not store the data to be processed.

Optionally, in the above embodiment of the present invention, an expiration time of the data to be processed is further stored in the target data table, where the second processing module is further configured to determine whether the data to be processed is out of date based on the expiration time; and if the data to be processed is judged not to be out of date, storing the data to be processed and the key words into the memory.

Optionally, in the foregoing embodiment of the present invention, in a case that the operation type of the operation request is an add type, the second processing module is further configured to perform a search in the target data table based on the keyword; if the keywords are retrieved from the target data table, deleting the original data and the keywords stored in the target data table, and storing the data to be processed and the keywords into the target data table; and if the key word is not retrieved in the target data table, storing the data to be processed and the key word in the target data table.

Optionally, in the foregoing embodiment of the present invention, the second processing module is further configured to perform retrieval in the operation queue based on the keyword; if the keywords are retrieved from the operation queue, judging whether the operation statement corresponding to the keywords is a deletion statement; if the operation statement is not a deletion statement, generating a deletion instruction based on the keyword, generating an insertion instruction based on the data to be processed and the keyword, and sequentially storing the deletion instruction and the insertion instruction to the tail of the operation queue; if the operation statement is a delete statement or no key is retrieved in the operation queue, a search is performed in the target data table based on the key.

Optionally, in the foregoing embodiment of the present invention, in a case that the operation type of the operation request is a modification type, the second processing module is further configured to perform a search in the target data table based on the keyword; if the keywords are retrieved from the target data table, modifying the data to be processed stored in the target data table based on the target data; and if the keywords are not searched in the target data table, deleting the data to be processed and the keywords stored in the memory, and outputting prompt information.

Optionally, in the foregoing embodiment of the present invention, the second processing module is further configured to perform retrieval in the operation queue based on the keyword; if the keywords are retrieved from the operation queue, judging whether the operation statement corresponding to the keywords is a deletion statement; if the operation statement is not a deletion statement, generating a modification instruction based on the target data and the keyword, and storing the modification instruction to the tail of the operation queue; if the operation statement is a delete statement or no key is retrieved in the operation queue, a search is performed in the target data table based on the key.

Optionally, in the foregoing embodiment of the present invention, the first processing module is further configured to store the target data and the keyword into the memory if the keyword is not retrieved from the memory and the keyword is retrieved from the target data table.

Optionally, in the above embodiment of the present invention, when the operation type of the operation request is a delete type, the second processing module is further configured to delete an operation instruction corresponding to a keyword stored in the operation queue; generating a deletion instruction based on the keyword; and storing the deleting instruction to the tail part of the operation queue.

Optionally, in the foregoing embodiment of the present invention, the second processing module includes: the calling unit is used for calling a target operation method of the embedded database based on the operation type of the operation request; and the processing unit is used for processing the target data table based on the target operation method.

Optionally, in the above embodiment of the present invention, the apparatus further includes: the deleting module is used for acquiring the time stamps of the target data table and the data stored in the memory according to a preset period, and deleting the data after the data expiration is determined based on the time stamps; or under the condition that the operation type of the operation request is the query type, acquiring the time stamp of the data to be processed, and deleting the target data table and the data to be processed stored in the memory after determining the expiration of the data to be processed based on the time stamp.

Example 3

According to an embodiment of the present invention, there is provided a storage medium including a stored program, wherein a device in which the storage medium is located is controlled to execute the data processing method in the above-described embodiment 1 when the program runs.

Example 4

According to an embodiment of the present invention, there is provided a processor configured to execute a program, where the program executes the data processing method in embodiment 1.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims

1. A data processing method, comprising:

receiving an operation request for processing data stored in an embedded database, wherein the operation request at least comprises: keywords corresponding to data to be processed;

processing a memory based on the operation request, wherein the memory is used for storing the data to be processed;

and after the memory processing is finished, processing a target data table in the embedded database based on the operation request, wherein the target data table is used for storing the data to be processed.

2. The method of claim 1, wherein the operation request further comprises: an operation type, wherein processing the memory based on the operation request comprises:

under the condition that the operation type is a query type, retrieving in the memory based on the keywords, and if the keywords are retrieved in the memory, outputting the to-be-processed data stored in the memory;

adding the data to be processed and the keywords in the memory under the condition that the operation type is an addition type, wherein the operation request further comprises the data to be processed;

under the condition that the operation type is a modification type, retrieving in the memory based on the keyword, and if the keyword is retrieved in the memory, modifying the data to be processed stored in the memory based on target data, wherein the operation request further comprises the target data;

and deleting the data to be processed and the keywords stored in the memory under the condition that the operation type is a deletion type.

3. The method of claim 2, wherein prior to adding the data to be processed and the key to the memory, the method further comprises:

retrieving in the memory based on the keyword;

if the keywords are retrieved from the memory, deleting the original data and the keywords stored in the memory, and adding the data to be processed and the keywords into the memory;

and if the key words are not retrieved in the memory, adding the data to be processed and the key words in the memory.

4. The method of claim 2, wherein if the key is not retrieved in the memory, processing the target data table based on the operation request.

5. The method of claim 1, wherein, in the case that the operation type of the operation request is a query type, processing the target data table based on the operation request comprises:

retrieving in an operation queue and the target data table based on the key;

if the keyword is retrieved from the operation queue, judging whether an operation statement corresponding to the keyword is a deletion statement;

if the keywords are retrieved from the target data table and the operation statement is not the deletion statement, storing the data to be processed and the keywords into the memory and outputting the data to be processed;

and if the keywords are not retrieved from the operation queue and the target data table or the operation statement is the deletion statement, outputting prompt information, wherein the prompt information is used for representing that the data to be processed is not stored in the target data table.

6. The method of claim 5, wherein the target data table further stores an expiration time of the pending data, and wherein before storing the pending data and the key to the memory, the method further comprises:

judging whether the data to be processed is out of date or not based on the expiration time;

and if the data to be processed is judged not to be out of date, storing the data to be processed and the key words into the memory.

7. The method of claim 6, wherein the expiration time is determined by a validity time carried in the operation request and a time at which the operation request is received.

8. The method according to claim 1, wherein in the case that the operation type of the operation request is an add type, processing the target data table based on the operation request comprises:

retrieving in the target data table based on the keyword;

if the keywords are searched in the target data table, deleting the original data and the keywords stored in the target data table, and storing the data to be processed and the keywords in the target data table;

and if the key word is not retrieved in the target data table, storing the data to be processed and the key word in the target data table.

9. The method of claim 8, wherein prior to retrieving in the target data table based on the key, the method further comprises:

retrieving in an operation queue based on the keyword;

if the operation statement is not the deletion statement, generating a deletion instruction based on the keyword, generating an insertion instruction based on the data to be processed and the keyword, and sequentially storing the deletion instruction and the insertion instruction to the tail of the operation queue;

and if the operation statement is the deletion statement or the keyword is not retrieved from the operation queue, retrieving in the target data table based on the keyword.

10. The method of claim 1, wherein, in the case that the operation type of the operation request is a modification type, processing the target data table based on the operation request comprises:

retrieving in the target data table based on the keyword;

if the keywords are retrieved from the target data table, modifying the data to be processed stored in the target data table based on target data;

and if the keywords are not retrieved in the target data table, deleting the data to be processed and the keywords stored in the memory, and outputting prompt information.

11. The method of claim 10, wherein prior to retrieving in the target data table based on the key, the method further comprises:

retrieving in an operation queue based on the keyword;

if the operation statement is not the deletion statement, generating a modification instruction based on the target data and the keyword, and storing the modification instruction to the tail of the operation queue;

12. The method of claim 10, wherein the target data and the key are stored to the memory if the key is not retrieved from the memory and the key is retrieved from the target data table.

13. The method according to claim 1, wherein, in the case that the operation type of the operation request is a delete type, processing the target data table based on the operation request comprises:

deleting the operation instruction corresponding to the keyword stored in the operation queue;

generating a deletion instruction based on the keyword;

and storing the deleting instruction to the tail part of the operation queue.

14. The method of claim 1, wherein the target data table is determined based on identification information corresponding to the keyword.

15. The method of claim 1, wherein processing the target data table based on the operation request comprises:

calling a target operation method of the embedded database based on the operation type of the operation request;

and processing the target data table based on the target operation method.

16. The method of claim 1,

acquiring the target data table and the time stamp of the data stored in the memory according to a preset period, and deleting the data after the data expiration is determined based on the time stamp; or

And under the condition that the operation type of the operation request is a query type, acquiring a time stamp of the data to be processed, and deleting the target data table and the data to be processed stored in the memory after determining that the data to be processed is out of date based on the time stamp.

17. A data processing apparatus, comprising:

a receiving module, configured to receive an operation request for processing data stored in an embedded database, where the operation request at least includes: keywords corresponding to data to be processed;

the first processing module is used for processing an internal memory based on the operation request, wherein the internal memory is used for storing the data to be processed;

and the second processing module is used for processing a target data table in the embedded database based on the operation request after the memory processing is finished, wherein the target data table is used for storing the data to be processed.

18. The apparatus of claim 17, wherein the first processing module comprises:

the calling unit is used for calling a target operation method of the embedded database based on the operation type of the operation request;

and the processing unit is used for processing the target data table based on the target operation method.

19. The apparatus of claim 17, further comprising:

the deleting module is used for acquiring the target data table and the time stamp of the data stored in the memory according to a preset period, and deleting the data after the data expiration is determined based on the time stamp; or

20. A storage medium, characterized in that the storage medium comprises a stored program, wherein when the program runs, a device in which the storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 16.

21. A processor, characterized in that the processor is configured to run a program, wherein the program is configured to execute the data processing method according to any one of claims 1 to 16 when running.