CN112559244A - Data processing method and device, electronic equipment and computer readable medium - Google Patents
Data processing method and device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN112559244A CN112559244A CN201910913390.4A CN201910913390A CN112559244A CN 112559244 A CN112559244 A CN 112559244A CN 201910913390 A CN201910913390 A CN 201910913390A CN 112559244 A CN112559244 A CN 112559244A
- Authority
- CN
- China
- Prior art keywords
- keyword
- database
- value
- data
- redis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 22
- 238000011084 recovery Methods 0.000 claims abstract description 65
- 238000000034 method Methods 0.000 claims abstract description 38
- 230000002688 persistence Effects 0.000 claims abstract description 27
- 238000012545 processing Methods 0.000 claims abstract description 22
- 230000002085 persistent effect Effects 0.000 claims description 31
- 238000004590 computer program Methods 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 4
- 230000009286 beneficial effect Effects 0.000 abstract description 4
- 230000002159 abnormal effect Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 10
- 230000004048 modification Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1471—Saving, restoring, recovering or retrying involving logging of persistent data for recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a data processing method, a data processing device, electronic equipment and a computer readable medium. An embodiment of the method comprises: receiving a data query request, wherein the data query request comprises keywords to be queried; inquiring a value corresponding to the keyword in the redis; if the query is found, the following persistence operations are executed: carrying out self-increment operation on the inquired value, and taking the self-increment operation result as a target value corresponding to the keyword; the keywords and target values are persisted to a database. The embodiment avoids the problem of data loss, and is beneficial to improving the data recovery speed after redis exception.
Description
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to a data processing method, a data processing device, electronic equipment and a computer readable medium.
Background
redis is a storage system in the form of key-value. Redis typically periodically persists updated data to disk or records data modifications to a file for data recovery in case of system anomalies.
In the existing method, data in the redis is usually processed in an RDB (RDB) manner or an AOF (appended-only-file) manner. The RDB mode is to generate a snapshot of data in the redis memory and persist the snapshot to the disk within a specified time period, so that recovery is performed based on the data in the disk when the redis is abnormal. However, when an exception occurs to redis before the persistence operation, this approach will result in the loss of data for the current cycle. For example, in the previous time period, the current document number (value) corresponding to a certain document (key) is 1000, in the current time period, the question number corresponding to the document is 1001, and if redis is abnormal before the persistence operation, the value 1001 is lost.
In addition, the AOF method records data modification operations of redis in a file form, and then performs data recovery through the file when the redis is abnormal. However, the amount of file data generated in this way is large, resulting in a slow data recovery speed.
Disclosure of Invention
The embodiment of the application provides a data processing method, a data processing device, electronic equipment and a computer readable medium, avoids data loss and is beneficial to improving the data recovery speed after redis exception.
In a first aspect, an embodiment of the present application provides a data processing method, where the method includes: receiving a data query request, wherein the data query request comprises keywords to be queried; inquiring a value corresponding to the keyword in the redis; if the query is found, the following persistence operations are executed: performing self-increment operation on the inquired value, and taking a self-increment operation result as a target value corresponding to the keyword; persisting the key and the target value to a database.
In a second aspect, an embodiment of the present application provides a data processing apparatus, including: a receiving unit configured to receive a data query request, wherein the data query request includes a keyword to be queried; a first query unit configured to query a value corresponding to the keyword in redis; the persistence unit is configured to execute the following persistence operations if the query is found: performing self-increment operation on the inquired value, and taking a self-increment operation result as a target value corresponding to the keyword; persisting the key and the target value to a database.
In a third aspect, an embodiment of the present application provides an electronic device, including at least one processor, and at least one memory and a bus connected to the processor; the processor and the memory complete mutual communication through a bus; the processor is used for calling the program instructions in the memory so as to execute the data processing method.
In a fourth aspect, the present application provides a computer-readable medium, on which a computer program is stored, which when executed by a processor implements the data processing method as described in the first aspect.
According to the data processing method, the data processing device, the electronic equipment and the computer readable medium, after a data query request comprising keywords is received, a value corresponding to the keywords is queried in a redis; if the query is found, the following persistence operations are executed: performing self-increment operation on the inquired value, and taking a self-increment operation result as a target value corresponding to the keyword; persisting the key and the target value to a database. Therefore, the value corresponding to the keyword can be directly updated after data query is carried out, the updated data is persisted, the problem of data loss caused by the RDB mode is solved, and meanwhile, the data recovery speed after redis exception is improved in comparison with the AOF mode.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is a flow diagram of one embodiment of a data processing method according to the present application;
FIG. 2 is a flow diagram of yet another embodiment of a data processing method according to the present application;
FIG. 3 is a schematic block diagram of one embodiment of a data processing apparatus according to the present application;
FIG. 4 is a block diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Referring to FIG. 1, a flow 100 of one embodiment of a data processing method according to the present application is shown. The data processing method comprises the following steps:
In the present embodiment, an execution subject (a device such as a server) of the data processing method may be installed with redis. Among them, redis is a storage system in the form of key-value. In practice, by using the key, the value (value) corresponding to the key can be searched from the redis.
In this embodiment, the execution subject may receive a data query request. The data query request comprises keywords to be queried. For example, when storing the number of a document by redis, the number name (e.g., "subject word") may be used as a key (key) to query the number (value) corresponding to the key.
And 102, inquiring a value corresponding to the keyword in the redis.
In this embodiment, after receiving a data query request, the execution main body may query a value corresponding to a keyword carried in the request in redis. In practice, a query may be performed using a query command of redis (e.g., mget command, etc.).
It should be noted that, if the value corresponding to the keyword is queried in redis, the persistence operations from step 103 to step 104 may be performed.
And 103, if the keyword is inquired, performing self-increment operation on the inquired value, and taking the self-increment operation result as a target value corresponding to the keyword.
In this embodiment, if a value corresponding to the keyword is queried in redis, a self-increment operation may be performed on the queried value to obtain a self-increment operation result, and the self-increment operation result is used as a target value corresponding to the keyword. The target value may be considered the most recent value of the key.
Here, the auto-increment operation is an operation of adding 1 to the original value. For example, when a number of a certain document (e.g., "receiving case") is searched for in redis, if a value of 1000 is found, 1001 can be obtained by performing a self-increment operation on the value.
It can be understood that the text number, the code, etc. are usually generated sequentially, and after being used once, a new text number should be returned in the next query, so that after the value corresponding to the keyword is queried, the value can be updated by performing a self-increment operation on the value. Thus, when the user makes a query for the keyword, the latest value can be returned.
In addition, it can be understood that, after obtaining the target value corresponding to the keyword, the execution subject may return the value, so as to facilitate the user to view or other program calls.
In this embodiment, the execution subject may persist the keyword and the target value corresponding to the keyword to a database. The database may be any relational database (e.g., Oracle, SQLServer, etc.).
In one scenario, the database may be a local database installed in the execution entity. In this case, the execution agent may directly combine the key and the target value into a key value pair, and store the key value pair in the local database.
In another scenario, the database may be a database installed in another device communicatively connected to the execution entity. In this case, the execution agent may combine the keyword and the target value into a key value pair, and then store the key value pair in the database by wired connection or wireless connection. It should be noted that the wireless connection means may include, but is not limited to, a 3G/4G connection, a WiFi connection, a bluetooth connection, a WiMAX connection, a Zigbee connection, a uwb (ultra wideband) connection, and other wireless connection means now known or developed in the future.
When a value corresponding to a certain keyword is found in redis, the value is updated (i.e., a self-increment operation), and thus the latest query result (i.e., a target value) can be obtained. Meanwhile, after a data query request is received, the updated target value can be persisted to the database, and the problem of data loss is avoided compared with an RDB mode of persisting after the current time period is over. In addition, the data is directly persisted, and compared with an AOF mode of recording data modification operation of the redis in a file form, the method is beneficial to improving the data recovery speed after the redis is abnormal.
In some optional implementations of this embodiment, the execution principal may persist the keywords and the target values to the database by:
first, a first update message including the keyword and the target value is sent to a first message queue.
The first message queue is a queue for storing update messages. The update message is used to indicate that the latest data (i.e., the key and value in the update message) is persisted to the database. The messages in the first message queue can follow a first-in first-out rule, and the messages which are preferentially entered into the first message queue can be preferentially consumed, so that the problem of data exception during concurrence of the persistence tasks can be avoided.
And secondly, persisting the keywords and the target values in the first updating message in the first message queue to a database through a message queue processing program. The message queue processing program can be various open-source message queue service software (such as RabbitMQ and the like). The message queue handler may handle messages of multiple message queues. And for each message queue, the messages in the message queue can be processed in turn.
Optionally, when processing the first update message in the first message queue, the message queue processing program may specifically execute the following steps:
first, the database is queried for the above-mentioned keywords ("case-accepted words" as exemplified by step 103).
If not, the key word (i.e., "subject word") and the target value (i.e., 1001) are persisted to the database. This makes it possible to persist the keyword and the latest value corresponding to the keyword. When the redis is abnormal, the data in the redis can be recovered through the data which is persisted to the database.
If so, it may be determined whether the value corresponding to the keyword in the database is less than the target value (i.e., 1001). If the value is less than (e.g., the value corresponding to the keyword in the database is 1000), the value (1000) corresponding to the keyword in the database may be updated to the target value (1001). Therefore, the data which is persisted can be updated, and the data which is persisted in the database is the latest data.
In the method provided by the above embodiment of the present application, after receiving a data query request including a keyword, a value corresponding to the keyword is queried in redis; if the query is found, the following persistence operations are executed: carrying out self-increment operation on the inquired value, and taking the self-increment operation result as a target value corresponding to the keyword; and persisting the keywords and the target value to a database. Therefore, the value corresponding to the keyword can be directly updated after data query is carried out, the updated data is persisted, and the problem of data loss is avoided compared with an RDB mode of persisting after the current time period is finished. In addition, the data is directly persisted, and compared with an AOF mode of recording data modification operation of the redis in a file form, the method is beneficial to improving the data recovery speed after the redis is abnormal.
With further reference to FIG. 2, a flow 200 of yet another embodiment of a data processing method is shown. The flow 200 of the data processing method comprises the following steps:
Step 201 in this embodiment can refer to step 101 in the embodiment shown in fig. 1, and is not described herein again.
Step 202 in this embodiment can refer to step 102 in the embodiment shown in fig. 1, and is not described here again.
It should be noted that, when the value corresponding to the keyword is queried in the redis, the persistence operations of steps 203 to 204 may be performed. When the value corresponding to the key is not queried in redis, step 205 may be performed.
Step 203 in this embodiment can refer to step 103 in the embodiment shown in fig. 1, and is not described herein again.
In this embodiment, the execution agent may first send a first update message including the keyword and the target value to a first message queue. The first message queue is a queue for storing update messages. The update message is used to indicate that the latest data (i.e., the key and value in the update message) is persisted to the database. The messages in the first message queue can follow a first-in first-out rule, and the messages which are preferentially entered into the first message queue can be preferentially consumed, so that the problem of data exception during concurrence of the persistence tasks can be avoided.
Then, the execution subject may persist the keyword and the target value in the first update message in the first message queue to a database through a message queue processing program. The message queue processing program can be various open-source message queue service software (such as RabbitMQ and the like). The message queue handler may handle messages of multiple message queues. And for each message queue, the messages in the message queue can be processed in turn.
When processing the first update message in the first message queue, the message queue processing program may specifically execute the following steps:
first, the database is queried for the above-mentioned keyword (e.g., "case-accepted word").
If not, the key word (i.e., the "subject word") and the target value (i.e., 1001) are persisted to the database. This makes it possible to persist the keyword and the latest value corresponding to the keyword. When the redis is abnormal, the data in the redis can be recovered through the data which is persisted to the database.
If the query is found in the database, it can be determined whether the value corresponding to the keyword in the database is less than the target value (i.e., 1001). If the value is less than (e.g., the value corresponding to the keyword in the database is 1000), the value (1000) corresponding to the keyword in the database may be updated to the target value (1001). Therefore, the data which is persisted can be updated, and the data which is persisted in the database is the latest data.
In this embodiment, when the value corresponding to the keyword is not queried in the redis, the value corresponding to the keyword may be queried from the database.
When the value corresponding to the key is not found in the database, step 206 may be performed. When the database is searched for the value corresponding to the keyword, step 207 may be executed.
In this embodiment, if the database does not search for the value corresponding to the keyword, it indicates that neither redis nor the database stores the data, and the keyword appears for the first time. Thus, a key and a value corresponding to the key may be created in redis, and the created value is an initial value of the key. After the keywords and the values corresponding to the keywords are created, the execution main body can persist the created keywords and the values to the database, so that data in the redis can be recovered through the data persisted to the database when the redis is abnormal.
In some optional implementations of this embodiment, the execution subject may send a second update message including the created key and value to the first message queue, and persist, by the message queue handler, the key and value in the second update message in the first message queue to the database. Here, the operation of persisting the key and the value in the second update message to the database is substantially the same as the operation of persisting the key and the target value in the first update message to the database in step 204, and details are not repeated here.
Step 207, if the value corresponding to the keyword is queried in the database, sending a data recovery message containing the keyword and the queried value to the second message queue, and storing the keyword and the value in the data recovery message in the second message queue to redis through a message queue processing program.
In this embodiment, if the database is searched for a value corresponding to the keyword, it means that redis has lost data due to an abnormal cause. At this time, the keyword and the value corresponding to the keyword queried from the database may be stored in redis to recover the data in the redis.
Therefore, the embodiment of the application is recovery on demand when data recovery is performed, that is, when a data query request is received, if the data does not exist in the redis, the data recovery is performed from the database. By the method, only the keywords and the values related to the current request are recovered, the current data query requirement can be met, the redis is not required to be used after all data are replied, and the recovery speed of the redis is improved.
In this embodiment, data recovery may be specifically performed through the following steps:
first, a data recovery message containing the above-described key and the queried value is sent to a second message queue. The second message queue is a queue for storing data recovery messages. The data recovery message is used to instruct recovery of the extracted data from the database (i.e., the key and value in the data recovery message) to redis. The messages in the second message queue can follow a first-in first-out rule, and the messages which are preferentially entered into the second message queue can be preferentially consumed, so that the problem of data exception when the data recovery task is concurrent can be avoided.
And then, storing the key words and the values in the data recovery messages in the second message queue to redis through the message queue processing program. When processing the data recovery message in the second message queue, the following steps may be specifically executed by the message queue processing program: firstly, inquiring whether keywords in the data recovery message exist in the redis; if the key does not exist, storing the key and the value in the data recovery message in the redis.
It should be noted that, since the message queue processing program sequentially processes the data recovery messages in the second message queue, if there is an earlier data recovery message before the sent data recovery message, it is necessary to wait until the earlier data recovery message is processed, and then perform processing on the currently sent data recovery message. Thus, after sending the data recovery message containing the key and the queried value to the second message queue, the executing entity may also perform the querying operation of steps 208-210 as follows.
And step 208, re-querying the value corresponding to the keyword in the redis, and recording the query times.
In this embodiment, after redis re-query, if the value corresponding to the keyword is queried, the persistence operations in steps 203 to 204 may be performed. Firstly, carrying out self-increment operation on the inquired value, and taking the self-increment operation result as a target value corresponding to the keyword; and then sending a first updating message containing the keywords and the target value to the first message queue, and persisting the keywords and the target value in the first updating message in the first message queue to a database through a message queue processing program.
It can be understood that the text number, the code, etc. are usually generated sequentially, and after being used once, a new text number should be returned in the next query, so that after the value corresponding to the keyword is queried, the value can be updated by performing a self-increment operation on the value. Thus, when the user makes a query for the keyword, the latest value can be returned. In addition, the result of the self-increment operation is taken as the latest value and persisted to the database, so that the persisted data can be updated, and the data persisted to the database is the latest data.
In this embodiment, after redis re-querying, if the value corresponding to the keyword is not queried, it indicates that the message queue processing program has not processed the data recovery message sent in step 207, or that the data recovery message sent in step 207 fails to be processed. Thus, the number of queries may be set, and when the number of queries is greater than a certain preset number (e.g., 5) and the value corresponding to the keyword has not been queried in redis, it may be regarded that the data recovery message sent in step 207 fails to be processed. When the number of queries is less than or equal to the preset number (e.g., 5) and the value corresponding to the keyword has not been queried in redis, it may be considered that the message queue handler has not processed the data recovery message sent in step 207.
Thus, in the present embodiment, it may be determined whether the number of inquiries is less than or equal to a preset number of times (e.g., 5 times). If the number of times is greater than the preset number of times, step 210 may be executed; if the number of times is less than or equal to the predetermined number of times, step 211 may be executed.
And step 210, if the number of times is more than the preset number of times, returning prompt information for indicating data recovery failure.
In this embodiment, if the number of queries is greater than the preset number, the executing entity may determine that the data recovery message sent in step 207 fails to be processed. Thus, a hint information indicating a failure of data recovery may be returned.
In step 211, if the number of times is less than or equal to the preset number of times, after waiting for a preset duration, the query operation is executed again.
In this embodiment, if the number of queries is less than or equal to the predetermined number, the executing entity may wait for a predetermined time (e.g., 500 ms) before re-executing the query operations of steps 209 to 210. Therefore, whether the redis data is successfully recovered or not can be determined by setting the waiting time length and the query times when the data is recovered.
As can be seen from fig. 2, compared with the embodiment corresponding to fig. 1, the flow 200 of the data processing method in this embodiment relates to an operation of data persistence or redis recovery through message communication. The message queue follows the first-in first-out rule, so that the data exception problem when the data recovery task is concurrent and the data exception problem when the data recovery task is concurrent can be avoided.
With further reference to fig. 3, as an implementation of the methods shown in the above-mentioned figures, the present application provides an embodiment of a data processing apparatus, which corresponds to the embodiment of the method shown in fig. 1, and which may be applied in various devices.
As shown in fig. 3, the data processing apparatus 300 according to the present embodiment includes: a receiving unit 301 configured to receive a data query request, where the data query request includes a keyword to be queried; a first querying unit 302 configured to query a value corresponding to the keyword in redis; a persistence unit 303 configured to, if queried, perform a persistence operation as follows: carrying out self-increment operation on the inquired value, and taking the self-increment operation result as a target value corresponding to the keyword; and persisting the keywords and the target value to a database.
In some optional implementations of the present embodiment, the persistence unit 303 is further configured to: and sending a first updating message containing the keywords and the target value to a first message queue, and persisting the keywords and the target value in the first updating message in the first message queue to a database through a message queue processing program.
In some optional implementations of the present embodiment, the persistence unit 303 is further configured to: the message queue processing program executes the following steps: inquiring the keywords in a database; if not, persisting the keywords and the target value to the database; if yes, determining whether the value corresponding to the keyword in the database is smaller than the target value; if the value is smaller than the target value, updating the value corresponding to the keyword in the database to the target value.
In some optional implementations of this embodiment, the apparatus further includes: the second query unit is configured to query the database for the value corresponding to the keyword if the value is not queried; and a data recovery unit configured to send a data recovery message including the keyword and the queried value to a second message queue if the database queries the value corresponding to the keyword, and store the keyword and the value in the data recovery message in the second message queue to redis through the message queue processing program.
In some optional implementations of this embodiment, the data recovery unit is further configured to: the message queue processing program executes the following steps: inquiring whether keywords in the data recovery message exist in the redis; and if the data recovery message does not exist, storing the key words and the values in the data recovery message in the redis.
In some optional implementations of this embodiment, the apparatus further includes: a query operation execution unit configured to execute a query operation as follows: re-inquiring the value corresponding to the keyword in the redis, and recording the inquiry times; if yes, executing the persistence operation; if not, determining whether the query times are less than or equal to preset times; if not, returning prompt information for indicating data recovery failure; and the repeated execution unit is configured to execute the query operation again after waiting for a preset time length if the number of times is less than or equal to the preset number of times.
In some optional implementations of this embodiment, the apparatus further includes: and the creating unit is configured to create the keyword and the value corresponding to the keyword in redis and persist the created keyword and the value to the database if the value corresponding to the keyword is not searched in the database.
In some optional implementations of this embodiment, the creating unit is further configured to: and sending a second updating message comprising the created key words and the created values to the first message queue, and persisting the key words and the values in the second updating message in the first message queue to a database through a message queue processing program.
According to the data processing device provided by the embodiment of the application, after a data query request comprising a keyword is received, a value corresponding to the keyword is queried in redis; if the query is found, the following persistence operations are executed: performing self-increment operation on the inquired value, and taking a self-increment operation result as a target value corresponding to the keyword; persisting the key and the target value to a database. Therefore, the value corresponding to the keyword can be directly updated after data query is carried out, the updated data is persisted, the problem of data loss caused by the RDB mode is solved, and meanwhile, the data recovery speed after redis exception is improved in comparison with the AOF mode.
In addition, the data processing apparatus includes a processor and a memory, the receiving unit, the first querying unit, the persistence unit, and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more than one, the problem of data loss is avoided by adjusting kernel parameters, and meanwhile, the data recovery speed after redis is abnormal is improved.
An embodiment of the present invention provides a computer-readable medium on which a computer program is stored, which, when executed by a processor, implements the data processing method.
The embodiment of the invention provides a processor, which is used for running a program, wherein the data processing method is executed when the program runs.
An embodiment of the present invention provides an electronic device 40, which includes at least one processor 401, at least one memory 403 connected to the processor, and a bus 402; the processor and the memory complete mutual communication through a bus; the processor is used for calling the program instructions in the memory so as to execute the data processing method. The device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device: receiving a data query request, wherein the data query request comprises keywords to be queried; inquiring a value corresponding to the keyword in the redis; if the query is found, the following persistence operations are executed: performing self-increment operation on the inquired value, and taking a self-increment operation result as a target value corresponding to the keyword; persisting the key and the target value to a database.
Optionally, the persisting the keyword and the target value to a database includes: and sending a first updating message containing the keywords and the target value to a first message queue, and persisting the keywords and the target value in the first updating message in the first message queue to a database through a message queue processing program.
Optionally, the persisting, by the message queue handler, the key and the target value in the first update message in the first message queue to a database includes: executing the following steps by the message queue processing program: querying the keyword in a database; if the keyword and the target value are not inquired, the keyword and the target value are persisted to the database; if yes, determining whether the value corresponding to the keyword in the database is smaller than the target value; and if the value is smaller than the target value, updating the value corresponding to the keyword in the database to the target value.
Optionally, the program, when executed on a data processing device, is further adapted to perform a procedure for initializing the following method steps: if the keyword is not searched, searching a value corresponding to the keyword in a database; if the value corresponding to the keyword is inquired in the database, sending a data recovery message containing the keyword and the inquired value to a second message queue, and storing the keyword and the value in the data recovery message in the second message queue to redis through the message queue processing program.
Optionally, the storing, by the message queue processing program, the key and the value in the data recovery message in the second message queue to redis includes: executing the following steps by the message queue processing program: inquiring whether keywords in the data recovery message exist in the redis; and if not, storing the key words and the values in the data recovery message in the redis.
Optionally, the program, when executed on a data processing device, is further adapted to perform a procedure for initializing the following method steps: the following query operations are performed: re-querying the value corresponding to the keyword in the redis, and recording the query times; if yes, executing the persistence operation; if not, determining whether the query times are less than or equal to preset times; if not, returning prompt information for indicating data recovery failure; and if the number of times is less than or equal to the preset number of times, after waiting for a preset time length, re-executing the query operation.
Optionally, the program, when executed on a data processing device, is further adapted to perform a procedure for initializing the following method steps: if the value corresponding to the keyword is not inquired in the database, the keyword and the value corresponding to the keyword are created in redis, and the created keyword and the value are persisted to the database.
Optionally, the persisting the created key and value to the database includes: sending a second update message comprising the created key and value to the first message queue, and persisting the key and value in the second update message in the first message queue to a database through a message queue handler.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a device includes one or more processors (CPUs), memory, and a bus. The device may also include input/output interfaces, network interfaces, and the like.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A method of data processing, the method comprising:
receiving a data query request, wherein the data query request comprises keywords to be queried;
inquiring a value corresponding to the keyword in the redis;
if the query is found, the following persistence operations are executed:
performing self-increment operation on the inquired value, and taking a self-increment operation result as a target value corresponding to the keyword;
persisting the key and the target value to a database.
2. The method of claim 1, wherein the persisting the keyword and the target value to a database comprises:
and sending a first updating message containing the keywords and the target value to a first message queue, and persisting the keywords and the target value in the first updating message in the first message queue to a database through a message queue processing program.
3. The method of claim 2, wherein said persisting, by a message queue handler, the key and the target value in the first update message in the first message queue to a database comprises:
executing the following steps by the message queue processing program:
querying the keyword in a database;
if the keyword and the target value are not inquired, the keyword and the target value are persisted to the database;
if yes, determining whether the value corresponding to the keyword in the database is smaller than the target value; and if the value is smaller than the target value, updating the value corresponding to the keyword in the database to the target value.
4. The method according to claim 1, wherein after said querying a value corresponding to the keyword in redis, the method further comprises:
if the keyword is not searched, searching a value corresponding to the keyword in a database;
if the value corresponding to the keyword is inquired in the database, sending a data recovery message containing the keyword and the inquired value to a second message queue, and storing the keyword and the value in the data recovery message in the second message queue to redis through the message queue processing program.
5. The method of claim 4, wherein storing, by the message queue handler, the key and the value in the data recovery message in the second message queue to redis comprises:
executing the following steps by the message queue processing program:
inquiring whether keywords in the data recovery message exist in the redis;
and if not, storing the key words and the values in the data recovery message in the redis.
6. The method of claim 4, wherein after said sending a data recovery message containing said key and queried value to a second message queue, the method further comprises:
the following query operations are performed: re-querying the value corresponding to the keyword in the redis, and recording the query times; if yes, executing the persistence operation; if not, determining whether the query times are less than or equal to preset times; if not, returning prompt information for indicating data recovery failure;
and if the number of times is less than or equal to the preset number of times, after waiting for a preset time length, re-executing the query operation.
7. The method of claim 4, wherein after the querying the database for the value corresponding to the keyword, the method further comprises:
if the value corresponding to the keyword is not inquired in the database, the keyword and the value corresponding to the keyword are created in redis, and the created keyword and the value are persisted to the database;
wherein the persisting the created keys and values to a database comprises: sending a second update message comprising the created key and value to the first message queue, and persisting the key and value in the second update message in the first message queue to a database through a message queue handler.
8. A data processing apparatus, characterized in that the apparatus comprises:
a receiving unit configured to receive a data query request, wherein the data query request includes a keyword to be queried;
a first query unit configured to query a value corresponding to the keyword in redis;
the persistence unit is configured to execute the following persistence operations if the query is found:
performing self-increment operation on the inquired value, and taking a self-increment operation result as a target value corresponding to the keyword;
persisting the key and the target value to a database.
9. An electronic device comprises at least one processor, at least one memory connected with the processor, and a bus; the processor and the memory complete mutual communication through the bus; the processor is configured to invoke program instructions in the memory to perform the data processing method of any of claims 1-7.
10. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910913390.4A CN112559244A (en) | 2019-09-25 | 2019-09-25 | Data processing method and device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910913390.4A CN112559244A (en) | 2019-09-25 | 2019-09-25 | Data processing method and device, electronic equipment and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112559244A true CN112559244A (en) | 2021-03-26 |
Family
ID=75029447
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910913390.4A Pending CN112559244A (en) | 2019-09-25 | 2019-09-25 | Data processing method and device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112559244A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102542034A (en) * | 2011-12-23 | 2012-07-04 | 北京人大金仓信息技术股份有限公司 | Database interface-based result set cache method |
| CN102867070A (en) * | 2012-09-29 | 2013-01-09 | 瑞庭网络技术（上海）有限公司 | Method for updating cache of key-value distributed memory system |
| CN103886079A (en) * | 2014-03-26 | 2014-06-25 | 北京京东尚科信息技术有限公司 | Data processing method and system |
| US8898126B1 (en) * | 2007-04-04 | 2014-11-25 | Emc Corporation | Method and apparatus for providing concurrent data insertion and updating |
| CN108829740A (en) * | 2018-05-23 | 2018-11-16 | 摇了购（武汉）电子商务有限公司 | Date storage method and device |
| CN109165216A (en) * | 2018-08-02 | 2019-01-08 | 杭州启博科技有限公司 | A kind of generation method and system, storage medium of Redis distributed data base major key id |
| CN109241072A (en) * | 2018-08-31 | 2019-01-18 | 携程计算机技术（上海）有限公司 | Buffering updating method and system based on Canal |
| CN110019179A (en) * | 2017-07-31 | 2019-07-16 | 北京嘀嘀无限科技发展有限公司 | Update method and device, the electronic equipment, storage medium of index database |
-
2019
- 2019-09-25 CN CN201910913390.4A patent/CN112559244A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8898126B1 (en) * | 2007-04-04 | 2014-11-25 | Emc Corporation | Method and apparatus for providing concurrent data insertion and updating |
| CN102542034A (en) * | 2011-12-23 | 2012-07-04 | 北京人大金仓信息技术股份有限公司 | Database interface-based result set cache method |
| CN102867070A (en) * | 2012-09-29 | 2013-01-09 | 瑞庭网络技术（上海）有限公司 | Method for updating cache of key-value distributed memory system |
| CN103886079A (en) * | 2014-03-26 | 2014-06-25 | 北京京东尚科信息技术有限公司 | Data processing method and system |
| CN110019179A (en) * | 2017-07-31 | 2019-07-16 | 北京嘀嘀无限科技发展有限公司 | Update method and device, the electronic equipment, storage medium of index database |
| CN108829740A (en) * | 2018-05-23 | 2018-11-16 | 摇了购（武汉）电子商务有限公司 | Date storage method and device |
| CN109165216A (en) * | 2018-08-02 | 2019-01-08 | 杭州启博科技有限公司 | A kind of generation method and system, storage medium of Redis distributed data base major key id |
| CN109241072A (en) * | 2018-08-31 | 2019-01-18 | 携程计算机技术（上海）有限公司 | Buffering updating method and system based on Canal |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10439937B2 (en) | Service addressing in distributed environment | |
| CN107832406B (en) | Method, device, equipment and storage medium for removing duplicate entries of mass log data | |
| US20170054808A1 (en) | Rapid client-side component processing based on component relationships | |
| CN107590207B (en) | Data synchronization method and device and electronic equipment | |
| US11474990B2 (en) | Priority queue for exclusive locks | |
| CN111414389B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN106326309B (en) | Data query method and device | |
| CN111930848B (en) | Data partition storage method, device and system | |
| US9659021B1 (en) | Client based backups and backup indexing | |
| CN102780603B (en) | Web traffic control method and device | |
| US11507277B2 (en) | Key value store using progress verification | |
| US10642530B2 (en) | Global occupancy aggregator for global garbage collection scheduling | |
| CN111737564A (en) | Information query method, device, equipment and medium | |
| CN107451203B (en) | Database access method and device | |
| CN112559244A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| CN110019295B (en) | Database retrieval method, device, system and storage medium | |
| US20150106363A1 (en) | Computer system, data management method, and recording medium storing program | |
| CN109918178B (en) | Transaction submitting method and related device | |
| CN110333883B (en) | Method and device for updating persistent data | |
| CN112699129A (en) | Data processing system, method and device | |
| CN107844491B (en) | Method and equipment for realizing strong consistency read operation in distributed system | |
| CN114650287B (en) | Breakpoint continuous method, device and system for data file | |
| US20200125570A1 (en) | Optimization of first set of ordered items | |
| CN117632544A (en) | Task processing method, device and equipment based on stock data call | |
| CN117350805A (en) | Order management method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |