CN116860714A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116860714A CN116860714A CN202310521999.3A CN202310521999A CN116860714A CN 116860714 A CN116860714 A CN 116860714A CN 202310521999 A CN202310521999 A CN 202310521999A CN 116860714 A CN116860714 A CN 116860714A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- version number
- data version
- queried
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 41
- 238000012216 screening Methods 0.000 claims abstract description 36
- 239000007787 solid Substances 0.000 claims abstract description 31
- 238000000034 method Methods 0.000 claims description 28
- 238000004590 computer program Methods 0.000 claims description 15
- 238000012544 monitoring process Methods 0.000 claims description 8
- 230000006399 behavior Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention provides a data processing method, a device, electronic equipment and a storage medium, comprising the following steps: acquiring keywords and a first data version number of batch of data to be processed, which are sent by a client; for each piece of data to be processed, inquiring a second data version number corresponding to the keyword from a cache database stored based on the solid state disk; comparing the queried second data version number with the first data version number, and screening the batch of data to be processed based on the comparison result; and updating the data stored in the background database according to the data to be processed obtained by screening, wherein the updated data in the background database is used for performing task processing. The invention adopts the database stored on the solid state disk to query data, and stores the key words and the data version number fields of the data on the solid state disk.
Description
Technical Field
The present invention relates to the field of computer technology, and in particular, to a data processing method, a data processing apparatus, an electronic device, and a computer readable storage medium.
Background
With the continuous development of intelligent terminals, users can watch videos through video clients on the intelligent terminals, comment, barrage and the like can be issued for video content, and the data can be delivered to the background through a message queue for processing.
Redundant data is easily generated due to network delay and the like, and data stored in a memory is generally required to be queried to intercept the redundant data in delivery data. However, in a high concurrency scenario, a large amount of data is delivered to the background, so that a large amount of data needs to be stored in the memory for inquiry, which easily results in high memory occupancy rate and further increases storage cost.
Disclosure of Invention
In view of the above problems, embodiments of the present invention have been made to provide a data processing method, a data processing apparatus, an electronic device, and a computer-readable storage medium that overcome or at least partially solve the above problems.
In order to solve the above problems, an embodiment of the present invention discloses a data processing method, including:
acquiring keywords and a first data version number of batch of data to be processed, which are sent by a client;
for each piece of data to be processed, inquiring a second data version number corresponding to the keyword from a cache database stored based on the solid state disk;
comparing the queried second data version number with the first data version number, and screening the batch of data to be processed based on the comparison result;
and updating the data stored in the background database according to the data to be processed obtained by screening, wherein the updated data in the background database is used for performing task processing.
Optionally, the screening the batch of data to be processed based on the comparison result includes:
discarding the data to be processed if the queried second data version number is greater than the first data version number;
and if the queried second data version number is smaller than or equal to the first data version number, reserving the data to be processed, and updating the second data version number recorded in the cache database into the first data version number.
Optionally, after the querying the second data version number corresponding to the keyword, the method further includes:
and if the second data version number corresponding to the keyword is not queried, storing the data to be processed into the cache database.
Optionally, the updating the data stored in the background database according to the data to be processed obtained by screening includes:
carrying out service logic processing on the data to be processed obtained through screening;
and updating the data stored in the background database according to the data processed by the business logic.
Optionally, the updating the data in the background database according to the data processed by the service logic includes:
inquiring a third data version number corresponding to a keyword of the data processed by the business logic from a background database aiming at the data processed by each business logic;
and comparing the queried third data version number with the first data version number of the data processed by the business logic, and updating the data corresponding to the key words of the data processed by the business logic in the background database based on the comparison result.
Optionally, the updating the data corresponding to the keyword of the data processed by the business logic in the background database based on the comparison result includes:
if the queried third data version number is smaller than or equal to the first data version number of the data processed by the business logic, updating the data corresponding to the key words of the data processed by the business logic in a background database by adopting the data processed by the business logic;
and if the queried third data version number is greater than the first data version number of the data processed by the business logic, keeping the data in the background database unchanged.
Optionally, the obtaining the keyword and the first data version number of the user behavior data sent by the client includes:
monitoring batch data to be processed sent by a client;
analyzing the batch of data to be processed to obtain keywords and a first data version number of the batch of data to be processed.
The embodiment of the invention also discloses a data processing device, which comprises:
the acquisition module is used for acquiring keywords and a first data version number of batch of data to be processed, which are sent by the client;
the query module is used for querying a second data version number corresponding to the keyword from a cache database stored based on the solid state disk for each piece of data to be processed;
the comparison module is used for comparing the queried second data version number with the first data version number and screening the batch of data to be processed based on the comparison result;
the updating module is used for updating the data stored in the background database according to the data to be processed obtained through screening, and the updated data in the background database is used for performing task processing.
Optionally, the comparison module includes:
the discarding sub-module is used for discarding the data to be processed if the queried second data version number is greater than the first data version number;
and the reservation sub-module is used for reserving the data to be processed and updating the second data version number recorded in the cache database into the first data version number if the queried second data version number is smaller than or equal to the first data version number.
Optionally, after the querying the second data version number corresponding to the keyword, the method further includes:
and the inserting module is used for storing the data to be processed into the cache database if the second data version number corresponding to the keyword is not queried.
Optionally, the updating module includes:
the service logic processing sub-module is used for carrying out service logic processing on the data to be processed obtained through screening;
and the data updating sub-module is used for updating the data stored in the background database according to the data processed by the business logic.
Optionally, the data updating sub-module includes:
the version number inquiry unit is used for inquiring a third data version number corresponding to the keyword of the data processed by the business logic from the background database aiming at the data processed by each business logic;
and the version number comparison unit is used for comparing the queried third data version number with the first data version number of the data processed by the business logic, and updating the data corresponding to the key words of the data processed by the business logic in the background database based on the comparison result.
Optionally, the version number comparison unit includes:
a background data updating subunit, configured to update, with the data processed by the service logic, data in a background database corresponding to a keyword of the data processed by the service logic, if the queried third data version number is less than or equal to the first data version number of the data processed by the service logic;
and the background data does not update the subunit, and is used for keeping the data in the background database unchanged if the queried third data version number is greater than the first data version number of the data processed by the service logic.
Optionally, the acquiring module includes:
the monitoring sub-module is used for monitoring batch data to be processed sent by the client;
and the analysis submodule is used for analyzing the batch of data to be processed to obtain keywords and a first data version number of the batch of data to be processed.
The embodiment of the invention also discloses an electronic device, which comprises: a processor, a memory and a computer program stored on the memory and capable of running on the processor, which when executed by the processor implements the steps of the data processing method as described above.
The embodiment of the invention also discloses a computer readable storage medium, wherein the computer readable storage medium stores a computer program which realizes the steps of the data processing method when being executed by a processor.
The embodiment of the invention has the following advantages:
in the embodiment of the invention, a keyword and a first data version number of batch data to be processed, which are sent by a client, are acquired; for each piece of data to be processed, inquiring a second data version number corresponding to the keyword from a cache database stored based on the solid state disk; comparing the queried second data version number with the first data version number, and screening the batch of data to be processed based on the comparison result; according to the data to be processed obtained by screening, the data stored in the background database is updated, the updated data in the background database is used for task processing, and compared with the data query in the prior art by adopting a database stored based on a memory, the data query in the invention is performed by adopting the database stored based on the solid state disk, and the key words and the data version number fields of the data are stored on the solid state disk.
Drawings
FIG. 1 is a flow chart of steps of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating steps of another data processing method according to an embodiment of the present invention;
FIGS. 2A-2E are flowcharts of sub-steps of another data processing method provided by an embodiment of the present invention;
FIG. 3 is a flow chart of a data processing method according to an embodiment of the present invention;
fig. 4 is a block diagram of a data processing apparatus according to an embodiment of the present invention.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Redundant data usually exists in data delivered to the background, and the data stored in the memory needs to be queried first to intercept the redundant data in the delivered data. However, in a high concurrency scenario, a large amount of data is delivered to the background, so that a large amount of data needs to be stored in the memory for inquiry, which easily results in high memory occupancy rate and further results in increased memory cost.
One of the core concepts of the embodiment of the invention is that a key word and a first data version number of batch of data to be processed sent by a client are obtained; for each piece of data to be processed, inquiring a second data version number corresponding to the keyword from a cache database stored based on the solid state disk; comparing the queried second data version number with the first data version number, and screening the batch of data to be processed based on the comparison result; according to the data to be processed obtained by screening, the data stored in the background database is updated, the updated data in the background database is used for task processing, and compared with the data query in the prior art by adopting a database stored based on a memory, the data query in the invention is performed by adopting the database stored based on the solid state disk, and the key words and the data version number fields of the data are stored on the solid state disk.
Referring to fig. 1, a step flowchart of a data processing method provided by an embodiment of the present invention is shown, where the method specifically may include the following steps:
step 101, obtaining a keyword and a first data version number of batch of to-be-processed data sent by a client.
The data processing method of the embodiment of the invention can be applied to a scene of task processing, and a user can watch videos through a video client, when a large number of users need to issue a barrage and comment on the videos being watched, a large number of data to be processed can be generated, the data to be processed can be delivered to a content security auditing platform through a message queue, a data version number recorded in the data to be processed is inquired from a cache database stored on a solid state disk, the data version number of each piece of data to be processed is compared with the inquired data version number, and therefore, the data to be processed in batches are screened on the basis of comparison results, and the data stored in a background database is updated on the basis of the screened data to enable the content security auditing platform to conduct content security auditing processing on the updated data.
In practical application, the data to be processed can be delivered to a customer service platform, and task processing is carried out on the updated data; the data to be processed may also be delivered to an OA (Office Automation ) management platform, and OA task work orders may be performed on the updated data, which is not limited herein.
The key may be a data identifier, may be a single field, or may be a combination of multiple fields. The data version number may be a numerical value for indicating the number of times of data update, for example, when the user inputs comment information, the same piece of comment data may be delivered twice due to network katon delay or the like, the data version number of comment data delivered for the first time may be 1, and the data version number of comment data delivered for the second time may be 2; for another example, the user may modify the comment information that has been published, the data version number of the comment data before modification may be 1, and the data version number of the comment data after modification may be 2.
And 102, inquiring a second data version number corresponding to the keyword from a cache database stored based on the solid state disk for each piece of data to be processed.
The cache database can be a database based on solid state disk storage data, and can be used for caching key words and data version number fields of the data. In the embodiment of the invention, the database based on the solid state disk for storing the data is adopted for inquiry, and the key words and the data version number fields of the data are stored on the solid state disk in a high concurrency scene because the storage capacity of the solid state disk is large, so that the occupancy rate of the memory can be reduced, and the storage cost is reduced.
And step 103, comparing the queried second data version number with the first data version number, and screening the batch of data to be processed based on the comparison result.
For each piece of data to be processed, by comparing the queried second data version number with the first data version number of the data, whether the piece of data to be processed is redundant data can be determined based on the comparison result, so that redundant data in the batch of data to be processed is filtered. In the embodiment of the invention, the problems of overlarge storage pressure of a background database, waste of calculation resources and increase of operation pressure during task processing can be avoided by screening the batch data to be processed.
And 104, updating the data in the background database according to the data to be processed obtained by screening, wherein the updated data in the background database is used for performing task processing.
After the data screening is completed, the data stored in the background database can be updated, so that task processing is performed based on the updated data.
In the embodiment of the invention, a keyword and a first data version number of batch data to be processed, which are sent by a client, are acquired; for each piece of data to be processed, inquiring a second data version number corresponding to the keyword from a cache database stored based on the solid state disk; comparing the queried second data version number with the first data version number, and screening the batch of data to be processed based on the comparison result; according to the data to be processed obtained by screening, the data stored in the background database is updated, the updated data in the background database is used for task processing, and compared with the data query in the prior art by adopting a database stored based on a memory, the data query in the invention is performed by adopting the database stored based on the solid state disk, and the key words and the data version number fields of the data are stored on the solid state disk.
Referring to fig. 2, a step flowchart of a data processing method provided by an embodiment of the present invention is shown, where the method specifically may include the following steps:
step 201, obtaining a keyword and a first data version number of batch of to-be-processed data sent by a client.
In an alternative embodiment, referring to FIG. 2A, the step 201 may include the following sub-steps S11-S12:
and S11, monitoring batch data to be processed sent by the client.
And S12, analyzing the batch of data to be processed to obtain keywords and a first data version number of the batch of data to be processed.
In the embodiment of the invention, the data stream sent by the client can be monitored, the data stream can comprise a large amount of data to be processed, and each piece of data to be processed can be analyzed to obtain the unique key and the first data version number.
Step 202, for each piece of data to be processed, inquiring a second data version number corresponding to the keyword from a cache database stored based on the solid state disk.
And 203, if the second data version number corresponding to the keyword is not queried, storing the data to be processed into the cache database.
When data query is performed on each piece of data to be processed, if the second data version number corresponding to the key is not queried from the cache database stored based on the solid state disk, that is, the data version number corresponding to the key does not exist in the cache database, the piece of data to be processed can be reserved, and the piece of data to be processed can be newly added into the cache database. In the service scene, the user correspondingly generates the data to be processed when posting comments for the first time, and the corresponding second data version number cannot be queried from the cache database, so that the data to be processed can be newly added into the cache database for storage.
And 204, comparing the queried second data version number with the first data version number, and screening the batch of data to be processed based on the comparison result.
In an alternative embodiment, referring to fig. 2B, the step of screening the batch of data to be processed based on the comparison result in step 204 may include the following substeps S21-S22:
and a substep S21, discarding the data to be processed if the queried second data version number is greater than the first data version number.
And step S22, if the queried second data version number is smaller than or equal to the first data version number, reserving the data to be processed, and updating the second data version number recorded in the cache database into the first data version number.
The data version number may be a numerical value indicating the number of data updates, and the larger the data version number, the later the generation time of the data may be indicated, and the newer the data. When the queried second data version number is larger than the first data version number, the delivered data is the expired data, and the data to be processed is discarded, so that the expired data in the data stream is filtered. When the queried second data version number is equal to the first data version number, the delivered data is repeated data, the data recorded in the cache database can be updated, and the data version number field and other fields of the data are updated in the cache database. When the queried second data version number is smaller than the first data version number, the delivered data is new data, the data recorded in the cache database can be updated, and the data version number field and other fields of the data are updated in the cache database.
And 205, updating the data stored in the background database according to the data to be processed obtained by screening, wherein the updated data in the background database is used for performing task processing.
In an alternative embodiment, referring to fig. 2C, the step of updating the data stored in the background database according to the filtered pending data in step 205 may include the following substeps S31-S32:
and S31, carrying out service logic processing on the data to be processed obtained through screening.
And a substep S32, updating the data stored in the background database according to the data processed by the business logic.
After the data to be processed is screened, service logic processing such as filtering sensitive words, anti-garbage and the like can be performed on the screened data to be processed, so that the data stored in the background database is updated based on the data processed by the service logic processing.
In an alternative embodiment, referring to FIG. 2D, the substep S32 may include the following substeps S321-S322:
in the substep S321, for each piece of data processed by the service logic, a third data version number corresponding to the keyword of the data processed by the service logic is queried from the background database.
And sub-step S322, comparing the queried third data version number with the first data version number of the data processed by the business logic, and updating the data corresponding to the key words of the data processed by the business logic in the background database based on the comparison result.
After the business logic processing is performed on the data, a third data version number corresponding to the key word of the data can be queried from the background database for each piece of data, the third data version number of the data is compared with the first data version number, and the data of the background database is updated based on the comparison result.
In an alternative embodiment, referring to fig. 2E, the step of updating, in the sub-step S322, the data corresponding to the key of the data processed by the service logic in the background database based on the comparison result may include the following sub-steps S3221-S3222:
and substep S3221, if the queried third data version number is less than or equal to the first data version number of the service logic processed data, updating the data corresponding to the key words of the service logic processed data in the background database by adopting the service logic processed data.
And substep S3222, if the queried third data version number is greater than the first data version number of the data processed by the service logic, maintaining the data in the background database unchanged.
When the third data version number corresponding to the data key word is not queried from the background database, or the queried third data version number is smaller than or equal to the first data version number of the data processed by the business logic, the delivered data is the new data which needs to be updated to the background database for storage, and the data can be updated to the background database. When the queried third data version number is larger than the first data version number of the data processed by the business logic, the delivered data is the expired data, and the data is not updated to the background database.
In the embodiment of the invention, a data version number recording mechanism is adopted and can be used for realizing optimistic locking, and in the prior art, when filtering a large amount of data to be processed, a pessimistic locking mechanism is generally adopted for improving the query efficiency due to the large redundant data quantity; before the data is updated to the background database, the second data version number in the cache database is compared with the first data version number of the data to be processed, so that batch of data to be processed is screened, the redundant data amount is greatly reduced when the data is updated to the background database, and the query efficiency can be improved by adopting an optimistic locking mechanism.
After updating the background data, other storage modules, such as data lakes, elastic search indexes, etc., may continue to be updated.
In the embodiment of the invention, a keyword and a first data version number of batch data to be processed, which are sent by a client, are acquired; for each piece of data to be processed, inquiring a second data version number corresponding to the keyword from a cache database stored based on the solid state disk; comparing the queried second data version number with the first data version number, and screening the batch of data to be processed based on the comparison result; according to the data to be processed obtained by screening, the data stored in the background database is updated, the updated data in the background database is used for task processing, and compared with the data query in the prior art by adopting a database stored based on a memory, the data query in the invention is performed by adopting the database stored based on the solid state disk, and the key words and the data version number fields of the data are stored on the solid state disk.
In order that those skilled in the art will better understand the embodiments of the present invention, the following description of the embodiments of the present invention will be provided. Referring to fig. 3, a flowchart of a data processing method according to an embodiment of the present invention is provided.
S1, starting a flow;
s2, monitoring a data stream to be processed;
s3, analyzing the data to obtain a keyword and a first data version number of the data;
s4, inquiring a second data version number corresponding to the keyword from the cache database, and comparing the second data version number with the first data version number;
s5, if the second data version number is not queried, namely the second data version does not exist in the cache database, the piece of data is newly added into the cache database;
s6, discarding the data if the queried version number of the second data is greater than the version number of the first data;
s7, if the queried second data version number is smaller than or equal to the first data version number, updating the data corresponding to the data key word in the cache database;
s8, carrying out service logic processing on the filtered data;
s9, inquiring a third data version number corresponding to the keyword of the data processed by the business logic from a background database, comparing the third data version number with the first data version number, and updating the data based on the comparison result so as to update the data based on an optimistic lock mechanism;
s10, updating data in other storage modules such as a data lake and the like;
s11, ending the flow.
It should be noted that, for simplicity of description, the method embodiments are shown as a series of acts, but it should be understood by those skilled in the art that the embodiments are not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred embodiments, and that the acts are not necessarily required by the embodiments of the invention.
Referring to fig. 4, a block diagram of a data processing apparatus according to an embodiment of the present invention is shown, which may specifically include the following modules:
an obtaining module 401, configured to obtain a keyword and a first data version number of batch of to-be-processed data sent by a client;
the query module 402 is configured to query, for each piece of data to be processed, a second data version number corresponding to the keyword from a cache database stored based on the solid state disk;
a comparison module 403, configured to compare the queried second data version number with the first data version number, and screen the batch of data to be processed based on a comparison result;
and the updating module 404 is configured to update data stored in a background database according to the data to be processed obtained by screening, where the updated data in the background database is used for performing task processing.
In an alternative embodiment, the comparison module includes:
the discarding sub-module is used for discarding the data to be processed if the queried second data version number is greater than the first data version number;
and the reservation sub-module is used for reserving the data to be processed and updating the second data version number recorded in the cache database into the first data version number if the queried second data version number is smaller than or equal to the first data version number.
In an alternative embodiment, after the querying the second data version number corresponding to the keyword, the method further includes:
and the inserting module is used for storing the data to be processed into the cache database if the second data version number corresponding to the keyword is not queried.
In an alternative embodiment, the update module includes:
the service logic processing sub-module is used for carrying out service logic processing on the data to be processed obtained through screening;
and the data updating sub-module is used for updating the data stored in the background database according to the data processed by the business logic.
In an alternative embodiment, the data update sub-module includes:
the version number inquiry unit is used for inquiring a third data version number corresponding to the keyword of the data processed by the business logic from the background database aiming at the data processed by each business logic;
and the version number comparison unit is used for comparing the queried third data version number with the first data version number of the data processed by the business logic, and updating the data corresponding to the key words of the data processed by the business logic in the background database based on the comparison result.
In an alternative embodiment, the version number comparison unit includes:
a background data updating subunit, configured to update, with the data processed by the service logic, data in a background database corresponding to a keyword of the data processed by the service logic, if the queried third data version number is less than or equal to the first data version number of the data processed by the service logic;
and the background data does not update the subunit, and is used for keeping the data in the background database unchanged if the queried third data version number is greater than the first data version number of the data processed by the service logic.
In an alternative embodiment, the acquiring module includes:
the monitoring sub-module is used for monitoring batch data to be processed sent by the client;
and the analysis submodule is used for analyzing the batch of data to be processed to obtain keywords and a first data version number of the batch of data to be processed.
In the embodiment of the invention, a keyword and a first data version number of batch data to be processed, which are sent by a client, are acquired; for each piece of data to be processed, inquiring a second data version number corresponding to the keyword from a cache database stored based on the solid state disk; comparing the queried second data version number with the first data version number, and screening the batch of data to be processed based on the comparison result; according to the data to be processed obtained by screening, the data stored in the background database is updated, the updated data in the background database is used for task processing, and compared with the data query in the prior art by adopting a database stored based on a memory, the data query in the invention is performed by adopting the database stored based on the solid state disk, and the key words and the data version number fields of the data are stored on the solid state disk.
For the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points.
The embodiment of the invention also provides electronic equipment, which comprises:
the system comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program realizes the processes of the data processing method embodiment when being executed by the processor, and can achieve the same technical effects, and the repetition is avoided, and the description is omitted here.
The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, realizes the processes of the above data processing method embodiment, and can achieve the same technical effects, and in order to avoid repetition, the description is omitted here.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.
It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the invention may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the embodiments of the invention.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The foregoing has described in detail a data processing method, apparatus, electronic device and storage medium according to the present invention, and specific examples have been used herein to illustrate the principles and embodiments of the present invention, where the above examples are only for aiding in the understanding of the method and core idea of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (10)
1. A method of data processing, the method comprising:
acquiring keywords and a first data version number of batch of data to be processed, which are sent by a client;
for each piece of data to be processed, inquiring a second data version number corresponding to the keyword from a cache database stored based on the solid state disk;
comparing the queried second data version number with the first data version number, and screening the batch of data to be processed based on the comparison result;
and updating the data stored in the background database according to the data to be processed obtained by screening, wherein the updated data in the background database is used for performing task processing.
2. The method of claim 1, wherein the screening the batch of data to be processed based on the comparison result comprises:
discarding the data to be processed if the queried second data version number is greater than the first data version number;
and if the queried second data version number is smaller than or equal to the first data version number, reserving the data to be processed, and updating the second data version number recorded in the cache database into the first data version number.
3. The method of claim 1, further comprising, after the querying a second data version number corresponding to the key:
and if the second data version number corresponding to the keyword is not queried, storing the data to be processed into the cache database.
4. The method according to claim 1, wherein updating the data stored in the background database according to the filtered data to be processed comprises:
carrying out service logic processing on the data to be processed obtained through screening;
and updating the data stored in the background database according to the data processed by the business logic.
5. The method of claim 4, wherein updating the data stored in the background database based on the business logic processed data comprises:
inquiring a third data version number corresponding to a keyword of the data processed by the business logic from a background database aiming at the data processed by each business logic;
and comparing the queried third data version number with the first data version number of the data processed by the business logic, and updating the data corresponding to the key words of the data processed by the business logic in the background database based on the comparison result.
6. The method of claim 5, wherein updating the data in the background database corresponding to the key of the business logic processed data based on the comparison result comprises:
if the queried third data version number is smaller than or equal to the first data version number of the data processed by the business logic, updating the data corresponding to the key words of the data processed by the business logic in a background database by adopting the data processed by the business logic;
and if the queried third data version number is greater than the first data version number of the data processed by the business logic, keeping the data in the background database unchanged.
7. The method of claim 1, wherein the obtaining the key and the first data version number of the user behavior data sent by the client comprises:
monitoring batch data to be processed sent by a client;
analyzing the batch of data to be processed to obtain keywords and a first data version number of the batch of data to be processed.
8. A data processing apparatus, the apparatus comprising:
the acquisition module is used for acquiring keywords and a first data version number of batch of data to be processed, which are sent by the client;
the query module is used for querying a second data version number corresponding to the keyword from a cache database stored based on the solid state disk for each piece of data to be processed;
the comparison module is used for comparing the queried second data version number with the first data version number and screening the batch of data to be processed based on the comparison result;
the updating module is used for updating the data stored in the background database according to the data to be processed obtained through screening, and the updated data in the background database is used for performing task processing.
9. An electronic device, comprising: a processor, a memory and a computer program stored on the memory and capable of running on the processor, which when executed by the processor carries out the steps of the data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the steps of the data processing method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310521999.3A CN116860714A (en) | 2023-05-10 | 2023-05-10 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310521999.3A CN116860714A (en) | 2023-05-10 | 2023-05-10 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116860714A true CN116860714A (en) | 2023-10-10 |
Family
ID=88217805
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310521999.3A Pending CN116860714A (en) | 2023-05-10 | 2023-05-10 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116860714A (en) |
-
2023
- 2023-05-10 CN CN202310521999.3A patent/CN116860714A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11392416B2 (en) | Automated reconfiguration of real time data stream processing | |
| US10129118B1 (en) | Real time anomaly detection for data streams | |
| CA2880349C (en) | Aggregating data in a mediation system | |
| CN111859132A (en) | Data processing method and device, intelligent equipment and storage medium | |
| CN110445828B (en) | Data distributed processing method based on Redis and related equipment thereof | |
| CN108509313B (en) | Service monitoring method, platform and storage medium | |
| CN111241177B (en) | Data acquisition method, system and network equipment | |
| CN111782692B (en) | Frequency control method and device | |
| CN111061758B (en) | Data storage method, device and storage medium | |
| CN112506870B (en) | Data warehouse increment updating method and device and computer equipment | |
| CN107040576A (en) | Information-pushing method and device, communication system | |
| CN111177237B (en) | Data processing system, method and device | |
| CN111563199A (en) | Data processing method and device | |
| EP2224381A1 (en) | Method and apparatus for case-based service composition | |
| CN114969441A (en) | Knowledge mining engine system based on graph database | |
| CN107330031B (en) | Data storage method and device and electronic equipment | |
| CN116860714A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN109284833B (en) | Method, apparatus and storage medium for obtaining feature data for machine learning model | |
| CN113986942B (en) | Message queue management method and device based on man-machine conversation | |
| CN111274316A (en) | Execution method and device of multi-level data flow task, electronic equipment and storage medium | |
| CN117390106B (en) | Data processing method, device, storage medium and computing equipment | |
| CN117009094B (en) | Data oblique scattering method and device, electronic equipment and storage medium | |
| CN112732757B (en) | Method, system, device, equipment and storage medium for processing degraded data | |
| CN110007899B (en) | Storm-based universal window frame system | |
| CN113672593A (en) | Data storage method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |