CN111858576A - Real-time message processing method, system, device, storage medium and electronic equipment - Google Patents
Real-time message processing method, system, device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN111858576A CN111858576A CN201910338181.1A CN201910338181A CN111858576A CN 111858576 A CN111858576 A CN 111858576A CN 201910338181 A CN201910338181 A CN 201910338181A CN 111858576 A CN111858576 A CN 111858576A
- Authority
- CN
- China
- Prior art keywords
- identification data
- message
- content data
- data
- check value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 230000008859 change Effects 0.000 claims abstract description 66
- 238000012795 verification Methods 0.000 claims abstract description 58
- 238000012545 processing Methods 0.000 claims abstract description 40
- 238000000034 method Methods 0.000 claims abstract description 27
- 238000005538 encapsulation Methods 0.000 claims abstract description 18
- 238000004806 packaging method and process Methods 0.000 claims abstract description 16
- 238000004590 computer program Methods 0.000 claims description 9
- 238000004519 manufacturing process Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012856 packing Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the invention provides a real-time message processing method, a real-time message processing system, a real-time message processing device, a computer readable storage medium and electronic equipment, and relates to the technical field of computers. The method comprises the following steps: acquiring a database change message reflecting the change condition of the database and sending the database change message to an encapsulation thread; generating identification data and content data according to the database change message by using the packaging thread, and sending the identification data and the content data to corresponding threads of a message queue according to the identification data; performing time sequence verification on the content data according to the identification data; and generating an index according to the identification data and the content data which pass the time sequence verification. According to the technical scheme of the embodiment of the invention, the identification data and the content data reflecting the change condition of the database are sent by using multiple threads, and the time sequence of the content data is checked, so that the high throughput and low delay processing when the information is modified in real time for massive commodities are realized on the premise of ensuring the time sequence of the information.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, a system, an apparatus, a computer-readable storage medium, and an electronic device for processing a real-time message.
Background
In internet applications, search engines are the main means by which people quickly obtain information. In the e-commerce industry, there may be instances where goods are modified in real time. The real-time commodity modification comprises real-time modification of data such as commodity names, inventory, prices and comments of commodities.
Particularly, during the commodity sales promotion activity, the message volume is huge, if the throughput of the search engine is insufficient, the sales promotion price is not changed in time, and the commodities are still searched and displayed under the condition of no inventory. If the message time sequence is incorrect, the goods on-shelf and off-shelf states are incorrect, so that the user experience and the sales volume of the goods are influenced.
How to realize high throughput and low delay processing when a large amount of commodities modify messages in real time on the premise of ensuring message time sequence is a problem which needs to be solved at present.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present invention and therefore may include information that does not constitute prior art known to a person of ordinary skill in the art.
Disclosure of Invention
Embodiments of the present invention provide a method, a system, a device, a computer-readable storage medium, and an electronic device for processing a real-time message, so as to achieve high throughput and low delay processing when modifying a message in real time for a large amount of goods on the premise of ensuring a message timing sequence at least to a certain extent.
Additional features and advantages of the invention will be set forth in the detailed description which follows, or may be learned by practice of the invention.
According to a first aspect of the embodiments of the present invention, a method for processing a real-time message is provided, which includes: acquiring a database change message reflecting the change condition of the database and sending the database change message to an encapsulation thread; generating identification data and content data according to the database change message by using the packaging thread, and sending the identification data and the content data to corresponding threads of a message queue according to the identification data; performing time sequence verification on the content data according to the identification data; and generating an index according to the identification data and the content data which pass the time sequence verification.
In some embodiments, before performing the timing check on the content data according to the identification data, the method further comprises: obtaining the identification data and the content data from the message queue; determining a target verification thread according to the identification data; and sending the identification data and the content data to the target verification thread.
In some embodiments, after performing a timing check on the content data according to the identification data, the method further comprises: and sending the identification data and the content data which pass the time sequence verification to corresponding threads of the message queue according to the identification data.
In some embodiments, said sending said identification data and said content data to respective threads of a message queue in dependence on identification data comprises: carrying out hash processing on the identification data to obtain a hash value; taking the remainder of the hash value relative to the bus thread number of the message queue; and determining the thread of the message queue corresponding to the identification data according to the remainder.
In some embodiments, said performing a timing check on said content data according to said identification data comprises: generating an information check value according to the content data; comparing the information check value with a cache check value; and if the information check value is larger than the cache check value, determining that the check is passed and updating the cache check value into the information check value.
In some embodiments, the information check value includes a first portion and a second portion, and the cache check value includes a third portion and a fourth portion; the generating an information check value according to the content data includes: generating the first part according to the file name of the content data, and generating the second part according to the file offset of the content data; and when the first part is larger than or equal to the third part and the second part is larger than the fourth part, determining that the information check value is larger than the cache check value.
In some embodiments, said performing a timing check on said content data according to said identification data further comprises: and if the cache check value is a null value, determining that the check is passed and updating the cache check value into the information check value.
In some embodiments, the total number of packed threads is equal to the number of bus threads of the message queue.
According to a second aspect of the embodiments of the present invention, there is provided a real-time message processing system, comprising: the system comprises a change message sending module, a message queue and a message sending module, wherein the change message sending module is used for acquiring a database change message reflecting the change condition of a database and sending the database change message to a packaging thread, generating identification data and content data according to the database change message by using the packaging thread, and sending the identification data and the content data to corresponding threads of the message queue according to the identification data; a redis cache; and the data access module is used for carrying out time sequence verification on the content data according to the identification data in combination with the redis cache so as to produce an index according to the identification data and the content data which pass the time sequence verification.
According to a third aspect of the embodiments of the present invention, there is provided a real-time message processing apparatus, including: the obtaining unit is used for obtaining a database change message reflecting the change condition of the database and sending the database change message to the packaging thread; the generating unit is used for generating identification data and content data according to the database change message by using the packaging thread and sending the identification data and the content data to corresponding threads of a message queue according to the identification data; the verification unit is used for carrying out time sequence verification on the content data according to the identification data; and the production unit is used for producing the index according to the identification data and the content data which pass the time sequence verification.
According to a fourth aspect of embodiments of the present invention, there is provided a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, implements the real-time message processing method as described in the first aspect of the embodiments above.
According to a fifth aspect of embodiments of the present invention, there is provided an electronic apparatus, including: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the real-time message processing method as described in the first aspect of the embodiments above.
In the technical solutions provided in some embodiments of the present invention, after the identification data and the content data are generated according to the obtained database change message that reflects the database change condition, the identification data and the content data are transmitted in a message queue in a multi-thread manner, and the content data is subjected to timing verification, so that high throughput and low delay processing when a large amount of goods modify the message in real time are realized on the premise of ensuring the message timing.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:
FIG. 1 schematically illustrates a flow diagram of a method of real-time message processing according to one embodiment of the invention;
fig. 2 schematically shows a data transmission flow in step S102 and step S104;
fig. 3 schematically shows a data transmission flow in step S106;
FIG. 4 schematically illustrates a flow diagram of timing verification according to one embodiment of the invention;
FIG. 5 schematically illustrates a block diagram of a real-time message processing system according to one embodiment of the invention;
FIG. 6 schematically shows a block diagram of a real-time message processing apparatus according to an embodiment of the invention;
FIG. 7 illustrates a schematic structural diagram of a computer system suitable for use with the electronic device to implement an embodiment of the invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations or operations have not been shown or described in detail to avoid obscuring aspects of the invention.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
In the related art, the real-time data flow of e-commerce search includes a binlog (binary log) subscription module, a data preprocessing module, a kafka message queue module, an index production module, and a search engine module from a database. Specifically, the change operation of the commodity state is written into the database, a binlog message is generated, the data preprocessing module carries out data formatting work, the indexing module generates an index, and the search engine loads index data.
To ensure the timeliness of messages, current e-commerce searches are processed by a single threaded program with real-time indexing. The disadvantages of this approach are evident in low throughput and large message delays, especially during product promotion activities.
In order to solve the foregoing problems, an embodiment of the present invention provides a real-time message processing method, which generates identification data and content data according to an acquired database change message that reflects a database change condition, sends the identification data and the content data in a message queue in a multi-thread manner, and performs a timing check on the content data according to the identification data.
The technical solution of the present invention is described in detail below with reference to specific examples.
Fig. 1 schematically illustrates a real-time message processing method according to an exemplary embodiment of the present invention. Referring to fig. 1, the real-time message processing method may include the steps of:
and step S102, acquiring a database change message reflecting the database change condition and sending the database change message to the packaging thread.
And step S104, generating identification data and content data according to the database change message by using the packaging thread, and sending the identification data and the content data to corresponding threads of the message queue according to the identification data.
And step S106, performing time sequence verification on the content data according to the identification data.
And step S108, generating an index according to the identification data and the content data which pass the time sequence verification.
In the technical scheme of the embodiment of the invention, after the identification data and the content data are generated according to the acquired database change message reflecting the database change condition, the identification data and the content data are transmitted in the message queue in a multi-thread mode, and the content data are subjected to time sequence verification according to the identification data, so that the high-efficiency processing of the real-time message on the premise of ensuring the message time sequence is realized, and the user experience is optimized.
The modification of the commodity information is a source of the message in the embodiment of the present invention. After the commodity information is modified, the database modifies the related information stored in the corresponding database, and generates binlog information for recording modified contents. The database may be a mysql database, and the binlog message is a database change message. mysql is a relational database management system.
In step S102, the binlog message is sent to a plurality of packing threads for data packing. When sending the binlog message to a plurality of encapsulation threads, the binlog message is sent in a polling mode.
In the data transmission flow in step S102 and step S104 shown in fig. 2, the number of the encapsulation threads is 3, but the number of the encapsulation threads is not limited to this in practical applications.
As shown in fig. 2, the number of the message queue threads is 3, but the number of the message queue threads is not limited thereto in practical applications.
In the embodiment of the invention, the total number of the encapsulation threads is equal to the number of bus threads of the message queue. After the single-threaded binlog message is sent to multiple encapsulation threads, the binlog message in each encapsulation thread is individually encapsulated as an identification data and content data key-value pair. And then, sending the key value pair formed by each packaging thread to the corresponding message queue thread according to the identification data.
Specifically, in step S104, when the data encapsulation step is performed, the binlog message needs to be deserialized and encapsulated into a message id and a message key-value pair. Here, the message id is identification data, the message is content data, and the message id and the message respectively correspond to the key and the value in the key value pair. The message id may be a product code, and the message includes a file name logfile, a file offset logfile offset, and other service data such as a product name in the binlog message.
After the binlog message is encapsulated into the message id and the message key-value pair, the message id and the message key-value pair are asynchronously sent to a message queue. Here, the message queue may be a kafka message queue, but the present invention is not limited thereto, and may be any other type of message queue.
When the message id and the message key value pair are asynchronously sent to the kafka message queue, the thread of the target kafka message queue needs to be determined according to the commodity code.
Specifically, when the thread of the target kafka message queue is determined according to the commodity code, that is, the identification data, may be subjected to hash processing to obtain a hash value; then taking the remainder of the hash value relative to the bus thread number of the message queue; and finally, determining the thread of the message queue corresponding to the identification data according to the remainder.
For example, when the hash value of the identification data of the message is 232 and the total thread of the message queue is 10, the obtained remainder is 2, and at this time, the thread of the kafka message queue corresponding to the message is 2, that is, the identification data and the content data of the message need to be sent to the thread with the number of 2 of the kafka message queue.
In step S106, the identification data and the content data sent to the kafka message queue in step S104 are subjected to timing verification by the storm distributed streaming framework. And when data processing is carried out in storm, timing sequence check is carried out when the data passes through redis cache.
The throughput is improved by processing the data in a multithread mode, the redis cache is used as temporary storage for time sequence verification, and the high-speed read-write performance of the redis cache guarantees that the throughput of the system is still high even in the face of data flood peaks.
As shown in fig. 3, in the data transmission flow in step S106, the messages in the message queue are received into different access threads, i.e., access thread 1, access thread 2, and access thread 3. Before the content data is subjected to time sequence verification according to the identification data, verification threads to which the identification data and the content verification data are to enter need to be determined. The determination process comprises the following steps: and acquiring the identification data and the content data from the message queue, determining a target verification thread according to the identification data, and sending the identification data and the content data to the target verification thread.
Specifically, when the target verification thread is determined according to the identification data, hash processing may be performed on the commodity code, that is, the identification data, to obtain a hash value; then taking the remainder of the hash value relative to the bus thread number of the check thread; and finally, determining a target verification thread of the identification data according to the remainder.
In step S106, when performing timing verification on content data based on identification data, after acquiring the identification data and the content data in the kafka message queue, an information verification value needs to be generated based on the content data.
In the embodiment of the present invention, the information check value includes a first part and a second part, and the cache check value includes a third part and a fourth part. When an information check value is generated according to content data, a first part is generated according to the file name logfilename of the content data, and a second part is generated according to the file offset logfileoffset of the content data.
Specifically, the file name in the content data is in the format mysql-bin.123, and the file offset is in the format 456. When the information check value is generated according to the file name and the file offset of the content data, the tail number in the file name is extracted, that is, the file code is taken, the file code and the file offset are connected, and the format of the obtained information check value is file code _ file offset, such as 123_ 456. The format of the cache check value is the same as that of the information check value.
After the cache check value corresponding to the identification data of the message is obtained from the redis cache, if the cache check value is null, the message corresponding to the identification data arrives for the first time, the check is determined to be passed, and the cache check value is updated to the information check value.
If the cache check value is not a null value, comparing the information check value with the cache check value; and if the information check value is larger than the cache check value, determining that the check is passed and updating the cache check value into the information check value. In the embodiment of the present invention, when the first portion is greater than or equal to the third portion and the second portion is greater than the fourth portion, it is determined that the information check value is greater than the cache check value.
Specifically, as shown in fig. 4, the step of performing timing verification on the content data includes:
step S401, a cache check value is obtained. Here, the cache check value is a cache check value corresponding to the identification data of the message in redis.
Step S402, judging whether the buffer check value is empty. If yes, step S403 is executed to construct an information check value, that is, to generate an information check value according to the content data.
After step S403, step S404 is executed to write the information check value into redis, i.e. to update the cache check value to the information check value.
If the determination result in step S402 is no, step S405 is executed to construct an information check value. Here, the content of step S405 is the same as that of step S403.
After step S405 is executed, step S406 is executed to determine whether the information check value is greater than the cache check value.
When the information check value is greater than the cache check value, step S407 is performed to confirm that the check is passed, and write the information check value into redis.
When the information check value is not greater than the buffer check value, step S408 is performed to confirm that the check is not passed, and the information is discarded.
After the content data is subjected to time sequence verification according to the identification data, the identification data and the content data which pass the time sequence verification are sent to corresponding threads of the message queue according to the identification data.
Specifically, when the thread of the target kafka message queue is determined according to the commodity code, that is, the identification data, may be subjected to hash processing to obtain a hash value; then taking the remainder of the hash value relative to the bus thread number of the message queue; and finally, determining the thread of the message queue corresponding to the identification data according to the remainder.
After the verified message is sent through the kafka message queue, step S108 is performed to acquire the message in the kafka message queue, and after an index is generated from the message, the index is sent to the search engine through the kafka message queue.
In the real-time message processing method provided in some embodiments of the present invention, after the identification data and the content data are generated according to the obtained database change message that reflects the database change condition, the identification data and the content data are transmitted in a message queue in a multi-thread manner, and the content data are subjected to timing verification, so that high throughput and low delay processing when a large amount of commodities modify the message in real time are realized on the premise of ensuring the message timing.
As shown in fig. 5, an embodiment of the present invention provides a real-time message processing system, including: a change messaging module 502, a data access module 503, and a redis cache 506.
The change message sending module 502 is configured to obtain a database change message reflecting a database change condition, send the database change message to the encapsulation thread, generate identification data and content data according to the database change message by using the encapsulation thread, and send the identification data and the content data to corresponding threads of the message queue according to the identification data. And the data access module 503 is configured to perform timing verification on the content data according to the identification data in combination with the redis cache, so as to generate an index according to the identification data and the content data that pass the timing verification.
In an embodiment of the present invention, as shown in fig. 5, the system further includes a database 501, an index production module 504, and a search engine 505.
When the commodity information is modified, the data in the database 501 is changed, and a database change message, i.e., binlog message, is generated. After the binlog message is single threaded to change messaging module 502, the binlog message is multi-threaded and after being encapsulated into identification data and content data, is multi-threaded to the kafka message queue. The data access module 503 acquires the messages in the kafka message queue, performs timing check on the content data in the messages in combination with the redis cache 506, and then sends the messages passing the timing check to the kafka message queue. The index generation module 504 generates an index from the message retrieved from the kafka message queue and sends the index to the search engine 505 via the kafka message queue, completing the message processing process.
In the real-time message processing system provided in some embodiments of the present invention, by sending identification data and content data reflecting changes in the database using multiple threads and performing timing verification on the content data, high throughput and low latency processing are achieved when modifying messages in real time for a large number of commodities on the premise of ensuring message timing.
The following describes embodiments of the apparatus of the present invention, which can be used to perform the above-mentioned real-time message processing method of the present invention. Referring to fig. 6, an embodiment of the present invention provides a real-time message processing apparatus 600, which may include: an acquisition unit 602, a generation unit 604, a verification unit 606, and a production unit 608.
The obtaining unit 602 may be configured to obtain a database change message reflecting a database change and send the database change message to the encapsulation thread.
The generating unit 604 may be configured to generate the identification data and the content data according to the database change message using the encapsulation thread, and send the identification data and the content data to corresponding threads of the message queue according to the identification data.
The verification unit 606 may be configured to perform timing verification on the content data according to the identification data.
The production unit 608 may be configured to produce an index based on the identification data and the content data that pass the timing verification.
For example, when the commodity information is modified, data in the database is changed, and a database change message, i.e., binlog message, is generated. Fetch unit 602 single threads fetch binlog messages and multi-threads pass into generation unit 604. After the binlog message is encapsulated into the identification data and the content data, the generating unit 604 multithreadally sends the identification data and the content data to the kafka message queue. The checking unit 606 acquires the message in the kafka message queue, performs timing check on content data in the message in combination with the redis cache, and then sends the message passing the timing check to the kafka message queue. The production unit 608 produces an index from the message acquired from the kafka message queue and transmits the index to the search engine through the kafka message queue, completing the message processing process.
In the embodiment of the invention, the total number of the encapsulation threads is equal to the number of bus threads of the message queue. After the single-threaded binlog message is sent to multiple encapsulation threads, generation unit 604 individually encapsulates the binlog message in each encapsulation thread as an identification data and content data key-value pair. And then, sending the key value pair formed by each packaging thread to the corresponding message queue thread according to the identification data.
When the key value pair formed by each encapsulation thread is sent to the corresponding message queue thread according to the identification data, the generating unit 604 performs hash processing on the identification data to obtain a hash value; taking the remainder of the hash value relative to the bus thread number of the message queue; and determining the thread of the message queue corresponding to the identification data according to the remainder.
The real-time message processing device provided by the embodiment of the invention further comprises a first sending unit and a second sending unit, wherein the first sending unit can be used for acquiring the identification data and the content data from the message queue; determining a target verification thread according to the identification data; and sending the identification data and the content data to a target verification thread.
The throughput is improved by processing the data in a multithread mode, the redis cache is used as temporary storage for time sequence verification, and the high-speed read-write performance of the redis cache guarantees that the throughput of the system is still high even in the face of data flood peaks.
The information check value includes a first portion and a second portion, and the cache check value includes a third portion and a fourth portion.
The checking unit 606 may be further configured to generate an information checking value according to the content data; the information check value is compared to the cache check value. And if the information check value is larger than the cache check value, determining that the check is passed and updating the cache check value into the information check value. And if the cache check value is a null value, determining that the check is passed and updating the cache check value into an information check value.
When the verification unit 606 generates an information verification value according to the content data, a first part is generated according to the file name of the content data, and a second part is generated according to the file offset of the content data; and when the first part is larger than or equal to the third part and the second part is larger than the fourth part, determining that the information check value is larger than the cache check value.
And the second sending unit can be used for sending the identification data and the content data which pass the time sequence verification to corresponding threads of the message queue according to the identification data.
For details that are not disclosed in the embodiments of the apparatus of the present invention, reference is made to the above-described embodiments of the real-time message processing method of the present invention for the details that are not disclosed in the embodiments of the apparatus of the present invention.
In the real-time message processing apparatus provided in the embodiment of the present invention, after the identification data and the content data are generated according to the acquired database change message that reflects the database change condition, the identification data and the content data are transmitted in the message queue in a multi-thread manner, and the content data is subjected to timing verification, so that high throughput and low delay processing when a large amount of commodities modify the message in real time are realized on the premise of ensuring the message timing.
Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use with the electronic device implementing an embodiment of the present invention. The computer system 700 of the electronic device shown in fig. 7 is only an example, and should not bring any limitation to the function and the scope of use of the embodiments of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for system operation are also stored. The CPU701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.
In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the invention include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program executes the above-described functions defined in the system of the present application when executed by the Central Processing Unit (CPU) 701.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present invention may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.
As another aspect, the present application also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to implement the real-time message processing method as described in the above embodiments.
For example, the electronic device may implement the following as shown in fig. 1: step S102, obtaining a database change message reflecting the change condition of the database and sending the database change message to a plurality of packaging threads; step S104, generating identification data and content data according to the database change message, and sending the identification data and the content data to corresponding threads of a message queue according to the identification data; step S106, carrying out time sequence verification on the content data according to the identification data; and S108, generating an index according to the identification data and the content data which pass the time sequence verification.
As another example, the electronic device may implement the steps shown in fig. 4.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the invention. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (12)
1. A method for real-time message processing, comprising:
acquiring a database change message reflecting the change condition of the database and sending the database change message to an encapsulation thread;
generating identification data and content data according to the database change message by using the packaging thread, and sending the identification data and the content data to corresponding threads of a message queue according to the identification data;
Performing time sequence verification on the content data according to the identification data;
and generating an index according to the identification data and the content data which pass the time sequence verification.
2. The method of claim 1, wherein before performing the timing check on the content data according to the identification data, the method further comprises:
obtaining the identification data and the content data from the message queue;
determining a target verification thread according to the identification data;
and sending the identification data and the content data to the target verification thread.
3. The method of claim 1, wherein after performing the timing check on the content data according to the identification data, the method further comprises:
and sending the identification data and the content data which pass the time sequence verification to corresponding threads of the message queue according to the identification data.
4. The method of claim 1, wherein sending the identification data and the content data to respective threads of a message queue based on identification data comprises:
carrying out hash processing on the identification data to obtain a hash value;
taking the remainder of the hash value relative to the bus thread number of the message queue;
And determining the thread of the message queue corresponding to the identification data according to the remainder.
5. The method of claim 1, wherein the performing the timing check on the content data according to the identification data comprises:
generating an information check value according to the content data;
comparing the information check value with a cache check value;
and if the information check value is larger than the cache check value, determining that the check is passed and updating the cache check value into the information check value.
6. The method of claim 5, wherein the information check value comprises a first portion and a second portion, and wherein the buffer check value comprises a third portion and a fourth portion;
the generating an information check value according to the content data includes:
generating the first part according to the file name of the content data, and generating the second part according to the file offset of the content data;
and when the first part is larger than or equal to the third part and the second part is larger than the fourth part, determining that the information check value is larger than the cache check value.
7. The method of claim 6, wherein the time-sequence checking the content data according to the identification data further comprises:
And if the cache check value is a null value, determining that the check is passed and updating the cache check value into the information check value.
8. The method of claim 3, wherein the total number of packed threads is equal to the number of bus threads of the message queue.
9. A real-time message processing system, the system comprising:
the system comprises a change message sending module, a message queue and a message sending module, wherein the change message sending module is used for acquiring a database change message reflecting the change condition of a database and sending the database change message to a packaging thread, generating identification data and content data according to the database change message by using the packaging thread, and sending the identification data and the content data to corresponding threads of the message queue according to the identification data;
a redis cache;
and the data access module is used for carrying out time sequence verification on the content data according to the identification data in combination with the redis cache so as to produce an index according to the identification data and the content data which pass the time sequence verification.
10. A real-time message processing apparatus, comprising:
the obtaining unit is used for obtaining a database change message reflecting the change condition of the database and sending the database change message to the packaging thread;
The generating unit is used for generating identification data and content data according to the database change message by using the packaging thread and sending the identification data and the content data to corresponding threads of a message queue according to the identification data;
the verification unit is used for carrying out time sequence verification on the content data according to the identification data;
and the production unit is used for producing the index according to the identification data and the content data which pass the time sequence verification.
11. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the real-time message processing method according to any one of claims 1 to 8.
12. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the real-time message processing method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910338181.1A CN111858576A (en) | 2019-04-25 | 2019-04-25 | Real-time message processing method, system, device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910338181.1A CN111858576A (en) | 2019-04-25 | 2019-04-25 | Real-time message processing method, system, device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111858576A true CN111858576A (en) | 2020-10-30 |
Family
ID=72951243
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910338181.1A Pending CN111858576A (en) | 2019-04-25 | 2019-04-25 | Real-time message processing method, system, device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858576A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113139081A (en) * | 2021-04-27 | 2021-07-20 | 中山亿联智能科技有限公司 | Method for reporting and reading user online playing information with high efficiency and low delay |
| CN113379431A (en) * | 2021-06-29 | 2021-09-10 | 上海中商网络股份有限公司 | Method and system for real-time verification of collected data |
| CN113779311A (en) * | 2020-11-04 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Data processing method, device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150310044A1 (en) * | 2014-02-03 | 2015-10-29 | Codefutures Corporation | Database device and processing of data in a database |
| CN108121730A (en) * | 2016-11-30 | 2018-06-05 | 北京京东尚科信息技术有限公司 | A kind of device and method by data update Fast synchronization to operation system |
| CN108334641A (en) * | 2018-03-21 | 2018-07-27 | 上海精数信息科技有限公司 | The method of acquisition user behavior data, system, electronic equipment, storage medium |
| CN109145060A (en) * | 2018-07-20 | 2019-01-04 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| CN109492012A (en) * | 2018-10-31 | 2019-03-19 | 厦门安胜网络科技有限公司 | A kind of method, apparatus and storage medium of data real-time statistics and retrieval |
-
2019
- 2019-04-25 CN CN201910338181.1A patent/CN111858576A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150310044A1 (en) * | 2014-02-03 | 2015-10-29 | Codefutures Corporation | Database device and processing of data in a database |
| CN108121730A (en) * | 2016-11-30 | 2018-06-05 | 北京京东尚科信息技术有限公司 | A kind of device and method by data update Fast synchronization to operation system |
| CN108334641A (en) * | 2018-03-21 | 2018-07-27 | 上海精数信息科技有限公司 | The method of acquisition user behavior data, system, electronic equipment, storage medium |
| CN109145060A (en) * | 2018-07-20 | 2019-01-04 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| CN109492012A (en) * | 2018-10-31 | 2019-03-19 | 厦门安胜网络科技有限公司 | A kind of method, apparatus and storage medium of data real-time statistics and retrieval |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113779311A (en) * | 2020-11-04 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Data processing method, device and storage medium |
| CN113139081A (en) * | 2021-04-27 | 2021-07-20 | 中山亿联智能科技有限公司 | Method for reporting and reading user online playing information with high efficiency and low delay |
| CN113139081B (en) * | 2021-04-27 | 2023-10-27 | 中山亿联智能科技有限公司 | Method for reporting online playing information of reading user with high efficiency and low delay |
| CN113379431A (en) * | 2021-06-29 | 2021-09-10 | 上海中商网络股份有限公司 | Method and system for real-time verification of collected data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9619491B2 (en) | Streamlined system to restore an analytic model state for training and scoring | |
| CN108959279B (en) | Data processing method, data processing device, readable medium and electronic equipment | |
| CN112800095B (en) | Data processing method, device, equipment and storage medium | |
| US20140115610A1 (en) | System and method for batch evaluation programs | |
| CN111858576A (en) | Real-time message processing method, system, device, storage medium and electronic equipment | |
| CN111125107A (en) | Data processing method, device, electronic equipment and medium | |
| CN111209310A (en) | Service data processing method and device based on stream computing and computer equipment | |
| CN110895534A (en) | Data splicing method, device, medium and electronic equipment | |
| CN109597810B (en) | Task segmentation method, device, medium and electronic equipment | |
| CN110866040A (en) | User portrait generation method, device and system | |
| CN109960212B (en) | Task sending method and device | |
| CN114780564A (en) | Data processing method, data processing apparatus, electronic device, and storage medium | |
| CN113836235B (en) | Data processing method based on data center and related equipment thereof | |
| CN113111078B (en) | Resource data processing method and device, computer equipment and storage medium | |
| CN113723893A (en) | Method and device for processing orders | |
| CN110688355A (en) | Method and device for changing container state | |
| CN116703629A (en) | Service processing method, device, electronic equipment and computer readable medium | |
| CN113296912B (en) | Task processing method, device, system, storage medium and electronic equipment | |
| CN115454971A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN115168358A (en) | Database access method and device, electronic equipment and storage medium | |
| CN115269431A (en) | Interface testing method and device, electronic equipment and storage medium | |
| CN114218313A (en) | Data management method, device, electronic equipment, storage medium and product | |
| CN108805593A (en) | A kind of method of data processing, system, electronic equipment and readable storage medium storing program for executing | |
| CN110309121B (en) | Log processing method and device, computer readable medium and electronic equipment | |
| CN113760240A (en) | Method and device for generating data model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |