CN113377795A - Message processing method and device - Google Patents

Message processing method and device Download PDF

Info

Publication number
CN113377795A
CN113377795A CN202110700492.5A CN202110700492A CN113377795A CN 113377795 A CN113377795 A CN 113377795A CN 202110700492 A CN202110700492 A CN 202110700492A CN 113377795 A CN113377795 A CN 113377795A
Authority
CN
China
Prior art keywords
instance
data
processing
distributed
database
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110700492.5A
Other languages
Chinese (zh)
Inventor
余欢
凌磊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Wodong Tianjun Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Wodong Tianjun Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Wodong Tianjun Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN202110700492.5A priority Critical patent/CN113377795A/en
Publication of CN113377795A publication Critical patent/CN113377795A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2308—Concurrency control
    • G06F16/2336—Pessimistic concurrency control approaches, e.g. locking or multiple versions without time stamps
    • G06F16/2343—Locking methods, e.g. distributed locking or locking implementation details
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/548—Queue

Abstract

The invention discloses a method and a device for processing messages, and relates to the technical field of computers. One embodiment of the method comprises: respectively receiving a plurality of messages to be processed triggered by a database by utilizing one or more instances in a distributed cluster, wherein the messages to be processed correspond to the same data identifier; performing distributed locking processing on the one or more instances according to the data identification to obtain an attribute value of a distributed lock; processing the received messages to be processed by utilizing the locked first instance so as to process the data stored in the database according to the data in the database; updating the attribute values of the distributed lock with the second instance that is not locked; synchronizing data in the first instance and the second instance when the attribute values of the distributed locks of the first instance and the second instance are different. The implementation mode reduces the retry times, avoids the distributed lock conflict and improves the message processing efficiency.

Description

Message processing method and device
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for processing a message.
Background
In a distributed cluster, for a change of service data of a certain data table in a database, each instance in the distributed cluster may receive a change message triggered by the database and related to the data table, and when each instance processes the change messages, a general manner is as follows: for multiple messages with the same service ID, each instance retries the multiple messages one by one according to the service ID, that is, each instance repeatedly processes multiple change messages corresponding to the same service ID, which results in low message processing efficiency.
Further, when the instances process the change messages, if the data is changed again, each instance will receive the change messages again, and after the messages are superimposed, more invalid retry times are added, thereby further reducing the message processing efficiency.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for message processing, which can lock an instance in a distributed cluster according to a data identifier (service ID), and then process multiple received messages to be processed by using the locked instance, so as to effectively reduce retry times, avoid distributed lock conflicts, improve message processing efficiency, and improve message processing accuracy by setting an attribute value of a distributed lock.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a message processing method, including:
respectively receiving a plurality of messages to be processed triggered by a database by utilizing one or more instances in a distributed cluster, wherein the messages to be processed correspond to the same data identifier;
performing distributed locking processing on the one or more instances according to the data identification to obtain an attribute value of a distributed lock;
processing the received messages to be processed by utilizing the locked first instance so as to process the data stored in the database according to the data in the database;
updating the attribute values of the distributed lock with the second instance that is not locked;
synchronizing data in the first instance and the second instance when the attribute values of the distributed locks of the first instance and the second instance are different.
Optionally, the processing the received plurality of messages to be processed by using the locked first instance includes:
and carrying out duplicate removal processing on the received messages to be processed by utilizing the first instance according to the data identification, and processing the data stored in the first instance according to the duplicate removal processing result and the data in the database.
Optionally, the processing the data stored in the database according to the deduplication processing result and the data in the database includes:
acquiring a data table corresponding to the data identifier from the database according to the data identifier indicated by the duplicate removal processing result;
and updating the data stored by the data table according to the acquired data table.
Optionally, performing distributed locking processing on the one or more instances according to the data identifier to obtain an attribute value of a distributed lock, including:
and generating a random number aiming at the data identification, performing distributed locking processing on the one or more instances according to the random number, and taking the random number as the attribute value.
Optionally, when the second instance receives the database-triggered data update message again, the attribute value of the distributed lock is updated again by using the second instance.
Optionally, updating the attribute value of the distributed lock with the second instance includes:
incrementing the distributed attribute value using the second instance.
Optionally, the synchronizing data in the first instance and the second instance includes:
and determining a data synchronization source from the first instance and the second instance according to the sizes of the attribute values of the distributed locks respectively corresponding to the first instance and the second instance, and synchronizing data according to the data in the data synchronization source.
According to a second aspect of the embodiments of the present invention, there is provided an apparatus for processing a message, including:
the device comprises a receiving module, a first processing module, a determining module, a second processing module, a third processing module and a fourth processing module; wherein the content of the first and second substances,
the receiving module is used for receiving a plurality of messages to be processed triggered by the database respectively received by one or more instances in the distributed cluster;
the first processing module performs distributed locking processing on the one or more instances according to the data identifier to obtain an attribute value of a distributed lock;
the second processing module is used for processing the received messages to be processed by utilizing the locked first instance so as to process the data stored in the second processing module according to the data in the database;
the third processing module updates the attribute value of the distributed lock by using the second instance which is not locked;
the determining module is configured to determine whether attribute values of distributed locks of the first instance and the second instance are different. If yes, triggering the fourth processing module; if not, ending the process;
and the fourth processing module is used for synchronizing the data in the first instance and the second instance.
According to a third aspect of the embodiments of the present invention, there is provided an electronic device for message processing, including:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a method as in any one of the methods of message processing as provided in the first aspect above.
According to a fourth aspect of embodiments of the present invention, there is provided a computer readable medium, on which a computer program is stored, which when executed by a processor implements the method of any one of the methods of message processing as provided in the first aspect above.
One embodiment of the above invention has the following advantages or benefits: the method has the advantages that locking processing is carried out on the examples in the distributed cluster according to the data identification (service ID), and then the received messages to be processed are processed by utilizing the locked examples, so that retry times are effectively reduced, distributed lock conflict is avoided, message processing efficiency is improved, and meanwhile, the accuracy of message processing is improved through setting of the attribute value of the distributed lock.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a main schematic flow chart of a message processing method according to a first embodiment of the present invention;
fig. 2 is a main schematic flow chart of a message processing method according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of the main modules of an apparatus for message processing according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a main schematic flow chart of a message processing method corresponding to a first embodiment of the present invention, and as shown in fig. 1, the method includes the following steps S101 to S104:
step S101: and respectively receiving a plurality of messages to be processed triggered by the database by utilizing one or more instances in the distributed cluster, wherein the plurality of messages to be processed correspond to the same data identifier.
For a situation where multiple data sources exist, complete service data may be stored in multiple data tables, multiple messages to be processed may be different types of messages, but subsequent processing procedures are the same, and therefore, in an embodiment of the present invention, a message processing method provided in an embodiment of the present invention is described in detail by taking an example where database change messages sent by multiple data tables are received.
In addition, pending messages having the same data identification are received by one or more instances in the distributed cluster, respectively. In one embodiment of the invention, the data identification of the message to be processed may be a service ID.
In addition, for multiple messages received by the same instance, duplicate removal is performed according to the data identifier of the message to be processed, that is, multiple messages are deduplicated according to the service ID.
For example, when the message processing method provided by the embodiment of the present invention is applied to an e-commerce scenario, in a certain message processing process, one service ID is usually associated with many commodities, the commodities have the same service ID, and when many commodities are added at one time, messages inserted by many commodities are received at the same time, that is, messages for the same service ID are changed. At this point, multiple messages may be deduplicated according to the service ID. Through the duplication and processing removal, the number of messages to be processed is effectively reduced, and the message processing efficiency is improved.
The instance may be a server in a distributed cluster, or may be a program running on the server, and configured to receive the message to be processed, acquire the distributed lock, and process the message to be processed.
Step S102: and carrying out distributed locking processing on one or more instances according to the data identification to obtain the attribute value of the distributed lock.
In one embodiment of the invention, performing distributed locking processing on one or more instances according to data identification comprises: and aiming at the data identification, generating any random number (the random number can be set according to actual requirements), performing distributed locking processing on one or more instances according to the random number, and taking the random number as an attribute value.
For example, a distributed lock for a service ID has been acquired during a certain message processing. At this time, a random number within 1 ten thousand (for example only) is acquired, the generated random number value is used as an attribute value of the distributed lock, and the instance of the lock acquiring the service ID is subjected to locking processing. The attribute value of the distributed lock can be used for representing the update state information of the data corresponding to the current service ID, so that the subsequent comparison of the attribute values obtained by different instances aiming at the same service ID is facilitated, and the data in the first instance and the second instance are synchronized according to the comparison result.
Step S103: and processing the received messages to be processed by utilizing the locked first instance so as to process the data stored in the first instance according to the data in the database.
In one embodiment of the present invention, processing a plurality of received messages to be processed includes:
and acquiring a data table corresponding to the data identifier from the database according to the data identifier indicated by the duplicate removal processing result by using the duplicate removal processing result of the first example and the data in the database, and updating the data stored in the database according to the acquired data table.
For example, in a certain message processing process, a certain instance has removed the duplicate of the message to be processed according to the service ID, acquires complete service information from the database according to the service ID corresponding to the duplicate removal result, and updates the data stored in itself according to the acquired complete service information. In the steps, the database is queried according to the service ID to acquire complete service information, which is beneficial to acquiring all changed data and improves the accuracy of message processing.
Step S104: updating the attribute values of the distributed lock with the second instance that is not locked;
in one embodiment of the invention, when the second instance receives the database-triggered data update message again, the attribute values of the distributed lock are updated again with the second instance.
Specifically, when the second instance receives the data update message of the database again, the attribute value of the distributed lock is incremented.
For example, in a certain message processing process, after a first instance performs distributed lock locking processing, 1 adding processing is performed on a current cache value of an attribute value of a distributed lock corresponding to a second instance that is not locked (it should be noted that the above 1 adding processing is only an example, and is intended to distinguish the attribute values of the distributed locks corresponding to the second instance and the first instance so as to facilitate subsequent synchronous updating of local data corresponding to the second instance with different attribute values). In the process that the first instance processes the data stored in itself, if the second instance receives the update message corresponding to the service ID again, the attribute value of the distributed lock of the second instance is incremented again (similarly, the incrementing again is also exemplified, and the purpose of the incrementing again is to generate a new change message again in the process that the first instance processes the data stored in itself in order to represent the data corresponding to the service ID). Through the setting, the data synchronization source and the second instance which needs to be synchronized are determined by comparing the attribute values of the distributed locks respectively corresponding to the first instance and the second instance, data synchronization is performed on the second instance, and meanwhile, whether the service ID generates a new change message (namely a new adding to-be-processed message) or not is judged in the process that the service ID updates corresponding data in the first instance and the second instance synchronizes the updated data of the first instance.
Step S105: data in the first instance and the second instance are synchronized when the attribute values of the distributed locks of the first instance and the second instance are different.
Through the arrangement, only the locked first instance is needed to process the message to be processed, the local message is updated, and for the second instances (the quantity of the understandable second instances can be multiple) with the service IDs unlocked, the data sources needing to be synchronized can be determined by comparing the attribute values of the distributed instances, so that data synchronization is realized.
In one embodiment of the invention, synchronizing data in a first instance and a second instance comprises:
because the attribute value of the distributed lock corresponding to the second instance is subjected to 1 adding processing (or incremental processing) at least once, before the data in the second instance is synchronized, the attribute values of the distributed locks corresponding to the first instance and the second instance are different, at this time, a data synchronization source is determined from the first instance and the second instance according to the attribute values of the distributed locks corresponding to the first instance and the second instance respectively, and the data is synchronized according to the data in the data synchronization source.
Specifically, the first instance is used as a synchronous data source, the data of the first instance is synchronized to the second data source, and the distributed lock attribute value of the first instance is also synchronized to the second instance.
Further, in a certain message processing process, after normal service logic is processed, the attribute value cache value of the current distributed lock is obtained again, and if the difference value between the attribute value of the distributed lock and the attribute value of the distributed lock in the previous locking process is greater than 1 (the attribute value of the distributed lock corresponding to the second instance which is not locked is subjected to at least two times of 1 adding processes), it indicates that new data enters during the process of changing the message, and the current message is retried.
Fig. 2 is a main schematic flow chart of a message processing method corresponding to a second embodiment of the present invention, and as shown in fig. 2, the method includes the following steps S201 to S206:
step S201: and respectively receiving a plurality of messages to be processed triggered by the database by utilizing one or more instances in the distributed cluster, wherein the plurality of messages to be processed correspond to the same data identifier.
Specifically, according to the embodiment of the present invention, the complete data of one service ID is stored in a plurality of data tables. For example, in a shopping mall, one service ID is usually associated with a plurality of commodities, and a data table corresponding to complete data of the service ID is divided into two types, i.e., a type a data table and a type B data table, wherein the type B data table is a plurality of data tables, such as detail data including unit price detail tables, sales detail tables, and the like; the class a data table is a summary table (a one-to-many relationship exists between a and B, and both the class a data table and the class B data table correspond to the same service ID) and is used for storing summarized data of total sales amount, profits and the like of commodities. Therefore, when one message is added (for example, one new is added to the sales specification), a plurality of change messages (i.e., the above-mentioned pending messages) are generated. At this time, the generated plurality of change information is sent to one or more instances in the distributed cluster by the database, so that the one or more instances in the distributed cluster synchronize the local data according to the change information.
Step S202: and generating a random number aiming at the data identification, performing distributed locking processing on one or more instances according to the random number, and storing the random number as an attribute value.
For example, during a message process, an instance has acquired a distributed lock for a service ID. At this time, a random number within 1 ten thousand is obtained (the random number can be adjusted correspondingly according to actual requirements), locking processing is performed on the instance of the lock of the obtained service ID according to the random number, and the generated random number value is used as an attribute value of the distributed lock.
Step S203: and carrying out deduplication processing on the received multiple messages to be processed according to the data identification by using the first example.
Specifically, the first instance is used for carrying out duplicate removal processing on a plurality of data to be processed according to the data identification, the first instance only needs to process the messages after the duplicate removal, and through the duplicate removal processing, the number of the messages to be processed is effectively reduced, and the message processing efficiency is improved.
Step S204: acquiring a data table corresponding to the data identifier from a database according to the data identifier indicated by the duplicate removal processing result; and updating the data stored by the self according to the acquired data table.
In the steps, the database is queried according to the service ID to acquire complete service information, which is beneficial to acquiring all changed data and improves the accuracy of message processing.
Step S205: the attribute value of the distributed lock is incremented using the second instance.
For example, in a certain message processing process, after the first instance performs distributed lock locking processing, 1 adding processing is performed on a current cache value of the attribute value of the distributed lock corresponding to the second instance that is not locked (the above-mentioned 1 adding processing is only an example, and is intended to distinguish the attribute values of the distributed locks corresponding to the second instance and the first instance so as to facilitate subsequent synchronous update on local data corresponding to the second instance with different attribute values). In the process that the first instance processes the data stored in the first instance, if the second instance receives the update message corresponding to the service ID again, the attribute value of the distributed lock of the second instance is incremented again (similarly, the incrementing again is also exemplified, and the purpose of the incrementing again is to generate a new change message again in the process that the first instance processes the data stored in the first instance in order to represent the data corresponding to the service ID).
Step S206: and determining a data synchronization source from the first instance and the second instance according to the sizes of the attribute values of the distributed locks respectively corresponding to the first instance and the second instance, and synchronizing the data according to the data in the data synchronization source.
For example, in a certain message processing process, since the attribute value of the distributed lock corresponding to the second instance is subjected to at least one processing of adding 1 (or incremental processing), before the data in the second instance is synchronized, the attribute values of the distributed locks corresponding to the first instance and the second instance are different, the first instance is used as a synchronized data source, the data of the first instance is synchronized to the second data source, the attribute value of the distributed lock of the first instance is also synchronized to the second instance, and if the difference value between the attribute value of the distributed lock and the attribute value of the distributed lock in the previous locking is greater than 1 (the attribute value of the distributed lock corresponding to the second instance which is not locked is subjected to at least two processing of adding 1), it is indicated that new data enters during the processing of the change message, and the current message is subjected to retry processing.
Fig. 3 is a schematic diagram of main modules of a message processing apparatus according to an embodiment of the present invention.
As shown in fig. 3, an embodiment of the present invention provides a data processing apparatus, including: a receiving module 301, a determining module 302, a first processing module 303, a second processing module 304, a third processing module 305 and a fourth processing module 306; wherein the content of the first and second substances,
a receiving module 301, configured to receive a message to be processed;
the first processing module 303 is configured to perform distributed locking processing on one or more instances according to the data identifier to obtain an attribute value of a distributed lock;
a second processing module 304, configured to process the received multiple messages to be processed by using the locked first instance, so as to process data stored in the second processing module according to data in the database;
a third processing module 305 for updating the attribute values of the distributed lock with the second instance that is not locked;
a determining module 302, configured to determine whether the attribute values of the distributed locks of the first instance and the second instance are different. If yes, triggering a fourth processing module 306;
a fourth processing module 306, configured to synchronize data in the first instance and the second instance.
In an embodiment of the present invention, the first processing module 303 is configured to perform deduplication processing on the received multiple messages to be processed according to the data identifier by using the first instance, and process data stored in the first processing module according to a deduplication processing result and data in the database.
In an embodiment of the present invention, the second processing module 304 is configured to obtain, according to the data identifier indicated by the deduplication processing result, a data table corresponding to the data identifier from a database;
and updating the data stored by the self according to the acquired data table.
In an embodiment of the present invention, the triple processing module 305 is configured to generate a random number for the data identifier, perform distributed locking processing on one or more instances according to the random number, and use the random number as an attribute value.
In an embodiment of the present invention, the third processing module 305 is configured to update the attribute value of the distributed lock again with the second instance when the second instance receives the database-triggered data update message again.
In one embodiment of the invention, a tri-processing module 305 is used to increment the distributed attribute values using the second instance.
In an embodiment of the present invention, the determining module 302 is configured to determine a data synchronization source from the first instance and the second instance according to sizes of attribute values of the distributed lock corresponding to the first instance and the second instance, respectively, and synchronize data according to the data in the data synchronization source.
In an embodiment of the present invention, the fourth processing module 306 is configured to determine a data synchronization source from the first instance and the second instance according to the size of the attribute value of the distributed lock corresponding to the first instance and the second instance, respectively, and synchronize the data according to the data in the data synchronization source.
According to the data storage device provided by the embodiment of the invention, when a plurality of messages with the same service ID are processed, the service ID is used for locking, the plurality of messages are subjected to de-duplication according to the service ID, and the plurality of received messages to be processed are processed only for the locked example, so that when one application example receives the messages with the same service ID, the processing times are reduced, the distributed lock conflict processing is optimized, the invalid retry is reduced, and the message processing efficiency is improved. In addition, the accuracy of message processing is improved through comparison of the distributed lock attribute values.
An embodiment of the present invention further provides a server, including: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the message processing method provided by any one of the embodiments.
Embodiments of the present invention further provide a computer-readable medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the message processing method provided in any of the above embodiments.
Fig. 4 shows an exemplary system architecture 400 to which the message processing method or the message processing apparatus of the embodiments of the present invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have various communication client applications installed thereon, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, and the like.
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 401, 402, 403. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.
It should be noted that the message processing method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the message processing device is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a receiving module, a determining module, a first storage module, a second storage module, a third processing module, and a fourth processing module. The names of these modules do not in some cases constitute a limitation on the module itself, and for example, a receiving module may also be described as a "module that receives data to be stored".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: respectively receiving a plurality of messages to be processed triggered by a database by utilizing one or more instances in a distributed cluster, wherein the plurality of messages to be processed correspond to the same data identifier; performing distributed locking processing on one or more instances according to the data identification to obtain an attribute value of a distributed lock; processing the received messages to be processed by utilizing the locked first instance so as to process the data stored in the database according to the data in the database; updating the attribute values of the distributed lock with the second instance that is not locked; data in the first instance and the second instance are synchronized when the attribute values of the distributed locks of the first instance and the second instance are different.
According to the technical scheme of the embodiment of the invention, one or more instances in the distributed cluster are adopted to respectively receive a plurality of messages to be processed triggered by the database, and the plurality of messages to be processed correspond to the same data identifier; performing distributed locking processing on one or more instances according to the data identification to obtain an attribute value of a distributed lock; processing the received messages to be processed by utilizing the locked first instance so as to process the data stored in the database according to the data in the database; updating the attribute values of the distributed lock with the second instance that is not locked; when the attribute values of the distributed locks of the first instance and the second instance are different, the technical means for synchronizing the data in the first instance and the second instance overcomes the technical problems of low message processing efficiency, distributed lock conflict and low message processing accuracy rate in the prior art, and further achieves the technical effects of locking the instances in the distributed cluster according to data identification (service ID), processing a plurality of received messages to be processed by using the locked instances, effectively reducing retry times, avoiding distributed lock conflict and improving message processing efficiency, and simultaneously improving the message processing accuracy rate by setting the attribute values of the distributed locks.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A message processing method is applied to a distributed cluster; the method comprises the following steps:
respectively receiving a plurality of messages to be processed triggered by a database by utilizing one or more instances in a distributed cluster, wherein the messages to be processed correspond to the same data identifier;
performing distributed locking processing on the one or more instances according to the data identification to obtain an attribute value of a distributed lock;
processing the received messages to be processed by utilizing the locked first instance so as to process the data stored in the database according to the data in the database;
updating the attribute values of the distributed lock with the second instance that is not locked;
synchronizing data in the first instance and the second instance when the attribute values of the distributed locks of the first instance and the second instance are different.
2. The method of claim 1, wherein processing the received plurality of pending messages using the locked first instance comprises: and carrying out duplicate removal processing on the received messages to be processed by utilizing the first instance according to the data identification, and processing the data stored in the first instance according to the duplicate removal processing result and the data in the database.
3. The method according to claim 2, wherein the processing the data stored in the database according to the deduplication processing result and the data in the database comprises: acquiring a data table corresponding to the data identifier from the database according to the data identifier indicated by the duplicate removal processing result;
and updating the data stored by the data table according to the acquired data table.
4. The method of claim 1, wherein performing distributed locking processing on the one or more instances according to the data identifier to obtain an attribute value of a distributed lock, comprises:
and generating a random number aiming at the data identification, performing distributed locking processing on the one or more instances according to the random number, and taking the random number as the attribute value.
5. The method of claim 1, further comprising:
and when the second instance receives the data updating message triggered by the database again, updating the attribute value of the distributed lock again by using the second instance.
6. The method of claim 5, wherein updating the attribute value of the distributed lock with the second instance comprises:
incrementing the distributed attribute value using the second instance.
7. The method of claim 1, wherein synchronizing the data in the first instance and the second instance comprises:
and determining a data synchronization source from the first instance and the second instance according to the sizes of the attribute values of the distributed locks respectively corresponding to the first instance and the second instance, and synchronizing data according to the data in the data synchronization source.
8. An apparatus for data processing, comprising:
the device comprises a receiving module, a first processing module, a determining module, a second processing module, a third processing module and a fourth processing module; wherein the content of the first and second substances,
the receiving module is used for receiving a plurality of messages to be processed triggered by the database respectively received by one or more instances in the distributed cluster;
the first processing module performs distributed locking processing on the one or more instances according to the data identifier to obtain an attribute value of a distributed lock;
the second processing module is used for processing the received messages to be processed by utilizing the locked first instance so as to process the data stored in the second processing module according to the data in the database;
the third processing module updates the attribute value of the distributed lock by using the second instance which is not locked;
the first determination module is configured to determine whether attribute values of distributed locks of the first instance and the second instance are different. If yes, triggering the fourth processing module; if not, ending the process;
and the fourth processing module is used for synchronizing the data in the first instance and the second instance.
9. An electronic device for message processing, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.
CN202110700492.5A 2021-06-23 2021-06-23 Message processing method and device Pending CN113377795A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110700492.5A CN113377795A (en) 2021-06-23 2021-06-23 Message processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110700492.5A CN113377795A (en) 2021-06-23 2021-06-23 Message processing method and device

Publications (1)

Publication Number Publication Date
CN113377795A true CN113377795A (en) 2021-09-10

Family

ID=77578759

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110700492.5A Pending CN113377795A (en) 2021-06-23 2021-06-23 Message processing method and device

Country Status (1)

Country Link
CN (1) CN113377795A (en)

Similar Documents

Publication Publication Date Title
CN110858194A (en) Method and device for expanding database
CN109150929B (en) Data request processing method and device under high concurrency scene
CN111782235A (en) Data upgrading and querying method and device
CN107844488B (en) Data query method and device
CN110909022A (en) Data query method and device
CN110795443A (en) Method, device, equipment and computer readable medium for data synchronization
CN112597126A (en) Data migration method and device
CN112948498A (en) Method and device for generating global identification of distributed system
CN111241189A (en) Method and device for synchronizing data
CN113641706A (en) Data query method and device
CN113761565A (en) Data desensitization method and apparatus
CN112100168A (en) Method and device for determining data association relationship
CN109144991B (en) Method and device for dynamic sub-metering, electronic equipment and computer-storable medium
CN113377795A (en) Message processing method and device
CN113760861A (en) Data migration method and device
CN110705935B (en) Logistics document processing method and device
CN113742321A (en) Data updating method and device
CN114116675A (en) Data archiving method and device
CN107665241B (en) Real-time data multi-dimensional duplicate removal method and device
CN113760841A (en) Method and device for realizing distributed lock
CN113760860B (en) Data reading method and device
CN113704242A (en) Data processing method and device
CN112711572A (en) Online capacity expansion method and device suitable for sub-warehouse and sub-meter
CN112579615A (en) Method and device for realizing distributed lock
CN113495891A (en) Data processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination