CN108833443B - Message transmission method and system and computer equipment - Google Patents
Message transmission method and system and computer equipment Download PDFInfo
- Publication number
- CN108833443B CN108833443B CN201810833542.5A CN201810833542A CN108833443B CN 108833443 B CN108833443 B CN 108833443B CN 201810833542 A CN201810833542 A CN 201810833542A CN 108833443 B CN108833443 B CN 108833443B
- Authority
- CN
- China
- Prior art keywords
- message
- security
- configuration
- consumer
- certificate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 230000005540 biological transmission Effects 0.000 title claims abstract description 53
- 238000012545 processing Methods 0.000 claims abstract description 65
- 238000012795 verification Methods 0.000 claims abstract description 30
- 238000005192 partition Methods 0.000 claims description 24
- 230000008569 process Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 3
- 238000011161 development Methods 0.000 abstract description 6
- 238000012423 maintenance Methods 0.000 abstract description 3
- 230000009286 beneficial effect Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/20—Network architectures or network communication protocols for network security for managing network security; network security policies in general
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
- H04L63/0823—Network architectures or network communication protocols for network security for authentication of entities using certificates
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/16—Implementing security features at a particular protocol layer
- H04L63/168—Implementing security features at a particular protocol layer above the transport layer
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to the technical field of computers, in particular to a message transmission method, a message transmission system and computer equipment, wherein the method comprises the following steps: after the program is started, the message system loads a security configuration table, and when the configuration data needs security processing, configuration information is obtained from the security configuration table; receiving a message which is sent by a message producer with a certificate and is subjected to security processing; the message is encrypted according to the configuration information so that the message consumer with the certificate receives the encrypted message. The invention supports distributed deployment by adopting the message system as a medium for message transmission, greatly improves the message transmission efficiency, can flexibly configure a safety processing and safety verification mode, a message format and a verification method, and reduces the development difficulty and maintenance cost of the system after new requirements are added.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a message transmission method and system, and a computer device.
Background
Data interaction is generally completed by adopting webservice service or IBM MQ message queue technology aiming at message passing methods between heterogeneous systems. When the webservice technology is adopted, the transmission parties complete service release and request calling according to the appointed interface specification document, and then data interaction between the transmission parties can be completed. When using the IBM MQ message queue technology, the MQ product acts as an important intermediary for message delivery for message transport. The product provides APIs in multiple development languages. Both parties of the transmission need to send and receive messages according to the specific API provided by the MQ.
However, messaging systems employing webservice technology and IBM MQ technology both suffer from inefficient messaging and processing in highly concurrent scenarios.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a message transmission method, a message transmission system, and a computer device, which can improve the efficiency of message transmission and processing in a concurrent scene, in view of the defects of the prior art.
The technical scheme for solving the technical problems is as follows: a method of message transmission, comprising:
after a program is started, a message system loads a security configuration table, and when configuration data needs to be processed safely, configuration information is obtained from the security configuration table, wherein the security configuration table comprises: whether to perform message system security processing, a message system security processing mode, message system certificate position configuration, whether to perform message security processing and a message password;
receiving a message which is sent by a message producer with a certificate and is subjected to security processing;
and encrypting the message according to the configuration information so that the message consumer with the certificate receives the encrypted message.
The invention has the beneficial effects that: by adopting the message system as a medium for message transmission, distributed deployment is supported, message transmission efficiency is greatly improved, a security processing and security verification mode, a message format and a verification method can be flexibly configured, and development difficulty and maintenance cost of the system after new requirements are added are reduced.
On the basis of the technical scheme, the invention can be further improved as follows.
Further, still include: and the message producer converts the data to be sent into a message with a preset format according to the configuration information and sends the message to the message system.
Further, still include:
the message consumer checks the received message according to a check message, wherein the check message is a configured message with the preset format;
and when the verification result is failed, processing according to the configured failure processing mode.
The beneficial effect of adopting the further scheme is that: the configurable security processing and security verification, and the configurable message format conversion and verification are adopted, so that the message sender and the message receiver are flexibly matched, and the maximum flexibility of the message transmission system is achieved. And the method is suitable for message transmission under the scene of large data volume and high concurrency, when a message sender and a message receiver are newly added, the system can transmit new service data only by adjusting the message format, the security policy and the kafka message system configuration, and a large amount of development work caused by demand change is greatly reduced.
Further, the predetermined format includes: message type, message label, fixed value, message format and failure processing mode, the method also includes:
and storing the preset format and the check message in a cache.
The beneficial effect of adopting the further scheme is that: configuration information such as a preset format, a check message and the like is stored in a local cache, so that reading from a database every time is avoided, and the message transmission speed can be effectively increased.
Further, the message producer and the message consumer are both multiple, the method further comprising:
the message system carries out kafka theme classification according to the message types;
wherein, the receiving of the message which is sent by the message producer with the certificate and is processed by security comprises:
the message system receives the messages sent by a plurality of message producers simultaneously and stores each message in a partition mode in a theme corresponding to the corresponding message type;
and each message consumer simultaneously subscribes messages of a plurality of message types and reads corresponding messages from the partition of the theme corresponding to each message type.
The beneficial effect of adopting the further scheme is that: the bottleneck of message transmission in a high concurrency scene is solved by a message type partition storage mode, message consumers can process messages in the queue in batches, the processing speed is higher, and the messages can be repeatedly consumed by a plurality of consumers.
Further, the method further comprises:
retrying to receive when the message consumer fails to receive the message;
and when the retry receiving times of the message consumers exceed the configured retry receiving times, sending the message to the error message theme of the corresponding theme, and recording the log information.
The beneficial effect of adopting the further scheme is that: by adopting a retry strategy, the reliability of the system can be improved.
Further, the configuration information includes: a security protocol, a certificate location, and a password configuration field, wherein the security processing mode is determined by the security protocol, and the security protocol comprises: SSL protocol, SASL _ plain ext protocol and SASL _ SSL protocol.
Further, the method further comprises: and the message consumer stores the received message in a memory database and sets the expiration time of the message.
The beneficial effect of adopting the further scheme is that: by adopting the strategy of message file backup, the reliability of the system can be improved.
Another technical solution of the present invention for solving the above technical problems is as follows: a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the message transmission method as in the previous embodiments when executing the program.
Another technical solution of the present invention for solving the above technical problems is as follows: a message transmission system comprising: a plurality of message producers, a message system, and a plurality of message consumers, wherein,
the message system is used for loading a security configuration table after a program is started, and acquiring configuration information from the security configuration table when configuration data needs to be processed safely, wherein the security configuration table comprises: whether to perform message system security processing, a message system security processing mode, message system certificate position configuration and password configuration, whether to perform message security processing and message passwords;
each message producer is used for converting data to be sent into a message with a preset format according to the configuration information and sending the message to the message system;
the message system is also used for receiving a message which is sent by the message producer with the certificate and is subjected to security processing, and encrypting the message according to the configuration information;
each message consumer is used for verifying the received message according to a verification message, wherein the verification message is a configured message with the preset format, and when the verification result is failed, the message consumer processes the message according to a configured failure processing mode.
The invention has the beneficial effects that: the message transmission system refers to the kafka message component, the bottleneck of message transmission in a high concurrency scene is solved through the distributed kafka message system supporting multiple partitions, message consumers can process messages in the queue in batch, and the processing speed is higher. The system adopts configurable security processing and security verification, and configurable message format conversion and verification, and flexibly matches the message sender and the message receiver to achieve the maximum flexibility of the message transmission system.
Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments of the present invention or in the description of the prior art will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a message transmission method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a message transmission method according to another embodiment of the present invention;
fig. 3 is a schematic flow chart of a message transmission method according to another embodiment of the present invention;
fig. 4 is a schematic flow chart of a message transmission method according to another embodiment of the present invention;
fig. 5 is a schematic flow chart of a message transmission method according to another embodiment of the present invention;
fig. 6 is a schematic flow chart of a message transmission method according to another embodiment of the present invention;
fig. 7 is a schematic structural block diagram of a message transmission system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, shall fall within the scope of protection of the present invention.
As shown in fig. 1, a message transmission method 100 includes:
101. after the program is started, the message system loads the security configuration table, and when the configuration data needs to be processed safely, the configuration information is obtained from the security configuration table.
Wherein, the security configuration table may include: whether to perform message system security processing, a message system security processing mode, message system certificate position configuration, whether to perform message security processing, and a message password.
102. And receiving a message which is sent by a message producer with a certificate and is subjected to security processing.
103. The message is encrypted according to the configuration information so that the message consumer with the certificate receives the encrypted message.
Specifically, in this embodiment, after the program is started, the message system loads all the security configuration tables, and when the configuration data for configuring the message system needs to be processed securely, the configuration information is obtained from the security configuration tables. The security configuration table includes contents such as whether to perform security processing of the message system, a security processing mode of the message system, certificate location configuration of the message system, whether to perform security processing of the message, and a message password.
The configuration data is configuration table information in a software database and is used for configuring the system; messages and messages are information that the software is to process.
The message producer and the message consumer are separate. The message producer needs to configure the certificate path of the message producer in a security configuration table; the message consumer needs to configure the message consumer's certificate path in the security configuration table. The certificate content includes: a username and password.
It should be noted that, in this embodiment, the configuration information may include: the security protocol, the certificate location, the password configuration field, and the like, it should be noted that the configuration information may include more parameters, and the embodiment of the present invention is not limited thereto.
The security processing mode is determined by a security protocol, for example: the security protocol may include: SSL, SASL _ plain ext and SASL _ SSL, which are not limited in this embodiment of the present invention.
It should be appreciated that in this embodiment, the messaging system may employ a kafka messaging component.
The message transmission method provided in the above embodiment supports distributed deployment by using the message system as a medium for message transmission, greatly improves message transmission efficiency, can flexibly configure a security processing and security verification mode, a message format, and a verification method, and reduces development difficulty and maintenance cost of a system after new requirements are added.
Optionally, in an embodiment, as shown in fig. 2, before 102, the method 100 may further include:
104. and the message producer converts the data to be sent into a message with a preset format according to the configuration information and sends the message to the message system.
Specifically, in this embodiment, after the program is started, the message system loads the security configuration table, and when the configuration data needs to be processed securely, the configuration information is obtained from the security configuration table. The message producer with the certificate obtains the configuration information from the message system, converts the data to be sent into a message with a preset format according to the configuration information, and sends the message to the message system. The message system receives the message and encrypts the message according to the configuration information so that the message consumer with the certificate receives the encrypted message.
It should be understood that, in this embodiment, a message producer may convert the same batch of data to be transmitted into a message in a predetermined format, automatically compress the message and send the message to a message system, and a message consumer may receive and decompress the message in batch, thereby implementing batch processing and reducing network load.
Optionally, in another embodiment, as shown in fig. 3, after 103, the method 100 may further include:
105. and the message consumer checks the received message according to the check message, wherein the check message is a configured message with a preset format.
106. And when the verification result is failed, processing according to the configured failure processing mode.
Specifically, in this embodiment, after the program is started, the message system loads the security configuration table, and when the configuration data needs to be processed securely, the configuration information is obtained from the security configuration table. The message producer with the certificate obtains the configuration information from the message system, converts the data to be sent into a message with a preset format according to the configuration information, and sends the message to the message system. The message system receives the message and encrypts the message according to the configuration information, and the message consumer with the certificate verifies the received message according to the verification message after receiving the encrypted message, wherein the verification message is the configured message with the preset format.
It should be noted that, the checking of the message here specifically refers to checking of the format of the message, such as: whether the message tag names are correct or not, whether the number is consistent or not, and the like.
And when the verification result is failed, processing according to the configured failure processing mode. For example: log or discard, etc. And when the verification result is that the message passes, performing corresponding service processing on the message, and storing the data into a database.
The message transmission method provided in the above embodiment flexibly matches the message sender and the message receiver by adopting configurable security processing and security verification, and configurable message format conversion and verification, thereby achieving the maximum flexibility of the message transmission system. And the method is suitable for message transmission under the scene of large data volume and high concurrency, when a message sender and a message receiver are newly added, the system can transmit new service data only by adjusting the message format, the security policy and the kafka message system configuration, and a large amount of development work caused by demand change is greatly reduced.
Optionally, in another embodiment, the predetermined format includes: the method 100 may further include the following steps:
and storing the preset format and the check message in a cache.
In the embodiment, the configuration information such as the preset format, the check message and the like is stored in the local cache, so that the reading from the database every time is avoided, and the message transmission speed can be effectively increased.
Optionally, in another embodiment, as shown in fig. 4, the message producer and the message consumer are both multiple, and the method 100 may further include:
107. the message system classifies kafka topics according to message category.
Then, step 104 specifically includes:
and the message producers convert the respective data to be sent into messages with a preset format according to the configuration information and simultaneously send the messages to the message system.
Then, step 102 specifically includes:
the message system receives messages sent by a plurality of message producers at the same time, and stores each message in a partition mode in a theme corresponding to the corresponding message type.
108. Each message consumer subscribes to messages of multiple message types at the same time, and reads corresponding messages from the partition of the theme corresponding to each message type.
Specifically, in this embodiment, the message producer and the message consumer are both multiple, the message system may perform kafka topic classification according to the message type, when multiple message producers send messages to the message system at the same time, the message system performs partition storage on each received message, where the message type of each message corresponds to one topic, and different topics are stored in different partitions. Each message consumer can subscribe to messages of multiple message types at the same time, and reads corresponding messages from the partition of the theme corresponding to each message type.
For example: if the number of message producers is 6, the number of message consumers is 2, the number of message types is 5, and the kafka theme has 5 classifications. 6 message producers send messages to the message system at the same time, and the messages are respectively recorded as: message 1, message 2, message 3, message 4, message 5, and message 6. The message 1, the message 2 and the message 3 are messages of the same kind; the message 4 and the message 5 are messages of the same type; the message 6 is a kind of message.
The message system stores each received message in a partition mode, the message type of each message corresponds to one theme, and different themes are stored in different partitions. That is to say: storing the message 1, the message 2 and the message 3 in a subarea A corresponding to the theme A; the message 4 and the message 5 are stored in a partition B corresponding to the theme B; the message 6 is stored in a partition C corresponding to its topic C.
Each message consumer can subscribe to messages of multiple message types at the same time, and reads corresponding messages from the partition of the theme corresponding to each message type. For example: the message consumer 1 subscribes 2 message types at the same time, corresponding to a theme A and a theme C; the message consumer 2 has subscribed to 2 message categories simultaneously, corresponding to topic B and topic C. Then, the message consumer 1 reads corresponding messages from the partition a corresponding to the topic a and the partition C corresponding to the topic C respectively; and the message consumer 2 respectively reads corresponding messages from the partition B corresponding to the theme B and the partition C corresponding to the theme C.
The message transmission method provided in the above embodiment solves the bottleneck of message transmission in a high concurrency scene through a message type partition storage manner, and a message consumer can process messages in a queue in batch, so that the processing speed is higher, and the messages can be repeatedly consumed by a plurality of consumers.
It should be noted that, in this embodiment, the number of message producers and the number of message consumers may be the same or different, and this is not limited in this embodiment of the present invention.
It should be understood that, in this embodiment, step 104, or step 109 and step 110, or step 111, or a combination of the above steps may also be included, and for brevity of description, the description is not repeated herein.
Optionally, in another embodiment, as shown in fig. 5, the method 100 may further include:
109. when the message consumer fails to receive the message, the message consumer retries the reception.
110. And when the retry receiving times of the message consumers exceed the configured retry times, sending the message to the error message theme of the corresponding theme, and recording the log information.
To address the problem of batch processing being blocked by retry processing, the present solution employs a reliable retry strategy between sending and receiving (i.e., message producer and message consumer). Specifically, in this embodiment, when the message consumer fails to receive the message, the message consumer may retry receiving, and when the number of retries of receiving exceeds the configured number of retries, the message is sent to the error message topic of the corresponding topic, and log information is recorded, so that the message consumer may continue to receive the next message.
It should be noted that, in this embodiment, step 104, or steps 107, 104, 108, or step 111, or a combination of the foregoing steps may also be included, and details are not described herein again for brevity of description.
Optionally, in another embodiment, as shown in fig. 6, the method 100 may further include:
111. the message consumer stores the received message in the memory database and sets the expiration time of the message.
In order to improve the reliability of the system, the scheme also adopts a strategy of message file backup. Specifically, in this embodiment, the message consumer saves the received message to the in-memory database and sets the expiration time. When abnormal conditions occur, the problem reason can be searched by checking the original message received in the memory database. When the expiration time is reached, the message is discarded so as to store the subsequently received message by utilizing the memory space occupied by the vacated message.
It should be understood that in the above embodiments, the content of the message and the message is essentially the same, and the message is only one specific form in the message transmission process. In addition, the message producer and the message consumer in the above embodiments are both provided with certificates, and only with certificates, the message can be transmitted through the message system.
It should be noted that, in this embodiment, the method may further include step 104, or steps 104 and 105 and 106, or step 107 and 108, or step 109 and 110, or a combination of the above steps, and for brevity of description, the description is not repeated herein.
It should be understood that, in the embodiments of the present invention, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the program, the steps of the message transmission method in the foregoing embodiments are implemented.
The above describes a technical solution of a message transmission method in detail with reference to fig. 1 to 6, and the following describes a technical solution of a message transmission system in detail with reference to fig. 7.
A message transmission system as shown in fig. 7 includes: a plurality of message producers 210, a message system 220, and a plurality of message consumers 230, wherein,
and the message system 220 is configured to load the security configuration table after the program is started, and obtain the configuration information from the security configuration table when the configuration data needs to be processed securely. Wherein, the security configuration table includes: whether to perform message system security processing, a message system security processing mode, message system certificate position configuration and password configuration, whether to perform message security processing and message passwords.
Each message producer 210 is configured to convert data to be sent into a message in a predetermined format according to the configuration information, and send the message to the message system 220.
The message system 220 is further configured to receive the message that is sent by the message producer 210 with the certificate and that is subjected to security processing, and encrypt the message according to the configuration information.
Each message consumer 230 is configured to verify the received message according to the verification message, where the verification message is a configured message with a predetermined format, and when the verification result is failed, perform processing according to a configured failure processing manner.
In particular, in this embodiment, each message producer 210 and each message consumer 230 may have a certificate configured by the message system 220 to ensure that message transfers between the two parties are possible. In addition, each message producer 210 can convert the same batch of data to be transmitted into a message with a predetermined format, compress the message and send the message to the message system 220, and the message consumer 230 can receive and decompress the message in batches, so that batch processing is realized, and network load is reduced.
The message transmission system in the above embodiment refers to the kafka message component, and solves the message transmission bottleneck in a high concurrency scenario through the distributed kafka message system supporting multiple partitions, so that the message consumers can process the messages in the queue in batch, and the processing speed is higher. The system adopts configurable security processing and security verification, and configurable message format conversion and verification, and flexibly matches the message sender and the message receiver to achieve the maximum flexibility of the message transmission system.
Optionally, in an embodiment, the message system 220 may further perform kafka topic classification according to the message category, receive messages sent by multiple message producers at the same time, and store each message in a topic corresponding to its corresponding message category in a partition manner. Each message consumer 230 may subscribe to messages of multiple message categories simultaneously and read the corresponding message from the partition of the topic corresponding to each message category.
Optionally, in another embodiment, each message consumer 230 is further configured to retry receiving when receiving the message fails, and send the message to the error message topic of the corresponding topic and record log information when the number of retries of receiving exceeds the configured number of retries.
In addition, each message consumer 230 is further configured to store the received message in the memory database, and set the expiration time of the message, so that when an abnormal condition occurs, the problem cause can be found by checking the original message received in the memory database.
In addition, the term "and/or" herein is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
While the invention has been described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (9)
1. A method for message transmission, comprising:
after a program is started, a message system loads a security configuration table, and when configuration data for configuring the message system needs to be processed safely, configuration information is obtained from the security configuration table, wherein the security configuration table comprises: whether to perform message system security processing, a message system security processing mode, message system certificate position configuration, whether to perform message security processing and a message password;
receiving a message which is sent by a message producer with a certificate and is subjected to security processing;
encrypting the message according to the configuration information so that the message consumer with the certificate receives the encrypted message;
further comprising: and the message producer converts the data to be sent into a message with a preset format according to the configuration information and sends the message to the message system.
2. The method of claim 1, further comprising:
the message consumer checks the received message according to a check message, wherein the check message is a configured message with the preset format;
and when the verification result is failed, processing according to the configured failure processing mode.
3. The method of claim 2, wherein the predetermined format comprises: message type, message label, fixed value, message format and failure processing mode, the method also includes:
and storing the preset format and the check message in a cache.
4. The method of any of claims 1-3, wherein the message producer and the message consumer are each in plurality, the method further comprising:
the message system carries out kafka theme classification according to the message types;
wherein, the receiving of the message which is sent by the message producer with the certificate and is processed by security comprises:
the message system receives the messages sent by a plurality of message producers simultaneously and stores each message in a partition mode in a theme corresponding to the corresponding message type;
and each message consumer simultaneously subscribes messages of a plurality of message types and reads corresponding messages from the partition of the theme corresponding to each message type.
5. The method of claim 4, further comprising:
retrying to receive when the message consumer fails to receive the message;
and when the retry receiving times of the message consumers exceed the configured retry receiving times, sending the message to the error message theme of the corresponding theme, and recording the log information.
6. The method according to any of claims 1-3, wherein the configuration information comprises: a security protocol, a certificate location, and a password configuration field, wherein the security processing mode is determined by the security protocol, and the security protocol comprises: SSL protocol, SASL _ plain ext protocol and SASL _ SSL protocol.
7. The method according to any one of claims 1-3, further comprising:
and the message consumer stores the received message in a memory database and sets the expiration time of the message.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the message transmission method according to any of claims 1-7 are implemented when the program is executed by the processor.
9. A message transmission system, comprising: a plurality of message producers, a message system, and a plurality of message consumers, wherein,
the message system is used for loading a security configuration table after a program is started, and acquiring configuration information from the security configuration table when configuration data for configuring the message system needs to be processed safely, wherein the security configuration table comprises: whether to perform message system security processing, a message system security processing mode, message system certificate position configuration and password configuration, whether to perform message security processing and message passwords;
each message producer is used for converting data to be sent into a message with a preset format according to the configuration information and sending the message to the message system; the message system is also used for receiving a message which is sent by the message producer with the certificate and is subjected to security processing, and encrypting the message according to the configuration information;
each message consumer is used for verifying the received message according to a verification message, wherein the verification message is a configured message with the preset format, and when the verification result is failed, the message consumer processes the message according to a configured failure processing mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810833542.5A CN108833443B (en) | 2018-07-26 | 2018-07-26 | Message transmission method and system and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810833542.5A CN108833443B (en) | 2018-07-26 | 2018-07-26 | Message transmission method and system and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108833443A CN108833443A (en) | 2018-11-16 |
| CN108833443B true CN108833443B (en) | 2021-02-02 |
Family
ID=64140058
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810833542.5A Active CN108833443B (en) | 2018-07-26 | 2018-07-26 | Message transmission method and system and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108833443B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110806958A (en) * | 2019-10-24 | 2020-02-18 | 长城计算机软件与系统有限公司 | Monitoring method, monitoring device, storage medium and electronic equipment |
| CN110995851B (en) * | 2019-12-11 | 2021-12-24 | 贝壳找房(北京)科技有限公司 | Message processing method, device, storage medium and equipment |
| CN111262915B (en) * | 2020-01-10 | 2020-09-22 | 北京东方金信科技有限公司 | Kafka cluster-crossing data conversion system and method |
| CN111866189A (en) * | 2020-09-03 | 2020-10-30 | 中国银行股份有限公司 | Batch file transmission method and device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106910146A (en) * | 2017-02-28 | 2017-06-30 | 东北师范大学 | A kind of isomery educational data switching plane and method based on Stream Processing technology |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080083026A1 (en) * | 2006-10-02 | 2008-04-03 | Bea Systems, Inc. | Kerberos Protocol Security Provider for a Java Based Application Server |
| CN104869174A (en) * | 2015-06-15 | 2015-08-26 | 北京邮电大学 | File transmission method based on third party message-oriented middle-ware |
| CN107438061B (en) * | 2016-05-27 | 2020-03-03 | 北京京东尚科信息技术有限公司 | Method and device for authenticating kafka client |
-
2018
- 2018-07-26 CN CN201810833542.5A patent/CN108833443B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106910146A (en) * | 2017-02-28 | 2017-06-30 | 东北师范大学 | A kind of isomery educational data switching plane and method based on Stream Processing technology |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108833443A (en) | 2018-11-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108833443B (en) | Message transmission method and system and computer equipment | |
| US11082419B2 (en) | System and method for cloud-based analytics | |
| CN100433735C (en) | Communication system | |
| CN111935227A (en) | Method for uploading file through browser, browser and electronic equipment | |
| CN105791399B (en) | More trunk interconnection net big data method for pushing and system | |
| CN114338651A (en) | File transmission method and device, electronic equipment and readable storage medium | |
| CN103259797A (en) | Data file transmission method and platform | |
| CN107688503A (en) | A kind of message treatment method based on ActiveMQ data/address bus, device and electronic equipment | |
| CN106603258A (en) | Data transmission method and device | |
| CN113037787A (en) | Data processing method and device | |
| CN101795243A (en) | Method, device and system for accessing attachment files in electronic mails | |
| US11050719B2 (en) | Systems and methods for controlling email access | |
| CN112865927B (en) | Message delivery verification method, device, computer equipment and storage medium | |
| CN113992352B (en) | Message pushing method and device, electronic equipment and storage medium | |
| CN109600722B (en) | Short message receiving verification method and device, electronic equipment and storage medium | |
| CN113347245B (en) | File fragment breakpoint transmission method and device based on message queue | |
| CN113076380B (en) | Data synchronization method, device, system, equipment and storage medium | |
| CN112333262A (en) | Data updating prompting method and device, computer equipment and readable storage medium | |
| CN114217932A (en) | Third-party docking exception handling method and system | |
| CN110213322B (en) | Content publishing processing method, client and server for community | |
| CN113645248A (en) | Data exchange system and method under cross-network environment | |
| CN107704557B (en) | Processing method and device for operating mutually exclusive data, computer equipment and storage medium | |
| CN106789272A (en) | A kind of server set group managing means and system | |
| CN104427003A (en) | Transmission apparatus, transmission method, and relay system | |
| CN111240867A (en) | Information communication system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100190 17-19 / F, building a 1, 66 Zhongguancun East Road, Haidian District, Beijing Patentee after: New Great Wall Technology Co.,Ltd. Address before: 100190 17-19 / F, building a 1, 66 Zhongguancun East Road, Haidian District, Beijing Patentee before: GREAT WALL COMPUTER SOFTWARE & SYSTEMS Inc. |
|
| CP01 | Change in the name or title of a patent holder |