CN113961632A - Message data processing method and device and electronic equipment - Google Patents
Message data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN113961632A CN113961632A CN202111229412.9A CN202111229412A CN113961632A CN 113961632 A CN113961632 A CN 113961632A CN 202111229412 A CN202111229412 A CN 202111229412A CN 113961632 A CN113961632 A CN 113961632A
- Authority
- CN
- China
- Prior art keywords
- message
- data
- service data
- database
- storing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/06—Buying, selling or leasing transactions
- G06Q30/0601—Electronic shopping [e-shopping]
- G06Q30/0633—Lists, e.g. purchase orders, compilation or processing
- G06Q30/0635—Processing of requisition or of purchase orders
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Finance (AREA)
- Data Mining & Analysis (AREA)
- Accounting & Taxation (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application provides a message data processing method and device, electronic equipment and a computer readable storage medium, and relates to the field of data processing. Acquiring a first message and a message queue identifier; storing the first message and the message queue identification to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the original message; storing the first service data to a second database; therefore, distributed processing of acquiring the original message, storing the original message, analyzing the original message and storing the analysis result in the message data processing process is realized, and the coupling among all sub-processes of the distributed processing is low, so that all sub-processes can be started, stopped and repeatedly executed at any time, and the flexibility of the message processing process is improved; and by storing the original message and the analysis result of the original message, the trace-following of each sub-process of message processing is realized, and the original message is conveniently checked at any time when the message data processing process is abnormal.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a message data processing method and apparatus, an electronic device, and a computer-readable storage medium.
Background
With the development of informatization, more and more information systems for processing information are provided, and in order to strengthen the interaction of each information system and further realize information intercommunication, a plurality of systems can be integrated in an integrated platform so as to provide comprehensive information service for users.
At present, in an information processing scenario, information consumers often need to acquire message data from messages generated by various information systems of an integrated platform. In the related art, the process of message data processing usually has high coupling, which results in lack of flexibility and controllability of message data processing.
Disclosure of Invention
The present application aims to solve at least one of the above technical drawbacks, in particular the technical drawback of high coupling in the message processing.
According to an aspect of the present application, there is provided a message data processing method, including: acquiring a first message and a message queue identifier; the message queue identification comprises an identification of a message queue of the first message in a server;
storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message;
and storing the first service data to a second database.
Optionally, the performing, by the parsing module, an analysis processing operation on the first message includes:
determining a first data analysis rule corresponding to the message queue according to the message queue identification;
and analyzing the first message according to the first data analysis rule to obtain second service data.
Optionally, the acquiring the first service data corresponding to the first message includes:
and performing data conversion on the second service data to obtain the first service data, wherein the first service data corresponds to a first data standard, and the second service data corresponds to a second data standard.
Optionally, the storing the first service data to a second database includes:
determining the data type of the first service data;
and storing the first service data to a target data table corresponding to the data type in the second database.
Optionally, the method further includes:
and if first indication information indicating that the first message is analyzed abnormally is received, or second indication information indicating that the first service data is stored abnormally when being stored in a corresponding target data table of the second database is received, adding an abnormal identifier for the first message.
Optionally, the method further includes:
and when a preset condition is met, re-executing the analysis of the first message.
Optionally, the preset condition includes at least one of the following:
the quantity of the first indication information and/or the second indication information reaches a preset value;
and reaching the analysis processing operation time point of the preset time period.
Optionally, the message queue identifier includes at least one of:
the identifier of the server corresponding to the message queue;
the identifier of the manager corresponding to the message queue;
and the identifier of the channel corresponding to the message queue.
According to another aspect of the present application, there is provided a message data processing apparatus, including:
the acquisition module is used for acquiring a first message and a message queue identifier; the message queue identification comprises an identification of a message queue of the first message in a server;
the first processing module is used for storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message;
and the storage module is used for storing the first service data to a second database.
According to another aspect of the present application, there is provided an electronic device including:
one or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to: executing the message data processing method of any one of the first aspect of the present application.
For example, in a third aspect of the present application, there is provided a computing device comprising: the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the corresponding operation of the message data processing method as shown in the first aspect of the application.
According to yet another aspect of the present application, there is provided a computer readable storage medium, which when executed by a processor implements the message data processing method of any one of the first aspects of the present application.
For example, in a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program, when executed by a processor, implements the message data processing method shown in the first aspect of the present application.
According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in the various alternative implementations of the first aspect described above.
The beneficial effect that technical scheme that this application provided brought is:
the embodiment of the application can be realized by acquiring a first message and a message queue identifier; storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message; storing the first service data to a second database; therefore, distributed processing of acquiring the original message (namely the first message), storing the original message, analyzing the original message and storing the analysis result in the message data processing process is realized, and the coupling among all sub-processes of the distributed processing is low, so that all sub-processes can be started, stopped and repeatedly executed at any time, and the flexibility and controllability of the message processing process are improved; and by storing the original message and the analysis result of the original message, the trace-following of each sub-process of message processing is realized, and the original message is conveniently checked at any time when the message data processing process is abnormal.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments of the present application will be briefly described below.
Fig. 1 is a schematic flowchart of a message data processing method according to an embodiment of the present application;
fig. 2 is a schematic diagram of a log list of a message data processing method according to an embodiment of the present application;
fig. 3 is a schematic diagram of message details of a message data processing method according to an embodiment of the present application;
fig. 4 is a schematic view of a data parsing rule configuration interface in a message data processing method according to an embodiment of the present application;
fig. 5 is a second schematic flowchart of a message data processing method according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a message data processing apparatus according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an electronic device for processing message data according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
The terms referred to in this application will first be introduced and explained:
message: a message is a unit of data that is transmitted between computers. The message may be very simple, e.g. containing only text strings; or may be more complex, possibly containing embedded objects.
Message queue: a message queue is a container that holds messages during their transmission. The message is generated and then sent to a message queue. The main function of the message queue is to provide routing and guarantee the delivery of the message; if the recipient is not available when the message is sent, the message queue will hold the message until it can be successfully delivered.
In a practical scenario, a message generated by each server or each information system contains various service data, and a message consumer often needs to process the message, acquire the service data from the message for data analysis, and the like. However, in the related art, the message processing procedure often has high coupling, so that once an error or an abnormality occurs in the message processing procedure, no trace is found for the error point or the reason, so that the whole processing procedure needs to be executed again.
The application provides a message data processing method, a message data processing device, an electronic device and a computer-readable storage medium, which aim to solve the above technical problems in the prior art.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
The scheme provided by the embodiment of the application can be executed by any electronic device, such as a terminal device, or a server, wherein the server can be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing service. The terminal may be, but is not limited to, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal and the server may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein. For technical problems in the prior art, the message data processing method, device, electronic device and storage medium provided by the present application aim to solve at least one of the technical problems in the prior art.
A possible implementation manner is provided in the embodiment of the present application, and as shown in fig. 1, a flowchart of a message data processing method is provided, where the scheme may be executed by any electronic device, and optionally, the method may be applied to a server or a terminal device, and for convenience of description, the method provided in the embodiment of the present application is described below with a server as an execution subject.
The embodiment of the application can be applied to the technical field of data processing and databases; for example, for a message in a plurality of message queues in an information processing system, a first message and a message queue identifier may be acquired, the first message and the message queue identifier are stored in a first database, and an analysis processing operation is performed on the first message to acquire first service data corresponding to the first message; then storing the first service data into a second database, thereby realizing distributed processing of acquiring an original message (namely, a first message), storing the original message, analyzing the original message and storing an analysis result in the message data processing process; therefore, the coupling among the sub-processes of the distributed processing is low, and the sub-processes can be started, stopped and repeatedly executed at any time, so that the flexibility and controllability of the message processing process are improved; in addition, by storing the original message and the analysis result of the original message, the trace-following of each sub-process of message processing is realized, for example, when the message data processing process is abnormal, the original message is convenient to check at any time.
Specifically, as shown in fig. 1, the message data processing method according to the embodiment of the present application may include the following steps:
s101: and acquiring a first message and a message queue identifier.
The message queue identification comprises an identification of a message queue of the first message in a server.
In particular, the first message may comprise any message in a message queue. In different scenarios, the message producer is different, and the message generated is also different.
As an example one, in the scenario of online shopping, an order message may be generated by an order system in response to a user placing an order to purchase a good; the order message may include various order service data, for example, the order message may include order service data such as a user account number, an order placing time, a payment amount, and the like of an order placing user.
As an example two, in the medical treatment scene, after the patient is registered, a registration list message of the patient can be generated through the registration system; the registration list message may include registration service data such as patient name, age, sex, office and doctor. For another example, in the medical treatment scene, after the doctor sets up the order, an order message can be generated through the order system; the order information may include the medical order business information such as the patient's condition, the prepared traditional Chinese medicine and western medicine, and the medication caution items.
In summary, the message data processing method according to the embodiment of the present application may be applied to various different scenarios, and the first message may include various types of messages, which is not limited in the embodiment of the present application.
The message queue comprises a container for storing messages in the message transmission process; the message is generated and then sent to a message queue. The message queue identification comprises an identification of a message queue of the first message in the server. The message queue is the message queue where the first message is located.
The message queue identifier may specifically include: at least one of an identifier of a server corresponding to the message queue, an identifier of a manager corresponding to the message queue, an identifier of a channel corresponding to the message queue, or a name of the message queue.
When the first message is acquired, a connection channel can be established with a server or an information system where the message queue is located; monitoring the first message through the connection channel; and when the generation of the first message is monitored, acquiring the first message and acquiring a message queue identifier.
Specifically, the first message may be obtained in the following ways:
the first method is as follows: real-time monitoring can be carried out, and when each first message is generated, the first message is acquired.
The second method comprises the following steps: in order to save the overhead of the processing action of acquiring the message, the monitored first message may also be periodically acquired. For example, acquiring the monitored first message is performed every 5 minutes, or when a preset number of first messages are monitored (for example, 100 first messages are monitored), the monitored first messages are acquired.
The third method comprises the following steps: the first message may also be retrieved in response to a preset trigger condition. For example, the preset trigger condition may include receiving an instruction to perform data analysis on the first message, and the like.
S102: and storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message.
In the embodiment of the present application, in the first message processing process, if an error or an abnormality occurs, in order to ensure that the original first message can be checked at any time, the first message and the message queue identifier may be stored in the first database for backup. Therefore, once an error or an abnormality occurs in the subsequent first message processing process, only the original first message needs to be acquired from the first database for checking, or the processing of the first message needs to be executed again, and the first message does not need to be acquired again from the server or the information system generating the first message, so that the message processing efficiency is improved.
It should be noted that, if an abnormality occurs when the first message and the message queue identifier are stored in the first database, which results in that the first message and the message queue identifier cannot be successfully stored in the first database, the first message and the message queue identifier may also be stored in the hard disk for standby.
In addition, after the first message is stored, the first message may be analyzed to obtain first service data corresponding to the first message.
The first service data may include structured service data included in the first message, and in a scenario of online shopping in combination with the first example and the second example, when the first message is an order message, the first message may include order service data such as a user account, an order number, an order placing time, a payment amount, and the like of an order placing user; in the medical treatment scene, when the first message is a registration form message, the first message may include the name, age, sex, department and doctor of the patient and other registration form business data.
It should be noted that, the process of storing the first message and the message queue identifier in the first database and performing the parsing operation on the first message may be understood as an asynchronous process, that is, when the first message is parsed, the process of storing the subsequent first message and the parsing operation may be continuously performed without waiting for the parsing result of the first message.
In addition, in this embodiment of the application, after the first message is analyzed, whether the first message analysis is successful may be further recorded in a manner of recording a log, and if the first message analysis fails, an analysis processing operation may be further performed on the first message again, for example, the first message may be obtained again from the first database according to an identifier of the first message recorded in the log, and the analysis processing operation may be performed on the first message.
Optionally, in this embodiment of the application, a recorded log list may be as shown in fig. 2, where the log may include a message Identifier (ID) of the first message, a type of the first message, a parsing time and a processing result of the first message, and the like. In addition, the message details of the first message with the parsing exception may be as shown in fig. 3, for example, the message details of the first message with the parsing exception may include a message identifier, a message type, and a message content of the first message with the parsing exception.
S103: and storing the first service data to a second database.
Specifically, in order to ensure that the message consumer can acquire or check the first service data at any time, the embodiment of the present application may further store the first service data in the second database. When the first service data is stored, the first service data of different types can be stored in corresponding target data tables of the second database according to the type of the first service data.
For example, in an actual scenario, when the first message is a medical statement message in a medical visit, the first message is analyzed to obtain first service data such as registration fee data, western medicine fee data, traditional Chinese medicine fee data and inspection fee data; for each type of first service data, the first service data may be stored in a corresponding target data table in the second database, such as a registration fee data table, a western medicine fee data table, a traditional Chinese medicine fee data table, and the like.
In addition, in this embodiment of the application, after the first service data is stored in the second database, whether the first service data is successfully stored may also be recorded in a manner of recording a log, and if the first service data is unsuccessfully stored, the first message may also be re-acquired from the first database according to an identifier of the first message corresponding to the first service data that is recorded in the log and failed to be stored, and the first message is analyzed and processed, and the analyzed first service data is stored in the second database.
The second database may be the same as or different from the first database. For example, to save the storage resource overhead, the first message and the first service data may be stored in different data tables of the same database; in addition, for the purpose of classified checking, the first message and the first service data may be stored in the first database and the second database, respectively. The present embodiment is not limited to this.
The embodiment of the application can be realized by acquiring a first message and a message queue identifier; storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message; storing the first service data to a second database; therefore, distributed processing of acquiring the original message (namely the first message), storing the original message, analyzing the original message and storing the analysis result in the message data processing process is realized, and the coupling among all sub-processes of the distributed processing is low, so that all sub-processes can be started, stopped and repeatedly executed at any time, and the flexibility and controllability of the message processing process are improved; and by storing the original message and the analysis result of the original message, the trace-following of each sub-process of message processing is realized, and the original message is conveniently checked at any time when the message data processing process is abnormal.
In another embodiment of the present application, the performing a parsing operation on the first message includes:
determining a first data analysis rule corresponding to the message queue according to the message queue identification;
and analyzing the first message according to the first data analysis rule to obtain second service data.
Specifically, in this embodiment of the present application, after the first message is stored, the first message may be analyzed.
In a practical scenario, the first message may be obtained from a message queue of any one of the plurality of information systems of the integrated platform, where the generated message may correspond to different data standards for each message queue, for example, the messages in message queue a and message queue B correspond to different data types and data formats, respectively, and so on.
Therefore, the data parsing rules of the messages are different for the messages of different data standards in different message queues. In the embodiment of the present application, a corresponding relationship between each message queue and a data parsing rule of a message in the message queue may be predetermined, for example, the data parsing rule of the message in the message queue a is a data parsing rule 1; the data analysis rule of the message in the message queue B is a data analysis rule 2; the data parsing rule of the message in the message queue C is data parsing rule 3, and so on.
Optionally, in this embodiment of the application, a data parsing rule corresponding to each message queue may be configured through a data parsing rule configuration interface shown in fig. 4, for example, and corresponding data parsing rules are configured for different message queues, so that flexibility of message parsing is improved.
In this way, after the first message is acquired, the data parsing rule corresponding to the message queue can be determined according to the message queue identifier, and for convenience of description, the data parsing rule can be called as a first data parsing rule; further, according to a first data parsing rule, parsing processing operation may be performed on the first message to obtain second service data corresponding to the first message.
Optionally, in this embodiment of the application, the first message may include a first message of an Extensible Markup Language (XML), and when the first message is parsed, the second service data may be extracted from the first message item by using an XML Path Language (XPATH) wildcard.
Wherein the second service data may include the structured service data extracted from the first message.
It should be noted that, in an actual scenario, since a data standard required to be consumed by a message consumer may be different from a data standard of the parsed second service data, in another embodiment of the present application, the acquiring the first service data corresponding to the first message may include:
and performing data conversion on the second service data to obtain the first service data, wherein the first service data corresponds to a first data standard, and the second service data corresponds to a second data standard.
Specifically, according to a first data standard of first service data that a message consumer needs to consume, the embodiment of the present application may further convert second service data to obtain the first service data.
It can be understood that the first service data and the second service data are structured service data obtained by parsing the first message, and the difference is that data standards corresponding to the first service data and the second service data are different.
Determining a first data analysis rule corresponding to the message queue according to the message queue identifier; and analyzing the first message according to the first data analysis rule to obtain second service data, analyzing the first message according to the data analysis rule corresponding to the message queue to obtain corresponding second service data, and performing data conversion on the second service data obtained by analysis to obtain first service data of a first data standard corresponding to a message consumer.
In another embodiment of the present application, the storing the first service data to a second database includes:
determining the data type of the first service data;
and storing the first service data to a target data table corresponding to the data type in the second database.
Specifically, when storing the first service data, the first service data may be classified according to service requirements of message consumers, and the first service data of different data types are respectively stored in corresponding target data tables of the second database.
For example, in an actual scenario, when the first message is a medical statement message in a medical visit, the first message is analyzed to obtain first service data such as registration fee data, western medicine fee data, traditional Chinese medicine fee data, examination fee data, and the like. According to the business requirements of message consumers, registration fee data, western medicine fee data, traditional Chinese medicine fee data, inspection fee data and the like can be respectively stored into a corresponding registration fee data table, a corresponding western medicine fee data table, a corresponding traditional Chinese medicine fee data table and a corresponding inspection fee data table; or, according to the service requirement of the message consumer, the western medicine charge data and the traditional Chinese medicine charge data can be merged and then stored in a charge data table and the like.
The data type of the first service data is determined; and storing the first service data into a corresponding target data table of the second database according to the data type of the first service data, so that the classified storage of the first service data is realized, and a message consumer can look up the first service data of a corresponding type.
In another embodiment of the present application, the method further comprises:
and if first indication information indicating that the first message is analyzed abnormally is received, or second indication information indicating that the first service data is stored abnormally when being stored in a corresponding target data table of the second database is received, adding an abnormal identifier for the first message.
Specifically, in the embodiment of the present application, after the first message is analyzed and the first service data is stored in the second database, whether the first message analysis is successful and whether the first service data is successfully stored may also be recorded in a log recording manner.
If first indication information indicating that the first message is analyzed abnormally is received, or second indication information indicating that the first service data is stored abnormally when being stored in a corresponding target data table of the second database is received, an abnormal identifier can be added to the first message, so that analysis processing operation can be re-executed on the first message which is analyzed abnormally and/or the first message corresponding to the first service data which is abnormal when the first service data is stored in the second database.
For example, in another embodiment of the present application, when a preset condition is satisfied, the parsing processing operation may be executed again on the first message that is parsed for the exception and/or the first message corresponding to the first service data that is parsed for the exception when the first service data is stored in the second database.
Optionally, in some embodiments of the present application, the preset condition includes at least one of: the quantity of the first indication information and/or the second indication information reaches a preset value; and reaching the analysis processing operation time point of the preset time period.
For example, when the number of the first indication information and/or the second indication information reaches a preset value, that is, the number of the first messages corresponding to the first service data with abnormal analysis and/or the first service data stored to the second database reaches a preset value, for example, when the number of the first messages corresponding to the first service data with abnormal analysis and/or the first service data stored to the second database reaches 100, the analysis processing operation is performed again on the first messages with abnormal analysis and/or the first messages corresponding to the first service data with abnormal first service data stored to the second database.
In addition, the analysis processing operation time point of the preset time period may also be reached, for example, the analysis processing operation is executed again on the first message with abnormal analysis and/or the first message corresponding to the first service data with abnormal first service data stored in the second database every ten minutes.
Specifically, when the analysis processing operation is re-executed on the first message with the abnormal analysis and/or the first message corresponding to the first service data with the abnormal first service data stored in the second database, the first message may be re-acquired from the first database according to the identifier of the first message with the failed analysis recorded in the log, and the analysis processing operation may be performed on the first message.
The embodiment of the application can be realized by acquiring a first message and a message queue identifier; storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message; storing the first service data to a second database; therefore, distributed processing of acquiring the original message (namely the first message), storing the original message, analyzing the original message and storing the analysis result in the message data processing process is realized, and the coupling among all sub-processes of the distributed processing is low, so that all sub-processes can be started, stopped and repeatedly executed at any time, and the flexibility and controllability of the message processing process are improved; and by storing the original message and the analysis result of the original message, the trace-following of each sub-process of message processing is realized, and the original message is conveniently checked at any time when the message data processing process is abnormal.
The following describes the embodiments of the present application in a medical settlement scenario as an example, with reference to fig. 5 as a whole:
in a medical settlement scene, if a message consumer needs to consume a medical settlement statement message, the message consumer can firstly subscribe a message queue of the medical settlement statement message, establish a connection channel through a server or a medical settlement information system corresponding to the message queue, and monitor the medical settlement statement message generated by the medical settlement information system.
When the generation of the medical statement message is monitored, acquiring the medical statement message, wherein the medical statement message can be monitored in real time, and when each medical statement message is monitored to be generated, acquiring the medical statement message; the monitored medical statement of account message may also be periodically retrieved. For example, acquiring the monitored medical statement message is performed every 5 minutes, or when a preset number of medical statement messages are monitored (for example, 100 medical statement messages are monitored), the monitored medical statement messages are acquired.
After the medical statement of account message is acquired, the medical statement of account message and the message queue identification thereof can be stored in the first database in order to ensure that the originally acquired medical statement of account message can be checked at any time if an error or an abnormality occurs in the subsequent medical statement of account message processing process. If the medical statement of account message cannot be successfully stored in the first database, the medical statement of account message which is not successfully stored can be stored in the hard disk.
In addition, a data analysis rule for analyzing the medical statement message is determined according to the message queue identifier of the medical statement message and the corresponding relationship between the predetermined message queue identifier and the data analysis rule, and the medical statement message is analyzed according to the determined data analysis rule, so that service data in the medical statement message, for example, service data such as registration fee data, western medicine fee data, traditional Chinese medicine fee data, inspection fee data, and the like, that is, second service data in the embodiment of the present application is obtained. In addition, the second service data can be converted according to the data standard corresponding to the service data required to be consumed by the message consumer, so that the first service data is obtained.
Furthermore, the first service data can be classified according to the service requirements of the message consumers, and the classified first service data is stored in a corresponding data table of the second database.
Wherein, when the medical settlement list message is not successfully analyzed or the analyzed first service data is not successfully stored in the second database, the identifier of the corresponding medical statement of account message may be recorded, and when a preset condition is satisfied, for example, when the number of medical statement of account messages corresponding to the abnormal medical statement of account message and/or the abnormal first service data stored in the second database reaches 100, or every ten minutes, the analysis processing operation is executed again on the medical statement message which is analyzed abnormally and/or the medical statement message which is stored in the second database and corresponds to the abnormal first business data, the medical statement of account message can be obtained from the first database again according to the identifier of the medical statement of account message with failed analysis of the log record, and the analysis processing operation is carried out on the medical statement of account message.
The embodiment of the application can be realized by acquiring a first message and a message queue identifier; storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message; storing the first service data to a second database; therefore, distributed processing of acquiring the original message (namely the first message), storing the original message, analyzing the original message and storing the analysis result in the message data processing process is realized, and the coupling among all sub-processes of the distributed processing is low, so that all sub-processes can be started, stopped and repeatedly executed at any time, and the flexibility and controllability of the message processing process are improved; and by storing the original message and the analysis result of the original message, the trace-following of each sub-process of message processing is realized, and the original message is conveniently checked at any time when the message data processing process is abnormal.
An embodiment of the present application provides a message data processing apparatus, and as shown in fig. 6, the message data processing apparatus 60 may include: an acquisition module 601, a first processing module 602, and a storage module 603, wherein,
an obtaining module 601, configured to obtain a first message and a message queue identifier; the message queue identification comprises an identification of a message queue of the first message in a server;
a first processing module 602, configured to store the first message and the message queue identifier in a first database, and perform an analysis processing operation on the first message to obtain first service data corresponding to the first message;
the storage module 603 is configured to store the first service data in a second database.
In another embodiment of the present application, the first processing module 602 is specifically configured to determine, according to the message queue identifier, a first data parsing rule corresponding to the message queue;
and analyzing the first message according to the first data analysis rule to obtain second service data.
In another embodiment of the present application, the first processing module 602 is specifically configured to perform data conversion on the second service data to obtain the first service data, where the first service data corresponds to a first data standard, and the second service data corresponds to a second data standard.
In another embodiment of the present application, the storage module 603 is specifically configured to,
determining the data type of the first service data;
and storing the first service data to a target data table corresponding to the data type in the second database.
In another embodiment of the present application, the apparatus further includes a recording module, configured to add an exception identifier to the first message when receiving first indication information indicating that the first message is abnormal in parsing or receiving second indication information indicating that the first service data is abnormal when being stored in a corresponding target data table of the second database.
In another embodiment of the present application, the apparatus further includes a second processing module, which re-executes parsing of the first message when a preset condition is satisfied.
In another embodiment of the present application, the preset condition includes at least one of:
the quantity of the first indication information and/or the second indication information reaches a preset value;
and reaching the analysis processing operation time point of the preset time period.
In another embodiment of the present application, the message queue identification includes at least one of:
the identifier of the server corresponding to the message queue;
the identifier of the manager corresponding to the message queue;
and the identifier of the channel corresponding to the message queue.
The message data processing apparatus of this embodiment can execute the message data processing method shown in the above embodiments of this application, and the implementation principles thereof are similar, and are not described herein again.
The embodiment of the application can be realized by acquiring a first message and a message queue identifier; storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message; storing the first service data to a second database; therefore, distributed processing of acquiring the original message (namely the first message), storing the original message, analyzing the original message and storing the analysis result in the message data processing process is realized, and the coupling among all sub-processes of the distributed processing is low, so that all sub-processes can be started, stopped and repeatedly executed at any time, and the flexibility and controllability of the message processing process are improved; and by storing the original message and the analysis result of the original message, the trace-following of each sub-process of message processing is realized, and the original message is conveniently checked at any time when the message data processing process is abnormal.
An embodiment of the present application provides an electronic device, including: a memory and a processor; at least one program stored in the memory for execution by the processor, which when executed by the processor, implements: the embodiment of the application can be realized by acquiring a first message and a message queue identifier; storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message; storing the first service data to a second database; therefore, distributed processing of acquiring the original message (namely the first message), storing the original message, analyzing the original message and storing the analysis result in the message data processing process is realized, and the coupling among all sub-processes of the distributed processing is low, so that all sub-processes can be started, stopped and repeatedly executed at any time, and the flexibility and controllability of the message processing process are improved; and by storing the original message and the analysis result of the original message, the trace-following of each sub-process of message processing is realized, and the original message is conveniently checked at any time when the message data processing process is abnormal.
In an alternative embodiment, an electronic device is provided, as shown in fig. 7, the electronic device 4000 shown in fig. 7 comprising: a processor 4001 and a memory 4003. Processor 4001 is coupled to memory 4003, such as via bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, and the transceiver 4004 may be used for data interaction between the electronic device and other electronic devices, such as transmission of data and/or reception of data. In addition, the transceiver 4004 is not limited to one in practical applications, and the structure of the electronic device 4000 is not limited to the embodiment of the present application.
The Processor 4001 may be a CPU (Central Processing Unit), a general-purpose Processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other Programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor 4001 may also be a combination that performs a computational function, including, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, or the like.
The Memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical Disc storage, optical Disc storage (including Compact Disc, laser Disc, optical Disc, digital versatile Disc, blu-ray Disc, etc.), a magnetic Disc storage medium or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to these.
The memory 4003 is used for storing application program codes (computer programs) for executing the present scheme, and is controlled by the processor 4001 to execute. Processor 4001 is configured to execute application code stored in memory 4003 to implement what is shown in the foregoing method embodiments.
Among them, electronic devices include but are not limited to: mobile phones, notebook computers, multimedia players, desktop computers, and the like.
The present application provides a computer-readable storage medium, on which a computer program is stored, which, when running on a computer, enables the computer to execute the corresponding content in the foregoing method embodiments.
The embodiment of the application can be realized by acquiring a first message and a message queue identifier; storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message; storing the first service data to a second database; therefore, distributed processing of acquiring the original message (namely the first message), storing the original message, analyzing the original message and storing the analysis result in the message data processing process is realized, and the coupling among all sub-processes of the distributed processing is low, so that all sub-processes can be started, stopped and repeatedly executed at any time, and the flexibility and controllability of the message processing process are improved; and by storing the original message and the analysis result of the original message, the trace-following of each sub-process of message processing is realized, and the original message is conveniently checked at any time when the message data processing process is abnormal.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The foregoing is only a partial embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (11)
1. A message data processing method, comprising:
acquiring a first message and a message queue identifier; the message queue identification comprises an identification of a message queue of the first message in a server;
storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message;
and storing the first service data to a second database.
2. The message processing method according to claim 1, wherein the performing a parsing operation on the first message comprises:
determining a first data analysis rule corresponding to the message queue according to the message queue identification;
and analyzing the first message according to the first data analysis rule to obtain second service data.
3. The message processing method according to claim 2, wherein the acquiring the first service data corresponding to the first message comprises:
performing data conversion on the second service data to obtain the first service data; the first service data corresponds to a first data standard, and the second service data corresponds to a second data standard.
4. The message processing method of claim 1, wherein the storing the first service data to a second database comprises:
determining the data type of the first service data;
and storing the first service data to a target data table corresponding to the data type in the second database.
5. The message processing method of claim 1, wherein the method further comprises:
and if first indication information indicating that the first message is analyzed abnormally is received, or second indication information indicating that the first service data is stored abnormally when being stored in a corresponding target data table of the second database is received, adding an abnormal identifier for the first message.
6. The message processing method according to claim 5, wherein after adding the anomaly identification to the first message, the method further comprises:
and when a preset condition is met, re-executing the analysis of the first message.
7. The message processing method according to claim 6, wherein the preset condition comprises at least one of:
the quantity of the first indication information and/or the second indication information reaches a preset value;
and reaching the analysis processing operation time point of the preset time period.
8. The message processing method according to any of claims 1 to 7,
the message queue identification comprises at least one of:
the identifier of the server corresponding to the message queue;
the identifier of the manager corresponding to the message queue;
and the identifier of the channel corresponding to the message queue.
9. A message data processing apparatus, comprising:
the acquisition module is used for acquiring a first message and a message queue identifier; the message queue identification comprises an identification of a message queue of the first message in a server;
the first processing module is used for storing the first message and the message queue identifier to a first database, and performing analysis processing operation on the first message to acquire first service data corresponding to the first message;
and the storage module is used for storing the first service data to a second database.
10. An electronic device, characterized in that the electronic device comprises:
one or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to: executing the message data processing method according to any of claims 1 to 8.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the message data processing method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111229412.9A CN113961632A (en) | 2021-10-21 | 2021-10-21 | Message data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111229412.9A CN113961632A (en) | 2021-10-21 | 2021-10-21 | Message data processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113961632A true CN113961632A (en) | 2022-01-21 |
Family
ID=79465979
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111229412.9A Pending CN113961632A (en) | 2021-10-21 | 2021-10-21 | Message data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113961632A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024113644A1 (en) * | 2022-11-30 | 2024-06-06 | 三一海洋重工有限公司 | Data storage method, automation system, and mechanical equipment |
-
2021
- 2021-10-21 CN CN202111229412.9A patent/CN113961632A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024113644A1 (en) * | 2022-11-30 | 2024-06-06 | 三一海洋重工有限公司 | Data storage method, automation system, and mechanical equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111245900B (en) | Distributed message sending processing system and processing method thereof | |
| CN108959279B (en) | Data processing method, data processing device, readable medium and electronic equipment | |
| CN110838071A (en) | Policy data processing method and device and server | |
| CN107133233B (en) | Processing method and device for configuration data query | |
| CN112636957A (en) | Early warning method and device based on log, server and storage medium | |
| CN112445866A (en) | Data processing method and device, computer readable medium and electronic equipment | |
| CN112084179B (en) | Data processing method, device, equipment and storage medium | |
| CN112631924A (en) | Automatic testing method and device, computer equipment and storage medium | |
| CN111367531B (en) | Code processing method and device | |
| CN111400170A (en) | Data permission testing method and device | |
| CN112445860B (en) | Method and device for processing distributed transaction | |
| CN111966653A (en) | Data processing method, device, server and storage medium for micro-service call link | |
| CN115328759A (en) | Form verification method and device | |
| CN113111078B (en) | Resource data processing method and device, computer equipment and storage medium | |
| CN110046086B (en) | Expected data generation method and device for test and electronic equipment | |
| CN113961632A (en) | Message data processing method and device and electronic equipment | |
| US10372762B1 (en) | Systems and methods for improved transactional mainframes | |
| CN108429636B (en) | Method and device for positioning abnormal system and electronic equipment | |
| CN112765280B (en) | Block data storage method and device, computer readable medium and electronic equipment | |
| CN114153703A (en) | Micro-service exception positioning method and device, electronic equipment and program product | |
| CN113011848A (en) | Method, device, equipment and storage medium for accessing approval service | |
| CN111427878B (en) | Data monitoring alarm method, device, server and storage medium | |
| CN113053531B (en) | Medical data processing method, medical data processing device, computer readable storage medium and equipment | |
| CN114218313A (en) | Data management method, device, electronic equipment, storage medium and product | |
| CN111643892A (en) | Message processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |