CN110825538A - MQ-based method for dynamically interacting data by self-defining data types - Google Patents
MQ-based method for dynamically interacting data by self-defining data types Download PDFInfo
- Publication number
- CN110825538A CN110825538A CN201911065780.7A CN201911065780A CN110825538A CN 110825538 A CN110825538 A CN 110825538A CN 201911065780 A CN201911065780 A CN 201911065780A CN 110825538 A CN110825538 A CN 110825538A
- Authority
- CN
- China
- Prior art keywords
- data
- equipment
- processing
- communication service
- types
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The invention relates to a method for dynamically carrying out data interaction based on self-defined data types of MQ. The method comprises the following steps: 1. adding new custom data types needing data interaction; 2. the equipment accesses the system and sends data to the system; 3. after receiving the data sent by the equipment, the system checks the correctness of the data; searching the MQ producer program through the equipment type code and the communication service type code; writing the received data to the MQ; 4. after receiving the data, the MQ sends the data to an MQ consumer program for data processing; 5. after receiving specific equipment data, the MQ analyzes and processes the data according to an agreed format; and writing the processing result to a specified data storage structure, and feeding back the data service. The problem caused by multiple data types and large data quantity of data exchange between the App and the server is solved; the method can be used for data interaction between the terminal device and the server.
Description
Technical Field
The invention relates to the technical field of data transmission, in particular to a method for dynamically carrying out data interaction based on self-defined data types of MQ.
Background
With the development of the internet and terminal equipment, more and more terminal equipment need to be accessed into the network to realize real-time information communication; for example, data exchange between apps and a server side on a mobile phone of each person, automatic uploading of data of various hardware devices such as an electric meter and a water meter, heartbeat detection of whether monitoring devices are on line or not, and the like. The terminal devices have a large amount of data, and each device has a plurality of data request types; for example, the water meter has data types such as sign-in, heartbeat, data reporting and sign-off, and the data types ensure that the equipment automatically contacts with the server, performs data interaction and the like; the whole interactive process has multiple data types and large data volume.
With the development of the technology, various devices have the possibility of continuously expanding data interaction types, such as a water meter, and the data interaction types of shutdown, startup and the like can be increased subsequently. According to the traditional method, after the added types need to be modified and debugged on the system, the deployment and the updating of the system are carried out; the range involved in the process is large, and in the case that the process of updating and deploying involves shutdown, the data reporting of other devices is affected. The influence on equipment which particularly requires data to run in real time is great; moreover, the equipment is continuously expanded, and if the sub-operation is required each time, the operation effect of the whole system of the equipment is greatly influenced; there is a need for a fast response, downtime-free response method that satisfies the growing data types and solves the problem of handling large amounts of data.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a method for dynamically carrying out data interaction based on self-defined data types of MQ; the data interaction method which is efficient and can expand the data interaction type is realized.
The technical scheme for solving the technical problems is as follows:
the method comprises the following steps:
step 1, adding new custom data types needing data interaction;
step 2, the equipment accesses the system and sends data to the system;
step 3, after the system receives the data sent by the equipment, checking the correctness of the data; searching the MQ producer program through the equipment type code and the communication service type code; writing the received data to the MQ;
step 4, after receiving the data, the MQ sends the data to an MQ consumer program for data processing;
step 5, after receiving specific equipment data, the MQ analyzes and processes the data according to an agreed format; and writing the processing result to a specified data storage structure, and feeding back the data service.
The step 1 specifically comprises the following steps:
1) creating contents of data interaction information of various communication service types, including a service data head, service data contents and a service data check code;
2) the service data head comprises a service type code and the total length of the service data content;
the service type code is expressed by two bytes, the first byte represents the type of the equipment, and the second byte represents various communication service types of the equipment; in addition, two bytes are added for representing the specific equipment code and one content with the length of 4 bytes is used for representing the length of the whole data communication information content;
3) designing the specific number of each byte and the meaning represented by the byte content according to the requirement of the communication service type of the equipment by the service data content, and performing specific analysis and data processing on the subsequent MQ consumer by the number of each byte and the byte content;
4) the service data check code is arranged behind the service data content and is used for checking the integrity of the whole data interaction information content; the checking method is that all the previous byte contents are checked according to the CRC method, and the CRC16 checking result is stored;
5) after the design is finished, inputting equipment type codes, communication service type codes, equipment codes, total length of data information and processing names of corresponding message queues MQ on the information.
The method constructs a message queue MQ available with high load after step 1, and distributes data to each processing system based on the MQ when various communication service type codes of various device types are received.
The step 3 specifically comprises the following steps:
1) after receiving the information, the UDP frame acquires the type of equipment, the communication service type of the equipment, the equipment code, the length of the whole data communication information content and the check result of the CRC16 of the whole information content based on the service data header;
2) checking the whole information content according to CRC16, and comparing the result with the received CRC16 result; if the comparison is wrong, discarding the data and not processing the data; if the comparison is consistent, acquiring the processing name of the corresponding message queue MQ according to the type of the acquired equipment and the communication service type of the equipment;
3) and based on the acquired processing name of the corresponding message queue MQ, directly writing the received data into the corresponding MQ queue.
The MQ-based consumer program of step 4 can be distributed on one or more servers; or implement multiple data storage means.
If a device or a communication service type of the device needs to be added, corresponding device type codes, communication service type codes, device codes, the total length of data information and the processing name of a corresponding message queue MQ are added to configuration information, an MQ producer processing program after data report is updated, a new processing program of the corresponding message queue MQ is written and registered to the MQ.
If new equipment exists or new equipment communication types are increased, new business of business data interaction can be rapidly increased only by configuring and realizing corresponding MQ consumer programs; the process does not need system halt, redeployment or modification of the original program; meanwhile, the MQ consumer program can be deployed in a distributed manner, and distributed operation of processing and storing specific data is realized; the load of the whole system is greatly dispersed, and an efficient and available expansion method is provided for a large amount of equipment data processing and the subsequent large-scale new service type increase.
Drawings
The invention is further described below with reference to the accompanying drawings:
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
Referring to fig. 1, a method for data interaction dynamically based on MQ (message queue) customizable data types is disclosed. The method of the invention comprises the following steps:
step 1: based on the types of various terminal devices and the communication service types of the devices, the data interaction length, the length between bytes and the representation method of various communication service types are established, and the processing name of the corresponding message queue MQ is configured.
The foregoing steps are specifically as follows:
1) and creating the content of the data interaction information of various communication service types, wherein the content comprises a service data head, service data content and a service data check code.
2) The service data header comprises a service type code and the total length of the service data content, the service type code is expressed by using fixed bytes, the specific number of the bytes is designed according to the service data, and the specific number of the bytes can be generally designed to be expressed by two bytes; the first byte represents the type of the equipment, and the second byte represents various communication service types of the equipment; two bytes are added simultaneously for representing specific equipment codes; additionally, a 4-byte length content is included to indicate the length of the entire data communication content; the total traffic data header is 8 bytes.
3) The method comprises the steps that business data content is designed according to the requirements of various communication business types of various devices and the design convention, the specific byte quantity and the meaning expressed by the byte content are designed, and the byte quantity and the byte content are subjected to specific analysis and data processing on subsequent MQ consumers.
4) The service data Check code is behind the service data content and is used for checking the integrity of the whole data interaction information content, the checking method is that all the preceding byte contents are checked according to a Cyclic Redundancy Check (CRC) method, and the checking result of the CRC16 is stored at the rearmost of the whole service data content.
5) After the data interaction length, the length between bytes and the representation method of each communication service type are designed, the equipment type code, the communication service type code, the equipment code, the total length of the data information and the processing name of the corresponding message queue MQ need to be input into the information, and then a new self-defined data type is added for data interaction; the device type code, the communication service type code and the processing name of the corresponding message queue MQ are unique contents.
Step 2: and constructing a message queue MQ available with high load, wherein the MQ can also rapidly expand or reduce the maximum load capacity at any time according to the traffic.
The message queue MQ is an important component for carrying out quick response and service decoupling on the whole system; and when various communication service type codes of various device types are received, distributing the data to each processing system based on the MQ, wherein the distribution rule is carried out according to the configuration of the step 1, and the real-time rule is kept updated. The message queue MQ needs to adjust the load according to the traffic volume of the system.
And step 3: and constructing a UDP framework capable of receiving the device data with high load, wherein the UDP framework is used for receiving the communication service type data sent by various types of devices with high concurrency and the received various types of communication service type data are used for the input of a producer of the MQ.
The detailed steps are as follows:
1) after receiving the information, the UDP framework obtains, based on the service data header, the type of the device, the communication service type of the device, the device code, the length of the entire data communication information content, and the check result of the CRC16 of the entire information content;
2) checking the whole information content according to CRC16, and comparing the result with the received CRC16 result; if the comparison is wrong, discarding the data and not processing the data; and if the comparison is consistent, acquiring the processing name of the corresponding message queue MQ according to the type of the acquired equipment and the communication service type of the equipment.
3) And directly writing the received data into the corresponding MQ queue based on the acquired processing name of the corresponding message queue MQ. The various corresponding message queues, MQ, are initialized at system startup and updated in real-time as new rules are created.
And 4, step 4: based on various communication service types in the step 1, the consumers based on the MQ establish service methods for processing various data, and based on the processing names of the consistent message queues MQ, the appointed service data is processed in a unified way; the service type processing method can be distributed on a plurality of servers for processing.
In this step, the MQ-based consumer program may be distributed on one or more servers for processing, and may also implement multiple data storage modes to reduce the data storage pressure of the entire system.
And 5: based on the configured communication service types of the devices and the processing names corresponding to the MQ, after the system receives the device data report, the UDP frame receives the device data report, the data verification is completed, and the device types and the communication service types of the devices are obtained; acquiring a corresponding MQ producer processing program according to convention, writing data into a specified MQ generation producer, and submitting the data to the MQ; and the MQ consumer acquires the data from the MQ, and stores or processes the acquired data according to the information represented by each byte in the configured data, thereby completing the process of data processing based on the MQ.
Step 6: if a device or a communication service type of the device needs to be added, the data interaction process of a new device communication service type can be realized only by adding corresponding device type codes, communication service type codes, device codes, the total length of data information and the processing name of a corresponding message queue MQ on configuration information, updating an MQ producer processing program after data reporting, writing a new processing program of the corresponding message queue MQ and registering the new processing program to the MQ. The newly added message queue MQ processing program does not depend on the original existing service and can be independently operated, so that the rapid and efficient expansion of the operation and the load of the whole system is realized, the stable operation of the whole data system is facilitated, and an efficient processing method is provided for large-scale data reporting.
Claims (9)
1. A method for dynamically performing data interaction based on self-defined data types of MQ is characterized in that: the method comprises the following steps:
step 1, adding new custom data types needing data interaction;
step 2, the equipment accesses the system and sends data to the system;
step 3, after the system receives the data sent by the equipment, checking the correctness of the data; searching the MQ producer program through the equipment type code and the communication service type code; writing the received data to the MQ;
step 4, after receiving the data, the MQ sends the data to an MQ consumer program for data processing;
step 5, after receiving specific equipment data, the MQ analyzes and processes the data according to an agreed format; and writing the processing result to a specified data storage structure, and feeding back the data service.
2. The method of claim 1, wherein: the step 1 specifically comprises the following steps:
1) creating contents of data interaction information of various communication service types, including a service data head, service data contents and a service data check code;
2) the service data head comprises a service type code and the total length of the service data content;
the service type code is expressed by two bytes, the first byte represents the type of the equipment, and the second byte represents various communication service types of the equipment; in addition, two bytes are added for representing the specific equipment code and one content with the length of 4 bytes is used for representing the length of the whole data communication information content;
3) designing the specific number of each byte and the meaning represented by the byte content according to the requirement of the communication service type of the equipment by the service data content, and performing specific analysis and data processing on the subsequent MQ consumer by the number of each byte and the byte content;
4) the service data check code is arranged behind the service data content and is used for checking the integrity of the whole data interaction information content; the checking method is that all the previous byte contents are checked according to the CRC method, and the CRC16 checking result is stored;
5) after the design is finished, inputting equipment type codes, communication service type codes, equipment codes, total length of data information and processing names of corresponding message queues MQ on the information.
3. The method of claim 1, wherein: the method constructs a message queue MQ available with high load after step 1, and distributes data to each processing system based on the MQ when various communication service type codes of various device types are received.
4. The method of claim 2, wherein: the method constructs a message queue MQ available with high load after step 1, and distributes data to each processing system based on the MQ when various communication service type codes of various device types are received.
5. The method according to claim 2 or 4, characterized in that: the step 3 specifically comprises the following steps:
1) after receiving the information, the UDP frame acquires the type of equipment, the communication service type of the equipment, the equipment code, the length of the whole data communication information content and the check result of the CRC16 of the whole information content based on the service data header;
2) checking the whole information content according to CRC16, and comparing the result with the received CRC16 result; if the comparison is wrong, discarding the data and not processing the data; if the comparison is consistent, acquiring the processing name of the corresponding message queue MQ according to the type of the acquired equipment and the communication service type of the equipment;
3) and based on the acquired processing name of the corresponding message queue MQ, directly writing the received data into the corresponding MQ queue.
6. The method according to any one of claims 1 to 4, characterized in that: the MQ-based consumer program of step 4 can be distributed on one or more servers; or implement multiple data storage means.
7. The method of claim 5, wherein: the MQ-based consumer program of step 4 can be distributed on one or more servers; or implement multiple data storage means.
8. The method according to any one of claims 1 to 4, characterized in that: if a device or a communication service type of the device needs to be added, corresponding device type codes, communication service type codes, device codes, the total length of data information and the processing name of a corresponding message queue MQ are added to configuration information, an MQ producer processing program after data report is updated, a new processing program of the corresponding message queue MQ is written and registered to the MQ.
9. The method of claim 7, wherein: if a device or a communication service type of the device needs to be added, corresponding device type codes, communication service type codes, device codes, the total length of data information and the processing name of a corresponding message queue MQ are added to configuration information, an MQ producer processing program after data report is updated, a new processing program of the corresponding message queue MQ is written and registered to the MQ.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911065780.7A CN110825538A (en) | 2019-11-04 | 2019-11-04 | MQ-based method for dynamically interacting data by self-defining data types |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911065780.7A CN110825538A (en) | 2019-11-04 | 2019-11-04 | MQ-based method for dynamically interacting data by self-defining data types |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110825538A true CN110825538A (en) | 2020-02-21 |
Family
ID=69552586
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911065780.7A Pending CN110825538A (en) | 2019-11-04 | 2019-11-04 | MQ-based method for dynamically interacting data by self-defining data types |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110825538A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113726845A (en) * | 2021-07-14 | 2021-11-30 | 深圳市有为信息技术发展有限公司 | Data transmission method and device of vehicle-mounted terminal, vehicle-mounted terminal and commercial vehicle |
| CN113923248A (en) * | 2021-09-30 | 2022-01-11 | 深圳海智创科技有限公司 | Universal MQ interaction method, system, device and storage medium |
-
2019
- 2019-11-04 CN CN201911065780.7A patent/CN110825538A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113726845A (en) * | 2021-07-14 | 2021-11-30 | 深圳市有为信息技术发展有限公司 | Data transmission method and device of vehicle-mounted terminal, vehicle-mounted terminal and commercial vehicle |
| CN113923248A (en) * | 2021-09-30 | 2022-01-11 | 深圳海智创科技有限公司 | Universal MQ interaction method, system, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108897691A (en) | Data processing method, device, server and medium based on interface analog service | |
| US10884826B2 (en) | System and method for establishing common request processing | |
| CN105335293A (en) | Automatic testing system and method based on parallel ports | |
| US9639444B2 (en) | Architecture for end-to-end testing of long-running, multi-stage asynchronous data processing services | |
| CN111683066B (en) | Heterogeneous system integration method, heterogeneous system integration device, computer equipment and storage medium | |
| CN108241576B (en) | Interface testing method and system | |
| CN112988485A (en) | Simulation test method and device for power Internet of things equipment | |
| CN113760565A (en) | Data processing platform, data processing method, storage medium and electronic equipment | |
| CN110688828A (en) | File processing method and device, file processing system and computer equipment | |
| CN112732572A (en) | Service testing method, device and system, storage medium and electronic device | |
| CN110932918A (en) | Log data acquisition method and device and storage medium | |
| CN113806037A (en) | Service calling method and device, storage medium and electronic equipment | |
| CN113505520A (en) | Method, device and system for supporting heterogeneous federated learning | |
| CN114205274A (en) | Network equipment testing method and device | |
| CN110825538A (en) | MQ-based method for dynamically interacting data by self-defining data types | |
| CN113127108A (en) | Service request processing method and device, storage medium and electronic equipment | |
| CN112631884A (en) | Pressure measurement method and device based on data synchronization, computer equipment and storage medium | |
| CN107168844B (en) | Performance monitoring method and device | |
| CN110334147A (en) | A kind of method of data synchronization and device | |
| CN111752820A (en) | Pressure testing method of gPC interface, computer equipment and storage medium | |
| CN103561113B (en) | The generation method and device of Web Service interface | |
| CN103944779B (en) | A kind of WAP service features monitoring method and system | |
| CN106535221A (en) | Resource management method and device, and multimode soft base station unified webmaster | |
| CN110022323A (en) | A kind of method and system of the cross-terminal real-time, interactive based on WebSocket and Redux | |
| CN102594587B (en) | Embedded WEB debugging and testing maintenance method and debugging and testing maintenance system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |