CN114553962B - Mobile equipment data processing method and system based on message queue - Google Patents
Mobile equipment data processing method and system based on message queue Download PDFInfo
- Publication number
- CN114553962B CN114553962B CN202210155081.7A CN202210155081A CN114553962B CN 114553962 B CN114553962 B CN 114553962B CN 202210155081 A CN202210155081 A CN 202210155081A CN 114553962 B CN114553962 B CN 114553962B
- Authority
- CN
- China
- Prior art keywords
- data
- message
- message queue
- qualified
- messages
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0227—Filtering policies
- H04L63/0245—Filtering by information in the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
- H04L63/123—Applying verification of the received information received data contents, e.g. message integrity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
- H04L63/126—Applying verification of the received information the source of the received data
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
- Telephonic Communication Services (AREA)
Abstract
The invention discloses a mobile equipment data processing method and a mobile equipment data processing system based on a message queue, which relate to the technical field of data processing and solve the technical problem that the data processing efficiency is low because the data cannot be screened before being sent to the message queue in the prior art; the message queue service end screens message data produced by a producer to obtain qualified messages; after receiving message data at a message queue server, the invention carries out primary screening and secondary screening on the message data, and then inputs qualified messages into a message queue after statistical analysis on the qualified messages; according to the invention, invalid or illegal data can be screened and removed from the message data, so that the data processing efficiency is improved; when the data is screened, the message data is screened through primary screening and secondary screening, and the authority of a producer and the effectiveness of the message data are considered in the screening process, so that the safety of the data is ensured from the perspective of a user.
Description
Technical Field
The invention belongs to the field of data processing, relates to a mobile equipment data processing technology based on a message queue, and particularly relates to a mobile equipment data processing method and a mobile equipment data processing system based on the message queue.
Background
Message queues are important components in distributed systems, and can be understood as containers for holding messages during message transmission, and can be used in many scenarios where concurrency control is required. It is very important to apply the message queue to data processing of the mobile device.
In the prior art (patent invention with publication number CN 110737535A), data input is controlled by an external program, subsequent actions are triggered by slice data distributed by a message queue, and each operation server queries and processes the flow direction of control data; in the prior art, automatic distribution and parallel operation of data can be realized through sliced data, but before the data are sent to a message queue, the data cannot be screened, so that invalid data exist in the message queue, and the data processing efficiency is low. Therefore, a method and system for processing data of a mobile device based on a message queue are needed.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art; the invention provides a mobile equipment data processing method and a mobile equipment data processing system based on a message queue, which are used for solving the technical problem that the data processing efficiency is low because the data cannot be screened before being sent to the message queue in the prior art.
To achieve the above object, a first aspect of the present invention provides a message queue based mobile device data processing system, which includes a producer, a message queue server and a consumer;
the message queue service end screens message data produced by a producer to obtain qualified messages; and
and performing statistical analysis on the qualified messages every set period, and inputting the qualified messages into a message queue according to the analysis result.
Preferably, the producer produces message data and sends the message data to the message queue server; wherein the producer is in communication and/or electrical connection with the message queue server.
Preferably, the message queue server sends the qualified message to the consumer through a distribution mode; the distribution mode comprises a point-to-point mode and a publish/subscribe mode, and the consumer is in communication and/or electrical connection with the message queue server.
Preferably, the message queue server performs a screening on the message data according to the authority, including:
acquiring the authority of a producer and the data type corresponding to each authority; wherein the data category includes location data;
and identifying the data type of the message data, and when the message data is the same as the data type corresponding to the authority, marking the message data as intermediate data through one-time screening.
Preferably, the authority of the producer is set through the message queue server.
Preferably, the secondary screening of the message data by the message queue server according to the content includes:
identifying data content of the intermediate data;
carrying out anomaly detection on the data content, and marking the intermediate data passing the anomaly detection as qualified information; wherein the anomaly detection comprises an integrity detection.
Preferably, the obtaining, by the message queue server, the analysis result every set period includes:
acquiring the information quantity of qualified messages in a set period; wherein the set period comprises 1 second or 1 minute;
when the number of the messages exceeds a number threshold, numbering the qualified messages, and meanwhile, acquiring rejection duration according to the number of the messages; wherein the quantity threshold is an integer greater than 0 and is set empirically;
and controlling the message queue server to receive message data according to the rejection time length.
Preferably, the qualified messages are numbered in sequence according to the marking time of the qualified messages;
when more than one qualified message is marked at the same time, the qualified messages at the time are not numbered back.
Preferably, the rejection duration is obtained according to the information quantity and the duration of inputting each qualified message into the message queue;
and sequentially inputting the corresponding qualified messages into a message queue according to the numbers of the qualified messages.
The second aspect of the present invention provides a message queue-based mobile device data processing method, including:
producing message data through the producer and sending the message data to the message queue server;
the message queue service end screens message data produced by a producer to obtain qualified messages;
the message queue server side carries out statistical analysis on qualified messages every other set period and inputs the qualified messages into the message queue according to the analysis result;
and the message queue server side sends qualified messages to the consumers in a distribution mode.
Compared with the prior art, the invention has the beneficial effects that:
1. after receiving message data at a message queue server, the invention carries out primary screening and secondary screening on the message data, and then inputs qualified messages into a message queue after statistical analysis of the qualified messages; the invention can screen and remove invalid or illegal data by the message data, thereby improving the data processing efficiency.
2. When the data is screened, the message data is screened through primary screening and secondary screening, and the authority of a producer and the effectiveness of the message data are considered in the screening process, so that the safety of the data is ensured from the perspective of a user.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic diagram of the working steps of the present invention.
Detailed Description
The technical solutions of the present invention will be described below clearly and completely in conjunction with the embodiments, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the prior art (patent invention with publication number CN 110737535A), data input is controlled by an external program, subsequent actions are triggered by slice data distributed by a message queue, and each operation server queries and processes the flow direction of control data; in the prior art, automatic distribution and parallel operation of data can be realized through slicing data, but the data cannot be screened before being sent to a message queue, so that invalid data exists in the message queue, the space of the message queue is wasted, and the data processing efficiency is low; after receiving the message data at the message queue service end, the invention carries out primary screening and secondary screening on the message data, and then inputs the qualified message into the message queue after statistical analysis of the qualified message, so as to solve the problems.
Referring to fig. 1, a first embodiment of the present application provides a message queue-based mobile device data processing system, which includes a producer, a message queue server, and a consumer;
the message queue service end screens message data produced by a producer to obtain qualified messages; and
and performing statistical analysis on the qualified messages every set period, and inputting the qualified messages into a message queue according to the analysis result.
The mobile device comprises a movable server and an intelligent terminal, wherein the intelligent terminal comprises an intelligent mobile phone, a notebook computer and the like. The producer in the present application is used to produce message data, i.e. the sender of the message data; the consumer is used to receive data, i.e., the recipient of the data.
In the application, message data are produced by a producer and sent to a message queue server; and the message queue server sends the qualified message to the consumer in a distribution mode. The producer is communicated and/or electrically connected with the message queue server, and the consumer is communicated and/or electrically connected with the message queue server.
Notably, the distribution mode includes a point-to-point mode and a publish/subscribe mode. Point-to-point mode means that there is only one recipient of each eligible message in the message queue, i.e., once received, the eligible message is removed from the message queue. In publish/subscribe mode, there may be multiple subscribers (recipients) for each eligible message in the message queue, to which the eligible message may be sent.
In one embodiment, the message queue server performs a screening on the message data according to the authority, including:
acquiring the authority of a producer and the data type corresponding to each authority;
and identifying the data type of the message data, and when the message data is the same as the data type corresponding to the authority, marking the message data as intermediate data through one-time screening.
The data category in the embodiment corresponds to the authority, such as position authority acquisition position data, camera authority corresponding image data and the like; the data category corresponding to each authority is not unique, e.g., a camera authority may correspond to both image data and video data.
In the embodiment, the message data is matched and compared with the data type corresponding to the authority of the producer, so that the producer is prevented from acquiring the data under the condition that the authority is not allowed, and the safety of the data is ensured.
In an alternative embodiment, the rights of the producer are set by the message queue server.
This embodiment defines the rights of the producer. It can be understood that, in this embodiment, the authority of the producer is preset by a user or in other manners, and is stored in the message queue server, or the message queue server can call the authority in time.
It should be noted that the rights of the producer in this embodiment are different from the rights acquired by the producer itself; by setting the authority of the producer by the user, the authority provided by the user for the producer can be clear, and the producer can obtain the authority by the producer. Therefore, the authorities acquired by the two modes are not necessarily the same, and the data security is indirectly ensured.
In a specific embodiment, the secondary screening of the message data by the message queue server according to the content includes:
identifying data content of the intermediate data;
and carrying out anomaly detection on the data content, and marking the intermediate data passing the anomaly detection as qualified messages.
The embodiment detects the validity of the intermediate data, such as integrity detection and correlation detection, avoids content violation or invalidity of the intermediate data, can ensure the quality of qualified messages, can screen invalid data, and further improves the data processing efficiency.
In one embodiment, the message queue service end obtains the analysis result every set period, including:
acquiring the information quantity of qualified messages in a set period;
when the number of the messages exceeds a number threshold, numbering the qualified messages, and meanwhile, acquiring rejection duration according to the number of the messages;
and controlling the message queue server to receive message data according to the rejection time length.
In the embodiment, the number of the messages in the set period is obtained, then the qualified messages are numbered according to the number of the messages, and the reject duration is obtained. And when the information quantity is less than the quantity threshold value, the qualified messages do not need to be numbered, and the qualified messages are input into the message queue according to the marking time of the qualified messages in sequence.
It is worth noting that when the message queue server side obtains qualified messages of the next set period through statistical analysis, the qualified messages of the current set period are being input into the message queue, that is, the statistical analysis of the qualified messages and the input of the qualified messages into the message queue are not the same set period; for example, the following steps are carried out:
when the qualified message in the current set period starts to be input into the message queue, the message queue server starts to perform statistical analysis on the qualified message in the next set period.
In a specific embodiment, the qualified messages are numbered in sequence according to the marking time of the qualified messages;
when more than one qualified message is marked at the same time, the qualified messages at the time are not numbered back.
The unreturned numbers in this embodiment refer to that the same qualified message is not numbered repeatedly, which is illustrated as follows:
the first time is marked with 1 qualified message, the second time is marked with 2 qualified messages, the third time is marked with 1 qualified message, and the first time is earlier than the second time which is earlier than the third time.
The number of the qualified message at the first time is 1, the number of any one qualified message at the second time is 2, the number of the other qualified message is 3, and the number of the qualified message at the third time is 4.
In a specific embodiment, the rejection duration is obtained according to the information quantity and the duration of inputting each qualified message into the message queue;
and sequentially inputting the corresponding qualified messages into the message queue according to the numbers of the qualified messages.
The information amount in this embodiment refers to the total amount of qualified messages that need to be written into the message queue.
The reject duration in this embodiment is obtained by the number of messages and the duration required for the qualified message to be input into the message queue, and the duration required for the qualified message to be input into the message queue can be estimated. The reject duration is mainly to avoid data backlog, for example:
assuming that each set period is 10 seconds, and the time length required for writing each qualified message into the message queue is 0.1 second;
when 900 qualified messages need to be written into the message queue in the current set period and 100 qualified messages are not written into the message queue in the last set period, 1000 qualified messages need to be written into the message queue;
the reject duration can be calculated to be 1000 × 0.1=100 seconds, and it can be understood that 10 set periods are required to input all eligible messages into the message queue.
The message queue server can refuse to acquire the message data in the future 10 set periods, and can also temporarily store the message data; in other preferred embodiments, a threshold value may be set for the future set period, and if the future set period does not exceed 5, the message data is not rejected.
An embodiment of a second aspect of the present application provides a method for processing data of a mobile device based on a message queue, including:
message data are produced by a producer and are sent to a message queue server;
the message queue service end screens message data produced by a producer to obtain qualified messages;
the message queue server side carries out statistical analysis on qualified messages every other set period and inputs the qualified messages into the message queue according to the analysis result;
and the message queue server side sends the qualified message to the consumer in a distribution mode.
The working principle of the invention is as follows:
message data are produced by a producer and are sent to a message queue server;
the message queue service end screens message data produced by a producer to obtain qualified messages;
the message queue server side carries out statistical analysis on qualified messages every other set period and inputs the qualified messages into the message queue according to the analysis result;
and the message queue server side sends the qualified message to the consumer in a distribution mode.
Although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the present invention.
Claims (4)
1. A mobile equipment data processing system based on a message queue comprises a producer, a message queue server and a consumer, and is characterized in that the message queue server screens message data produced by the producer to obtain qualified messages; and
performing statistical analysis on qualified messages every set period, and inputting the qualified messages into a message queue according to an analysis result;
the message queue server side performs one-time screening on message data according to the authority, and the method comprises the following steps:
acquiring the authority of a producer and the data type corresponding to each authority;
identifying the data type of the message data, and when the message data is the same as the data type corresponding to the authority, marking the message data as intermediate data through one-time screening;
the authority of the producer is set through the message queue server;
the message queue server side performs secondary screening on message data according to the content, and the secondary screening comprises the following steps:
identifying data content of the intermediate data;
carrying out anomaly detection on data content, and marking intermediate data passing the anomaly detection as qualified messages; wherein the anomaly detection comprises an integrity detection;
the message queue server side obtains analysis results every other set period, and the analysis results comprise:
acquiring the information quantity of qualified messages in a set period; wherein the set period comprises 1 second or 1 minute;
when the number of the messages exceeds a number threshold, numbering the qualified messages, and meanwhile, acquiring rejection duration according to the number of the messages; wherein the number threshold is an integer greater than 0 and is set empirically; controlling a message queue server to receive message data according to the rejection duration;
numbering qualified messages in sequence according to the marking time of the qualified messages;
when more than one qualified message is marked at the same moment, the qualified messages at the moment are not numbered back;
acquiring rejection duration according to the information quantity and the duration of inputting each qualified message into the message queue; corresponding qualified messages are sequentially input into a message queue according to the numbers of the qualified messages;
after receiving the message data at the message queue server, performing primary screening and secondary screening on the message data, and inputting qualified messages into a message queue after statistical analysis of the qualified messages; according to the invention, invalid or illegal data can be screened and removed from the message data, so that the data processing efficiency is improved;
when data is screened, the message data is screened through primary screening and secondary screening, the authority of a producer and the effectiveness of the message data are considered in the screening process, and the safety of the data is guaranteed from the perspective of a user.
2. The message queue-based mobile device data processing system of claim 1, wherein message data is generated by the producer and sent to the message queue server; wherein the producer is in communication and/or electrical connection with the message queue server.
3. The message queue-based mobile device data processing system of claim 1, wherein the message queue server sends qualified messages to consumers by distribution; the distribution mode comprises a point-to-point mode and a publish/subscribe mode, and the consumer is in communication and/or electrical connection with the message queue server.
4. A message queue based mobile device data processing system as claimed in any one of claims 1 to 3, characterised in that the data processing method of the system comprises:
producing message data through the producer and sending the message data to the message queue server;
the message queue service end screens message data produced by a producer to obtain qualified messages;
the message queue server side carries out statistical analysis on qualified messages every other set period and inputs the qualified messages into the message queue according to the analysis result;
and the message queue server side sends qualified messages to the consumers in a distribution mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210155081.7A CN114553962B (en) | 2022-02-21 | 2022-02-21 | Mobile equipment data processing method and system based on message queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210155081.7A CN114553962B (en) | 2022-02-21 | 2022-02-21 | Mobile equipment data processing method and system based on message queue |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114553962A CN114553962A (en) | 2022-05-27 |
| CN114553962B true CN114553962B (en) | 2023-01-06 |
Family
ID=81676163
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210155081.7A Active CN114553962B (en) | 2022-02-21 | 2022-02-21 | Mobile equipment data processing method and system based on message queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114553962B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111818065A (en) * | 2020-07-13 | 2020-10-23 | 易盼红 | User terminal information control system and method based on big data |
| CN112559488A (en) * | 2020-12-09 | 2021-03-26 | 中铁第四勘察设计院集团有限公司 | Escalator full life cycle data management method and system based on data center station |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150281126A1 (en) * | 2014-03-31 | 2015-10-01 | Plx Technology, Inc. | METHODS AND APPARATUS FOR A HIGH PERFORMANCE MESSAGING ENGINE INTEGRATED WITHIN A PCIe SWITCH |
| CN108132845A (en) * | 2016-12-01 | 2018-06-08 | 阿里巴巴集团控股有限公司 | Message storage, delivering method and device and electronic equipment |
| CN108270662A (en) * | 2017-12-26 | 2018-07-10 | 微梦创科网络科技（中国）有限公司 | A kind of message distributing method, device and equipment |
| CN109656921B (en) * | 2018-11-26 | 2023-09-01 | 平安科技（深圳）有限公司 | Organization architecture data processing method, device, computer equipment and storage medium |
| CN113778818A (en) * | 2020-09-25 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Method, apparatus, device and computer readable medium for optimizing system |
| CN112667414A (en) * | 2020-12-23 | 2021-04-16 | 平安普惠企业管理有限公司 | Message queue-based message consumption method and device, computer equipment and medium |
-
2022
- 2022-02-21 CN CN202210155081.7A patent/CN114553962B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111818065A (en) * | 2020-07-13 | 2020-10-23 | 易盼红 | User terminal information control system and method based on big data |
| CN112559488A (en) * | 2020-12-09 | 2021-03-26 | 中铁第四勘察设计院集团有限公司 | Escalator full life cycle data management method and system based on data center station |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114553962A (en) | 2022-05-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10185479B2 (en) | Declassifying of suspicious messages | |
| CN102769549B (en) | The method and apparatus of network security monitoring | |
| CN113157466A (en) | Message pushing method, device, system, electronic equipment and storage medium | |
| AU2009284000B2 (en) | A communication device | |
| CN111447399B (en) | Commanding and scheduling method, device and system | |
| CN114553962B (en) | Mobile equipment data processing method and system based on message queue | |
| CN105430623A (en) | Monitoring method, device and system for RCS junk message | |
| CN115410580A (en) | Voice recognition method, device, equipment and medium for command scheduling system | |
| CN108810829B (en) | Multimedia message interception processing method and device | |
| CN112217944B (en) | Online ticket processing method, device, equipment and storage medium | |
| CN110557324B (en) | Unread IM message processing method and device | |
| CN111092922B (en) | Information sending method and device | |
| CN108427615A (en) | A kind of message monitoring method and device | |
| CN106708706A (en) | Task program exception alarm information processing method and apparatus | |
| CN112988417A (en) | Message processing method and device, electronic equipment and computer readable medium | |
| CN101753358B (en) | Alarming information notification method and system | |
| CN112004161A (en) | Processing method and device of address resources, terminal equipment and storage medium | |
| CN110569673A (en) | Data file processing method, device, equipment and storage medium | |
| CN110943921A (en) | Intelligent route configuration method | |
| CN115633187B (en) | Chat message processing method and device, computer equipment and storage medium | |
| CN112468471B (en) | Remote authorization method and device | |
| CN113259862B (en) | Short message centralized forwarding system and method | |
| CN108712312B (en) | Online authentication request scheduling method and device and computer equipment | |
| CN109889590B (en) | Message processing method, device, client and computer readable medium | |
| CN112181789B (en) | Equipment end log collection method based on AWS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |