WO2020177384A1 - Method and apparatus for reporting and processing user message status of message pushing, and storage medium - Google Patents
Method and apparatus for reporting and processing user message status of message pushing, and storage medium Download PDFInfo
- Publication number
- WO2020177384A1 WO2020177384A1 PCT/CN2019/117979 CN2019117979W WO2020177384A1 WO 2020177384 A1 WO2020177384 A1 WO 2020177384A1 CN 2019117979 W CN2019117979 W CN 2019117979W WO 2020177384 A1 WO2020177384 A1 WO 2020177384A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- message
- data
- messages
- message push
- background
- Prior art date
Links
- 238000012545 processing Methods 0.000 title claims abstract description 23
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000004140 cleaning Methods 0.000 claims description 18
- 230000005540 biological transmission Effects 0.000 claims description 13
- 238000003672 processing method Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 5
- 238000012827 research and development Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 4
- 238000005192 partition Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/725—Cordless telephones
Definitions
- This application relates to research and development management, and specifically to a method, device and storage medium for processing user message status reporting for message push.
- the reported information includes the message arrives on the mobile phone, the message is displayed in the notification bar or banner, click on the message, APP message notification shielding, APP uninstalled, etc. status information.
- the message push background After the message push background receives these status report requests, it first verifies whether the reported identity is legal, then decrypts the reported content, and then obtains which MsgID and which APP user reported the message status according to the decrypted content, and then saves it to the hard disk of the message push background In the relational database above, this is an external storage.
- the message push background receives the reported information and saves it to a relational database (such as mysql, oracle) for storage, which will result in slow service processing and eventually unavailability of the service.
- this application provides a user message status report processing method for message push, which is applied to an electronic device, and includes:
- the message push background After the message push background pushes messages to the APPs on each mobile terminal, the message push background receives the reported information sent by each APP.
- the reported information includes at least the arrival of the message on the mobile terminal, the message displayed in the notification bar or banner, user clicks, and APP Whether the message notification is blocked and whether the APP has been uninstalled;
- the message push background stores the received report information in the local server memory queue of the message push background, and the message push background reports to each mobile terminal that the reported information has been received;
- the message push background While storing the reported information in the memory queue of the local server in the message push background, the message push background also starts multiple workers threads to read the reported information stored in the memory queue in real time and store it in the MQ storage component;
- the message push background reads each message in the reported information in the consumer MQ storage component, verifies whether the reported identity is legal, and decrypts the content of the message. If the reported identity is legal, the message is stored in HDFS.
- the present application also provides an electronic device, the electronic device comprising: a memory and a processor, the memory stores a user message status report processing program for message push, and the user message status report processing program for message push is processed by the The following steps are implemented when the device is executed:
- each APP After the message push background pushes messages to the APPs on each mobile terminal, each APP sends report information to the message push background, and the report information includes multiple pieces of information returned by the APP from which the message is pushed from each mobile terminal.
- the report The information includes at least the arrival of the message on the mobile terminal, the display of the message in the notification bar or banner, user clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled;
- the message push background directly stores the received report information in the local server memory queue of the message push background, and then the message push background sends feedback to each mobile terminal that the reported information has been received;
- the message push background While storing the reported information in the memory queue of the local server in the message push background, the message push background also starts multiple workers threads to read the reported information stored in the memory queue in real time and store it in the MQ storage component;
- the message push background reads each message in the reported information in the consumer MQ storage component, and verifies whether the reported identity is legal, and decrypts the message content. If the reported identity is legal, the message is stored in HDFS.
- This application also provides a user message status report processing device for message push, including:
- the report message receiving module after the message push background pushes the message to the APP on each mobile terminal, the message push background receives the report information sent by each APP, and the report information includes at least the message reached the mobile terminal, the message is displayed in the notification bar or banner, User clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled;
- the report message memory queue storage module is used for the message push background to directly store the received report information in the local server memory queue of the message push background, and then the message push background reports to each mobile terminal that the reported information has been received;
- the memory queue information reading module is used to store the reported information in the local server memory queue of the message push background.
- the message push background also starts multiple workers thread to read the reported information stored in the memory queue in real time and store it in MQ Storage components;
- the message middleware reading module is used for the message push background to read each message in the reported information in the consumer MQ storage component, and verify whether the reported identity is legal, and decrypt the message content. If the reported identity is legal, keep the message To HDFS.
- the present application also provides a computer non-volatile readable storage medium, the computer non-volatile readable storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, The user message status report processing method for message push as described above is implemented.
- the message status information is processed asynchronously.
- the message status information is first stored in the server memory queue.
- the storage speed is fast.
- it also uses multiple worker threads to read the memory queue in real time.
- the reported information is stored in the MQ storage component, which will not cause a surge in access pressure to the message push background at a certain moment.
- the message push background then reads each message in the reported information in the consumer MQ storage component, and saves the message in HDFS after the reported identity is legal.
- this application speeds up the storage speed and the client responds quickly.
- FIG. 1 is a schematic flowchart of a method for processing user message status reporting in message push according to an embodiment of the present application
- FIG. 2 is a schematic diagram of the hardware architecture of an electronic device according to an embodiment of the present application.
- Fig. 3 is a block diagram of a user message status report processing program for message push in an embodiment of the present application.
- FIG. 1 is a schematic flowchart of a method for processing user message status reporting for message push provided by an embodiment of the application. The method is applied to an electronic device and includes the following steps:
- Step S10 after the message push background pushes messages to the APPs on each mobile terminal, each APP sends report information to the message push background, and the report information includes multiple pieces of information returned by the APPs that have pushed messages from each mobile terminal.
- the reported information includes at least the arrival of the message on the mobile terminal, the display of the message in the notification bar or banner, the user clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled.
- Step S20 The message push background directly stores the received report information in the local server memory queue of the message push background, and then the message push background feeds back the received report information to each mobile terminal.
- Step S30 While storing the reported information in the local server memory queue of the message push background, the message push background also starts multiple workers threads (worker threads) to read the reported information stored in the memory queue in real time and store it in the MQ storage component (Message middleware). In worker Thread mode, each message in the reported information is taken out one by one for processing.
- work threads worker threads
- Kafka distributed, partitioned, and reproducible submission log service
- Kafka categorizes reported messages according to topics when they are saved. Topics can be considered as a type of message. Each topic is divided into multiple partitions, and each partition is an append.log file at the storage level.
- the person who sends the message is called the Producer, where the worker thread is equivalent to the producer.
- the Producer publishes the reported message to the specified topic, and the Producer can also determine which partition the reported message belongs to. Any message posted to the partition will be directly appended to the end of the log file.
- the position of each message in the file is called the offset.
- the offset is a long number, and the offset uniquely marks a message.
- Step S40 the message push background reads each message in the reported information in the consumer MQ storage component, and verifies whether the reported identity is legal, and decrypts the message content. If the reported identity is legal, the message is saved to HDFS (distributed File system).
- HDFS distributed File system
- a big data platform is used to read each message in the reported information in the consumption kafka, which can speed up the processing speed.
- HashMap uses the key-value interface of Map.
- Entry class is:
- the consumption buffer threshold For the consumer side of Kafka, set the consumption buffer threshold. When the message in Kafka reaches the consumption buffer threshold, the message push background will fetch multiple messages in batches.
- data cleaning is performed using an ETL (data cleaning) cleaning program to remove dirty data, where the dirty data includes duplicate data, incomplete data, invalid data, and inconsistent data, wherein, before data cleaning First, the messages are clustered according to repeated data, incomplete data, invalid data, and inconsistent data.
- ETL data cleaning
- repeated data cleaning includes judging whether it is duplicated and deleting duplicate data, wherein a multi-segment comparison method is used to sequentially compare the same length fields in the same position of no less than 3 two messages.
- a multi-segment comparison method is used to sequentially compare the same length fields in the same position of no less than 3 two messages.
- judging whether to repeat can be based on extracting a segment of the same position of the two messages, and comparing whether the two segments are exactly the same, if they are exactly the same, extract the two messages at a certain interval (for example, one character) Another field in the same position of the two messages, and compare whether the two fields are exactly the same. If they are exactly the same, extract another field in the same position of the two messages at a certain interval and compare them further.
- Segment messages are repeated data. If they are not the same, they are deemed not to be repeated data. Wherein, preferably, the same positions of the messages extracted three times are the head of the message field, the tail of the message field, and the middle of the message field, respectively.
- the above method is first used to determine whether the keys are completely consistent, and then whether the values are completely consistent.
- the key is first judged whether it is completely consistent, and then the value is judged whether it is completely consistent, so as to judge whether the message is a duplicate message.
- the cosine similarity is calculated according to the key-value pair vector composed of the key and value of the two messages, and the preset similarity is calculated
- the threshold is used to determine the degree of similarity. If it is higher than the threshold of similarity, it is considered a duplicate message.
- first cluster all the messages in the reported information for example, classify all messages from the same mobile terminal into one category, and then group the key and value of each message into a key-value pair vector to calculate two messages If the cosine similarity of the key-value pair vector is higher than the similarity threshold, the same length fields at the same position of the two messages at no less than 3 are used to compare in turn to determine whether the two messages are duplicate data. If the similarity is lower than the similarity threshold, it is considered not to be duplicate data.
- the vector of key-value pairs of message A and message B are as follows:
- d(A, B) represents the cosine similarity of message A and message B;
- key Ai is the i-th component of the key vector representing message A;
- value Ai is the i-th component of the value vector representing message A;
- key Bi is the i-th component of the key vector representing message B;
- value Bi is the i-th component of the value vector representing the message B;
- n the number of key-value pairs of message A and message B.
- the electronic device 2 is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions.
- it can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster composed of multiple servers).
- the electronic device 2 at least includes, but is not limited to, a memory 21, a processor 22, and a network interface 23 that can be communicatively connected to each other through a system bus.
- the memory 21 includes at least one type of computer non-volatile readable storage medium
- the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random Access memory (RAM), static random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, etc.
- the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or a memory of the electronic device 2.
- the memory 21 may also be an external storage device of the electronic device 2, for example, a plug-in hard disk equipped on the electronic device 2, a smart media card (SMC), a secure digital (Secure Digital, SD) card, flash card (Flash Card), etc.
- the memory 21 may also include both the internal storage unit of the electronic device 2 and its external storage device.
- the memory 21 is generally used to store the operating system and various application software installed in the electronic device 2, for example, the user message status report processing program code of the message push, etc.
- the memory 21 can also be used to temporarily store various types of data that have been output or will be output.
- the processor 22 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments.
- the processor 22 is generally used to control the overall operation of the electronic device 2, for example, to perform data interaction or communication-related control and processing with the electronic device 2.
- the processor 22 is configured to run the program code or processing data stored in the memory 21, for example, to run the user message status report processing program of the message push.
- the network interface 23 may include a wireless network interface or a wired network interface, and the network interface 23 is generally used to establish a communication connection between the electronic device 2 and other electronic devices.
- the network interface 23 is used to connect the electronic device 2 with a push platform through a network, and establish a data transmission channel and a communication connection between the electronic device 2 and the push platform.
- the network may be Intranet, Internet, Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network , Bluetooth (Bluetooth), Wi-Fi and other wireless or wired networks.
- the electronic device 2 may also include a display, and the display may also be called a display screen or a display unit.
- the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an organic light-emitting diode (OLED) display, etc.
- the display is used for displaying information processed in the electronic device 2 and for displaying a visualized user interface.
- FIG. 2 only shows the electronic device 2 with components 21-23, but it should be understood that it is not required to implement all of the illustrated components, and more or fewer components may be implemented instead.
- the memory 21 containing a readable storage medium may include an operating system, a user message status report processing program 50 for message push, and the like.
- the processor 22 implements the following steps when executing the user message status report processing program 50 of the message push in the memory 21:
- Step S10 after the message push background pushes messages to the APPs on each mobile terminal, each APP sends report information to the message push background, and the report information includes multiple pieces of information returned by the APPs that have pushed messages from each mobile terminal.
- the reported information includes at least the arrival of the message on the mobile terminal, the display of the message in the notification bar or banner, the user clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled.
- Step S20 The message push background directly stores the received report information in the local server memory queue of the message push background, and then the message push background feeds back the received report information to each mobile terminal.
- Step S30 while storing the reported information in the memory queue of the local server in the message push background, multiple worker threads are also started to read the reported information stored in the memory queue in real time and store it in the MQ storage component.
- worker Thread mode each message in the reported information is taken out one by one for processing.
- step S40 the message push background reads each message in the reported information in the consumer MQ storage component, verifies whether the reported identity is legal, and decrypts the message content. If the reported identity is legal, the message is stored in HDFS.
- the user message status report processing program of the message push stored in the memory 21 may be divided into one or more program modules, and the one or more program modules are stored in the memory 21, and It can be executed by one or more processors (the processor 22 in this embodiment) to complete the application.
- FIG. 3 shows a schematic diagram of the program modules of the user message status report processing program of the message push.
- the user message status report processing program 50 of the message push can be divided into a report message receiving module 501, Report message memory queue storage module 502, memory queue information reading module 503, and message middleware reading module 504.
- the program module referred to in this application refers to a series of computer program instruction segments that can complete specific functions, and is more suitable than a program to describe the execution process of the user message status report processing program of the message push in the electronic device 2 .
- the following description will specifically introduce the specific functions of the program modules.
- the report message receiving module 501 is configured to receive the report message when each APP sends the report information to the message push background after pushing the message to the APP on each mobile terminal in the message push background.
- the reported information includes multiple pieces of information returned by the APP from which the message is pushed from each mobile terminal.
- the reported information includes at least the arrival of the message on the mobile terminal, the display of the message in the notification bar or banner, the user click, and whether the APP message notification is Blocking, whether the APP has been uninstalled.
- the reported message memory queue storage module 502 is used to directly store the received reported information in the local server memory queue of the message push background after the message push background, and then the message push background feedbacks the received report information to each mobile terminal.
- the memory queue information reading module 503 is used to store the reported information in the memory queue of the local server in the message push background, and also start multiple worker threads to read the reported information stored in the memory queue in real time and store it in the MQ storage component . In worker Thread mode, each message in the reported information is taken out one by one for processing.
- the message middleware reading module 504 is used to read each message in the reported information in the consumer MQ storage component, and verify whether the reported identity is legal, and decrypt the message content. If the reported identity is legal, the message is stored in HDFS .
- HashMap uses the key-value interface of Map.
- Entry class is:
- a threshold comparison module 505. For the producer side of Kafka, a transmission buffer threshold is set, and messages are buffered. After the number of messages reaches the transmission buffer threshold, they are sent to Kafka in batches;
- the consumption buffer threshold For the consumer side of Kafka, set the consumption buffer threshold. When the message in Kafka reaches the consumption buffer threshold, the message push background will fetch multiple messages in batches.
- a data cleaning module 506 is further included.
- the data cleaning module 506 uses an ETL (data cleaning) cleaning program to clean data to remove dirty data, where the dirty data includes duplicate data, incomplete data, and invalid data. Data and inconsistent data. Before data cleaning, messages are clustered according to repeated data, incomplete data, invalid data, and inconsistent data.
- ETL data cleaning
- the repeated data judging module 507 is used for judging whether the data is repeated and deleting the repeated data.
- a multi-segment comparison method is adopted, and two of no less than 3 The same length fields in the same position of the message are compared sequentially to determine whether the two messages are duplicate data.
- judging whether to repeat can be based on extracting a segment of the same position of the two messages, and comparing whether the two segments are exactly the same, if they are exactly the same, extract the two messages at a certain interval (for example, one character) Another field in the same position of the two messages, and compare whether the two fields are exactly the same.
- Segment messages are repeated data. If they are not the same, they are deemed not to be repeated data.
- the same positions of the messages extracted three times are the head of the message field, the tail of the message field, and the middle of the message field, respectively.
- the above method is first used to determine whether the keys are completely consistent, and then whether the values are completely consistent.
- the key is first judged whether it is completely consistent, and then the value is judged whether it is completely consistent, so as to judge whether the message is a duplicate message.
- the repeated data judging module 507 also calculates the cosine similarity based on the key-value pair vector composed of the key and value of the two messages. Before comparing the two messages in a multi-segment comparison method, first The preset similarity threshold is used to determine the degree of similarity. If it is higher than the similarity threshold, it is considered as a duplicate message.
- first cluster all the messages in the reported information for example, classify all messages from the same mobile terminal into one category, and then group the key and value of each message into a key-value pair vector to calculate two messages If the cosine similarity of the key-value pair vector is higher than the similarity threshold, no less than 3 two messages with the same length at the same position are compared in sequence to determine whether the two messages are duplicate data. Can speed up the speed of message checking. If the similarity is lower than the similarity threshold, it is considered not to be duplicate data.
- This application also provides a user message status report processing device for message push, including:
- the report message receiving module 501 after the message push background pushes messages to the APPs on each mobile terminal, the message push background receives the report information sent by each APP, and the report information includes at least the message reached the mobile terminal and the message is displayed in the notification bar or banner , User clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled;
- the report message memory queue storage module 502 is used for the message push background to directly store the received report information in the local server memory queue of the message push background, and then the message push background reports to each mobile terminal that the report information has been received;
- the memory queue information reading module 503 is used to store the reported information in the memory queue of the local server in the message push background. At the same time, the message push background also starts multiple workers threads to read the reported information stored in the memory queue in real time and store it in MQ storage component;
- the message middleware reading module 504 is used for the message push background to read each message in the reported information in the consumer MQ storage component, verify whether the reported identity is legal, and decrypt the message content. If the reported identity is legal, the message Keep it in HDFS.
- the embodiment of the present application also proposes a computer non-volatile readable storage medium.
- the computer non-volatile readable storage medium may be a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, etc., or any combination of several.
- the computer non-volatile readable storage medium includes a user message status report processing program for message push, etc.
- the user message status report processing program 50 for message push is executed by the processor 22 to implement the following operations:
- Step S10 after the message push background pushes messages to the APPs on each mobile terminal, each APP sends report information to the message push background, and the report information includes multiple pieces of information returned by the APPs that have pushed messages from each mobile terminal.
- the reported information includes at least the arrival of the message on the mobile terminal, the display of the message in the notification bar or banner, the user clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled.
- Step S20 The message push background directly stores the received report information in the local server memory queue of the message push background, and then the message push background feeds back the received report information to each mobile terminal.
- Step S30 while storing the reported information in the memory queue of the local server in the message push background, the message push background also starts multiple workers threads to read the reported information stored in the memory queue in real time and store it in the MQ storage component.
- worker Thread mode each message in the reported information is taken out one by one for processing.
- step S40 the message push background reads each message in the reported information in the consumer MQ storage component, and verifies whether the reported identity is legal, and decrypts the message content. If the reported identity is legal, the message is stored in HDFS.
- the specific implementation manner of the computer non-volatile readable storage medium of the present application is substantially the same as the specific implementation manner of the user message status report processing method for message push and the electronic device 2 described above, and will not be repeated here.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Stored Programmes (AREA)
Abstract
The present solution relates to research and development management and provided thereby are a method and apparatus for reporting and processing a user message status of message pushing, and a storage medium. The method comprises: each app sending report information to a message pushing background; the message pushing background storing the received report information in a local server memory queue (MQ) of the message pushing background, and the message pushing background feeding the received report information back to a user client; when the report information is stored in the local server MQ of the message pushing background, further starting up a plurality of worker threads to read in real time the report information stored in the MQ and store said information in an MQ storage component; then, the message pushing background reading each message in the report information in the MQ storage component, and verifying whether a reporting identity is legal; and if the reporting identity is legal, keeping the messages in an HDFS. The present solution accelerates storage speed and does not cause a sudden increase in access pressure for the message pushing background at a given moment.
Description
本申请要求于2019年3月5日提交的中国专利申请号2019101649356的优先权益,上述案件全部内容以引用的方式并入本文中。This application claims the priority rights of Chinese Patent Application No. 2019101649356 filed on March 5, 2019. The entire contents of the above cases are incorporated herein by reference.
本申请涉及研发管理,具体地说,涉及一种消息推送的用户消息状态上报处理方法、装置及存储介质。This application relates to research and development management, and specifically to a method, device and storage medium for processing user message status reporting for message push.
通常的用户手机上收到APP推送的消息推送会触发上报信息传送给消息推送后台,上报信息包括消息到达到手机、消息展示在通知栏或横幅、点击消息、APP消息通知屏蔽、APP已卸载等状态信息。消息推送后台收到这些状态上报请求后,先验证上报身份是否合法,然后解密上报的内容,然后根据解密的内容获取是哪个MsgID、哪个APP用户上报的消息状态,然后保存到消息推送后台的硬盘上的关系型数据库中,这属于外部存储。发明人意识到,当用户规模到亿级时,且同时向用户推送消息时,APP用户会同时大量向消息推送后台上报消息状态,一条推送消息平均都2-3条状态要上报给消息推送改后台,消息推送后台收到上报信息保存到关系型数据库(如mysql、oracle)中存储,会导致服务处理较慢,最后导致服务不可用。Usually the user’s mobile phone receives the message push of the APP push will trigger the report information to be sent to the message push background. The reported information includes the message arrives on the mobile phone, the message is displayed in the notification bar or banner, click on the message, APP message notification shielding, APP uninstalled, etc. status information. After the message push background receives these status report requests, it first verifies whether the reported identity is legal, then decrypts the reported content, and then obtains which MsgID and which APP user reported the message status according to the decrypted content, and then saves it to the hard disk of the message push background In the relational database above, this is an external storage. The inventor realized that when the user scale reaches 100 million and simultaneously pushes messages to users, APP users will simultaneously report a large number of message statuses to the message push background, and an average of 2-3 statuses of a push message must be reported to the message push change In the background, the message push background receives the reported information and saves it to a relational database (such as mysql, oracle) for storage, which will result in slow service processing and eventually unavailability of the service.
发明内容Summary of the invention
为解决以上技术问题,本申请提供一种消息推送的用户消息状态上报处理方法,应用于电子装置,包括:To solve the above technical problems, this application provides a user message status report processing method for message push, which is applied to an electronic device, and includes:
在消息推送后台向各移动终端上的APP推送消息后,消息推送后台接收各APP发送的上报信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载;After the message push background pushes messages to the APPs on each mobile terminal, the message push background receives the reported information sent by each APP. The reported information includes at least the arrival of the message on the mobile terminal, the message displayed in the notification bar or banner, user clicks, and APP Whether the message notification is blocked and whether the APP has been uninstalled;
消息推送后台将接收到的上报信息存放在消息推送后台的本地服务器内 存队列中,消息推送后台向各个移动终端反馈已接收到上报信息;The message push background stores the received report information in the local server memory queue of the message push background, and the message push background reports to each mobile terminal that the reported information has been received;
在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,消息推送后台还启动多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中;While storing the reported information in the memory queue of the local server in the message push background, the message push background also starts multiple workers threads to read the reported information stored in the memory queue in real time and store it in the MQ storage component;
消息推送后台读取消费MQ存储组件中的上报信息中的每条消息,并且验证上报身份是否合法,以及对消息内容进行解密,若上报身份合法则将消息保持到HDFS中。The message push background reads each message in the reported information in the consumer MQ storage component, verifies whether the reported identity is legal, and decrypts the content of the message. If the reported identity is legal, the message is stored in HDFS.
本申请还提供一种电子装置,该电子装置包括:存储器和处理器,所述存储器中存储有消息推送的用户消息状态上报处理程序,所述消息推送的用户消息状态上报处理程序被所述处理器执行时实现如下步骤:The present application also provides an electronic device, the electronic device comprising: a memory and a processor, the memory stores a user message status report processing program for message push, and the user message status report processing program for message push is processed by the The following steps are implemented when the device is executed:
在消息推送后台向各移动终端上的APP推送消息后,各APP向消息推送后台发送上报信息,所述上报信息中包含多条来自各个移动终端的被推送消息的APP返回的信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载;After the message push background pushes messages to the APPs on each mobile terminal, each APP sends report information to the message push background, and the report information includes multiple pieces of information returned by the APP from which the message is pushed from each mobile terminal. The report The information includes at least the arrival of the message on the mobile terminal, the display of the message in the notification bar or banner, user clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled;
消息推送后台将接收到的上报信息直接存放在消息推送后台的本地服务器内存队列中,然后消息推送后台向各个移动终端反馈已接收到上报信息;The message push background directly stores the received report information in the local server memory queue of the message push background, and then the message push background sends feedback to each mobile terminal that the reported information has been received;
在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,消息推送后台还启动多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中;While storing the reported information in the memory queue of the local server in the message push background, the message push background also starts multiple workers threads to read the reported information stored in the memory queue in real time and store it in the MQ storage component;
然后消息推送后台读取消费MQ存储组件中的上报信息中的每条消息,并且验证上报身份是否合法,以及对消息内容进行解密,若上报身份合法则将消息保持到HDFS中。Then the message push background reads each message in the reported information in the consumer MQ storage component, and verifies whether the reported identity is legal, and decrypts the message content. If the reported identity is legal, the message is stored in HDFS.
本申请还提供一种消息推送的用户消息状态上报处理装置,包括:This application also provides a user message status report processing device for message push, including:
上报消息接收模块,在消息推送后台向各移动终端上的APP推送消息后,消息推送后台接收各APP发送的上报信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载;The report message receiving module, after the message push background pushes the message to the APP on each mobile terminal, the message push background receives the report information sent by each APP, and the report information includes at least the message reached the mobile terminal, the message is displayed in the notification bar or banner, User clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled;
上报消息内存队列存储模块,用于消息推送后台将接收到的上报信息直接存放在消息推送后台的本地服务器内存队列中,然后消息推送后台向各个 移动终端反馈已接收到上报信息;The report message memory queue storage module is used for the message push background to directly store the received report information in the local server memory queue of the message push background, and then the message push background reports to each mobile terminal that the reported information has been received;
内存队列信息读取模块,用于在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,消息推送后台还启动多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中;The memory queue information reading module is used to store the reported information in the local server memory queue of the message push background. At the same time, the message push background also starts multiple workers thread to read the reported information stored in the memory queue in real time and store it in MQ Storage components;
消息中间件读取模块,用于消息推送后台读取消费MQ存储组件中的上报信息中的每条消息,并且验证上报身份是否合法,以及对消息内容进行解密,若上报身份合法则将消息保持到HDFS中。The message middleware reading module is used for the message push background to read each message in the reported information in the consumer MQ storage component, and verify whether the reported identity is legal, and decrypt the message content. If the reported identity is legal, keep the message To HDFS.
本申请还提供一种计算机非易失性可读存储介质,所述计算机非易失性可读存储介质存储有计算机程序,所述计算机程序包括程序指令,所述程序指令被处理器执行时,实现如上所述的消息推送的用户消息状态上报处理方法。The present application also provides a computer non-volatile readable storage medium, the computer non-volatile readable storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, The user message status report processing method for message push as described above is implemented.
本申请把消息状态信息进行异步化处理,先将消息状态信息存放在服务器内存队列中,存储速度快,并且,在存储到内存队列的同时,又利用多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中,不会造成某时刻对消息推送后台的访问压力剧增。消息推送后台再读取消费MQ存储组件中的上报信息中的每条消息,并在上报身份合法后将消息保持到HDFS中。在大量的客户端需要上报状态到消息推送后台时,本申请加快了存储速度,客户端响应快。In this application, the message status information is processed asynchronously. The message status information is first stored in the server memory queue. The storage speed is fast. In addition, while storing it in the memory queue, it also uses multiple worker threads to read the memory queue in real time. The reported information is stored in the MQ storage component, which will not cause a surge in access pressure to the message push background at a certain moment. The message push background then reads each message in the reported information in the consumer MQ storage component, and saves the message in HDFS after the reported identity is legal. When a large number of clients need to report status to the message push background, this application speeds up the storage speed and the client responds quickly.
通过结合下面附图对其实施例进行描述,本申请的上述特征和技术优点将会变得更加清楚和容易理解。By describing its embodiments in conjunction with the following drawings, the above-mentioned features and technical advantages of the present application will become clearer and easier to understand.
图1是本申请实施例的消息推送的用户消息状态上报处理方法的流程示意图;FIG. 1 is a schematic flowchart of a method for processing user message status reporting in message push according to an embodiment of the present application;
图2是本申请实施例的电子装置的硬件架构示意图;2 is a schematic diagram of the hardware architecture of an electronic device according to an embodiment of the present application;
图3是本申请实施例的消息推送的用户消息状态上报处理程序的模块构成图。Fig. 3 is a block diagram of a user message status report processing program for message push in an embodiment of the present application.
下面将参考附图来描述本申请所述的消息推送的用户消息状态上报处理 方法、装置及存储介质的实施例。本领域的普通技术人员可以认识到,在不偏离本申请的精神和范围的情况下,可以用各种不同的方式或其组合对所描述的实施例进行修正。因此,附图和描述在本质上是说明性的,而不是用于限制权利要求的保护范围。此外,在本说明书中,附图未按比例画出,并且相同的附图标记表示相同的部分。Hereinafter, embodiments of the method, device, and storage medium for reporting user message status for message push in this application will be described with reference to the accompanying drawings. A person of ordinary skill in the art may realize that the described embodiments can be modified in various different ways or combinations thereof without departing from the spirit and scope of the present application. Therefore, the drawings and descriptions are illustrative in nature and are not used to limit the scope of protection of the claims. In addition, in this specification, the drawings are not drawn to scale, and the same reference numerals denote the same parts.
图1为本申请实施例提供的消息推送的用户消息状态上报处理方法的流程示意图。该方法应用于电子装置,包括以下步骤:FIG. 1 is a schematic flowchart of a method for processing user message status reporting for message push provided by an embodiment of the application. The method is applied to an electronic device and includes the following steps:
步骤S10,在消息推送后台向各移动终端上的APP推送消息后,各APP向消息推送后台发送上报信息,所述上报信息中包含多条来自各个移动终端的被推送消息的APP返回的信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载。Step S10, after the message push background pushes messages to the APPs on each mobile terminal, each APP sends report information to the message push background, and the report information includes multiple pieces of information returned by the APPs that have pushed messages from each mobile terminal. The reported information includes at least the arrival of the message on the mobile terminal, the display of the message in the notification bar or banner, the user clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled.
步骤S20,消息推送后台将接收到的上报信息直接存放在消息推送后台的本地服务器内存队列中,然后消息推送后台向各个移动终端反馈已接收到上报信息。Step S20: The message push background directly stores the received report information in the local server memory queue of the message push background, and then the message push background feeds back the received report information to each mobile terminal.
步骤S30,在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,消息推送后台还启动多个worker thread(工人线程)实时读取内存队列里存储的上报信息并存储在MQ存储组件(消息中间件)中。在worker Thread模式中,会逐个取出上报信息中的每一条消息进行处理。Step S30: While storing the reported information in the local server memory queue of the message push background, the message push background also starts multiple workers threads (worker threads) to read the reported information stored in the memory queue in real time and store it in the MQ storage component (Message middleware). In worker Thread mode, each message in the reported information is taken out one by one for processing.
当所有的消息全部读取完成后,工人线程会等待新的消息来。工人线程将读取的上报信息存储在MQ存储组件中,例如kafka(分布式、分区、可复制的提交日志服务)。kafka对上报消息保存时根据主题进行归类,其中,主题可以认为是一类消息,每个主题被分成多个分区,每个分区在存储层面是append.log文件。在kafka中,发送消息者称为Producer(生产者),在这里worker thread就相当于producer。Producer将上报消息发布到指定的主题中,同时Producer也能决定将此上报消息归属于哪个分区。任何发布到分区的消息都会被直接追加到log文件的尾部,每条消息在文件中的位置称为偏移量,偏移量为一个long型数字,偏移量唯一的标记一条消息。When all the messages are read, the worker thread will wait for new messages. The worker thread stores the read report information in the MQ storage component, such as kafka (distributed, partitioned, and reproducible submission log service). Kafka categorizes reported messages according to topics when they are saved. Topics can be considered as a type of message. Each topic is divided into multiple partitions, and each partition is an append.log file at the storage level. In Kafka, the person who sends the message is called the Producer, where the worker thread is equivalent to the producer. The Producer publishes the reported message to the specified topic, and the Producer can also determine which partition the reported message belongs to. Any message posted to the partition will be directly appended to the end of the log file. The position of each message in the file is called the offset. The offset is a long number, and the offset uniquely marks a message.
步骤S40,然后消息推送后台读取消费MQ存储组件中的上报信息中的每条消息,并且验证上报身份是否合法,以及对消息内容进行解密,若上报身 份合法则将消息保持到HDFS(分布式文件系统)中。Step S40, then the message push background reads each message in the reported information in the consumer MQ storage component, and verifies whether the reported identity is legal, and decrypts the message content. If the reported identity is legal, the message is saved to HDFS (distributed File system).
优选地,采用大数据平台来读取消费kafka中的上报信息中的每条消息,能够加快处理速度。Preferably, a big data platform is used to read each message in the reported information in the consumption kafka, which can speed up the processing speed.
进一步地,消息推送后台将接收到的上报信息直接存放在消息推送后台的本地服务器内存队列中的方法是:通过Map msgstats=new HashMap()将上报信息状态对象放到本地服务器内存的map内存集合中,其中,HashMap的内部是通过数组Entry[]来存放数据的,HashMap采用Map的key-value键值对接口,Entry类的定义是:Further, the method for the message push background to directly store the received report information in the local server memory queue of the message push background is: through Map msgstats=new HashMap(), the reported information status object is placed in the map memory collection of the local server memory Among them, HashMap internally stores data through the array Entry[]. HashMap uses the key-value interface of Map. The definition of Entry class is:
class Entry{class Entry{
Object keyObject key
Object valueObject value
Entry next;Entry next;
}}
若要添加到HashMap中的键值对对应的key已经存在HashMap中,则找到该键值对,用新的value取代旧的value,若要添加到HashMap中的键值对对应的key不在HashMap中,则将其添加到对应的链表中。If the key corresponding to the key-value pair added to the HashMap already exists in the HashMap, find the key-value pair and replace the old value with the new value. If the key corresponding to the key-value pair to be added to the HashMap is not in the HashMap , Then add it to the corresponding linked list.
进一步地,对于kafka的producer端,设置传送缓冲阈值,将消息缓存起来,待消息数量达到了传送缓冲阈值,才批量的发送给kafka;Further, for the producer side of Kafka, set the transmission buffer threshold, buffer the messages, and send them to Kafka in batches after the number of messages reaches the transmission buffer threshold;
对于kafka的consumer端,设置消费缓冲阈值,当kafka中的消息达到消费缓冲阈值,消息推送后台才批量取出多条消息。For the consumer side of Kafka, set the consumption buffer threshold. When the message in Kafka reaches the consumption buffer threshold, the message push background will fetch multiple messages in batches.
在一个可选实施例中,采用ETL(数据清洗)清洗程序进行数据清洗,以清除脏数据,其中,所述脏数据包括重复数据、残缺数据,无效数据、不一致数据,其中,在数据清洗之前先将消息按照重复数据、残缺数据,无效数据、不一致数据进行聚类。In an optional embodiment, data cleaning is performed using an ETL (data cleaning) cleaning program to remove dirty data, where the dirty data includes duplicate data, incomplete data, invalid data, and inconsistent data, wherein, before data cleaning First, the messages are clustered according to repeated data, incomplete data, invalid data, and inconsistent data.
在一个可选实施例中,重复数据清洗包括判断是否重复,删除重复数据,其中,采用多段比对的方式,以不少于3处的两个消息的相同位置的相同长度字段依次分别比对来判断两个消息是否是重复数据。具体说,判断是否重复可以是根据提取两个消息的相同位置的一段字段,并比对这两段字段是否完全相同,如果完全相同,则以一定的间隔(例如隔一个字符)提取两个消息的另一相同位置的一段字段,并比对这两段字段是否完全相同,如果完全相 同,则以一定的间隔提取两个消息的另一相同位置的一段字段,进一步比较,如相同则认为两段消息是重复数据,如不相同,则认定为不是重复数据。其中,优选地,三次提取的消息的相同位置分别为消息字段的首部,消息字段的尾部和消息字段的中部。In an optional embodiment, repeated data cleaning includes judging whether it is duplicated and deleting duplicate data, wherein a multi-segment comparison method is used to sequentially compare the same length fields in the same position of no less than 3 two messages. To determine whether the two messages are duplicate data. Specifically, judging whether to repeat can be based on extracting a segment of the same position of the two messages, and comparing whether the two segments are exactly the same, if they are exactly the same, extract the two messages at a certain interval (for example, one character) Another field in the same position of the two messages, and compare whether the two fields are exactly the same. If they are exactly the same, extract another field in the same position of the two messages at a certain interval and compare them further. If they are the same, the two fields are considered to be the same. Segment messages are repeated data. If they are not the same, they are deemed not to be repeated data. Wherein, preferably, the same positions of the messages extracted three times are the head of the message field, the tail of the message field, and the middle of the message field, respectively.
进一步地,对于以键值对的形式存储的数据,则首先采用以上方法判断键是否完全一致,然后再判断值是否完全一致。其中,键值对是以key=value的形式呈现。此处即是表示先对key来判断是否完全一致,然后在判断value是否完全一致,从而判断消息是否是重复消息。Further, for data stored in the form of key-value pairs, the above method is first used to determine whether the keys are completely consistent, and then whether the values are completely consistent. Among them, the key-value pair is presented in the form of key=value. Here it means that the key is first judged whether it is completely consistent, and then the value is judged whether it is completely consistent, so as to judge whether the message is a duplicate message.
在一个可选实施例中,在采用多段比对的方式比对两个消息之前,先根据两个消息的key和value组成的键值对向量来计算余弦相似度,并根据预设的相似度阈值来判断相似程度,如果高于相似度阈值,则认为是重复消息。具体说,先对所有的上报信息中的消息进行聚类,例如,把所有来自同一移动终端的消息归为一类,然后把每个消息的key和value组成键值对向量,计算两个消息的键值对向量的余弦相似度,如果高于相似度阈值,再采用不少于3处的两个消息的相同位置的相同长度字段依次分别比对来判断两个消息是否是重复数据,若相似度低于相似度阈值,则认为不是重复数据。In an optional embodiment, before the two messages are compared in a multi-segment comparison method, the cosine similarity is calculated according to the key-value pair vector composed of the key and value of the two messages, and the preset similarity is calculated The threshold is used to determine the degree of similarity. If it is higher than the threshold of similarity, it is considered a duplicate message. Specifically, first cluster all the messages in the reported information, for example, classify all messages from the same mobile terminal into one category, and then group the key and value of each message into a key-value pair vector to calculate two messages If the cosine similarity of the key-value pair vector is higher than the similarity threshold, the same length fields at the same position of the two messages at no less than 3 are used to compare in turn to determine whether the two messages are duplicate data. If the similarity is lower than the similarity threshold, it is considered not to be duplicate data.
例如,消息A和消息B的键值对组成的向量分别如下:For example, the vector of key-value pairs of message A and message B are as follows:
其中,d(A,B)表示消息A和消息B的余弦相似度;Among them, d(A, B) represents the cosine similarity of message A and message B;
key
Ai是表示消息A的键向量的第i个分量;
key Ai is the i-th component of the key vector representing message A;
value
Ai是表示消息A的值向量的第i个分量;
value Ai is the i-th component of the value vector representing message A;
key
Bi是表示消息B的键向量的第i个分量;
key Bi is the i-th component of the key vector representing message B;
value
Bi是表示消息B的值向量的第i个分量;
value Bi is the i-th component of the value vector representing the message B;
n表示消息A和消息B的键值对的数量。n represents the number of key-value pairs of message A and message B.
在计算了两个消息的余弦相似度后,若相似度高于相似度阈值,再采用不少于3处的两个消息的相同位置的相同长度字段依次分别比对来判断两个消息是否是重复数据。可以加快消息查重的速度。若相似度低于相似度阈值,则认为不是重复数据。After calculating the cosine similarity of the two messages, if the similarity is higher than the similarity threshold, no less than 3 two messages with the same length at the same position are compared in turn to determine whether the two messages are Duplicate data. Can speed up the speed of message checking. If the similarity is lower than the similarity threshold, it is considered not to be duplicate data.
参阅图2所示,是本申请电子装置的实施例的硬件架构示意图。本实施例中,所述电子装置2是一种能够按照事先设定或者存储的指令,自动进行数值计算和/或信息处理的设备。例如,可以是智能手机、平板电脑、笔记本电脑、台式计算机、机架式服务器、刀片式服务器、塔式服务器或机柜式服务器(包括独立的服务器,或者多个服务器所组成的服务器集群)等。如图2所示,所述电子装置2至少包括,但不限于,可通过系统总线相互通信连接的存储器21、处理器22、网络接口23。其中:所述存储器21至少包括一种类型的计算机非易失性可读存储介质,所述可读存储介质包括闪存、硬盘、多媒体卡、卡型存储器(例如,SD或DX存储器等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘等。在一些实施例中,所述存储器21可以是所述电子装置2的内部存储单元,例如该电子装置2的硬盘或内存。在另一些实施例中,所述存储器21也可以是所述电子装置2的外部存储设备,例如该电子装置2上配备的插接式硬盘,智能存储卡(Smart Media Card,SMC),安全数字(Secure Digital,SD)卡,闪存卡(Flash Card)等。当然,所述存储器21还可以既包括所述电子装置2的内部存储单元也包括其外部存储设备。本实施例中,所述存储器21通常用于存储安装于所述电子装置2的操作系统和各类应用软件,例如所述消息推送的用户消息状态上报处理程序代码等。此外,所述存储器21还可以用于暂时地存储已经输出或者将要输出的各类数据。Refer to FIG. 2, which is a schematic diagram of the hardware architecture of an embodiment of the electronic device of the present application. In this embodiment, the electronic device 2 is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions. For example, it can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster composed of multiple servers). As shown in FIG. 2, the electronic device 2 at least includes, but is not limited to, a memory 21, a processor 22, and a network interface 23 that can be communicatively connected to each other through a system bus. Wherein: the memory 21 includes at least one type of computer non-volatile readable storage medium, the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random Access memory (RAM), static random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. . In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or a memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, for example, a plug-in hard disk equipped on the electronic device 2, a smart media card (SMC), a secure digital (Secure Digital, SD) card, flash card (Flash Card), etc. Of course, the memory 21 may also include both the internal storage unit of the electronic device 2 and its external storage device. In this embodiment, the memory 21 is generally used to store the operating system and various application software installed in the electronic device 2, for example, the user message status report processing program code of the message push, etc. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or will be output.
所述处理器22在一些实施例中可以是中央处理器(Central Processing Unit,CPU)、控制器、微控制器、微处理器、或其他数据处理芯片。该处理器22通常用于控制所述电子装置2的总体操作,例如执行与所述电子装置2进行数据交互或者通信相关的控制和处理等。本实施例中,所述处理器22用于运行所述存储器21中存储的程序代码或者处理数据,例如运行所述的消息推送的用户消息状态上报处理程序等。The processor 22 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 22 is generally used to control the overall operation of the electronic device 2, for example, to perform data interaction or communication-related control and processing with the electronic device 2. In this embodiment, the processor 22 is configured to run the program code or processing data stored in the memory 21, for example, to run the user message status report processing program of the message push.
所述网络接口23可包括无线网络接口或有线网络接口,该网络接口23通常用于在所述电子装置2与其他电子装置之间建立通信连接。例如,所述网络接口23用于通过网络将所述电子装置2与推送平台相连,在所述电子装置2与推送平台之间建立数据传输通道和通信连接等。所述网络可以是企业内部网(Intranet)、互联网(Internet)、全球移动通讯系统(Global System of Mobile communication,GSM)、宽带码分多址(Wideband CodeDivision Multiple Access,WCDMA)、4G网络、5G网络、蓝牙(Bluetooth)、Wi-Fi等无线或有线网络。The network interface 23 may include a wireless network interface or a wired network interface, and the network interface 23 is generally used to establish a communication connection between the electronic device 2 and other electronic devices. For example, the network interface 23 is used to connect the electronic device 2 with a push platform through a network, and establish a data transmission channel and a communication connection between the electronic device 2 and the push platform. The network may be Intranet, Internet, Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network , Bluetooth (Bluetooth), Wi-Fi and other wireless or wired networks.
可选地,该电子装置2还可以包括显示器,显示器也可以称为显示屏或显示单元。在一些实施例中可以是LED显示器、液晶显示器、触控式液晶显示器以及有机发光二极管(Organic Light-Emitting Diode,OLED)显示器等。显示器用于显示在电子装置2中处理的信息以及用于显示可视化的用户界面。Optionally, the electronic device 2 may also include a display, and the display may also be called a display screen or a display unit. In some embodiments, it may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an organic light-emitting diode (OLED) display, etc. The display is used for displaying information processed in the electronic device 2 and for displaying a visualized user interface.
需要指出的是,图2仅示出了具有组件21-23的电子装置2,但是应理解的是,并不要求实施所有示出的组件,可以替代的实施更多或者更少的组件。It should be pointed out that FIG. 2 only shows the electronic device 2 with components 21-23, but it should be understood that it is not required to implement all of the illustrated components, and more or fewer components may be implemented instead.
包含可读存储介质的存储器21中可以包括操作系统、消息推送的用户消息状态上报处理程序50等。处理器22执行存储器21中消息推送的用户消息状态上报处理程序50时实现如下步骤:The memory 21 containing a readable storage medium may include an operating system, a user message status report processing program 50 for message push, and the like. The processor 22 implements the following steps when executing the user message status report processing program 50 of the message push in the memory 21:
步骤S10,在消息推送后台向各移动终端上的APP推送消息后,各APP向消息推送后台发送上报信息,所述上报信息中包含多条来自各个移动终端的被推送消息的APP返回的信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载。Step S10, after the message push background pushes messages to the APPs on each mobile terminal, each APP sends report information to the message push background, and the report information includes multiple pieces of information returned by the APPs that have pushed messages from each mobile terminal. The reported information includes at least the arrival of the message on the mobile terminal, the display of the message in the notification bar or banner, the user clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled.
步骤S20,消息推送后台将接收到的上报信息直接存放在消息推送后台的本地服务器内存队列中,然后消息推送后台向各个移动终端反馈已接收到上报信息。Step S20: The message push background directly stores the received report information in the local server memory queue of the message push background, and then the message push background feeds back the received report information to each mobile terminal.
步骤S30,在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,还启动多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中。在worker Thread模式中,会逐个取出上报信息中的每一条消息进行处理。Step S30, while storing the reported information in the memory queue of the local server in the message push background, multiple worker threads are also started to read the reported information stored in the memory queue in real time and store it in the MQ storage component. In worker Thread mode, each message in the reported information is taken out one by one for processing.
步骤S40,然后消息推送后台读取消费MQ存储组件中的上报信息中的每条消息,并且验证上报身份是否合法,以及对消息内容进行解密,若上报身 份合法则将消息保持到HDFS中。In step S40, the message push background reads each message in the reported information in the consumer MQ storage component, verifies whether the reported identity is legal, and decrypts the message content. If the reported identity is legal, the message is stored in HDFS.
在本实施例中,存储于存储器21中的所述消息推送的用户消息状态上报处理程序可以被分割为一个或者多个程序模块,所述一个或者多个程序模块被存储于存储器21中,并可由一个或多个处理器(本实施例为处理器22)所执行,以完成本申请。例如,图3示出了所述消息推送的用户消息状态上报处理程序的程序模块示意图,该实施例中,所述消息推送的用户消息状态上报处理程序50可以被分割为上报消息接收模块501、上报消息内存队列存储模块502、内存队列信息读取模块503、消息中间件读取模块504。其中,本申请所称的程序模块是指能够完成特定功能的一系列计算机程序指令段,比程序更适合于描述所述消息推送的用户消息状态上报处理程序在所述电子装置2中的执行过程。以下描述将具体介绍所述程序模块的具体功能。In this embodiment, the user message status report processing program of the message push stored in the memory 21 may be divided into one or more program modules, and the one or more program modules are stored in the memory 21, and It can be executed by one or more processors (the processor 22 in this embodiment) to complete the application. For example, FIG. 3 shows a schematic diagram of the program modules of the user message status report processing program of the message push. In this embodiment, the user message status report processing program 50 of the message push can be divided into a report message receiving module 501, Report message memory queue storage module 502, memory queue information reading module 503, and message middleware reading module 504. Among them, the program module referred to in this application refers to a series of computer program instruction segments that can complete specific functions, and is more suitable than a program to describe the execution process of the user message status report processing program of the message push in the electronic device 2 . The following description will specifically introduce the specific functions of the program modules.
其中,上报消息接收模块501用于在消息推送后台向各移动终端上的APP推送消息后,在各APP向消息推送后台发送上报信息时,接收上报消息。所述上报信息中包含多条来自各个移动终端的被推送消息的APP返回的信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载。The report message receiving module 501 is configured to receive the report message when each APP sends the report information to the message push background after pushing the message to the APP on each mobile terminal in the message push background. The reported information includes multiple pieces of information returned by the APP from which the message is pushed from each mobile terminal. The reported information includes at least the arrival of the message on the mobile terminal, the display of the message in the notification bar or banner, the user click, and whether the APP message notification is Blocking, whether the APP has been uninstalled.
上报消息内存队列存储模块502用于在消息推送后台将接收到的上报信息后将信息直接存放在消息推送后台的本地服务器内存队列中,然后消息推送后台向各个移动终端反馈已接收到上报信息。The reported message memory queue storage module 502 is used to directly store the received reported information in the local server memory queue of the message push background after the message push background, and then the message push background feedbacks the received report information to each mobile terminal.
内存队列信息读取模块503用于在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,还启动多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中。在worker Thread模式中,会逐个取出上报信息中的每一条消息进行处理。The memory queue information reading module 503 is used to store the reported information in the memory queue of the local server in the message push background, and also start multiple worker threads to read the reported information stored in the memory queue in real time and store it in the MQ storage component . In worker Thread mode, each message in the reported information is taken out one by one for processing.
消息中间件读取模块504用于读取消费MQ存储组件中的上报信息中的每条消息,并且验证上报身份是否合法,以及对消息内容进行解密,若上报身份合法则将消息保持到HDFS中。The message middleware reading module 504 is used to read each message in the reported information in the consumer MQ storage component, and verify whether the reported identity is legal, and decrypt the message content. If the reported identity is legal, the message is stored in HDFS .
进一步地,上报消息内存队列存储模块502将接收到的上报信息直接存放在消息推送后台的本地服务器内存队列中是通过Map msgstats=new HashMap()将上报信息状态对象放到本地服务器内存的map内存集合中,其中,HashMap的内部是通过数组Entry[]来存放数据的,HashMap采用Map的 key-value键值对接口,Entry类的定义是:Further, the report message memory queue storage module 502 directly stores the received report information in the local server memory queue of the message push background by using Map msgstats=new HashMap() to put the report information status object into the map memory of the local server memory In the collection, the inside of HashMap stores data through the array Entry[]. HashMap uses the key-value interface of Map. The definition of Entry class is:
class Entry{class Entry{
Object keyObject key
Object valueObject value
Entry next;Entry next;
}}
若要添加到HashMap中的键值对对应的key已经存在HashMap中,则找到该键值对,用新的value取代旧的value,若要添加到HashMap中的键值对对应的key不在HashMap中,则将其添加到对应的链表中。If the key corresponding to the key-value pair added to the HashMap already exists in the HashMap, find the key-value pair and replace the old value with the new value. If the key corresponding to the key-value pair to be added to the HashMap is not in the HashMap , Then add it to the corresponding linked list.
进一步地,还包括阈值比较模块505,对于kafka的producer端,设置传送缓冲阈值,将消息缓存起来,待消息数量达到了传送缓冲阈值,才批量的发送给kafka;Further, it also includes a threshold comparison module 505. For the producer side of Kafka, a transmission buffer threshold is set, and messages are buffered. After the number of messages reaches the transmission buffer threshold, they are sent to Kafka in batches;
对于kafka的consumer端,设置消费缓冲阈值,当kafka中的消息达到消费缓冲阈值,消息推送后台才批量取出多条消息。For the consumer side of Kafka, set the consumption buffer threshold. When the message in Kafka reaches the consumption buffer threshold, the message push background will fetch multiple messages in batches.
在一个可选实施例中,还包括数据清洗模块506,数据清洗模块506采用ETL(数据清洗)清洗程序进行数据清洗,以清除脏数据,其中,所述脏数据包括重复数据、残缺数据,无效数据、不一致数据,其中,在数据清洗之前先将消息按照重复数据、残缺数据,无效数据、不一致数据进行聚类。In an optional embodiment, a data cleaning module 506 is further included. The data cleaning module 506 uses an ETL (data cleaning) cleaning program to clean data to remove dirty data, where the dirty data includes duplicate data, incomplete data, and invalid data. Data and inconsistent data. Before data cleaning, messages are clustered according to repeated data, incomplete data, invalid data, and inconsistent data.
在一个可选实施例中,还包括重复数据判断模块507,重复数据判断模块507用于判断数据是否重复,删除重复数据,其中,采用多段比对的方式,以不少于3处的两个消息的相同位置的相同长度字段依次分别比对来判断两个消息是否是重复数据。具体说,判断是否重复可以是根据提取两个消息的相同位置的一段字段,并比对这两段字段是否完全相同,如果完全相同,则以一定的间隔(例如隔一个字符)提取两个消息的另一相同位置的一段字段,并比对这两段字段是否完全相同,如果完全相同,则以一定的间隔提取两个消息的另一相同位置的一段字段,进一步比较,如相同则认为两段消息是重复数据,如不相同,则认定为不是重复数据。其中,优选地,三次提取的消息的相同位置分别为消息字段的首部,消息字段的尾部和消息字段的中部。In an optional embodiment, it further includes a repeated data judging module 507. The repeated data judging module 507 is used for judging whether the data is repeated and deleting the repeated data. Among them, a multi-segment comparison method is adopted, and two of no less than 3 The same length fields in the same position of the message are compared sequentially to determine whether the two messages are duplicate data. Specifically, judging whether to repeat can be based on extracting a segment of the same position of the two messages, and comparing whether the two segments are exactly the same, if they are exactly the same, extract the two messages at a certain interval (for example, one character) Another field in the same position of the two messages, and compare whether the two fields are exactly the same. If they are exactly the same, extract another field in the same position of the two messages at a certain interval and compare them further. If they are the same, the two fields are considered to be the same. Segment messages are repeated data. If they are not the same, they are deemed not to be repeated data. Wherein, preferably, the same positions of the messages extracted three times are the head of the message field, the tail of the message field, and the middle of the message field, respectively.
进一步地,对于以键值对的形式存储的数据,则首先采用以上方法判断键是否完全一致,然后再判断值是否完全一致。其中,键值对是以key=value 的形式呈现。此处即是表示先对key来判断是否完全一致,然后在判断value是否完全一致,从而判断消息是否是重复消息。Further, for data stored in the form of key-value pairs, the above method is first used to determine whether the keys are completely consistent, and then whether the values are completely consistent. Among them, the key-value pair is presented in the form of key=value. Here it means that the key is first judged whether it is completely consistent, and then the value is judged whether it is completely consistent, so as to judge whether the message is a duplicate message.
在一个可选实施例中,重复数据判断模块507还根据两个消息的key和value组成的键值对向量来计算余弦相似度,在采用多段比对的方式比对两个消息之前,先根据预设的相似度阈值来判断相似程度,如果高于相似度阈值,则认为是重复消息。具体说,先对所有的上报信息中的消息进行聚类,例如,把所有来自同一移动终端的消息归为一类,然后把每个消息的key和value组成键值对向量,计算两个消息的键值对向量的余弦相似度,如果高于相似度阈值,再采用不少于3处的两个消息的相同位置的相同长度字段依次分别比对来判断两个消息是否是重复数据。可以加快消息查重的速度。若相似度低于相似度阈值,则认为不是重复数据。In an optional embodiment, the repeated data judging module 507 also calculates the cosine similarity based on the key-value pair vector composed of the key and value of the two messages. Before comparing the two messages in a multi-segment comparison method, first The preset similarity threshold is used to determine the degree of similarity. If it is higher than the similarity threshold, it is considered as a duplicate message. Specifically, first cluster all the messages in the reported information, for example, classify all messages from the same mobile terminal into one category, and then group the key and value of each message into a key-value pair vector to calculate two messages If the cosine similarity of the key-value pair vector is higher than the similarity threshold, no less than 3 two messages with the same length at the same position are compared in sequence to determine whether the two messages are duplicate data. Can speed up the speed of message checking. If the similarity is lower than the similarity threshold, it is considered not to be duplicate data.
本申请还提供一种消息推送的用户消息状态上报处理装置,包括:This application also provides a user message status report processing device for message push, including:
上报消息接收模块501,在消息推送后台向各移动终端上的APP推送消息后,消息推送后台接收各APP发送的上报信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载;The report message receiving module 501, after the message push background pushes messages to the APPs on each mobile terminal, the message push background receives the report information sent by each APP, and the report information includes at least the message reached the mobile terminal and the message is displayed in the notification bar or banner , User clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled;
上报消息内存队列存储模块502,用于消息推送后台将接收到的上报信息直接存放在消息推送后台的本地服务器内存队列中,然后消息推送后台向各个移动终端反馈已接收到上报信息;The report message memory queue storage module 502 is used for the message push background to directly store the received report information in the local server memory queue of the message push background, and then the message push background reports to each mobile terminal that the report information has been received;
内存队列信息读取模块503,用于在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,消息推送后台还启动多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中;The memory queue information reading module 503 is used to store the reported information in the memory queue of the local server in the message push background. At the same time, the message push background also starts multiple workers threads to read the reported information stored in the memory queue in real time and store it in MQ storage component;
消息中间件读取模块504,用于消息推送后台读取消费MQ存储组件中的上报信息中的每条消息,并且验证上报身份是否合法,以及对消息内容进行解密,若上报身份合法则将消息保持到HDFS中。The message middleware reading module 504 is used for the message push background to read each message in the reported information in the consumer MQ storage component, verify whether the reported identity is legal, and decrypt the message content. If the reported identity is legal, the message Keep it in HDFS.
此外,本申请实施例还提出一种计算机非易失性可读存储介质,所述计算机非易失性可读存储介质可以是硬盘、多媒体卡、SD卡、闪存卡、SMC、只读存储器(ROM)、可擦除可编程只读存储器(EPROM)、便携式紧致盘只读存储器(CD-ROM)、USB存储器等等中的任意一种或者几种的任意组合。所述计算机非易失性可读存储介质中包括消息推送的用户消息状态上报处理程序 等,所述消息推送的用户消息状态上报处理程序50被处理器22执行时实现如下操作:In addition, the embodiment of the present application also proposes a computer non-volatile readable storage medium. The computer non-volatile readable storage medium may be a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory ( ROM), erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, etc., or any combination of several. The computer non-volatile readable storage medium includes a user message status report processing program for message push, etc. The user message status report processing program 50 for message push is executed by the processor 22 to implement the following operations:
步骤S10,在消息推送后台向各移动终端上的APP推送消息后,各APP向消息推送后台发送上报信息,所述上报信息中包含多条来自各个移动终端的被推送消息的APP返回的信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载。Step S10, after the message push background pushes messages to the APPs on each mobile terminal, each APP sends report information to the message push background, and the report information includes multiple pieces of information returned by the APPs that have pushed messages from each mobile terminal. The reported information includes at least the arrival of the message on the mobile terminal, the display of the message in the notification bar or banner, the user clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled.
步骤S20,消息推送后台将接收到的上报信息直接存放在消息推送后台的本地服务器内存队列中,然后消息推送后台向各个移动终端反馈已接收到上报信息。Step S20: The message push background directly stores the received report information in the local server memory queue of the message push background, and then the message push background feeds back the received report information to each mobile terminal.
步骤S30,在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,消息推送后台还启动多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中。在worker Thread模式中,会逐个取出上报信息中的每一条消息进行处理。Step S30, while storing the reported information in the memory queue of the local server in the message push background, the message push background also starts multiple workers threads to read the reported information stored in the memory queue in real time and store it in the MQ storage component. In worker Thread mode, each message in the reported information is taken out one by one for processing.
步骤S40,然后消息推送后台读取消费MQ存储组件中的上报信息中的每条消息,并且验证上报身份是否合法,以及对消息内容进行解密,若上报身份合法则将消息保持到HDFS中。In step S40, the message push background reads each message in the reported information in the consumer MQ storage component, and verifies whether the reported identity is legal, and decrypts the message content. If the reported identity is legal, the message is stored in HDFS.
本申请之计算机非易失性可读存储介质的具体实施方式与上述消息推送的用户消息状态上报处理方法以及电子装置2的具体实施方式大致相同,在此不再赘述。The specific implementation manner of the computer non-volatile readable storage medium of the present application is substantially the same as the specific implementation manner of the user message status report processing method for message push and the electronic device 2 described above, and will not be repeated here.
以上所述仅为本申请的优选实施例,并不用于限制本申请,对于本领域的技术人员来说,本申请可以有各种更改和变化。凡在本申请的精神和原则之内,所作的任何修改、等同替换、改进等,均应包含在本申请的保护范围之内。The foregoing descriptions are only preferred embodiments of the application, and are not intended to limit the application. For those skilled in the art, the application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the protection scope of this application.
Claims (20)
- 一种消息推送的用户消息状态上报处理方法,应用于电子装置,其特征在于,包括:A user message status report processing method for message push, applied to an electronic device, and is characterized in that it includes:在消息推送后台向各移动终端上的APP推送消息后,消息推送后台接收各APP发送的上报信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载;After the message push background pushes messages to the APPs on each mobile terminal, the message push background receives the reported information sent by each APP. The reported information includes at least the arrival of the message on the mobile terminal, the message displayed in the notification bar or banner, user clicks, and APP Whether the message notification is blocked and whether the APP has been uninstalled;消息推送后台将接收到的上报信息存放在消息推送后台的本地服务器内存队列中,然后消息推送后台向各个移动终端反馈已接收到上报信息;The message push background stores the received report information in the local server memory queue of the message push background, and then the message push background sends feedback to each mobile terminal that the reported information has been received;在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,消息推送后台还启动多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中;While storing the reported information in the memory queue of the local server in the message push background, the message push background also starts multiple workers threads to read the reported information stored in the memory queue in real time and store it in the MQ storage component;消息推送后台读取消费MQ存储组件中的上报信息中的每条消息,并且验证上报身份是否合法,以及对消息内容进行解密,若上报身份合法则将消息保持到HDFS中。The message push background reads each message in the reported information in the consumer MQ storage component, verifies whether the reported identity is legal, and decrypts the content of the message. If the reported identity is legal, the message is stored in HDFS.
- 根据权利要求1所述的消息推送的用户消息状态上报处理方法,其特征在于:The user message status report processing method for message push according to claim 1, characterized in that:消息推送后台将接收到的上报信息存放在消息推送后台的本地服务器内存队列中的方法是:通过Map msgstats=new HashMap()将上报信息状态对象放到本地服务器内存的map内存集合中,The method for the message push background to store the received report information in the memory queue of the local server in the message push background is: through Map msgstats=new HashMap(), the reported information status object is placed in the map memory collection of the local server memory.其中,HashMap的内部是通过数组Entry[]来存放数据,HashMap采用Map的key-value键值对接口,Among them, HashMap internally stores data through the array Entry[], and HashMap uses Map’s key-value key-value interface,若要添加到HashMap中的键值对对应的key已经存在HashMap中,则找到该键值对,用新的value取代旧的value,若要添加到HashMap中的键值对对应的key不在HashMap中,则将其添加到对应的链表中。If the key corresponding to the key-value pair added to the HashMap already exists in the HashMap, find the key-value pair and replace the old value with the new value. If the key corresponding to the key-value pair to be added to the HashMap is not in the HashMap , Then add it to the corresponding linked list.
- 根据权利要求1所述的消息推送的用户消息状态上报处理方法,其特征在于:The user message status report processing method for message push according to claim 1, characterized in that:对于worker thread的producer端,设置传送缓冲阈值,将消息缓存起来,待消息数量达到了传送缓冲阈值,才批量的发送给MQ存储组件;For the producer side of the worker thread, set the transmission buffer threshold, buffer the messages, and send them to the MQ storage component in batches after the number of messages reaches the transmission buffer threshold;对于worker thread的consumer端,设置消费缓冲阈值,当MQ存储组件中的消息达到消费缓冲阈值,消息推送后台才批量取出多条消息。For the consumer side of the worker thread, set the consumption buffer threshold. When the message in the MQ storage component reaches the consumption buffer threshold, the message push background will fetch multiple messages in batches.
- 根据权利要求1所述的消息推送的用户消息状态上报处理方法,其特征在于:The user message status report processing method for message push according to claim 1, characterized in that:采用ETL清洗程序进行数据清洗,以清除脏数据,其中,所述脏数据包括重复数据、残缺数据,无效数据、不一致数据,其中,在数据清洗之前先将消息按照重复数据、残缺数据,无效数据、不一致数据进行聚类。The ETL cleaning program is used for data cleaning to remove dirty data. The dirty data includes duplicate data, incomplete data, invalid data, and inconsistent data. Before data cleaning, the message is classified as duplicate data, incomplete data, and invalid data. , Clustering inconsistent data.
- 根据权利要求4所述的消息推送的用户消息状态上报处理方法,其特征在于:The user message status report processing method for message push according to claim 4, characterized in that:先根据两个消息的key和value组成的键值对向量来计算余弦相似度,并根据预设的相似度阈值来判断相似程度,如果高于相似度阈值,再以两个消息的消息字段的首部、消息字段的尾部和消息字段的中部的相同长度字段依次分别比对来最终确定两个消息是否是重复数据。First calculate the cosine similarity according to the key-value pair vector composed of the key and value of the two messages, and judge the similarity according to the preset similarity threshold. If it is higher than the similarity threshold, then use the message field of the two messages. The header, the tail of the message field, and the same length fields in the middle of the message field are compared in sequence to finally determine whether the two messages are duplicate data.
- 一种消息推送的用户消息状态上报处理装置,其特征在于,包括:A user message status report processing device for message push, which is characterized in that it comprises:上报消息接收模块,在消息推送后台向各移动终端上的APP推送消息后,消息推送后台接收各APP发送的上报信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载;The report message receiving module, after the message push background pushes the message to the APP on each mobile terminal, the message push background receives the report information sent by each APP, and the report information includes at least the message reached the mobile terminal, the message is displayed in the notification bar or banner, User clicks, whether the APP message notification is blocked, and whether the APP has been uninstalled;上报消息内存队列存储模块,用于消息推送后台将接收到的上报信息直接存放在消息推送后台的本地服务器内存队列中,然后消息推送后台向各个移动终端反馈已接收到上报信息;The report message memory queue storage module is used for the message push background to directly store the received report information in the local server memory queue of the message push background, and then the message push background reports to each mobile terminal that the reported information has been received;内存队列信息读取模块,用于在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,消息推送后台还启动多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中;The memory queue information reading module is used to store the reported information in the local server memory queue of the message push background. At the same time, the message push background also starts multiple workers thread to read the reported information stored in the memory queue in real time and store it in MQ Storage components;消息中间件读取模块,用于消息推送后台读取消费MQ存储组件中的上报信息中的每条消息,并且验证上报身份是否合法,以及对消息内容进行解密,若上报身份合法则将消息保持到HDFS中。The message middleware reading module is used for the message push background to read each message in the reported information in the consumer MQ storage component, and verify whether the reported identity is legal, and decrypt the message content. If the reported identity is legal, keep the message To HDFS.
- 根据权利要求6所述的消息推送的用户消息状态上报处理装置,其特征在于,还包括阈值比较模块,The user message status report processing device for message push according to claim 6, characterized in that it further comprises a threshold comparison module,用于对于worker thread的producer端,设置传送缓冲阈值,将消息缓存起 来,待消息数量达到了传送缓冲阈值,才批量的发送给MQ存储组件;For the producer side of the worker thread, set the transmission buffer threshold and buffer the messages. After the number of messages reaches the transmission buffer threshold, they will be sent to the MQ storage component in batches;对于worker thread的consumer端,设置消费缓冲阈值,当MQ存储组件中的消息达到消费缓冲阈值,消息推送后台才批量取出多条消息。For the consumer side of the worker thread, set the consumption buffer threshold. When the message in the MQ storage component reaches the consumption buffer threshold, the message push background will fetch multiple messages in batches.
- 根据权利要求6所述的消息推送的用户消息状态上报处理装置,其特征在于,还包括数据清洗模块,用于采用ETL清洗程序进行数据清洗,以清除脏数据,其中,所述脏数据包括重复数据、残缺数据,无效数据、不一致数据,其中,在数据清洗之前先将消息按照重复数据、残缺数据,无效数据、不一致数据进行聚类。The user message status report processing device for message push according to claim 6, characterized in that it further comprises a data cleaning module, which is used to clean data by using an ETL cleaning program to remove dirty data, wherein the dirty data includes duplicates Data, incomplete data, invalid data, and inconsistent data. Before data cleaning, the message is clustered according to repeated data, incomplete data, invalid data, and inconsistent data.
- 根据权利要求8所述的消息推送的用户消息状态上报处理装置,其特征在于,还包括重复数据判断模块,判断是否重复采用多段比对的方式,以不少于3处的两个消息的相同位置的相同长度字段依次分别比对来判断两个消息是否是重复数据。The user message status report processing device for message push according to claim 8, characterized in that it further comprises a repeated data judging module, which judges whether to repeatedly use a multi-segment comparison method, and the two messages in no less than 3 places are identical. The positions of the same length fields are compared sequentially to determine whether the two messages are duplicate data.
- 根据权利要求9所述的消息推送的用户消息状态上报处理装置,先根据两个消息的key和value组成的键值对向量来计算余弦相似度,并根据预设的相似度阈值来判断相似程度,如果高于相似度阈值,再以两个消息的消息字段的首部、消息字段的尾部和消息字段的中部的相同长度字段依次分别比对来最终确定两个消息是否是重复数据。According to the user message status report processing device for message push according to claim 9, the cosine similarity is calculated according to the key-value pair vector composed of the key and value of the two messages, and the similarity is judged according to the preset similarity threshold If it is higher than the similarity threshold, the header of the message field, the tail of the message field, and the same length field in the middle of the message field of the two messages are compared in sequence to finally determine whether the two messages are duplicate data.
- 一种电子装置,其特征在于,该电子装置包括:存储器和处理器,所述存储器中存储有消息推送的用户消息状态上报处理程序,所述消息推送的用户消息状态上报处理程序被所述处理器执行时实现如下步骤:An electronic device, characterized in that the electronic device includes a memory and a processor, the memory stores a user message status report processing program for message push, and the user message status report processing program for message push is processed by the The following steps are implemented when the device is executed:在消息推送后台向各移动终端上的APP推送消息后,消息推送后台接收各APP发送的上报信息,所述上报信息至少包括消息到达移动终端、消息展示在通知栏或横幅、用户点击情况、APP消息通知是否被屏蔽、APP是否已卸载;After the message push background pushes messages to the APPs on each mobile terminal, the message push background receives the reported information sent by each APP. The reported information includes at least the arrival of the message on the mobile terminal, the message displayed in the notification bar or banner, user clicks, and APP Whether the message notification is blocked and whether the APP has been uninstalled;消息推送后台将接收到的上报信息直接存放在消息推送后台的本地服务器内存队列中,然后消息推送后台向各个移动终端反馈已接收到上报信息;The message push background directly stores the received report information in the local server memory queue of the message push background, and then the message push background sends feedback to each mobile terminal that the reported information has been received;在将上报信息存放在消息推送后台的本地服务器内存队列中的同时,消息推送后台还启动多个worker thread实时读取内存队列里存储的上报信息并存储在MQ存储组件中;While storing the reported information in the memory queue of the local server in the message push background, the message push background also starts multiple workers threads to read the reported information stored in the memory queue in real time and store it in the MQ storage component;然后消息推送后台读取消费MQ存储组件中的上报信息中的每条消息,并 且验证上报身份是否合法,以及对消息内容进行解密,若上报身份合法则将消息保持到HDFS中。Then the message push background reads each message in the reported information in the consumer MQ storage component, and verifies whether the reported identity is legal, and decrypts the message content. If the reported identity is legal, the message is stored in HDFS.
- 根据权利要求11所述的电子装置,其特征在于:所述消息推送的用户消息状态上报处理程序被所述处理器执行时还实现:The electronic device according to claim 11, wherein the user message status report processing program of the message push is also implemented when the processor is executed:对于worker thread的producer端,设置传送缓冲阈值,将消息缓存起来,待消息数量达到了传送缓冲阈值,才批量的发送给MQ存储组件;For the producer side of the worker thread, set the transmission buffer threshold, buffer the messages, and send them to the MQ storage component in batches after the number of messages reaches the transmission buffer threshold;对于worker thread的consumer端,设置消费缓冲阈值,当MQ存储组件中的消息达到消费缓冲阈值,消息推送后台才批量取出多条消息。For the consumer side of the worker thread, set the consumption buffer threshold. When the message in the MQ storage component reaches the consumption buffer threshold, the message push background will fetch multiple messages in batches.
- 根据权利要求11所述的电子装置,其特征在于:所述消息推送的用户消息状态上报处理程序被所述处理器执行时还实现:The electronic device according to claim 11, wherein the user message status report processing program of the message push is also implemented when the processor is executed:采用ETL清洗程序进行数据清洗,以清除脏数据,其中,所述脏数据包括重复数据、残缺数据,无效数据、不一致数据,其中,在数据清洗之前先将消息按照重复数据、残缺数据,无效数据、不一致数据进行聚类。The ETL cleaning program is used for data cleaning to remove dirty data. The dirty data includes duplicate data, incomplete data, invalid data, and inconsistent data. Before data cleaning, the message is classified as duplicate data, incomplete data, and invalid data. , Clustering inconsistent data.
- 根据权利要求13所述的电子装置,其特征在于:所述消息推送的用户消息状态上报处理程序被所述处理器执行时还实现:The electronic device according to claim 13, wherein the user message status report processing program of the message push is also implemented when the processor is executed:重复数据的清洗包括判断是否重复,删除重复数据,其中,判断是否重复采用多段比对的方式,以不少于3处的两个消息的相同位置的相同长度字段依次分别比对来判断两个消息是否是重复数据。The cleaning of duplicate data includes judging whether it is duplicated and deleting duplicate data. Among them, judging whether to repeatedly use multiple comparisons, and compare the same length fields in the same position of two messages at no less than 3 to determine the two. Whether the message is duplicate data.
- 根据权利要求14所述的电子装置,所述消息推送的用户消息状态上报处理程序被所述处理器执行时还实现:According to the electronic device of claim 14, the user message status report processing program of the message push is further implemented when being executed by the processor:先根据两个消息的key和value组成的键值对向量来计算余弦相似度,并根据预设的相似度阈值来判断相似程度,如果高于相似度阈值,再以两个消息的消息字段的首部、消息字段的尾部和消息字段的中部的相同长度字段依次分别比对来最终确定两个消息是否是重复数据。First calculate the cosine similarity according to the key-value pair vector composed of the key and value of the two messages, and judge the similarity according to the preset similarity threshold. If it is higher than the similarity threshold, then use the message field of the two messages. The header, the tail of the message field, and the same length fields in the middle of the message field are compared in sequence to finally determine whether the two messages are duplicate data.
- 一种计算机非易失性可读存储介质,其特征在于,所述计算机非易失性可读存储介质存储有计算机程序,所述计算机程序包括程序指令,所述程序指令被处理器执行时,实现权利要求1所述的消息推送的用户消息状态上报处理方法。A computer non-volatile readable storage medium, wherein the computer non-volatile readable storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, The user message status report processing method for message push of claim 1 is realized.
- 根据权利要求16所述的计算机非易失性可读存储介质,其特征在于:所述程序指令被处理器执行时,还实现:The computer non-volatile readable storage medium according to claim 16, wherein when the program instructions are executed by the processor, they further implement:对于worker thread的producer端,设置传送缓冲阈值,将消息缓存起来,待消息数量达到了传送缓冲阈值,才批量的发送给MQ存储组件;For the producer side of the worker thread, set the transmission buffer threshold, buffer the messages, and send them to the MQ storage component in batches after the number of messages reaches the transmission buffer threshold;对于worker thread的consumer端,设置消费缓冲阈值,当MQ存储组件中的消息达到消费缓冲阈值,消息推送后台才批量取出多条消息。For the consumer side of the worker thread, set the consumption buffer threshold. When the message in the MQ storage component reaches the consumption buffer threshold, the message push background will fetch multiple messages in batches.
- 根据权利要求16所述的计算机非易失性可读存储介质,其特征在于:所述程序指令被处理器执行时,还实现:The computer non-volatile readable storage medium according to claim 16, wherein when the program instructions are executed by the processor, they further implement:采用ETL清洗程序进行数据清洗,以清除脏数据,其中,所述脏数据包括重复数据、残缺数据,无效数据、不一致数据,其中,在数据清洗之前先将消息按照重复数据、残缺数据,无效数据、不一致数据进行聚类。The ETL cleaning program is used for data cleaning to remove dirty data. The dirty data includes duplicate data, incomplete data, invalid data, and inconsistent data. Before data cleaning, the message is classified as duplicate data, incomplete data, and invalid data. , Clustering inconsistent data.
- 根据权利要求16所述的计算机非易失性可读存储介质,其特征在于:所述程序指令被处理器执行时,还实现:The computer non-volatile readable storage medium according to claim 16, wherein when the program instructions are executed by the processor, they further implement:重复数据的清洗包括判断是否重复,删除重复数据,其中,判断是否重复采用多段比对的方式,以不少于3处的两个消息的相同位置的相同长度字段依次分别比对来判断两个消息是否是重复数据。The cleaning of duplicate data includes judging whether it is duplicated and deleting duplicate data. Among them, judging whether to repeatedly use multiple comparisons, and compare the same length fields in the same position of two messages at no less than 3 to determine the two. Whether the message is duplicate data.
- 根据权利要求19所述的计算机非易失性可读存储介质,其特征在于:所述程序指令被处理器执行时,还实现:The computer non-volatile readable storage medium according to claim 19, wherein when the program instruction is executed by the processor, it further implements:先根据两个消息的key和value组成的键值对向量来计算余弦相似度,并根据预设的相似度阈值来判断相似程度,如果高于相似度阈值,再以两个消息的消息字段的首部、消息字段的尾部和消息字段的中部的相同长度字段依次分别比对来最终确定两个消息是否是重复数据。First calculate the cosine similarity according to the key-value pair vector composed of the key and value of the two messages, and judge the similarity according to the preset similarity threshold. If it is higher than the similarity threshold, then use the message field of the two messages. The header, the tail of the message field, and the same length fields in the middle of the message field are compared in sequence to finally determine whether the two messages are duplicate data.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910164935.6 | 2019-03-05 | ||
CN201910164935.6A CN109873904B (en) | 2019-03-05 | 2019-03-05 | User message state reporting processing method, device and storage medium for message pushing |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020177384A1 true WO2020177384A1 (en) | 2020-09-10 |
Family
ID=66919780
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/117979 WO2020177384A1 (en) | 2019-03-05 | 2019-11-13 | Method and apparatus for reporting and processing user message status of message pushing, and storage medium |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN109873904B (en) |
WO (1) | WO2020177384A1 (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112131180A (en) * | 2020-09-25 | 2020-12-25 | 京东数字科技控股股份有限公司 | Data reporting method and device and storage medium |
CN112148387A (en) * | 2020-10-14 | 2020-12-29 | 中国平安人寿保险股份有限公司 | Method and device for preloading feedback information, computer equipment and storage medium |
CN112199495A (en) * | 2020-11-13 | 2021-01-08 | 北京明略软件系统有限公司 | Personnel message notification method, device, electronic equipment and computer readable storage medium |
CN114172956A (en) * | 2021-11-29 | 2022-03-11 | 上海金仕达软件科技有限公司 | Intelligent information pushing method and system |
CN114244798A (en) * | 2021-11-23 | 2022-03-25 | 彩讯科技股份有限公司 | Card message sending method, system, equipment and storage medium |
CN114979270A (en) * | 2022-05-25 | 2022-08-30 | 上海交通大学 | Message publishing method and system suitable for RDMA network |
US11922026B2 (en) | 2022-02-16 | 2024-03-05 | T-Mobile Usa, Inc. | Preventing data loss in a filesystem by creating duplicates of data in parallel, such as charging data in a wireless telecommunications network |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109873904B (en) * | 2019-03-05 | 2021-08-03 | 平安科技(深圳)有限公司 | User message state reporting processing method, device and storage medium for message pushing |
CN110809050B (en) * | 2019-11-08 | 2022-11-29 | 智者四海(北京)技术有限公司 | Personalized pushing system and method based on streaming computing |
CN111290696A (en) * | 2020-02-26 | 2020-06-16 | 平安银行股份有限公司 | Flow control method and device for application program assembly |
CN113347238A (en) * | 2021-05-26 | 2021-09-03 | 湖南大学 | Message partitioning method, system, device and storage medium based on block chain |
CN114268934A (en) * | 2021-12-24 | 2022-04-01 | 深圳市创维软件有限公司 | BLE GATT-based flow control method, device and storage medium |
CN116016652A (en) * | 2022-11-30 | 2023-04-25 | 雄狮汽车科技(南京)有限公司 | Message component connection method, medium, device and apparatus |
CN116719806A (en) * | 2023-08-11 | 2023-09-08 | 尚特杰电力科技有限公司 | Big data calculation analysis system |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104330973A (en) * | 2014-09-23 | 2015-02-04 | 小米科技有限责任公司 | Equipment control method and apparatus |
CN105468645A (en) * | 2014-09-10 | 2016-04-06 | 阿里巴巴集团控股有限公司 | Information push apparatus and push method and mobile terminal |
US20170126617A1 (en) * | 2014-04-04 | 2017-05-04 | Beijing Qihoo Technology Company Limited | Method, device and system for processing dns cache information |
CN109144992A (en) * | 2017-06-15 | 2019-01-04 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of data storage |
CN109873904A (en) * | 2019-03-05 | 2019-06-11 | 平安科技(深圳)有限公司 | User of message status report process method, device and the storage medium of message push |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8254528B2 (en) * | 2007-07-03 | 2012-08-28 | At&T Intellectual Property I, L.P. | System and method to push messages indicating status of trouble reports in a telecommunications network |
KR101272077B1 (en) * | 2011-05-16 | 2013-06-12 | 에스케이텔레콤 주식회사 | System and method for providing push service |
CN103297326B (en) * | 2013-05-29 | 2017-12-29 | 深圳Tcl新技术有限公司 | The method and apparatus of message push |
CN103856393B (en) * | 2013-07-09 | 2017-11-03 | 上海携程商务有限公司 | Distributed message middleware system and its operation method based on database |
US9325654B2 (en) * | 2014-02-28 | 2016-04-26 | Aol Inc. | Systems and methods for optimizing message notification timing based on electronic content consumption associated with a geographic location |
CN104023072A (en) * | 2014-06-13 | 2014-09-03 | 中国民航信息网络股份有限公司 | Device and method for pushing data |
CN104092767B (en) * | 2014-07-21 | 2017-06-13 | 北京邮电大学 | A kind of publish/subscribe system and its method of work for increasing message queue model |
US9342388B1 (en) * | 2015-12-02 | 2016-05-17 | International Business Machines Corporation | Dynamic queue alias |
CN105471714A (en) * | 2015-12-09 | 2016-04-06 | 百度在线网络技术(北京)有限公司 | Message processing method and device |
CN105761039A (en) * | 2016-02-17 | 2016-07-13 | 华迪计算机集团有限公司 | Method for processing express delivery information big data |
CN105978968A (en) * | 2016-05-11 | 2016-09-28 | 山东合天智汇信息技术有限公司 | Real-time transmission processing method, server and system of mass data |
CN106487902A (en) * | 2016-10-19 | 2017-03-08 | 华迪计算机集团有限公司 | A kind of method of data capture based on message-oriented middleware and system |
CN108021434A (en) * | 2017-12-06 | 2018-05-11 | 浪潮软件集团有限公司 | Data processing apparatus, method of processing data thereof, medium, and storage controller |
CN108874555A (en) * | 2018-05-23 | 2018-11-23 | 福建天泉教育科技有限公司 | A kind of method and device for writing message to message-oriented middleware |
-
2019
- 2019-03-05 CN CN201910164935.6A patent/CN109873904B/en active Active
- 2019-11-13 WO PCT/CN2019/117979 patent/WO2020177384A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170126617A1 (en) * | 2014-04-04 | 2017-05-04 | Beijing Qihoo Technology Company Limited | Method, device and system for processing dns cache information |
CN105468645A (en) * | 2014-09-10 | 2016-04-06 | 阿里巴巴集团控股有限公司 | Information push apparatus and push method and mobile terminal |
CN104330973A (en) * | 2014-09-23 | 2015-02-04 | 小米科技有限责任公司 | Equipment control method and apparatus |
CN109144992A (en) * | 2017-06-15 | 2019-01-04 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of data storage |
CN109873904A (en) * | 2019-03-05 | 2019-06-11 | 平安科技(深圳)有限公司 | User of message status report process method, device and the storage medium of message push |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112131180A (en) * | 2020-09-25 | 2020-12-25 | 京东数字科技控股股份有限公司 | Data reporting method and device and storage medium |
CN112131180B (en) * | 2020-09-25 | 2024-02-06 | 京东科技控股股份有限公司 | Data reporting method, device and storage medium |
CN112148387A (en) * | 2020-10-14 | 2020-12-29 | 中国平安人寿保险股份有限公司 | Method and device for preloading feedback information, computer equipment and storage medium |
CN112199495A (en) * | 2020-11-13 | 2021-01-08 | 北京明略软件系统有限公司 | Personnel message notification method, device, electronic equipment and computer readable storage medium |
CN114244798A (en) * | 2021-11-23 | 2022-03-25 | 彩讯科技股份有限公司 | Card message sending method, system, equipment and storage medium |
CN114244798B (en) * | 2021-11-23 | 2024-06-07 | 彩讯科技股份有限公司 | Card message sending method, system, equipment and storage medium |
CN114172956A (en) * | 2021-11-29 | 2022-03-11 | 上海金仕达软件科技有限公司 | Intelligent information pushing method and system |
CN114172956B (en) * | 2021-11-29 | 2024-03-15 | 上海金仕达软件科技股份有限公司 | Intelligent information pushing method and system |
US11922026B2 (en) | 2022-02-16 | 2024-03-05 | T-Mobile Usa, Inc. | Preventing data loss in a filesystem by creating duplicates of data in parallel, such as charging data in a wireless telecommunications network |
CN114979270A (en) * | 2022-05-25 | 2022-08-30 | 上海交通大学 | Message publishing method and system suitable for RDMA network |
CN114979270B (en) * | 2022-05-25 | 2023-08-25 | 上海交通大学 | Message publishing method and system suitable for RDMA network |
Also Published As
Publication number | Publication date |
---|---|
CN109873904A (en) | 2019-06-11 |
CN109873904B (en) | 2021-08-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020177384A1 (en) | Method and apparatus for reporting and processing user message status of message pushing, and storage medium | |
US10262032B2 (en) | Cache based efficient access scheduling for super scaled stream processing systems | |
US10409650B2 (en) | Efficient access scheduling for super scaled stream processing systems | |
CN111046034B (en) | Method and system for managing memory data and maintaining data in memory | |
US10140472B2 (en) | Multi-level privacy evaluation | |
CN107704539B (en) | Method and device for large-scale text information batch structuring | |
US20160378552A1 (en) | Automatic scaling of computing resources using aggregated metrics | |
US9882830B2 (en) | Architecture for metrics aggregation without service partitioning | |
US9910755B2 (en) | Retrieval of authoritative measurement data from in-memory datastores | |
US9882982B2 (en) | Datastore for aggregated measurements for metrics | |
CN111813573B (en) | Communication method of management platform and robot software and related equipment thereof | |
US10031901B2 (en) | Narrative generation using pattern recognition | |
CN110704381A (en) | Data analysis method, device and storage medium | |
WO2019153598A1 (en) | Customer risk level management method, server and computer readable storage medium | |
US20190332603A1 (en) | Model score recall for machine learning systems and applications | |
WO2016134639A1 (en) | Data block processing method and device | |
CN111258819A (en) | Data acquisition method, device and system for MySQL database backup file | |
CN107908525B (en) | Alarm processing method, equipment and readable storage medium | |
CN112969198A (en) | Data transmission method, terminal and storage medium | |
WO2019062020A1 (en) | Asynchronous task unified processing method and apparatus, and storage medium | |
CN113850664A (en) | Data anomaly detection method and data reporting service | |
US11055267B2 (en) | Handling time series index updates at ingestion | |
US9256634B2 (en) | Resuming big data transformations | |
CN113220530A (en) | Data quality monitoring method and platform | |
US20160378800A1 (en) | Aggregation of metrics data with fine granularity |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19918402 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19918402 Country of ref document: EP Kind code of ref document: A1 |