CN109873904B - User message state reporting processing method, device and storage medium for message pushing - Google Patents
User message state reporting processing method, device and storage medium for message pushing Download PDFInfo
- Publication number
- CN109873904B CN109873904B CN201910164935.6A CN201910164935A CN109873904B CN 109873904 B CN109873904 B CN 109873904B CN 201910164935 A CN201910164935 A CN 201910164935A CN 109873904 B CN109873904 B CN 109873904B
- Authority
- CN
- China
- Prior art keywords
- message
- information
- messages
- data
- message pushing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 5
- 238000000034 method Methods 0.000 claims abstract description 29
- 239000000872 buffer Substances 0.000 claims description 17
- 238000012545 processing Methods 0.000 claims description 15
- 239000013598 vector Substances 0.000 claims description 13
- 230000005540 biological transmission Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 5
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000005406 washing Methods 0.000 claims 1
- 238000012827 research and development Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 4
- 238000005192 partition Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/725—Cordless telephones
Abstract
The scheme relates to research and development management, and provides a user message state reporting processing method, a device and a storage medium for message pushing, wherein the method comprises the following steps: each APP sends reporting information to a message pushing background; the message pushing background stores the received reporting information in a local server memory queue of the message pushing background, and the message pushing background feeds back the received reporting information to the user client; when the reported information is stored in a memory queue of a local server of a message pushing background, a plurality of worker threads are started to read the reported information stored in the memory queue in real time and store the reported information in an MQ storage component, then the message pushing background reads each message in the reported information in a consumption MQ storage component and verifies whether the reported identity is legal, and if the reported identity is legal, the message is kept in the HDFS. The scheme accelerates the storage speed and cannot cause the sudden increase of the access pressure on the message pushing background at a certain moment.
Description
Technical Field
The present invention relates to research and development management, and in particular, to a method, an apparatus, and a storage medium for processing a message status report of a user in a message push.
Background
The message push received by the APP push on a common user mobile phone triggers the report information to be transmitted to a message push background, wherein the report information comprises state information of the message reaching the mobile phone, the message displayed on a notification bar or banner, a click message, APP message notification shielding, the unloaded APP and the like. After receiving the state reporting requests, the message pushing background verifies whether the reporting identity is legal or not, decrypts the reported content, acquires which MsgID and which APP user reported the message state according to the decrypted content, and stores the message state in a relational database on a hard disk of the message pushing background, wherein the relational database belongs to external storage. The method has the problems that when the user scale reaches hundred million level and the information is simultaneously pushed to the user, the APP user can simultaneously report the information state to the information pushing background in a large amount, one pushing information is reported to the information pushing background in an average of 2-3 states, the information pushing background receives the reported information and stores the reported information in a relational database (such as mysql and oracle), the service processing is slow, and finally the service is unavailable.
Disclosure of Invention
In order to solve the above technical problem, the present invention provides a method for reporting and processing a user message status of message push, which is applied to an electronic device, and comprises:
after a message pushing background pushes messages to APPs on each mobile terminal, each APP sends reporting information to the message pushing background, wherein the reporting information comprises information returned by the APPs of the pushed messages from each mobile terminal, and the reporting information at least comprises the information of the messages arriving at the mobile terminal, the information displayed on a notification bar or banner, the clicking condition of a user, whether APP message notification is shielded or not and whether the APP is unloaded or not;
the message pushing background stores the received reporting information in a local server memory queue of the message pushing background, and then the message pushing background feeds back the received reporting information to each mobile terminal;
when the reported information is stored in a memory queue of a local server of a message pushing background, the message pushing background also starts a plurality of worker threads to read the reported information stored in the memory queue in real time and store the reported information in an MQ storage component;
and then the message pushing background reads each message in the reported information in the consumption MQ storage component, verifies whether the reported identity is legal or not, decrypts the message content, and keeps the message in the HDFS if the reported identity is legal.
Preferably, the MQ storage component employs kafka.
Preferably, the method for storing the received report information in the memory queue of the local server of the message pushing background by the message pushing background is as follows: the reported information state object is put into the Map memory set of the local server memory through Map msgstats ═ new HashMap (),
the HashMap stores data through an array Entry [ ], the HashMap adopts a key-value key value of the Map to interface, and the definition of the Entry class is as follows:
class Entry{
Object key
Object value
Entry next；
}
if the key corresponding to the key pair to be added to the HashMap already exists in the HashMap, the key pair is found, the old value is replaced by the new value, and if the key corresponding to the key pair to be added to the HashMap does not exist in the HashMap, the key pair is added to the corresponding linked list.
Preferably, a transmission buffer threshold value is set for a producer end of the worker thread, messages are buffered, and the messages are sent to the kafka in batch when the number of the messages reaches the transmission buffer threshold value;
and (3) setting a consumption buffer threshold value for a consumer end of the worker thread, and when the messages in the kafka reach the consumption buffer threshold value, the message pushing background takes out a plurality of messages in batch.
Preferably, an ETL cleaning program is used to clean the dirty data, where the dirty data includes duplicate data, incomplete data, invalid data, and inconsistent data, and before the data cleaning, the messages are clustered according to the duplicate data, the incomplete data, the invalid data, and the inconsistent data.
Preferably, the data de-duplication comprises judging whether the two messages are duplicated or not and deleting the duplicated data, wherein the judging whether the two messages are duplicated or not is carried out by sequentially and respectively comparing fields with the same length at the same positions of the two messages which are not less than 3 positions in a multi-segment comparison mode.
Preferably, before the multi-segment comparison is adopted to judge whether the data is repeated, cosine similarity is calculated according to a key value pair vector consisting of the key and the value of the two messages, the similarity degree is judged according to a preset similarity threshold, and if the similarity degree is higher than the similarity threshold, the multi-segment comparison is further adopted to judge whether the data is repeated.
The present invention also provides an electronic device, comprising: the device comprises a memory and a processor, wherein a user message state reporting processing program pushed by a message is stored in the memory, and when the user message state reporting processing program pushed by the message is executed by the processor, the following steps are realized:
after a message pushing background pushes messages to APPs on each mobile terminal, each APP sends reporting information to the message pushing background, wherein the reporting information comprises information returned by the APPs of the pushed messages from each mobile terminal, and the reporting information at least comprises the information of the messages arriving at the mobile terminal, the information displayed on a notification bar or banner, the clicking condition of a user, whether APP message notification is shielded or not and whether the APP is unloaded or not;
the message pushing background directly stores the received reporting information in a local server memory queue of the message pushing background, and then the message pushing background feeds back the received reporting information to each mobile terminal;
when the reported information is stored in a memory queue of a local server of a message pushing background, the message pushing background also starts a plurality of worker threads to read the reported information stored in the memory queue in real time and store the reported information in an MQ storage component;
and then the message pushing background reads each message in the reported information in the consumption MQ storage component, verifies whether the reported identity is legal or not, decrypts the message content, and keeps the message in the HDFS if the reported identity is legal.
Preferably, the MQ storage component employs kafka.
The present invention further provides a computer-readable storage medium, where a computer program is stored, where the computer program includes program instructions, and when the program instructions are executed by a processor, the method for processing reporting of a user message status of message pushing as described above is implemented.
The message state information is asynchronously processed, the message state information is stored in the memory queue of the server, the storage speed is high, the reported information stored in the memory queue is read in real time by using a plurality of worker threads and is stored in the MQ storage component while the message state information is stored in the memory queue, and the sudden increase of the access pressure on the message pushing background at a certain moment can not be caused. And the message pushing background reads each message in the reported information in the consumption MQ storage component, and keeps the message in the HDFS after the reported identity is legal. When a large number of clients need to report states to a message pushing background, the method and the device accelerate the storage speed and enable the clients to respond quickly.
Drawings
The above features and technical advantages of the present invention will become more apparent and readily appreciated from the following description of the embodiments thereof taken in conjunction with the accompanying drawings.
Fig. 1 is a schematic flow chart of a method for reporting a user message status in message push according to an embodiment of the present invention;
FIG. 2 is a diagram of a hardware architecture of an electronic device according to an embodiment of the invention;
fig. 3 is a block diagram of a user message status reporting handler for message pushing according to an embodiment of the present invention.
Detailed Description
Embodiments of a method, an apparatus, and a storage medium for reporting a user message status according to the message push of the present invention are described below with reference to the accompanying drawings. Those of ordinary skill in the art will recognize that the described embodiments can be modified in various different ways, or combinations thereof, without departing from the spirit and scope of the present invention. Accordingly, the drawings and description are illustrative in nature and not intended to limit the scope of the claims. Furthermore, in the present description, the drawings are not to scale and like reference numerals refer to like parts.
Fig. 1 is a flowchart illustrating a method for reporting a user message status in message push according to an embodiment of the present invention. The method is applied to the electronic device and comprises the following steps:
step S10, after the message pushing background pushes messages to the APPs on each mobile terminal, each APP sends reporting information to the message pushing background, the reporting information comprises information returned by the APPs of the pushed messages from each mobile terminal, and the reporting information at least comprises the message arrival at the mobile terminal, the message display on a notification bar or banner, the user click condition, whether the APP message notification is shielded, and whether the APP is unloaded.
Step S20, the message pushing background directly stores the received reporting information in the local server memory queue of the message pushing background, and then the message pushing background feeds back the received reporting information to each mobile terminal.
Step S30, while storing the reporting information in the memory queue of the local server in the message pushing background, the message pushing background also starts multiple worker threads to read the reporting information stored in the memory queue in real time and store the information in the MQ storage component (message middleware). In the worker Thread mode, each message in the reported information is taken out one by one for processing.
When all the messages are read completely, the worker thread waits for a new message. The worker thread stores the read reported information in an MQ storage component, such as kafka (distributed, partitioned, replicable commit Log service). kafka classifies reported messages according to topics when the reported messages are stored, wherein the topics can be regarded as a type of messages, each topic is divided into a plurality of partitions, and each partition is an apend. In kafka, the sender is called Producer, where worker thread is equivalent to Producer. The Producer issues the report message to a specified topic, and the Producer can also decide to which partition the report message belongs. Any message issued to a partition is directly appended to the end of the log file, the position of each message in the file is called an offset, the offset is a long type number, and the offset uniquely marks one message.
And step S40, the message pushing background reads each message in the reported information in the consumption MQ storage component, verifies whether the reported identity is legal, decrypts the message content, and keeps the message in the HDFS (distributed file system) if the reported identity is legal.
Preferably, a big data platform is adopted to read each message in the reported information in the consumption kafka, so that the processing speed can be increased.
Further, the method for directly storing the received report information in the memory queue of the local server of the message pushing background by the message pushing background is as follows: the method comprises the following steps of putting a reported information state object into a Map memory set of a local server memory through Map msgstats (new HashMap ()), wherein data are stored in the HashMap through an array Entry [ ], the HashMap adopts a key-value key value of the Map to interface, and the Entry class is defined as follows:
class Entry{
Object key
Object value
Entry next；
}
if the key corresponding to the key pair to be added to the HashMap already exists in the HashMap, the key pair is found, the old value is replaced by the new value, and if the key corresponding to the key pair to be added to the HashMap does not exist in the HashMap, the key pair is added to the corresponding linked list.
Further, a transmission buffer threshold value is set for the producer end of the kafka, messages are buffered, and the messages are sent to the kafka in batch when the number of the messages reaches the transmission buffer threshold value;
and setting a consumption buffer threshold value for a consumer end of the kafka, and when the messages in the kafka reach the consumption buffer threshold value, the message pushing background takes out a plurality of messages in batch.
In an optional embodiment, an ETL (data cleansing) cleansing program is used to perform data cleansing to remove dirty data, wherein the dirty data includes duplicate data, incomplete data, invalid data, and inconsistent data, and wherein messages are clustered according to the duplicate data, the incomplete data, the invalid data, and the inconsistent data before data cleansing.
In an optional embodiment, the data de-duplication comprises judging whether the two messages are duplicated or not, and deleting the duplicated data, wherein the two messages are judged whether to be duplicated or not by sequentially and respectively comparing fields with the same length at the same position of not less than 3 positions in a multi-segment comparison mode. Specifically, the determining whether to repeat may be according to extracting a field at the same position of the two messages, comparing whether the two fields are completely the same, if the two fields are completely the same, extracting a field at another same position of the two messages at a certain interval (for example, every other character), comparing whether the two fields are completely the same, if the two fields are completely the same, extracting a field at another same position of the two messages at a certain interval, further comparing, if the two fields are the same, determining that the two messages are the repeated data, and if the two messages are not the same, determining that the two messages are not the repeated data. Wherein, preferably, the same positions of the messages extracted three times are respectively the head of the message field, the tail of the message field and the middle of the message field.
Further, for the data stored in the form of key value pairs, firstly, the above method is adopted to judge whether the keys are completely consistent, and then, whether the values are completely consistent is judged. Wherein the key-value pair is presented in the form of key value. Here, it means that whether the message is a duplicate message is determined by determining whether the key completely matches and then determining whether the value completely matches.
In an optional embodiment, before comparing the two messages in a multi-segment comparison manner, the cosine similarity is calculated according to a key-value pair vector consisting of the key and the value of the two messages, the similarity degree is judged according to a preset similarity threshold, and if the similarity degree is higher than the similarity threshold, the two messages are considered to be repeated messages. Specifically, clustering is performed on all messages in the reported information, for example, all messages from the same mobile terminal are classified into one class, then a key and a value of each message form a key-value pair vector, cosine similarity of the key-value pair vectors of the two messages is calculated, if the similarity is higher than a similarity threshold, fields with the same length at the same positions of the two messages which are not less than 3 are sequentially and respectively compared to judge whether the two messages are repeated data, and if the similarity is lower than the similarity threshold, the two messages are not the repeated data.
For example, the vectors composed of key-value pairs for message a and message B are as follows:
wherein d (A, B) represents the cosine similarity of message A and message B;
keyAiis the ith component of the key vector representing message a;
valueAiis the i-th component of the value vector representing message a;
keyBiis the ith component of the key vector representing message B;
valueBiis the i-th component of the value vector representing message B;
n represents the number of key-value pairs for message a and message B.
After the cosine similarity of the two messages is calculated, if the similarity is higher than a similarity threshold, the fields with the same length at the same positions of the two messages at least 3 positions are sequentially and respectively compared to judge whether the two messages are repeated data. The speed of checking the duplicate of the message can be accelerated. If the similarity is lower than the similarity threshold, the data is not the repeated data.
Fig. 2 is a schematic diagram of a hardware architecture of an electronic device according to an embodiment of the invention. In the present embodiment, the electronic device 2 is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction. For example, the server may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a rack server (including an independent server or a server cluster composed of a plurality of servers). As shown in fig. 2, the electronic device 2 includes at least, but is not limited to, a memory 21, a processor 22, and a network interface 23, which are communicatively connected to each other through a system bus. Wherein: the memory 21 includes at least one type of computer-readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 21 may be an internal storage unit of the electronic device 2, such as a hard disk or a memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic apparatus 2, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a flash Card (FlashCard), or the like, provided on the electronic apparatus 2. Of course, the memory 21 may also comprise both an internal memory unit of the electronic apparatus 2 and an external memory device thereof. In this embodiment, the memory 21 is generally configured to store an operating system and various types of application software installed in the electronic device 2, for example, a user message status reporting processing program code for the message push. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is generally configured to control the overall operation of the electronic apparatus 2, such as performing data interaction or communication related control and processing with the electronic apparatus 2. In this embodiment, the processor 22 is configured to run a program code or process data stored in the memory 21, for example, run the user message state reporting processing program for message pushing, and the like.
The network interface 23 may comprise a wireless network interface or a wired network interface, and the network interface 23 is generally used for establishing communication connection between the electronic device 2 and other electronic devices. For example, the network interface 23 is configured to connect the electronic device 2 to a push platform through a network, establish a data transmission channel and a communication connection between the electronic device 2 and the push platform, and the like. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), Wi-Fi, and the like.
Optionally, the electronic device 2 may further comprise a display, which may also be referred to as a display screen or a display unit. In some embodiments, the display device can be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an Organic Light-Emitting Diode (OLED) display, and the like. The display is used for displaying information processed in the electronic apparatus 2 and for displaying a visualized user interface.
It is noted that fig. 2 only shows the electronic device 2 with components 21-23, but it is to be understood that not all shown components are required to be implemented, and that more or less components may be implemented instead.
The memory 21 containing the readable storage medium may include an operating system, a user message status report handler 50 for message push, and the like. The processor 22 executes the user message status report processing program 50 for message push in the memory 21 to implement the following steps:
step S10, after the message pushing background pushes messages to the APPs on each mobile terminal, each APP sends reporting information to the message pushing background, the reporting information comprises information returned by the APPs of the pushed messages from each mobile terminal, and the reporting information at least comprises the message arrival at the mobile terminal, the message display on a notification bar or banner, the user click condition, whether the APP message notification is shielded, and whether the APP is unloaded.
Step S20, the message pushing background directly stores the received reporting information in the local server memory queue of the message pushing background, and then the message pushing background feeds back the received reporting information to each mobile terminal.
Step S30, while storing the reporting information in the memory queue of the local server in the message push background, a plurality of worker threads are started to read the reporting information stored in the memory queue in real time and store the reading information in the MQ storage component. In the worker Thread mode, each message in the reported information is taken out one by one for processing.
And step S40, the message pushing background reads each message in the reported information in the consumption MQ storage component, verifies whether the reported identity is legal, decrypts the message content, and keeps the message in the HDFS if the reported identity is legal.
In this embodiment, the user message status reporting processing program for message pushing stored in the memory 21 may be divided into one or more program modules, and the one or more program modules are stored in the memory 21 and can be executed by one or more processors (in this embodiment, the processor 22) to complete the present invention. For example, fig. 3 shows a program module schematic diagram of the user message status reporting processing program for message pushing, in this embodiment, the user message status reporting processing program 50 for message pushing may be divided into a reporting message receiving module 501, a reporting message memory queue storing module 502, a memory queue information reading module 503, and a message middleware reading module 504. The program module referred to in the present invention refers to a series of computer program instruction segments capable of performing specific functions, and is more suitable for describing the execution process of the user message status reporting processing program for message pushing in the electronic device 2 than a program. The following description will specifically describe specific functions of the program modules.
The report message receiving module 501 is configured to receive a report message when each APP sends a report message to the message push background after the message push background pushes the message to the APPs on each mobile terminal. The report information at least comprises information returned by APP of the pushed messages from each mobile terminal, the information arrives at the mobile terminal, the information is displayed in a notification bar or banner, the user clicks, the APP message notification is shielded, and the APP is unloaded.
The reported message memory queue storage module 502 is configured to store the received reported information in the local server memory queue of the message pushing background directly after the message pushing background stores the information in the local server memory queue of the message pushing background, and then the message pushing background feeds back the received reported information to each mobile terminal.
The memory queue information reading module 503 is configured to start a plurality of worker threads to read the reporting information stored in the memory queue in real time and store the reading information in the MQ storage component while storing the reporting information in the memory queue of the local server of the message push background. In the worker Thread mode, each message in the reported information is taken out one by one for processing.
The message middleware reading module 504 is configured to read each message in the reporting information in the consuming MQ storage component, verify whether the reporting identity is legal, decrypt the message content, and keep the message in the HDFS if the reporting identity is legal.
Further, the reporting message memory queue storage module 502 directly stores the received reporting information in the local server memory queue of the message pushing background, and places the reporting information state object into the Map memory set of the local server memory through Map msgstates ═ new HashMap (), where the HashMap stores data inside through array Entry [ ], and the HashMap uses the key-value key value pair interface of the Map, and the definition of the Entry class is:
class Entry{
Object key
Object value
Entry next；
}
if the key corresponding to the key pair to be added to the HashMap already exists in the HashMap, the key pair is found, the old value is replaced by the new value, and if the key corresponding to the key pair to be added to the HashMap does not exist in the HashMap, the key pair is added to the corresponding linked list.
Further, the system further comprises a threshold comparison module 505, which sets a transmission buffer threshold for the producer end of kafka, buffers the messages, and sends the messages to the kafka in batch when the number of the messages reaches the transmission buffer threshold;
and setting a consumption buffer threshold value for a consumer end of the kafka, and when the messages in the kafka reach the consumption buffer threshold value, the message pushing background takes out a plurality of messages in batch.
In an optional embodiment, the data cleansing module 506 further includes a data cleansing module 506, and the data cleansing module 506 performs data cleansing using an ETL (data cleansing) cleansing program to remove dirty data, where the dirty data includes duplicate data, incomplete data, invalid data, and inconsistent data, and where messages are clustered according to the duplicate data, the incomplete data, the invalid data, and the inconsistent data before the data cleansing.
In an optional embodiment, the system further includes a repeated data determining module 507, where the repeated data determining module 507 is configured to determine whether data is repeated and delete repeated data, where a multi-segment comparison manner is adopted, and fields with the same length at the same position of two messages at no less than 3 are sequentially and respectively compared to determine whether the two messages are repeated data. Specifically, the determining whether to repeat may be according to extracting a field at the same position of the two messages, comparing whether the two fields are completely the same, if the two fields are completely the same, extracting a field at another same position of the two messages at a certain interval (for example, every other character), comparing whether the two fields are completely the same, if the two fields are completely the same, extracting a field at another same position of the two messages at a certain interval, further comparing, if the two fields are the same, determining that the two messages are the repeated data, and if the two messages are not the same, determining that the two messages are not the repeated data. Wherein, preferably, the same positions of the messages extracted three times are respectively the head of the message field, the tail of the message field and the middle of the message field.
Further, for the data stored in the form of key value pairs, firstly, the above method is adopted to judge whether the keys are completely consistent, and then, whether the values are completely consistent is judged. Wherein the key-value pair is presented in the form of key value. Here, it means that whether the message is a duplicate message is determined by determining whether the key completely matches and then determining whether the value completely matches.
In an optional embodiment, the repeated data determining module 507 further calculates cosine similarity according to a key-value pair vector composed of the key and the value of the two messages, determines the similarity according to a preset similarity threshold before comparing the two messages in a multi-segment comparison manner, and determines that the two messages are repeated if the similarity is higher than the similarity threshold. Specifically, clustering is performed on all messages in the reported information, for example, all messages from the same mobile terminal are classified into one class, then a key and a value of each message form a key-value pair vector, cosine similarity of the key-value pair vectors of the two messages is calculated, and if the similarity is higher than a similarity threshold, fields with the same length at the same positions of the two messages at not less than 3 positions are sequentially and respectively compared to judge whether the two messages are repeated data. The speed of checking the duplicate of the message can be accelerated. If the similarity is lower than the similarity threshold, the data is not the repeated data.
Furthermore, the embodiment of the present invention also provides a computer-readable storage medium, which may be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), a portable compact disc read only memory (CD-ROM), a USB memory, and the like. The computer-readable storage medium includes a user message state reporting handler for message pushing, and the like, and when executed by the processor 22, the user message state reporting handler 50 for message pushing implements the following operations:
step S10, after the message pushing background pushes messages to the APPs on each mobile terminal, each APP sends reporting information to the message pushing background, the reporting information comprises information returned by the APPs of the pushed messages from each mobile terminal, and the reporting information at least comprises the message arrival at the mobile terminal, the message display on a notification bar or banner, the user click condition, whether the APP message notification is shielded, and whether the APP is unloaded.
Step S20, the message pushing background directly stores the received reporting information in the local server memory queue of the message pushing background, and then the message pushing background feeds back the received reporting information to each mobile terminal.
Step S30, while storing the reporting information in the memory queue of the local server in the message pushing background, the message pushing background also starts a plurality of worker threads to read the reporting information stored in the memory queue in real time and store the reading information in the MQ storage component. In the worker Thread mode, each message in the reported information is taken out one by one for processing.
And step S40, the message pushing background reads each message in the reported information in the consumption MQ storage component, verifies whether the reported identity is legal, decrypts the message content, and keeps the message in the HDFS if the reported identity is legal.
The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the above-mentioned user message status reporting processing method for message pushing and the specific implementation of the electronic device 2, and will not be described herein again.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A processing method for reporting user message state of message push is applied to an electronic device, and is characterized by comprising the following steps:
after a message pushing background pushes messages to APPs on each mobile terminal, each APP sends reporting information to the message pushing background, wherein the reporting information comprises information returned by the APPs of the pushed messages from each mobile terminal, and the reporting information at least comprises the information of the messages arriving at the mobile terminal, the information displayed on a notification bar or banner, the clicking condition of a user, whether APP message notification is shielded or not and whether the APP is unloaded or not;
the message pushing background stores the received reporting information in a local server memory queue of the message pushing background, and then the message pushing background feeds back the received reporting information to each mobile terminal;
when the reported information is stored in a memory queue of a local server of a message pushing background, the message pushing background also starts a plurality of worker threads to read the reported information stored in the memory queue in real time and store the reported information in an MQ storage component;
and then the message pushing background reads each message in the reported information in the consumption MQ storage component, verifies whether the reported identity is legal or not, decrypts the message content, and keeps the message in the HDFS if the reported identity is legal.
2. The method of claim 1, wherein the method comprises:
the MQ storage component employs kafka.
3. The method of claim 2, wherein the method comprises:
the method for storing the received reported information in the local server memory queue of the message pushing background by the message pushing background is as follows: the reported information state object is put into the Map memory set of the local server memory through Map msgstats ═ new HashMap (),
the HashMap stores data through an array Entry [ ], the HashMap adopts a key-value key value of the Map to interface, and the definition of the Entry class is as follows:
class Entry{
Object key
Object value
Entry next；
}
if the key corresponding to the key pair to be added to the HashMap already exists in the HashMap, the key pair is found, the old value is replaced by the new value, and if the key corresponding to the key pair to be added to the HashMap does not exist in the HashMap, the key pair is added to the corresponding linked list.
4. The method of claim 1, wherein the method comprises:
setting a transmission buffer threshold value for a producer end of worker thread, caching messages, and sending the messages to kafka in batches when the number of the messages reaches the transmission buffer threshold value;
and (3) setting a consumption buffer threshold value for a consumer end of the worker thread, and when the messages in the kafka reach the consumption buffer threshold value, the message pushing background takes out a plurality of messages in batch.
5. The method for processing the report of the status of the user message pushed by the message as claimed in claim 3, wherein:
and carrying out data cleaning by adopting an ETL cleaning program to remove dirty data, wherein the dirty data comprises repeated data, incomplete data, invalid data and inconsistent data, and clustering the messages according to the repeated data, the incomplete data, the invalid data and the inconsistent data before the data cleaning.
6. The method of claim 5, wherein the method comprises:
the repeated data washing comprises the steps of judging whether repeated data is obtained or not and deleting the repeated data, wherein the judgment whether repeated data is obtained or not is carried out in a multi-section comparison mode, and whether the two messages are the repeated data or not is judged by sequentially and respectively comparing fields with the same length at the same positions of the two messages which are not less than 3 positions.
7. The method for processing the report of the status of the user message pushed by the message as claimed in claim 6, wherein:
before the multi-section comparison mode is adopted to judge whether the data are repeated, cosine similarity is calculated according to a key value pair vector formed by the key and the value of the two messages, the similarity degree is judged according to a preset similarity threshold value, and if the similarity degree is higher than the similarity threshold value, the multi-section comparison mode is further adopted to judge whether the data are repeated.
8. An electronic device, comprising: the device comprises a memory and a processor, wherein a user message state reporting processing program pushed by a message is stored in the memory, and when the user message state reporting processing program pushed by the message is executed by the processor, the following steps are realized:
after a message pushing background pushes messages to APPs on each mobile terminal, each APP sends reporting information to the message pushing background, wherein the reporting information comprises information returned by the APPs of the pushed messages from each mobile terminal, and the reporting information at least comprises the information of the messages arriving at the mobile terminal, the information displayed on a notification bar or banner, the clicking condition of a user, whether APP message notification is shielded or not and whether the APP is unloaded or not;
the message pushing background directly stores the received reporting information in a local server memory queue of the message pushing background, and then the message pushing background feeds back the received reporting information to each mobile terminal;
when the reported information is stored in a memory queue of a local server of a message pushing background, the message pushing background also starts a plurality of worker threads to read the reported information stored in the memory queue in real time and store the reported information in an MQ storage component;
and then the message pushing background reads each message in the reported information in the consumption MQ storage component, verifies whether the reported identity is legal or not, decrypts the message content, and keeps the message in the HDFS if the reported identity is legal.
9. The electronic device of claim 8, wherein the MQ storage component employs kafka.
10. A computer-readable storage medium, storing a computer program, the computer program comprising program instructions, which when executed by a processor, implement the method for processing user message status report of message push in accordance with any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910164935.6A CN109873904B (en) | 2019-03-05 | 2019-03-05 | User message state reporting processing method, device and storage medium for message pushing |
| PCT/CN2019/117979 WO2020177384A1 (en) | 2019-03-05 | 2019-11-13 | Method and apparatus for reporting and processing user message status of message pushing, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910164935.6A CN109873904B (en) | 2019-03-05 | 2019-03-05 | User message state reporting processing method, device and storage medium for message pushing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109873904A CN109873904A (en) | 2019-06-11 |
| CN109873904B true CN109873904B (en) | 2021-08-03 |
Family
ID=66919780
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910164935.6A Active CN109873904B (en) | 2019-03-05 | 2019-03-05 | User message state reporting processing method, device and storage medium for message pushing |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109873904B (en) |
| WO (1) | WO2020177384A1 (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109873904B (en) * | 2019-03-05 | 2021-08-03 | 平安科技（深圳）有限公司 | User message state reporting processing method, device and storage medium for message pushing |
| CN110809050B (en) * | 2019-11-08 | 2022-11-29 | 智者四海（北京）技术有限公司 | Personalized pushing system and method based on streaming computing |
| CN112131180B (en) * | 2020-09-25 | 2024-02-06 | 京东科技控股股份有限公司 | Data reporting method, device and storage medium |
| CN112148387A (en) * | 2020-10-14 | 2020-12-29 | 中国平安人寿保险股份有限公司 | Method and device for preloading feedback information, computer equipment and storage medium |
| CN112199495A (en) * | 2020-11-13 | 2021-01-08 | 北京明略软件系统有限公司 | Personnel message notification method, device, electronic equipment and computer readable storage medium |
| CN113347238A (en) * | 2021-05-26 | 2021-09-03 | 湖南大学 | Message partitioning method, system, device and storage medium based on block chain |
| CN114244798A (en) * | 2021-11-23 | 2022-03-25 | 彩讯科技股份有限公司 | Card message sending method, system, equipment and storage medium |
| CN114172956B (en) * | 2021-11-29 | 2024-03-15 | 上海金仕达软件科技股份有限公司 | Intelligent information pushing method and system |
| US11922026B2 (en) | 2022-02-16 | 2024-03-05 | T-Mobile Usa, Inc. | Preventing data loss in a filesystem by creating duplicates of data in parallel, such as charging data in a wireless telecommunications network |
| CN114979270B (en) * | 2022-05-25 | 2023-08-25 | 上海交通大学 | Message publishing method and system suitable for RDMA network |
| CN116719806A (en) * | 2023-08-11 | 2023-09-08 | 尚特杰电力科技有限公司 | Big data calculation analysis system |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20120128013A (en) * | 2011-05-16 | 2012-11-26 | 에스케이텔레콤 주식회사 | System and method for providing push service |
| US8625746B2 (en) * | 2007-07-03 | 2014-01-07 | At&T Intellectual Property I, L.P. | System and method to push messages indicating status of trouble reports in a telecommunications network |
| CN103856393A (en) * | 2013-07-09 | 2014-06-11 | 携程计算机技术（上海）有限公司 | Distributed message-oriented middleware system based on database and operating method thereof |
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| EP2913982A1 (en) * | 2014-02-28 | 2015-09-02 | AOL, Inc. | Systems and methods for optimizing message notification timing based on electronic content consumption associated with a geographic location |
| CN105471714A (en) * | 2015-12-09 | 2016-04-06 | 百度在线网络技术（北京）有限公司 | Message processing method and device |
| CN105761039A (en) * | 2016-02-17 | 2016-07-13 | 华迪计算机集团有限公司 | Method for processing express delivery information big data |
| CN108874555A (en) * | 2018-05-23 | 2018-11-23 | 福建天泉教育科技有限公司 | A kind of method and device for writing message to message-oriented middleware |
| CN109144992A (en) * | 2017-06-15 | 2019-01-04 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of data storage |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103297326B (en) * | 2013-05-29 | 2017-12-29 | 深圳Tcl新技术有限公司 | The method and apparatus of message push |
| CN103957239B (en) * | 2014-04-04 | 2015-09-09 | 北京奇虎科技有限公司 | DNS cache information processing method, equipment and system |
| CN104023072A (en) * | 2014-06-13 | 2014-09-03 | 中国民航信息网络股份有限公司 | Device and method for pushing data |
| CN105468645A (en) * | 2014-09-10 | 2016-04-06 | 阿里巴巴集团控股有限公司 | Information push apparatus and push method and mobile terminal |
| CN104330973B (en) * | 2014-09-23 | 2017-02-15 | 小米科技有限责任公司 | Equipment control method and apparatus |
| US9342388B1 (en) * | 2015-12-02 | 2016-05-17 | International Business Machines Corporation | Dynamic queue alias |
| CN105978968A (en) * | 2016-05-11 | 2016-09-28 | 山东合天智汇信息技术有限公司 | Real-time transmission processing method, server and system of mass data |
| CN106487902A (en) * | 2016-10-19 | 2017-03-08 | 华迪计算机集团有限公司 | A kind of method of data capture based on message-oriented middleware and system |
| CN108021434A (en) * | 2017-12-06 | 2018-05-11 | 浪潮软件集团有限公司 | Data processing apparatus, method of processing data thereof, medium, and storage controller |
| CN109873904B (en) * | 2019-03-05 | 2021-08-03 | 平安科技（深圳）有限公司 | User message state reporting processing method, device and storage medium for message pushing |
-
2019
- 2019-03-05 CN CN201910164935.6A patent/CN109873904B/en active Active
- 2019-11-13 WO PCT/CN2019/117979 patent/WO2020177384A1/en active Application Filing
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8625746B2 (en) * | 2007-07-03 | 2014-01-07 | At&T Intellectual Property I, L.P. | System and method to push messages indicating status of trouble reports in a telecommunications network |
| KR20120128013A (en) * | 2011-05-16 | 2012-11-26 | 에스케이텔레콤 주식회사 | System and method for providing push service |
| CN103856393A (en) * | 2013-07-09 | 2014-06-11 | 携程计算机技术（上海）有限公司 | Distributed message-oriented middleware system based on database and operating method thereof |
| EP2913982A1 (en) * | 2014-02-28 | 2015-09-02 | AOL, Inc. | Systems and methods for optimizing message notification timing based on electronic content consumption associated with a geographic location |
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| CN105471714A (en) * | 2015-12-09 | 2016-04-06 | 百度在线网络技术（北京）有限公司 | Message processing method and device |
| CN105761039A (en) * | 2016-02-17 | 2016-07-13 | 华迪计算机集团有限公司 | Method for processing express delivery information big data |
| CN109144992A (en) * | 2017-06-15 | 2019-01-04 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of data storage |
| CN108874555A (en) * | 2018-05-23 | 2018-11-23 | 福建天泉教育科技有限公司 | A kind of method and device for writing message to message-oriented middleware |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020177384A1 (en) | 2020-09-10 |
| CN109873904A (en) | 2019-06-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109873904B (en) | User message state reporting processing method, device and storage medium for message pushing | |
| US20170242887A1 (en) | Efficient access scheduling for super scaled stream processing systems | |
| CN110019087B (en) | Data processing method and system | |
| CN111414389B (en) | Data processing method and device, electronic equipment and storage medium | |
| US11625315B2 (en) | Software regression recovery via automated detection of problem change lists | |
| CN112162965B (en) | Log data processing method, device, computer equipment and storage medium | |
| CN109981715B (en) | Session management method and device | |
| CN111770002A (en) | Test data forwarding control method and device, readable storage medium and electronic equipment | |
| CN111586126A (en) | Method, device and equipment for pre-downloading small program and storage medium | |
| CN110807050B (en) | Performance analysis method, device, computer equipment and storage medium | |
| CN110910249B (en) | Data processing method and device, node equipment and storage medium | |
| CN110955390B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN110599229A (en) | Hundred million-level flow advertisement real-time processing method, storage medium, electronic equipment and system | |
| CN115934414A (en) | Data backup method, data recovery method, device, equipment and storage medium | |
| CN112969198A (en) | Data transmission method, terminal and storage medium | |
| CN110442439B (en) | Task process processing method and device and computer equipment | |
| CN109213972B (en) | Method, device, equipment and computer storage medium for determining document similarity | |
| CN103778218A (en) | Cloud computation-based standard information consistency early warning system and method | |
| CN111078975B (en) | Multi-node incremental data acquisition system and acquisition method | |
| CN108268662B (en) | Social graph generation method based on H5 page, electronic device and storage medium | |
| CN116391177A (en) | Prioritized inactive memory device updates | |
| US8214846B1 (en) | Method and system for threshold management | |
| CN112148705A (en) | Data migration method and device | |
| CN111858579A (en) | Data storage method and device | |
| CN116932779B (en) | Knowledge graph data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |