CN111143415A - Data processing method and device and computer readable storage medium - Google Patents
Data processing method and device and computer readable storage medium Download PDFInfo
- Publication number
- CN111143415A CN111143415A CN201911367948.XA CN201911367948A CN111143415A CN 111143415 A CN111143415 A CN 111143415A CN 201911367948 A CN201911367948 A CN 201911367948A CN 111143415 A CN111143415 A CN 111143415A
- Authority
- CN
- China
- Prior art keywords
- data
- message queue
- information
- service line
- line data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 38
- 238000004458 analytical method Methods 0.000 claims abstract description 34
- 238000000034 method Methods 0.000 claims abstract description 16
- 238000001914 filtration Methods 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 13
- 238000005192 partition Methods 0.000 claims description 11
- 238000012163 sequencing technique Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000004880 explosion Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000010923 batch production Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000009960 carding Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000008521 reorganization Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Abstract
The embodiment of the invention discloses a data processing method, a device and a medium, which are used for recording various acquired service line data to a first message queue; and extracting effective data flow of various service line data in the first message queue. According to preset window time, counting each effective data stream by using a sliding window to obtain a data block to be analyzed; and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result to the second message queue. By adopting the message queue cache and the data reading mode of the sliding window, the direct processing of the real-time service line data can be realized, unnecessary time consumption caused in the data reading process is avoided, and therefore the value information of mass data can be more effectively mined. And the analysis result is stored in the second message queue, and the business side can intuitively acquire valuable data information by reading the second message queue.
Description
Technical Field
The present invention relates to the field of data technologies, and in particular, to a data processing method, an apparatus, and a computer-readable storage medium.
Background
The E-commerce platform generates data flow information at every moment, and the data flow information comprises real-time information flow data of the user login, the region to which the user belongs, the number of sold commodities, the amount of money, the commodity category of a platform supplier, commodity browsing information of a buyer, commodity purchasing information and the like. In the field of e-commerce platforms emphasizing data values, particularly real-time data values, the relationship among real-time data of a carding platform, the reorganization of data information structures and the potential value of data information mining are extremely important within the shortest time range and the maximum limit.
Many data processing frameworks on the market today read historical data from a database or data storage medium and then process the data in a batch process. This approach has some inherent drawbacks or deficiencies. The first disadvantage is that: reading data from a database or a storage medium requires a certain time, and in some scenes with high requirements on timeliness of the data, the value of the data is correspondingly discounted. The second disadvantage is that: the characteristics of batch processing can cause that a batch of data is processed completely and then the next batch of data is processed, so that the computing engine is not fully utilized, and the data processing delay is inevitable due to the switching between different batches of data in the whole data pool. The third disadvantage is that: mass data generated at a certain moment can cause data processing to be untimely and even data loss due to the computing capacity of a computing frame, and can cause system downtime seriously.
Therefore, how to effectively mine the value information of mass data is a problem to be solved by the technical personnel in the field.
Disclosure of Invention
The embodiment of the invention aims to provide a data processing method, a data processing device and a computer readable storage medium, which can effectively mine value information of mass data.
To solve the foregoing technical problem, an embodiment of the present invention provides a data processing method, including:
recording the acquired various service line data to a first message queue;
extracting effective data flow of various service line data in the first message queue;
according to preset window time, counting each effective data stream by using a sliding window to obtain a data block to be analyzed;
and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result to the second message queue.
Optionally, the recording the acquired various types of service line data to the first message queue includes:
adding label information to the obtained various service line data according to a preset classification rule;
and recording various service line data added with the label information to a first message queue.
Optionally, the extracting valid data streams of various types of service line data in the first message queue includes:
sequencing the target service line data according to the timestamp corresponding to the target service line data to obtain a data stream; the target service line data is any one of all service line data;
extracting effective data streams in the data streams according to data filtering rules corresponding to the target service line data; wherein, different label information has their own correspondent data filtering rule.
Optionally, the analyzing each data block to be analyzed according to the corresponding service processing rule, and storing the obtained analysis result in the second message queue includes:
when the data block to be analyzed is commodity transaction information, counting the sales quantity and the sales amount of different commodity categories under different divisions in the commodity transaction information according to preset division information and commodity category information;
and storing the first N commodity transaction information with the highest sales quantity and the first N commodity transaction information with the highest sales amount in a second message queue according to the corresponding relation among the divisions, the commodity categories, the sales quantities and the sales amounts.
Optionally, the analyzing each data block to be analyzed according to the corresponding service processing rule, and storing the obtained analysis result in the second message queue includes:
when the data block to be analyzed is user browsing information, counting click amounts of advertisement information provided with the same label information in different pre-divided time periods;
and storing the advertisement information to a second message queue according to the corresponding relation of the label information, the time period and the click rate.
Optionally, the analyzing each data block to be analyzed according to the corresponding service processing rule, and storing the obtained analysis result in the second message queue includes:
when the data block to be analyzed is user login information, counting the user login number under different partitions and in different time periods in the user login information according to preset partition information;
and storing the user login information to a second message queue according to the corresponding relation of the division, the time period and the user login number.
The embodiment of the invention also provides a data processing device, which comprises a recording unit, an extraction unit, a statistical unit and an analysis unit;
the recording unit is used for recording the acquired various service line data to a first message queue;
the extracting unit is used for extracting effective data streams of various service line data in the first message queue;
the statistical unit is used for performing statistics on each effective data stream by using a sliding window according to preset window time to obtain a data block to be analyzed;
and the analysis unit is used for analyzing each data block to be analyzed according to the corresponding service processing rule and storing the obtained analysis result to the second message queue.
Optionally, the recording unit is specifically configured to add tag information to the obtained various service line data according to a preset classification rule; and recording various service line data added with the label information to a first message queue.
Optionally, the extracting unit comprises a sorting subunit and a filtering subunit;
the sequencing subunit is configured to sequence the target service line data according to the timestamp corresponding to the target service line data to obtain a data stream; the target service line data is any one of all service line data;
the filtering subunit is configured to extract an effective data stream from the data stream according to a data filtering rule corresponding to the target service line data; wherein, different label information has their own correspondent data filtering rule.
Optionally, the analysis unit comprises a statistics subunit and a storage subunit;
the statistic subunit is configured to, when the data block to be analyzed is commodity transaction information, count sales numbers and sales amounts of different commodity categories in different divisions in the commodity transaction information according to preset division information and commodity category information;
the storage subunit is configured to store the first N-digit commodity transaction information with the highest sales quantity and the first N-digit commodity transaction information with the highest sales amount in the second message queue according to the correspondence between the divisions, the commodity categories, the sales quantities, and the sales amounts.
Optionally, the analysis unit comprises a statistics subunit and a storage subunit;
the statistical subunit is configured to, when the data block to be analyzed is user browsing information, calculate click amounts of advertisement information provided with the same tag information in different pre-divided time periods;
and the storage subunit is used for storing the advertisement information to a second message queue according to the corresponding relation among the label information, the time period and the click rate.
Optionally, the analysis unit comprises a statistics subunit and a storage subunit;
the statistical subunit is configured to, when the data block to be analyzed is user login information, count the user login number under different partitions and in different time periods in the user login information according to preset partition information;
the storage subunit is configured to store the user login information in a second message queue according to a correspondence between the partition, the time period, and the user login number.
An embodiment of the present invention further provides a data processing apparatus, including:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the data processing method as claimed in any one of the above.
An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the steps of the data processing method according to any one of the above items.
According to the technical scheme, the acquired various service line data are recorded to the first message queue; the message queue technology is adopted to help the data computing engine to cache and send real-time information stream data, so that massive information streams generated in an instant or short time range can be well processed, and information loss caused by information explosion is avoided. And extracting effective data flow of various service line data in the first message queue. According to preset window time, counting each effective data stream by using a sliding window to obtain a data block to be analyzed; and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result to the second message queue. In the technical scheme, the service line data generated by the online platform in real time does not need to be read in the database first, and the message queue cache and the sliding window data reading mode are adopted, so that the direct processing of the real-time service line data can be realized, unnecessary time consumption caused in the data reading process is avoided, and the value information of mass data can be more effectively excavated. And the analysis result is stored in the second message queue, and the business side can intuitively acquire valuable data information by reading the second message queue.
Drawings
In order to illustrate the embodiments of the present invention more clearly, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a hardware structure of a data processing apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without any creative work belong to the protection scope of the present invention.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Next, a data processing method provided by an embodiment of the present invention is described in detail. Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, where the method includes:
s101: and recording the acquired various service line data to a first message queue.
The service line data may be from user data, commodity data, etc. generated by the online trading platform. In the embodiment of the invention, the message queue is adopted to store the service line data, so that the data calculation engine can be effectively helped to cache and send the real-time information stream data, the massive information streams generated in the instant or short time range can be well processed, and the information loss caused by information explosion is avoided.
The service line data comprises different types of data, so that the data types possibly contained in the service line data can be classified in advance for the convenience of analysis and management of the data, and when the service line data is obtained, label information can be added to the obtained various types of service line data according to a preset classification rule; and recording various service line data added with the label information to a first message queue.
The tag information is used to distinguish different data types. In the embodiment of the present invention, the specific form of the tag information is not limited, and for example, a form of a combination of numbers or letters may be adopted as the tag information.
S102: and extracting effective data flow of various service line data in the first message queue.
Considering that some conventional data without analysis value often exist in the service line data, in order to improve the data analysis efficiency, the conventional data in the service line data can be filtered, so that effective data with analysis value can be extracted.
The processing mode of each service line data is similar, in the embodiment of the present invention, an introduction is performed by taking any one of all service line data, that is, a target service line data, as an example, and in a specific implementation, the target service line data may be sorted according to a timestamp corresponding to the target service line data to obtain a data stream; extracting effective data flow in the data flow according to a data filtering rule corresponding to the target service line data; wherein, different label information has their own correspondent data filtering rule.
Different types of data differ in their corresponding conventional data. In the embodiment of the invention, different types of data are distinguished by adopting the label information, so that the corresponding filtering rules can be preset for different label information. The filtering rules may include data content or data characteristics of the regular data, so as to filter out the regular data and obtain the effective data stream.
S103: and according to the preset window time, counting each effective data stream by using a sliding window to obtain a data block to be analyzed.
Sliding window is a technique used to improve throughput by allowing the sender to transmit additional packets before receiving any acknowledgement, and the receiver tells the sender how many packets to send at a certain time, thus effectively avoiding congestion in the network.
In view of the large data volume of the effective data stream, in order to ensure the ordered execution of data processing, in the embodiment of the present invention, a sliding window is adopted to process each effective data stream. Each of the valid data streams is processed in a similar manner, and the following description will be given by taking the processing of one valid data stream as an example.
The preset window time can be regarded as the time span of the sliding window for collecting data for multiple times, and the value of the preset window time is larger than the time corresponding to the sliding window.
S104: and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result to the second message queue.
In a preset window time, each valid data stream corresponds to one data block to be analyzed, and the corresponding business processing rules are different because the data types contained in the data blocks to be analyzed are different.
The data type corresponding to the data block to be analyzed may include commodity transaction information, user browsing information, user login information, and the like.
When the data block to be analyzed is the commodity transaction information, the sales quantity and the sales amount of different commodity categories in different divisions in the commodity transaction information can be counted according to preset division information and commodity category information.
The section refers to an area to which the commodity vendor belongs. In practical applications, the sections to which the commodities belong may be divided into different areas such as provincial, city, or county levels.
After the sales numbers and the sales amounts of the different zones and the different commodity categories in the commodity transaction information are counted, the first N-digit commodity transaction information with the highest sales number and the first N-digit commodity transaction information with the highest sales amount can be stored in the second message queue according to the corresponding relation among the zones, the commodity categories, the sales numbers and the sales amounts.
The value of N may be set according to actual requirements, and is not limited herein, for example, the value of N may be set to 10.
When the data block to be analyzed is user browsing information, the click rate of the advertisement information provided with the same label information in different pre-divided time periods can be counted; and storing the advertisement information to a second message queue according to the corresponding relation of the label information, the time period and the click rate.
The user browsing information may include advertisement browsing information, commodity browsing information, and the like.
When the data block to be analyzed is user login information, counting the user login number under different partitions and in different time periods in the user login information according to preset partition information; and storing the user login information into a second message queue according to the corresponding relation among the sections, the time periods and the user login number.
In the embodiment of the present invention, in order to facilitate distinguishing from a message queue storing service line data, the message queue storing service line data may be referred to as a first message queue, and the message queue storing an analysis result may be referred to as a second message queue.
According to the technical scheme, the acquired various service line data are recorded to the first message queue; the message queue technology is adopted to help the data computing engine to cache and send real-time information stream data, so that massive information streams generated in an instant or short time range can be well processed, and information loss caused by information explosion is avoided. And extracting effective data flow of various service line data in the first message queue. According to preset window time, counting each effective data stream by using a sliding window to obtain a data block to be analyzed; and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result to the second message queue. In the technical scheme, the service line data generated by the online platform in real time does not need to be read in the database first, and the message queue cache and the sliding window data reading mode are adopted, so that the direct processing of the real-time service line data can be realized, unnecessary time consumption caused in the data reading process is avoided, and the value information of mass data can be more effectively excavated. And the analysis result is stored in the second message queue, and the business side can intuitively acquire valuable data information by reading the second message queue.
Fig. 2 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention, which includes a recording unit 21, an extracting unit 22, a statistical unit 23, and an analyzing unit 24;
a recording unit 21, configured to record the obtained various service line data to a first message queue;
an extracting unit 22, configured to extract valid data streams of various types of service line data in the first message queue;
the statistical unit 23 is configured to perform statistics on each valid data stream by using a sliding window according to a preset window time to obtain a data block to be analyzed;
and the analysis unit 24 is configured to analyze each data block to be analyzed according to the corresponding service processing rule, and store an obtained analysis result in the second message queue.
Optionally, the recording unit is specifically configured to add tag information to the obtained various service line data according to a preset classification rule; and recording various service line data added with the label information to a first message queue.
Optionally, the extraction unit comprises a sorting subunit and a filtering subunit;
the sequencing subunit is used for sequencing the target service line data according to the timestamp corresponding to the target service line data to obtain a data stream; the target service line data is any one of all service line data;
the filtering subunit is used for extracting an effective data stream from the data stream according to a data filtering rule corresponding to the target service line data; wherein, different label information has their own correspondent data filtering rule.
Optionally, the analysis unit comprises a statistics subunit and a storage subunit;
the statistical subunit is used for counting the sales quantity and the sales amount of different commodity categories in different divisions in the commodity transaction information according to preset division information and commodity category information when the data block to be analyzed is the commodity transaction information;
and the storage subunit is used for storing the first N-digit commodity transaction information with the highest sales quantity and the first N-digit commodity transaction information with the highest sales amount into the second message queue according to the corresponding relation among the sections, the commodity categories, the sales quantities and the sales amounts.
Optionally, the analysis unit comprises a statistics subunit and a storage subunit;
the statistical subunit is used for counting the click rate of the advertisement information provided with the same label information in different pre-divided time periods when the data block to be analyzed is the user browsing information;
and the storage subunit is used for storing the advertisement information to the second message queue according to the corresponding relation among the label information, the time period and the click rate.
Optionally, the analysis unit comprises a statistics subunit and a storage subunit;
the statistical subunit is used for counting the user login number under different partitions and in different time periods in the user login information according to preset partition information when the data block to be analyzed is the user login information;
and the storage subunit is used for storing the user login information into the second message queue according to the corresponding relation among the sections, the time periods and the user login number.
The description of the features in the embodiment corresponding to fig. 2 may refer to the related description of the embodiment corresponding to fig. 1, and is not repeated here.
According to the technical scheme, the acquired various service line data are recorded to the first message queue; the message queue technology is adopted to help the data computing engine to cache and send real-time information stream data, so that massive information streams generated in an instant or short time range can be well processed, and information loss caused by information explosion is avoided. And extracting effective data flow of various service line data in the first message queue. According to preset window time, counting each effective data stream by using a sliding window to obtain a data block to be analyzed; and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result to the second message queue. In the technical scheme, the service line data generated by the online platform in real time does not need to be read in the database first, and the message queue cache and the sliding window data reading mode are adopted, so that the direct processing of the real-time service line data can be realized, unnecessary time consumption caused in the data reading process is avoided, and the value information of mass data can be more effectively excavated. And the analysis result is stored in the second message queue, and the business side can intuitively acquire valuable data information by reading the second message queue.
Fig. 3 is a schematic diagram of a hardware structure of a data processing apparatus 30 according to an embodiment of the present invention, including:
a memory 31 for storing a computer program;
a processor 32 for executing a computer program to implement the steps of any of the data processing methods described above.
The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the data processing methods described above are implemented.
The data processing method, the data processing device and the computer-readable storage medium provided by the embodiments of the present invention are described in detail above. The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Claims (10)
1. A data processing method, comprising:
recording the acquired various service line data to a first message queue;
extracting effective data flow of various service line data in the first message queue;
according to preset window time, counting each effective data stream by using a sliding window to obtain a data block to be analyzed;
and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result to the second message queue.
2. The method of claim 1, wherein the recording the obtained various types of service line data to the first message queue comprises:
adding label information to the obtained various service line data according to a preset classification rule;
and recording various service line data added with the label information to a first message queue.
3. The method of claim 2, wherein extracting the active data stream of each type of service line data in the first message queue comprises:
sequencing the target service line data according to the timestamp corresponding to the target service line data to obtain a data stream; the target service line data is any one of all service line data;
extracting effective data streams in the data streams according to data filtering rules corresponding to the target service line data; wherein, different label information has their own correspondent data filtering rule.
4. The method of claim 3, wherein analyzing each data block to be analyzed according to its corresponding business processing rule, and storing the obtained analysis result in the second message queue comprises:
when the data block to be analyzed is commodity transaction information, counting the sales quantity and the sales amount of different commodity categories under different divisions in the commodity transaction information according to preset division information and commodity category information;
and storing the first N commodity transaction information with the highest sales quantity and the first N commodity transaction information with the highest sales amount in a second message queue according to the corresponding relation among the divisions, the commodity categories, the sales quantities and the sales amounts.
5. The method of claim 3, wherein analyzing each data block to be analyzed according to its corresponding business processing rule, and storing the obtained analysis result in the second message queue comprises:
when the data block to be analyzed is user browsing information, counting click amounts of advertisement information provided with the same label information in different pre-divided time periods;
and storing the advertisement information to a second message queue according to the corresponding relation of the label information, the time period and the click rate.
6. The method of claim 3, wherein analyzing each data block to be analyzed according to its corresponding business processing rule, and storing the obtained analysis result in the second message queue comprises:
when the data block to be analyzed is user login information, counting the user login number under different partitions and in different time periods in the user login information according to preset partition information;
and storing the user login information to a second message queue according to the corresponding relation of the division, the time period and the user login number.
7. A data processing apparatus is characterized by comprising a recording unit, an extracting unit, a statistical unit and an analyzing unit;
the recording unit is used for recording the acquired various service line data to a first message queue;
the extracting unit is used for extracting effective data streams of various service line data in the first message queue;
the statistical unit is used for performing statistics on each effective data stream by using a sliding window according to preset window time to obtain a data block to be analyzed;
and the analysis unit is used for analyzing each data block to be analyzed according to the corresponding service processing rule and storing the obtained analysis result to the second message queue.
8. The device according to claim 7, wherein the recording unit is specifically configured to add tag information to the obtained various types of service line data according to a preset classification rule; and recording various service line data added with the label information to a first message queue.
9. A data processing apparatus, comprising:
a memory for storing a computer program;
a processor for executing the computer program to carry out the steps of the data processing method according to any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the data processing method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911367948.XA CN111143415B (en) | 2019-12-26 | 2019-12-26 | Data processing method, device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911367948.XA CN111143415B (en) | 2019-12-26 | 2019-12-26 | Data processing method, device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111143415A true CN111143415A (en) | 2020-05-12 |
| CN111143415B CN111143415B (en) | 2023-12-29 |
Family
ID=70520482
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911367948.XA Active CN111143415B (en) | 2019-12-26 | 2019-12-26 | Data processing method, device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111143415B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111901352A (en) * | 2020-07-30 | 2020-11-06 | 彩讯科技股份有限公司 | Message distribution processing method, device, server and storage medium |
| CN112035534A (en) * | 2020-09-18 | 2020-12-04 | 上海依图网络科技有限公司 | Real-time big data processing method and device and electronic equipment |
| CN112506978A (en) * | 2020-12-15 | 2021-03-16 | 中国联合网络通信集团有限公司 | Big data real-time processing method, device and equipment |
| CN112633904A (en) * | 2020-12-30 | 2021-04-09 | 中国平安财产保险股份有限公司 | Complaint behavior analysis method, complaint behavior analysis device, complaint behavior analysis equipment and computer-readable storage medium |
| CN112751726A (en) * | 2020-12-17 | 2021-05-04 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113360564A (en) * | 2021-07-12 | 2021-09-07 | 杭州安恒信息技术股份有限公司 | ETL-based data stream processing method, system, device and readable storage medium |
| CN113609202A (en) * | 2021-08-11 | 2021-11-05 | 湖南快乐阳光互动娱乐传媒有限公司 | Data processing method and device |
| CN113993001A (en) * | 2021-09-08 | 2022-01-28 | 四创电子股份有限公司 | Real-time streaming analysis alarm method based on sliding data window |
| WO2023109964A1 (en) * | 2021-12-16 | 2023-06-22 | 中移(苏州)软件技术有限公司 | Data analysis method and apparatus, and device and computer-readable storage medium |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050262515A1 (en) * | 2004-05-20 | 2005-11-24 | International Business Machines Corporation | Methods, systems, and media to enhance browsing of messages in a message queue |
| US20090063360A1 (en) * | 2007-08-30 | 2009-03-05 | Chicago Mercantile Exchange, Inc. | Dynamic market data filtering |
| US20130339473A1 (en) * | 2012-06-15 | 2013-12-19 | Zynga Inc. | Real time analytics via stream processing |
| CN104615777A (en) * | 2015-02-27 | 2015-05-13 | 浪潮集团有限公司 | Method and device for real-time data processing based on stream-oriented calculation engine |
| CN105512297A (en) * | 2015-12-10 | 2016-04-20 | 中国测绘科学研究院 | Distributed stream-oriented computation based spatial data processing method and system |
| CN105786941A (en) * | 2014-12-26 | 2016-07-20 | 中国移动通信集团上海有限公司 | Information mining method and device |
| CN106156026A (en) * | 2015-03-24 | 2016-11-23 | 中国人民解放军国防科学技术大学 | A kind of method based on the data online anomaly of stream fictitious assets |
| CN106528865A (en) * | 2016-12-02 | 2017-03-22 | 航天科工智慧产业发展有限公司 | Quick and accurate cleaning method of traffic big data |
| WO2017092582A1 (en) * | 2015-12-01 | 2017-06-08 | 阿里巴巴集团控股有限公司 | Data processing method and apparatus |
| WO2017185576A1 (en) * | 2016-04-25 | 2017-11-02 | 百度在线网络技术（北京）有限公司 | Multi-streaming data processing method, system, storage medium, and device |
| CN108287905A (en) * | 2018-01-26 | 2018-07-17 | 华南理工大学 | A kind of extraction of network flow feature and storage method |
| CN108874812A (en) * | 2017-05-10 | 2018-11-23 | 腾讯科技（北京）有限公司 | A kind of data processing method and server, computer storage medium |
| CN108874834A (en) * | 2017-05-16 | 2018-11-23 | 北京嘀嘀无限科技发展有限公司 | A kind of data processing method, processing system and computer installation |
| CN109471898A (en) * | 2018-12-19 | 2019-03-15 | 华迪计算机集团有限公司 | It is a kind of for data to be carried out with the method and system of shared distribution |
| CN109905412A (en) * | 2019-04-28 | 2019-06-18 | 山东渔翁信息技术股份有限公司 | A kind of parallel encrypting and deciphering processing method of network data, device and medium |
-
2019
- 2019-12-26 CN CN201911367948.XA patent/CN111143415B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050262515A1 (en) * | 2004-05-20 | 2005-11-24 | International Business Machines Corporation | Methods, systems, and media to enhance browsing of messages in a message queue |
| US20090063360A1 (en) * | 2007-08-30 | 2009-03-05 | Chicago Mercantile Exchange, Inc. | Dynamic market data filtering |
| US20130339473A1 (en) * | 2012-06-15 | 2013-12-19 | Zynga Inc. | Real time analytics via stream processing |
| CN105786941A (en) * | 2014-12-26 | 2016-07-20 | 中国移动通信集团上海有限公司 | Information mining method and device |
| CN104615777A (en) * | 2015-02-27 | 2015-05-13 | 浪潮集团有限公司 | Method and device for real-time data processing based on stream-oriented calculation engine |
| CN106156026A (en) * | 2015-03-24 | 2016-11-23 | 中国人民解放军国防科学技术大学 | A kind of method based on the data online anomaly of stream fictitious assets |
| WO2017092582A1 (en) * | 2015-12-01 | 2017-06-08 | 阿里巴巴集团控股有限公司 | Data processing method and apparatus |
| CN105512297A (en) * | 2015-12-10 | 2016-04-20 | 中国测绘科学研究院 | Distributed stream-oriented computation based spatial data processing method and system |
| WO2017185576A1 (en) * | 2016-04-25 | 2017-11-02 | 百度在线网络技术（北京）有限公司 | Multi-streaming data processing method, system, storage medium, and device |
| CN106528865A (en) * | 2016-12-02 | 2017-03-22 | 航天科工智慧产业发展有限公司 | Quick and accurate cleaning method of traffic big data |
| CN108874812A (en) * | 2017-05-10 | 2018-11-23 | 腾讯科技（北京）有限公司 | A kind of data processing method and server, computer storage medium |
| CN108874834A (en) * | 2017-05-16 | 2018-11-23 | 北京嘀嘀无限科技发展有限公司 | A kind of data processing method, processing system and computer installation |
| CN108287905A (en) * | 2018-01-26 | 2018-07-17 | 华南理工大学 | A kind of extraction of network flow feature and storage method |
| CN109471898A (en) * | 2018-12-19 | 2019-03-15 | 华迪计算机集团有限公司 | It is a kind of for data to be carried out with the method and system of shared distribution |
| CN109905412A (en) * | 2019-04-28 | 2019-06-18 | 山东渔翁信息技术股份有限公司 | A kind of parallel encrypting and deciphering processing method of network data, device and medium |
Non-Patent Citations (1)
| Title |
|---|
| 刘峰;鄂海红;: "基于海量数据的消息队列的性能对比与优化方案", 软件, no. 10 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111901352A (en) * | 2020-07-30 | 2020-11-06 | 彩讯科技股份有限公司 | Message distribution processing method, device, server and storage medium |
| CN111901352B (en) * | 2020-07-30 | 2023-08-25 | 彩讯科技股份有限公司 | Method, device, server and storage medium for message distribution processing |
| CN112035534A (en) * | 2020-09-18 | 2020-12-04 | 上海依图网络科技有限公司 | Real-time big data processing method and device and electronic equipment |
| CN112506978A (en) * | 2020-12-15 | 2021-03-16 | 中国联合网络通信集团有限公司 | Big data real-time processing method, device and equipment |
| CN112751726A (en) * | 2020-12-17 | 2021-05-04 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112633904A (en) * | 2020-12-30 | 2021-04-09 | 中国平安财产保险股份有限公司 | Complaint behavior analysis method, complaint behavior analysis device, complaint behavior analysis equipment and computer-readable storage medium |
| CN113360564A (en) * | 2021-07-12 | 2021-09-07 | 杭州安恒信息技术股份有限公司 | ETL-based data stream processing method, system, device and readable storage medium |
| CN113609202A (en) * | 2021-08-11 | 2021-11-05 | 湖南快乐阳光互动娱乐传媒有限公司 | Data processing method and device |
| CN113993001A (en) * | 2021-09-08 | 2022-01-28 | 四创电子股份有限公司 | Real-time streaming analysis alarm method based on sliding data window |
| CN113993001B (en) * | 2021-09-08 | 2024-04-12 | 四创电子股份有限公司 | Real-time stream analysis alarm method based on sliding data window |
| WO2023109964A1 (en) * | 2021-12-16 | 2023-06-22 | 中移(苏州)软件技术有限公司 | Data analysis method and apparatus, and device and computer-readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111143415B (en) | 2023-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111143415A (en) | Data processing method and device and computer readable storage medium | |
| CN107168854B (en) | Internet advertisement abnormal click detection method, device, equipment and readable storage medium | |
| US10482093B2 (en) | Data mining method | |
| CN106022900B (en) | User risk data mining method and device | |
| US20060074621A1 (en) | Apparatus and method for prioritized grouping of data representing events | |
| CN106453357A (en) | Network ticket buying abnormal behavior recognition method and system and equipment | |
| CN111210326A (en) | Method and system for constructing user portrait | |
| CN111639138B (en) | Data processing method, device, equipment and storage medium | |
| WO2021254027A1 (en) | Method and apparatus for identifying suspicious community, and storage medium and computer device | |
| CN110060087B (en) | Abnormal data detection method, device and server | |
| CN106327230B (en) | Abnormal user detection method and equipment | |
| CN106802904A (en) | Log processing method, apparatus and system | |
| CN113837617A (en) | Anti-bill-swiping risk management method and device | |
| CN110675078A (en) | Marketing company risk diagnosis method, system, computer terminal and storage medium | |
| CN109559058A (en) | A kind of e-commerce user behavioral data analytical technology based on cloud computing | |
| CN109658643B (en) | Payment risk warning method, server and intelligent POS (point of sale) | |
| CN114022051A (en) | Index fluctuation analysis method, storage medium and electronic equipment | |
| CN109919667A (en) | A kind of method and apparatus of the IP of enterprise for identification | |
| CN110223104B (en) | Big data-based customer model construction system | |
| KR20110093398A (en) | Device and method for managing mobile terminated service | |
| CN110928942A (en) | Index data monitoring and management method and device | |
| CN105868991A (en) | Method and device for identifying machine assisted cheating | |
| CN111429192A (en) | Invoice data analysis method and system | |
| CN114338429B (en) | Network bandwidth determining method and device and electronic equipment | |
| CN111967966B (en) | Automatic wake-up method and system for sleep clients of mobile phone banks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |