CN111143415B - Data processing method, device and computer readable storage medium - Google Patents

Data processing method, device and computer readable storage medium Download PDF

Info

Publication number
CN111143415B
CN111143415B CN201911367948.XA CN201911367948A CN111143415B CN 111143415 B CN111143415 B CN 111143415B CN 201911367948 A CN201911367948 A CN 201911367948A CN 111143415 B CN111143415 B CN 111143415B
Authority
CN
China
Prior art keywords
data
message queue
information
line data
analyzed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911367948.XA
Other languages
Chinese (zh)
Other versions
CN111143415A (en
Inventor
孟浩浩
张春鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhengcaiyun Co ltd
Original Assignee
Zhengcaiyun Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhengcaiyun Co ltd filed Critical Zhengcaiyun Co ltd
Priority to CN201911367948.XA priority Critical patent/CN111143415B/en
Publication of CN111143415A publication Critical patent/CN111143415A/en
Application granted granted Critical
Publication of CN111143415B publication Critical patent/CN111143415B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24568Data stream processing; Continuous queries
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2458Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2462Approximate or statistical queries
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/546Message passing systems or structures, e.g. queues
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G06Q40/04Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Fuzzy Systems (AREA)
  • Development Economics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • Technology Law (AREA)
  • General Business, Economics & Management (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The embodiment of the invention discloses a data processing method, a device and a medium, which are used for recording acquired various business line data to a first message queue; and extracting the effective data flow of various service line data in the first message queue. According to the preset window time, counting each effective data stream by utilizing a sliding window to obtain a data block to be analyzed; and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result into a second message queue. By adopting the message queue buffer and the sliding window data reading mode, the direct processing of the real-time business line data can be realized, and unnecessary time consumption caused in the data reading process is avoided, so that the value information of a large amount of data can be more effectively mined. And the analysis result is stored in the second message queue, and the service party can intuitively acquire valuable data information by reading the second message queue.

Description

Data processing method, device and computer readable storage medium
Technical Field
The present invention relates to the field of data technologies, and in particular, to a data processing method, apparatus, and computer readable storage medium.
Background
The electronic commerce platform can generate data stream information at every moment, and the data stream information comprises real-time information stream data such as user login and real-time information stream data such as the region to which the user belongs, the number, the amount and the category of the commodity sold by the platform provider, commodity browsing information of a buyer, commodity purchasing information and the like. In the field of electronic commerce platforms for emphasizing data value, especially real-time data value, the maximum sorting of the relationship between real-time data of the platform, the recombination of data information structures and the exploitation of the potential value of data information are extremely important within the shortest time range.
Many data processing frameworks in the market today use a batch process to process historical data read from a database or data storage medium. This approach has some inherent drawbacks or shortcomings. The method has the following defects: reading the data itself from the database or storage medium requires a certain amount of time, and in some scenarios where the timeliness requirements for the data are high, the value of the data may be correspondingly compromised. And the second disadvantage is that: the characteristics of the batch process itself may cause that a batch of data is processed completely and then the next batch of data is processed, which not only makes the computing engine underutilized, but also causes unavoidable data processing delay due to switching between different batches of data in the whole data pool. And the third disadvantage is: the massive data generated at a certain moment can cause untimely data processing and even data loss due to the self computing capacity of the computing framework, and seriously cause system downtime.
It can be seen that how to effectively mine the value information of the mass data is a problem to be solved by those skilled in the art.
Disclosure of Invention
The embodiment of the invention aims to provide a data processing method, a data processing device and a computer readable storage medium, which can effectively mine the value information of mass data.
In order to solve the above technical problems, an embodiment of the present invention provides a data processing method, including:
recording the acquired various service line data to a first message queue;
extracting effective data streams of various service line data in the first message queue;
according to the preset window time, counting each effective data stream by utilizing a sliding window to obtain a data block to be analyzed;
and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result into a second message queue.
Optionally, the recording the acquired various service line data to the first message queue includes:
adding tag information to the acquired various service line data according to a preset classification rule;
and recording various business line data added with the tag information to a first message queue.
Optionally, the extracting the valid data flow of the various service line data in the first message queue includes:
sequencing the target business line data according to the time stamp corresponding to the target business line data to obtain a data stream; wherein, the target service line data is any one service line data in all service line data;
extracting effective data flow in the data flow according to the data filtering rule corresponding to the target service line data; wherein, different tag information has data filtering rules corresponding to the tag information.
Optionally, the analyzing each data block to be analyzed according to the corresponding service processing rule, and storing the obtained analysis result in the second message queue includes:
when the data block to be analyzed is commodity transaction information, counting the sales quantity and sales amount of different commodity categories under different regions in the commodity transaction information according to preset region information and commodity category information;
and storing the top N-bit commodity transaction information with the highest sales quantity and the top N-bit commodity transaction information with the highest sales amount into a second message queue according to the corresponding relation among the region, the commodity category, the sales quantity and the sales amount.
Optionally, the analyzing each data block to be analyzed according to the corresponding service processing rule, and storing the obtained analysis result in the second message queue includes:
when the data block to be analyzed is browsing information of a user, counting click amounts of advertisement information provided with the same tag information in different time periods divided in advance;
and storing the advertisement information to a second message queue according to the corresponding relation among the label information, the time period and the click quantity.
Optionally, the analyzing each data block to be analyzed according to the corresponding service processing rule, and storing the obtained analysis result in the second message queue includes:
when the data block to be analyzed is user login information, counting the user login quantity in different time periods under different regions in the user login information according to preset region information;
and storing the user login information into a second message queue according to the corresponding relation of the region, the time period and the user login quantity.
The embodiment of the invention also provides a data processing device which comprises a recording unit, an extracting unit, a statistics unit and an analysis unit;
the recording unit is used for recording the acquired various service line data to a first message queue;
the extracting unit is used for extracting the effective data streams of various service line data in the first message queue;
the statistics unit is used for counting each effective data stream by utilizing a sliding window according to preset window time to obtain a data block to be analyzed;
the analysis unit is used for analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result into the second message queue.
Optionally, the recording unit is specifically configured to add tag information to each type of acquired service line data according to a preset classification rule; and recording various business line data added with the tag information to a first message queue.
Optionally, the extraction unit includes a sorting subunit and a filtering subunit;
the sequencing subunit is configured to sequence the target service line data according to a timestamp corresponding to the target service line data, so as to obtain a data stream; wherein, the target service line data is any one service line data in all service line data;
the filtering subunit is configured to extract an effective data stream from the data streams according to a data filtering rule corresponding to the target service line data; wherein, different tag information has data filtering rules corresponding to the tag information.
Optionally, the analysis unit includes a statistics subunit and a storage subunit;
the statistics subunit is used for counting sales quantity and sales amount of different commodity categories under different regions in the commodity transaction information according to preset region information and commodity category information when the data block to be analyzed is commodity transaction information;
the storage subunit is configured to store the top N bits of the sales amount and the top N bits of the sales amount to the second message queue according to the corresponding relationship among the division, the commodity category, the sales amount and the sales amount.
Optionally, the analysis unit includes a statistics subunit and a storage subunit;
the statistics subunit is used for counting the click rate of advertisement information provided with the same label information in different time periods divided in advance when the data block to be analyzed is user browsing information;
and the storage subunit is used for storing the advertisement information to a second message queue according to the corresponding relation among the label information, the time period and the click quantity.
Optionally, the analysis unit includes a statistics subunit and a storage subunit;
the statistics subunit is used for counting the number of user logins in different regions and different time periods in the user login information according to preset region information when the data block to be analyzed is the user login information;
the storage subunit is configured to store the user login information to a second message queue according to a corresponding relationship of a region, a time period and a user login number.
The embodiment of the invention also provides a data processing device, which comprises:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the data processing method as claimed in any one of the preceding claims.
Embodiments of the present invention also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the data processing method according to any of the preceding claims.
According to the technical scheme, the acquired various service line data are recorded to a first message queue; by adopting the message queue technology, the data calculation engine is helped to buffer and send real-time information stream data, so that massive information streams produced in a transient or short time range can be well processed, and information loss caused by information explosion is avoided. And extracting the effective data flow of various service line data in the first message queue. According to the preset window time, counting each effective data stream by utilizing a sliding window to obtain a data block to be analyzed; and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result into a second message queue. In the technical scheme, the service line data generated by the online platform in real time does not need to be read in the mode of first-in database, and the direct processing of the real-time service line data can be realized by adopting the mode of message queue caching and sliding window data reading, so that unnecessary time consumption in the data reading process is avoided, and the value information of a large amount of data can be more effectively mined. And the analysis result is stored in the second message queue, and the service party can intuitively acquire valuable data information by reading the second message queue.
Drawings
For a clearer description of embodiments of the present invention, the drawings that are required to be used in the embodiments will be briefly described, it being apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to the drawings without inventive effort for those skilled in the art.
FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic hardware structure of a data processing apparatus according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without making any inventive effort are within the scope of the present invention.
In order to better understand the aspects of the present invention, the present invention will be described in further detail with reference to the accompanying drawings and detailed description.
Next, a data processing method provided by the embodiment of the present invention is described in detail. Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, where the method includes:
s101: and recording the acquired various service line data to a first message queue.
The business line data may be from user data, merchandise data, etc. generated by the online transaction platform. In the embodiment of the invention, the message queue is used for storing the service line data, so that the data calculation engine can be effectively assisted to buffer and send the real-time information flow data, and massive information flows produced in a transient or short time range can be well processed, so that information loss caused by information explosion is avoided.
The service line data comprises different types of data, in order to facilitate analysis and management of the data, the data types possibly contained in the service line data can be classified in advance, and when the service line data is acquired, tag information can be added to the acquired various service line data according to a preset classification rule; and recording various business line data added with the tag information to a first message queue.
The tag information is used to distinguish between different data types. In the embodiment of the present invention, the specific form of the tag information is not limited, and for example, a form of a combination of numerals or letters may be used as the tag information.
S102: and extracting the effective data flow of various service line data in the first message queue.
Considering that the service line data often have some conventional data which does not have analysis value, in order to improve the data analysis efficiency, the conventional data in the service line data can be filtered, so that effective data with analysis value is extracted.
In the embodiment of the invention, any one business line data, namely, target business line data, is taken as an example to develop and introduce, and in the specific implementation, the target business line data can be sequenced according to the timestamp corresponding to the target business line data to obtain a data stream; extracting effective data flow in the data flow according to the data filtering rule corresponding to the target service line data; wherein, different tag information has data filtering rules corresponding to the tag information.
The different types of data differ in their corresponding regular data. In the embodiment of the invention, different types of data are distinguished by adopting the label information, so that the corresponding filtering rule can be preset aiming at different label information. The filtering rule may include data content or data characteristics of the regular data, so as to filter the regular data, and obtain an effective data stream.
S103: and counting each effective data stream by utilizing a sliding window according to the preset window time to obtain a data block to be analyzed.
Sliding windows are a technique used to improve throughput by allowing the sender to transmit additional packets before receiving any acknowledgement, and the receiver tells the sender how many packets can be transmitted at a time, thus effectively avoiding congestion in the network.
In order to ensure orderly execution of data processing, in the embodiment of the present invention, a sliding window is used to process each valid data stream in consideration of the larger data volume of the valid data stream. The processing manner of each effective data stream is similar, and in the following description, the processing of one effective data stream is taken as an example for development.
The preset window time can be regarded as the time span of the sliding window for collecting data for many times, and the value of the preset window time is larger than the time corresponding to the sliding window.
S104: and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result into a second message queue.
In the preset window time, each effective data stream corresponds to one data block to be analyzed, and the corresponding business processing rules are different because the data types contained in the data blocks to be analyzed are different.
The data types corresponding to the data blocks to be analyzed can comprise commodity transaction information, user browsing information, user login information and the like.
When the data block to be analyzed is commodity transaction information, the sales quantity and sales amount of different commodity categories under different regions in the commodity transaction information can be counted according to preset region information and commodity category information.
The division refers to an area to which a commodity seller belongs. In practical application, the regions to which the commodities belong can be divided according to different regions such as provincial level, municipal level or county level.
After the sales quantity and sales amount of different regions and different commodity categories in the commodity transaction information are counted, the top N positions with the highest sales quantity and the top N positions with the highest sales amount of the commodity transaction information can be stored in the second message queue according to the corresponding relation among the regions, the commodity categories, the sales quantity and the sales amount.
The value of N may be set according to actual requirements, and is not limited herein, for example, the value of N may be set to 10.
When the data block to be analyzed is browsing information of a user, the click rate of advertisement information provided with the same label information in different time periods divided in advance can be counted; and storing the advertisement information to a second message queue according to the corresponding relation among the label information, the time period and the click quantity.
The user browsing information may include advertisement browsing information, commodity browsing information, and the like.
When the data block to be analyzed is user login information, counting the number of user logins in different time periods under different regions in the user login information according to preset region information; and storing the user login information into a second message queue according to the corresponding relation of the region, the time period and the user login quantity.
In the embodiment of the invention, in order to facilitate distinguishing from the message queue storing the service line data, the message queue storing the service line data may be referred to as a first message queue, and the message queue storing the analysis result may be referred to as a second message queue.
According to the technical scheme, the acquired various service line data are recorded to a first message queue; by adopting the message queue technology, the data calculation engine is helped to buffer and send real-time information stream data, so that massive information streams produced in a transient or short time range can be well processed, and information loss caused by information explosion is avoided. And extracting the effective data flow of various service line data in the first message queue. According to the preset window time, counting each effective data stream by utilizing a sliding window to obtain a data block to be analyzed; and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result into a second message queue. In the technical scheme, the service line data generated by the online platform in real time does not need to be read in the mode of first-in database, and the direct processing of the real-time service line data can be realized by adopting the mode of message queue caching and sliding window data reading, so that unnecessary time consumption in the data reading process is avoided, and the value information of a large amount of data can be more effectively mined. And the analysis result is stored in the second message queue, and the service party can intuitively acquire valuable data information by reading the second message queue.
Fig. 2 is a schematic structural diagram of a data processing device according to an embodiment of the present invention, which includes a recording unit 21, an extracting unit 22, a statistics unit 23, and an analysis unit 24;
a recording unit 21, configured to record the acquired various service line data to a first message queue;
an extracting unit 22, configured to extract valid data flows of various service line data in the first message queue;
a statistics unit 23, configured to perform statistics on each valid data stream by using a sliding window according to a preset window time, so as to obtain a data block to be analyzed;
and the analysis unit 24 is configured to analyze each data block to be analyzed according to the corresponding service processing rule, and store the obtained analysis result in the second message queue.
Optionally, the recording unit is specifically configured to add tag information to each type of acquired service line data according to a preset classification rule; and recording various business line data added with the tag information to a first message queue.
Optionally, the extraction unit comprises a sorting subunit and a filtering subunit;
the sequencing subunit is used for sequencing the target service line data according to the time stamp corresponding to the target service line data to obtain a data stream; the target service line data is any one of all service line data;
the filtering subunit is used for extracting effective data streams in the data streams according to the data filtering rules corresponding to the target service line data; wherein, different tag information has data filtering rules corresponding to the tag information.
Optionally, the analysis unit includes a statistics subunit and a storage subunit;
the statistics subunit is used for counting the sales quantity and sales amount of different commodity categories under different regions in the commodity transaction information according to preset region information and commodity category information when the data block to be analyzed is commodity transaction information;
and the storage subunit is used for storing the top N-bit commodity transaction information with the highest sales quantity and the top N-bit commodity transaction information with the highest sales quantity into the second message queue according to the corresponding relation among the division, the commodity category, the sales quantity and the sales quantity.
Optionally, the analysis unit includes a statistics subunit and a storage subunit;
the statistics subunit is used for counting the click rate of the advertisement information provided with the same label information in different time periods divided in advance when the data block to be analyzed is user browsing information;
and the storage subunit is used for storing the advertisement information to the second message queue according to the corresponding relation among the label information, the time period and the click quantity.
Optionally, the analysis unit includes a statistics subunit and a storage subunit;
the statistics subunit is used for counting the user login quantity in different time periods under different regions in the user login information according to preset region information when the data block to be analyzed is the user login information;
and the storage subunit is used for storing the user login information into the second message queue according to the corresponding relation of the region, the time period and the user login quantity.
The description of the features in the embodiment corresponding to fig. 2 may be referred to the related description of the embodiment corresponding to fig. 1, and will not be repeated here.
According to the technical scheme, the acquired various service line data are recorded to a first message queue; by adopting the message queue technology, the data calculation engine is helped to buffer and send real-time information stream data, so that massive information streams produced in a transient or short time range can be well processed, and information loss caused by information explosion is avoided. And extracting the effective data flow of various service line data in the first message queue. According to the preset window time, counting each effective data stream by utilizing a sliding window to obtain a data block to be analyzed; and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result into a second message queue. In the technical scheme, the service line data generated by the online platform in real time does not need to be read in the mode of first-in database, and the direct processing of the real-time service line data can be realized by adopting the mode of message queue caching and sliding window data reading, so that unnecessary time consumption in the data reading process is avoided, and the value information of a large amount of data can be more effectively mined. And the analysis result is stored in the second message queue, and the service party can intuitively acquire valuable data information by reading the second message queue.
Fig. 3 is a schematic hardware structure of a data processing apparatus 30 according to an embodiment of the present invention, including:
a memory 31 for storing a computer program;
a processor 32 for executing a computer program to perform the steps of any of the data processing methods described above.
The embodiment of the invention also provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and when the computer program is executed by a processor, the steps of any one of the data processing methods are realized.
The foregoing describes in detail a data processing method, apparatus and computer readable storage medium provided by embodiments of the present invention. In the description, each embodiment is described in a progressive manner, and each embodiment is mainly described by the differences from other embodiments, so that the same similar parts among the embodiments are mutually referred. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the invention can be made without departing from the principles of the invention and these modifications and adaptations are intended to be within the scope of the invention as defined in the following claims.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

Claims (10)

1. A method of data processing, comprising:
recording the acquired various service line data to a first message queue;
extracting effective data streams of various service line data in the first message queue;
according to the preset window time, counting each effective data stream by utilizing a sliding window to obtain a data block to be analyzed; the sliding window is a number of packets that the receiver tells the sender to be able to transmit at each instant;
and analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result into a second message queue.
2. The method of claim 1, wherein the recording the acquired various types of service line data to the first message queue comprises:
adding tag information to the acquired various service line data according to a preset classification rule;
and recording various business line data added with the tag information to a first message queue.
3. The method of claim 2, wherein extracting valid data streams for various traffic line data in the first message queue comprises:
sequencing the target business line data according to the time stamp corresponding to the target business line data to obtain a data stream; wherein, the target service line data is any one service line data in all service line data;
extracting effective data flow in the data flow according to the data filtering rule corresponding to the target service line data; wherein, different tag information has data filtering rules corresponding to the tag information.
4. A method according to claim 3, wherein analyzing each data block to be analyzed according to the corresponding service processing rule, and storing the analysis result in the second message queue comprises:
when the data block to be analyzed is commodity transaction information, counting the sales quantity and sales amount of different commodity categories under different regions in the commodity transaction information according to preset region information and commodity category information;
and storing the top N-bit commodity transaction information with the highest sales quantity and the top N-bit commodity transaction information with the highest sales amount into a second message queue according to the corresponding relation among the region, the commodity category, the sales quantity and the sales amount.
5. A method according to claim 3, wherein analyzing each data block to be analyzed according to the corresponding service processing rule, and storing the analysis result in the second message queue comprises:
when the data block to be analyzed is browsing information of a user, counting click amounts of advertisement information provided with the same tag information in different time periods divided in advance;
and storing the advertisement information to a second message queue according to the corresponding relation among the label information, the time period and the click quantity.
6. A method according to claim 3, wherein analyzing each data block to be analyzed according to the corresponding service processing rule, and storing the analysis result in the second message queue comprises:
when the data block to be analyzed is user login information, counting the user login quantity in different time periods under different regions in the user login information according to preset region information;
and storing the user login information into a second message queue according to the corresponding relation of the region, the time period and the user login quantity.
7. The data processing device is characterized by comprising a recording unit, an extracting unit, a statistics unit and an analysis unit;
the recording unit is used for recording the acquired various service line data to a first message queue;
the extracting unit is used for extracting the effective data streams of various service line data in the first message queue;
the statistics unit is used for counting each effective data stream by utilizing a sliding window according to preset window time to obtain a data block to be analyzed; the sliding window is a number of packets that the receiver tells the sender to be able to transmit at each instant;
the analysis unit is used for analyzing each data block to be analyzed according to the corresponding business processing rule, and storing the obtained analysis result into the second message queue.
8. The apparatus of claim 7, wherein the recording unit is specifically configured to add tag information to each type of service line data according to a preset classification rule; and recording various business line data added with the tag information to a first message queue.
9. A data processing apparatus, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the data processing method according to any one of claims 1 to 6.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the data processing method according to any one of claims 1 to 6.
CN201911367948.XA 2019-12-26 2019-12-26 Data processing method, device and computer readable storage medium Active CN111143415B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911367948.XA CN111143415B (en) 2019-12-26 2019-12-26 Data processing method, device and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911367948.XA CN111143415B (en) 2019-12-26 2019-12-26 Data processing method, device and computer readable storage medium

Publications (2)

Publication Number Publication Date
CN111143415A CN111143415A (en) 2020-05-12
CN111143415B true CN111143415B (en) 2023-12-29

Family

ID=70520482

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911367948.XA Active CN111143415B (en) 2019-12-26 2019-12-26 Data processing method, device and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN111143415B (en)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111901352B (en) * 2020-07-30 2023-08-25 彩讯科技股份有限公司 Method, device, server and storage medium for message distribution processing
CN112035534A (en) * 2020-09-18 2020-12-04 上海依图网络科技有限公司 Real-time big data processing method and device and electronic equipment
CN112506978A (en) * 2020-12-15 2021-03-16 中国联合网络通信集团有限公司 Big data real-time processing method, device and equipment
CN112751726B (en) * 2020-12-17 2022-09-09 北京达佳互联信息技术有限公司 Data processing method and device, electronic equipment and storage medium
CN112633904B (en) * 2020-12-30 2024-04-30 中国平安财产保险股份有限公司 Complaint behavior analysis method, apparatus, device and computer readable storage medium
CN113360564A (en) * 2021-07-12 2021-09-07 杭州安恒信息技术股份有限公司 ETL-based data stream processing method, system, device and readable storage medium
CN113626218A (en) * 2021-07-30 2021-11-09 江苏苏宁物流有限公司 Data processing method, data processing device, storage medium and computer equipment
CN113609202B (en) * 2021-08-11 2024-09-06 湖南快乐阳光互动娱乐传媒有限公司 Data processing method and device
CN113993001B (en) * 2021-09-08 2024-04-12 四创电子股份有限公司 Real-time stream analysis alarm method based on sliding data window
CN116266183A (en) * 2021-12-16 2023-06-20 中移(苏州)软件技术有限公司 Data analysis method, device, equipment and computer storage medium
CN118350058B (en) * 2024-06-20 2024-08-27 江西省送变电工程有限公司 Electric power material data management method and system

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104615777A (en) * 2015-02-27 2015-05-13 浪潮集团有限公司 Method and device for real-time data processing based on stream-oriented calculation engine
CN105512297A (en) * 2015-12-10 2016-04-20 中国测绘科学研究院 Distributed stream-oriented computation based spatial data processing method and system
CN105786941A (en) * 2014-12-26 2016-07-20 中国移动通信集团上海有限公司 Information mining method and device
CN106156026A (en) * 2015-03-24 2016-11-23 中国人民解放军国防科学技术大学 A kind of method based on the data online anomaly of stream fictitious assets
CN106528865A (en) * 2016-12-02 2017-03-22 航天科工智慧产业发展有限公司 Quick and accurate cleaning method of traffic big data
WO2017092582A1 (en) * 2015-12-01 2017-06-08 阿里巴巴集团控股有限公司 Data processing method and apparatus
WO2017185576A1 (en) * 2016-04-25 2017-11-02 百度在线网络技术(北京)有限公司 Multi-streaming data processing method, system, storage medium, and device
CN108287905A (en) * 2018-01-26 2018-07-17 华南理工大学 A kind of extraction of network flow feature and storage method
CN108874812A (en) * 2017-05-10 2018-11-23 腾讯科技(北京)有限公司 A kind of data processing method and server, computer storage medium
CN108874834A (en) * 2017-05-16 2018-11-23 北京嘀嘀无限科技发展有限公司 A kind of data processing method, processing system and computer installation
CN109471898A (en) * 2018-12-19 2019-03-15 华迪计算机集团有限公司 It is a kind of for data to be carried out with the method and system of shared distribution
CN109905412A (en) * 2019-04-28 2019-06-18 山东渔翁信息技术股份有限公司 A kind of parallel encrypting and deciphering processing method of network data, device and medium

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7509650B2 (en) * 2004-05-20 2009-03-24 International Business Machines Corporation Enhance browsing of messages in a message queue
US9639895B2 (en) * 2007-08-30 2017-05-02 Chicago Mercantile Exchange, Inc. Dynamic market data filtering
US20130339473A1 (en) * 2012-06-15 2013-12-19 Zynga Inc. Real time analytics via stream processing

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105786941A (en) * 2014-12-26 2016-07-20 中国移动通信集团上海有限公司 Information mining method and device
CN104615777A (en) * 2015-02-27 2015-05-13 浪潮集团有限公司 Method and device for real-time data processing based on stream-oriented calculation engine
CN106156026A (en) * 2015-03-24 2016-11-23 中国人民解放军国防科学技术大学 A kind of method based on the data online anomaly of stream fictitious assets
WO2017092582A1 (en) * 2015-12-01 2017-06-08 阿里巴巴集团控股有限公司 Data processing method and apparatus
CN105512297A (en) * 2015-12-10 2016-04-20 中国测绘科学研究院 Distributed stream-oriented computation based spatial data processing method and system
WO2017185576A1 (en) * 2016-04-25 2017-11-02 百度在线网络技术(北京)有限公司 Multi-streaming data processing method, system, storage medium, and device
CN106528865A (en) * 2016-12-02 2017-03-22 航天科工智慧产业发展有限公司 Quick and accurate cleaning method of traffic big data
CN108874812A (en) * 2017-05-10 2018-11-23 腾讯科技(北京)有限公司 A kind of data processing method and server, computer storage medium
CN108874834A (en) * 2017-05-16 2018-11-23 北京嘀嘀无限科技发展有限公司 A kind of data processing method, processing system and computer installation
CN108287905A (en) * 2018-01-26 2018-07-17 华南理工大学 A kind of extraction of network flow feature and storage method
CN109471898A (en) * 2018-12-19 2019-03-15 华迪计算机集团有限公司 It is a kind of for data to be carried out with the method and system of shared distribution
CN109905412A (en) * 2019-04-28 2019-06-18 山东渔翁信息技术股份有限公司 A kind of parallel encrypting and deciphering processing method of network data, device and medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于海量数据的消息队列的性能对比与优化方案;刘峰;鄂海红;;软件(第10期);全文 *

Also Published As

Publication number Publication date
CN111143415A (en) 2020-05-12

Similar Documents

Publication Publication Date Title
CN111143415B (en) Data processing method, device and computer readable storage medium
CN107168854B (en) Internet advertisement abnormal click detection method, device, equipment and readable storage medium
CN111639138B (en) Data processing method, device, equipment and storage medium
US8849798B2 (en) Sampling analysis of search queries
US20210035126A1 (en) Data processing method, system and computer device based on electronic payment behaviors
US20060098647A1 (en) Monitoring and reporting enterprise data using a message-based data exchange
CN106815254B (en) Data processing method and device
US8949315B2 (en) System and method for generating web analytic reports
CN111311136A (en) Wind control decision method, computer equipment and storage medium
CN110060087B (en) Abnormal data detection method, device and server
CN106131083A (en) A kind of attack message detection and take precautions against method and switch
CN111062799A (en) Method and device for managing family client, electronic equipment and storage medium
CN106294676B (en) A kind of data retrieval method of ecommerce government system
CN110675078A (en) Marketing company risk diagnosis method, system, computer terminal and storage medium
CN112633842A (en) Task pushing method, device and system
CN102982048A (en) Method and device for assessing junk information mining rule
CN115357629A (en) Processing method, system, electronic device and storage medium for financial data stream
CN116887340B (en) Real-time pushing system for short message status report
CN114022051A (en) Index fluctuation analysis method, storage medium and electronic equipment
CN113225325B (en) IP (Internet protocol) blacklist determining method, device, equipment and storage medium
CN109919197B (en) Random forest model training method and device
CN110866241A (en) Evaluation model generation and equipment association method, device and storage medium
CN110032596A (en) Traffic Anomaly user identification method and system
CN112560992B (en) Method, device, electronic equipment and storage medium for optimizing picture classification model
CN116151670B (en) Intelligent evaluation method, system and medium for marketing project quality of marketing business

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant