WO2017080171A1 - 大数据的筛选分析方法及系统 - Google Patents

大数据的筛选分析方法及系统 Download PDF

Info

Publication number
WO2017080171A1
WO2017080171A1 PCT/CN2016/083187 CN2016083187W WO2017080171A1 WO 2017080171 A1 WO2017080171 A1 WO 2017080171A1 CN 2016083187 W CN2016083187 W CN 2016083187W WO 2017080171 A1 WO2017080171 A1 WO 2017080171A1
Authority
WO
WIPO (PCT)
Prior art keywords
screening
data
analysis
dimension
screening analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/083187
Other languages
English (en)
French (fr)
Inventor
张幼明
周猛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Le Holdings Beijing Co Ltd
LeCloud Computing Co Ltd
Original Assignee
Le Holdings Beijing Co Ltd
LeCloud Computing Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Le Holdings Beijing Co Ltd, LeCloud Computing Co Ltd filed Critical Le Holdings Beijing Co Ltd
Priority to US15/248,592 priority Critical patent/US20170139969A1/en
Publication of WO2017080171A1 publication Critical patent/WO2017080171A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/73—Querying
    • G06F16/735—Filtering based on additional data, e.g. user or group profiles
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9535—Search customisation based on user profiles and personalisation

Definitions

  • the invention relates to the field of data analysis, in particular to a method and system for screening and analyzing big data.
  • the method for screening and analyzing data is only to perform analysis on data in a single dimension, or to perform combined screening in multiple dimensions.
  • the screening defect in a single dimension is that if the data information points are hidden under multiple filtering dimensions, it is difficult to find;
  • the disadvantage of the combined filtering is that when a certain dimension sub-item is determined for data analysis, the selection of the sub-items depends largely on The experience of the person making the judgment leads to an erroneous judgment. Regardless of whether it is a single-dimension screening method or a combined dimension screening method, when the final screening result cannot be obtained due to the selection of the wrong screening dimension in the screening process, it is necessary to re-screen, which seriously affects the screening efficiency.
  • monitoring or analyzing the traffic or the situation of the target information is usually implemented on the operating platform through a combination of different screening dimensions, including: region, city, operating system, browser, gender, age range, etc.
  • the monitoring method of the prior art is to select the sub-items in each screening dimension according to the previous experience to perform combined screening analysis on the target information. If the target information happens to be the problem information point, the monitoring is completed, otherwise the screening dimension sub-item is re-selected. The other permutations and combinations were screened for analysis to complete the monitoring.
  • the method can monitor information such as video traffic and video jam, the processing amount of the entire processing process is large, which results in a large processor load and low processing efficiency, which is not conducive to popularization and application.
  • even if the information point of the suspected problem is found by the method, it is difficult to confirm that the information point is optimal because there are a large number of other arrangement combinations.
  • the embodiment of the invention provides a method and a system for screening and analyzing big data, which are used to solve the defect that the data can only be combined and filtered in multiple dimensions in the prior art, and realize multiple rounds of screening analysis of the data to obtain more accurate. Filter the results.
  • An embodiment of the present invention provides a method for screening and analyzing big data, including multiple rounds of screening analysis, and each round of screening analysis includes:
  • the number of rounds of the multiple rounds of screening analysis is determined according to the number of screening dimensions and the target requirements.
  • the embodiment of the present invention provides a screening and analysis system for big data, configured to perform multiple rounds of screening analysis, the system comprising:
  • Filtering the analysis unit configured to perform screening analysis on the data in the filtered data set according to an unselected screening dimension
  • the target requires a determination unit that is configured to provide a target requirement
  • a data group generating unit to be filtered configured to save data corresponding to at least one dimension sub-item under the screening dimension that meets a target requirement as a data group to be filtered in a next round;
  • the number of rounds of the multiple rounds of screening analysis is determined according to the number of screening dimensions and the target requirements.
  • the screening analysis method and system provided by the invention gradually screens the processed data through multiple screening dimensions to form multiple rounds of screening analysis, and each round of screening analysis uses the previous round of screening results as the current round of screening analysis to be filtered data. Group, so that each round of screening analysis is smaller than the amount of data in the previous round of screening analysis. Therefore, compared with the prior art combined screening under multiple screening conditions at one time, it is not easy to overburden the system due to excessive data volume. Therefore, the problem of collapse, and the target requirements to be met in each round of screening analysis are set according to the reference value of the data group to be screened under the screening sub-item of the round, which improves the accuracy of the screening analysis.
  • FIG. 1 is a flow chart of a screening analysis method according to an embodiment of the present invention.
  • FIG. 2 is a flow chart of a screening analysis method according to another embodiment of the present invention.
  • FIG. 3 is a schematic structural view of a screening analysis system according to an embodiment of the present invention.
  • FIG. 4 is a schematic structural diagram of a server for implementing a screening analysis method according to an embodiment of the present invention.
  • the invention is applicable to a wide variety of general purpose or special purpose computing system environments or configurations.
  • the invention may be described in the general context of computer-executable instructions executed by a computer, such as a program module.
  • program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types.
  • the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network.
  • program modules can be located in both local and remote computer storage media including storage devices.
  • FIG. 1 is a flow chart of a screening analysis method according to an embodiment of the present invention.
  • the screening analysis method includes multiple rounds of screening analysis processes. Among them, each round of screening analysis includes:
  • the screening analysis server performs screening analysis on the data in the screening data group according to an unselected filtering dimension
  • the screening analysis server saves the data corresponding to the target requirement and the at least one dimension sub-item corresponding to the screening dimension in the round screening analysis as the next round of the to-be-screened data group.
  • the number of rounds of multiple rounds of screening analysis in the screening analysis method is determined by the number of screening dimensions and target requirements.
  • the screening analysis server may set the attributes of the data in advance, and set the adapted attributes as the filterable attributes, thereby obtaining the screening dimension.
  • the screening dimensions may include: region, city, operating system, browser, gender, age range, and the like.
  • the dimension sub-item in each dimension is a specific classification item of the screening dimension.
  • the dimension sub-item whose filtering dimension is a region may be a geographically-divided partition (for example, a southern region or a northern region), and may be a residential cell.
  • the division, or the division by the business circle may also be the division of the administrative area (for example, Beijing area, Shanghai area, etc.).
  • the target requirement is the basis for screening and analyzing the filtered data. It can be understood as the screening result obtained by the screening analysis server, for example, the obtained data value is the largest, the smallest, the trend is the smoothest, and the like.
  • the screening analysis server can get the desired screening results from the data set to be filtered.
  • the number of rounds in the screening analysis process that is, in order to obtain the required screening result, several rounds of screening analysis of the data are required, which are determined by the number of screening dimensions and the target requirements. For example, the number of rounds of the screening analysis process does not exceed the screening dimension. Quantity; In the screening analysis process, the screening analysis server obtains the screening result that meets the target requirements, then the screening analysis process stops, and the number of rounds of the screening analysis process is also determined.
  • the screening analysis server performs screening analysis on multiple rounds of screening data through multiple screening dimensions, and each round of screening analysis uses the screening result of the previous round as the current round of screening analysis.
  • the data set to be screened except the first round, so that each round of screening analysis is smaller than the amount of data of the previous round of screening analysis, so the present invention is compared with the prior art in combination screening under multiple screening conditions at one time.
  • the screening analysis method shown in the embodiment is not easy to cause the system to be overburdened and collapsed due to the excessive amount of data, and the target requirements to be met in each round of screening analysis are The accuracy of the screening analysis is improved according to the reference value setting of the data group to be filtered under the screening sub-item of the round.
  • the screening analysis method includes multiple rounds of screening analysis processes. Among them, each round of screening analysis includes:
  • the screening analysis server performs screening analysis on the data in the screening data group according to an unselected screening dimension
  • the screening analysis server saves data corresponding to the target requirement and corresponding to at least one dimension sub-item in the screening dimension as the next round of data to be filtered;
  • S203 The screening analysis server generates and saves a corresponding screening path.
  • the number of rounds of multiple rounds of screening analysis in the screening analysis method is determined by the number of screening dimensions and target requirements.
  • the screening analysis method of the embodiment shown in FIG. 2 saves the data corresponding to the target requirement and the at least one dimension sub-item under the screening dimension as the next round to be screened in S202 with respect to the method shown in FIG. After the data group, the method further includes S203: the screening analysis server generates and saves the corresponding screening path.
  • the screening path is saved, and when the screening result of the pending data is queried later, the saved screening path is used as the entrance of the combined query, and the same screening is obtained by one screening. As a result, the burden of the system repeating multiple rounds of screening analysis is reduced.
  • the screening analysis server withdraws the erroneous screening analysis, and deletes the filtered and generated filtering paths under the reclaimed screening analysis.
  • the screening path is removed by withdrawing the round of screening analysis, and the screening path is removed, so that the round of screening analysis is removed from the multi-round screening analysis.
  • the data becomes the next round of data to be filtered, which can avoid the trouble of re-selecting the filtering dimension or its sub-items of the round dimension sub-item from the initial data for screening analysis.
  • the target requirements in the embodiment of the present invention include: the value corresponding to the data in the data group to be filtered is the largest, and the value corresponding to the data in the data group to be filtered is the smallest and largest.
  • the absolute value of the difference between the value and the minimum value is greater than a predetermined threshold; or the fluctuation range of the value corresponding to the data under each dimension sub-item relative to the reference value is greater than a predetermined range.
  • the predetermined threshold, the reference value, and the predetermined range are determined based on historical data in the history database.
  • a large amount of historical result data stored in the system may be used as a reference, and the threshold and the range are set by using the maximum value, the minimum value, and the predetermined threshold or reference value under the dimension sub-items in the data group to be filtered.
  • Screening analysis is carried out with the predetermined range, and the screening results obtained by each screening analysis are saved in the historical database to guide the subsequent screening analysis.
  • the historical database is continuously expanded and updated by more and more accurate data, compared with the prior art. The screening analysis based on the choices made by personal experience is more accurate.
  • the screening analysis system includes a screening analysis unit 301, a target requirement determination unit 302, and a to-be-screened data group generation unit 303.
  • the screening analysis unit 301 is configured to perform screening analysis on the data in the data group to be filtered generated by the data group to be filtered 303 to be filtered according to an unselected screening dimension.
  • the target requirement determining unit 302 is connected to the screening analyzing unit 301, and is configured to provide a target request to the screening analyzing unit 301, and the target requirements provided include: a maximum value corresponding to the data in the data group to be filtered, and data in the data group to be filtered. Corresponding value minimum requirement, and maximum value The absolute value of the difference between the minimum value and the minimum value is greater than a predetermined threshold; or the fluctuation range of the value corresponding to the data under each dimension sub-item relative to the reference value is greater than a predetermined range.
  • the to-be-screened data group generation unit 303 is connected to the screening analysis unit 301 for at least one of the screening dimensions in the round screening analysis performed by the screening analysis unit 301 to satisfy the target requirement provided by the target requirement determining unit 302.
  • the data of the dimension sub-item is saved as the data set to be filtered for the next round of screening analysis.
  • the screening analysis unit 301 performs screening results by performing multiple rounds of screening analysis on the data through multiple screening dimensions, and each round of screening analysis is the last round of the data to be filtered generating unit 303.
  • the screening results as the current round of screening analysis of the data to be screened (except the first round), so that each round of screening analysis is smaller than the amount of data in the previous round of screening analysis, so with the prior art one-time under multiple screening conditions Compared with the combined screening, the screening analysis method shown in the embodiment of the present invention is not easy to cause the system to be overburdened and collapsed due to excessive data volume, and the target required by the target requirement determining unit 302 to be satisfied in each round of screening analysis is The requirements are based on the reference value setting of the data group to be filtered under the screening sub-item of the round, which improves the accuracy of the screening analysis.
  • the screening analysis system in this embodiment is a server or a server cluster, wherein each unit may be a separate server or a server cluster.
  • the interaction between the units is represented by a server or a server cluster corresponding to each unit.
  • the plurality of servers or server clusters together form the screening analysis system of the present invention.
  • the plurality of servers or server clusters together constitute the screening analysis system of the present invention includes:
  • the target requires determining a server or server cluster to provide a target requirement to the screening analysis server or the server cluster.
  • the target requirements may include: the maximum value corresponding to the data in the data group to be filtered, and the data in the data group to be filtered corresponds to The minimum value requirement, and the absolute value of the difference between the maximum value and the minimum value is greater than a predetermined threshold; or the fluctuation range of the value corresponding to the data under each dimension sub-item relative to the reference value is greater than a predetermined range;
  • a data group generation server or server cluster to be filtered which is used to determine the target requirements provided by the server or server cluster to meet the target requirements, corresponding to the screening analysis server or server set
  • the data of at least one dimension sub-item under the screening dimension in the round of screening analysis performed by the group is saved as the to-be-screened data set of the next round of screening analysis.
  • the screening analysis unit and the data group to be filtered generating unit together comprise a first server or a first server cluster
  • the target requirement determining unit constitutes a second server or a second server cluster.
  • the interaction between the above units is represented by an interaction between each of the first server and the second server or an interaction between the first server cluster and the second server cluster, the first server and the second server or the first The server cluster and the second server cluster together constitute the screening analysis system of the present invention.
  • the screening analysis system in the embodiment shown in FIG. 3 may further include a screening path processing unit 304 connected to the to-be-screened data group generating unit 303 for determining that the target requirement will be met. After the data required by the target 302 and corresponding to the at least one dimension sub-item in the screening dimension is saved as the next round of the data group to be filtered, the corresponding screening path is generated and saved.
  • the screening path processing unit 304 saves the screening path after each round of screening analysis, and may query the saved screening path as the entrance of the combined query when querying the screening result of the pending data in the future.
  • the same screening results are obtained by one screening, reducing the burden of the system repeating multiple rounds of screening analysis.
  • the screening path processing unit in this embodiment may be a server or a server cluster.
  • the interaction between the screening path processing unit and all the units in the embodiment shown in FIG. 3 represents an interaction between servers or server clusters corresponding to the units, and the plurality of servers or server clusters together constitute the present invention. Screening analysis system.
  • the screening analysis unit and the data group to be filtered generating unit together comprise a first server or a first server cluster
  • the target requirement determining unit constitutes a second server or a second server cluster
  • the screening path processing unit constitutes a third server or a third server cluster.
  • the interaction between the above units is represented by an interaction between each of the first server to the third server or an interaction between the first server cluster and the third server cluster, the first server to the first The three servers or the first server cluster to the third server cluster together constitute the screening analysis system of the present invention.
  • the screening path processing unit 304 may be further configured to delete the generated and saved screenings of the retrieved screening analysis after each round of screening analysis is withdrawn. path.
  • the screening path is removed by withdrawing the round screening analysis and the filtering path is deleted by the filtering path processing unit 304, so that the multiple rounds of screening analysis are removed.
  • the data obtained by the round screening analysis becomes the next round of data to be filtered, which can avoid the trouble of re-selecting the screening dimension of the round dimension sub-item or its sub-items from the initial data for screening analysis.
  • the screening analysis system of the embodiment of the present invention may further include a predetermined threshold determining unit 305 and a history database 306 connected to the target determining unit 302.
  • the predetermined threshold determining unit 305 is configured to determine a predetermined threshold, a reference value, and a predetermined range based on historical data in the history database 306, and the history database 306 can be updated according to the screening result after the multiple rounds of screening analysis.
  • some data in the historical database can be uploaded through the network through the user equipment.
  • the predetermined threshold determining unit and the history database in this embodiment may each be a separate server or a server cluster.
  • the interaction between the predetermined threshold determining unit, the history database, and all the units in the foregoing embodiment is expressed as an interaction between servers or server clusters corresponding to the respective units, and the plurality of servers or server clusters collectively constitute the present invention. Screening analysis system.
  • the screening analysis unit and the data group to be filtered generating unit together comprise a first server or a first server cluster
  • the target requirement determining unit, the predetermined threshold determining unit and the historical database together constitute a second server or a second server cluster
  • the screening path processing unit Form a third server or a third server cluster.
  • the interaction between the above units is represented by an interaction between each of the first server to the third server or an interaction between the first server cluster to the third server cluster, the first server to the third server or the first The server cluster to the third server cluster together constitute the screening analysis system of the present invention System.
  • a related function module can be implemented by a hardware processor.
  • the server 400 can include:
  • a processor 410 a communications interface 420, a memory 430, and a communication bus 440. among them:
  • the processor 410, the communication interface 420, and the memory 430 complete communication with each other via the communication bus 440.
  • the communication interface 420 is configured to communicate with a network element such as a client.
  • the processor 410 is configured to execute the program 432, and specifically may perform the related steps in the foregoing method embodiments.
  • program 432 can include program code, the program code including computer operating instructions.
  • the processor 410 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
  • CPU central processing unit
  • ASIC Application Specific Integrated Circuit
  • a memory for storing computer operating instructions
  • a processor configured to execute the computer operating instructions of the memory storage to perform:
  • the data corresponding to the target requirement and corresponding to at least one dimension sub-item in the screening dimension is saved as the next round of data to be filtered.
  • the present invention will be further described by taking the use of video traffic of a user in the video field as an example.
  • filtering dimensions such as a region, an operating system, a browser, etc.
  • Each of the screening conditions has its own dimension sub-items.
  • the region includes some provinces in China such as Beijing, Shanghai, Tianjin, Guangdong, etc.
  • the operating system includes Windows, Android, IOS system, and the browser includes 360 browser and Baidu browsing. , Google Chrome.
  • the screening analysis system performs the first round of screening analysis as follows.
  • the data group to be filtered generating unit uses the data in the initial database, that is, the traffic used by the user to watch the video as the data group to be filtered.
  • a screening dimension such as a region, is randomly selected, and the screening analysis unit performs screening under the screening dimension.
  • the target requirement determining unit determines that the target requirement in the round screening analysis is to find the maximum value and the minimum value of the user usage flow under the sub-item of the geographic dimension, and the difference between the maximum value and the minimum value is greater than a predetermined threshold, and the predetermined threshold value is determined by the predetermined threshold value unit.
  • the historical database is determined to be 1000T.
  • the screening analysis unit Through the screening analysis unit, users in Beijing, Shanghai, Tianjin, Guangdong and other places watched the traffic used by the video: Beijing users used 568T, Shanghai users used 642T, Tianjin users used 295T, and Guangdong users used 1546T.
  • the maximum value is 1546T in Guangdong and the minimum value is 295T in Tianjin.
  • the difference between the maximum and minimum values is 1251T, which is greater than the predetermined threshold of 1000T.
  • the usage flow of the dimension sub-items in Guangdong and Tianjin meets the data requirements, so the data set generation unit to be filtered saves the usage traffic of Guangdong and Tianjin as the next round of data to be filtered.
  • the filtering path processing unit generates and saves a corresponding screening path.
  • the screening analysis system performs a second round of screening analysis.
  • the data set to be filtered has become the traffic of users watching videos in Tianjin and Guangdong.
  • the target requirement determining unit determines that the target requirement in the round screening analysis is to find the maximum value of the user usage flow under the sub-item of the operating system dimension, and calculate the minimum value, and the maximum value and the minimum value.
  • the difference is greater than a predetermined threshold, and the predetermined threshold in the current round of screening analysis is determined to be 50T by the predetermined threshold determining unit and the history database.
  • Steps 202 and 203 are repeated: the users in the Guangdong area use the Windows, Android, and IOS operating systems to watch video usage by the screening analysis unit are 658T, 423T, and 460T respectively, and the users in the Tianjin area use Windows, Android, and IOS operating systems to watch.
  • the traffic used by the video is 132T, 95T and 60T respectively.
  • the maximum user traffic usage in Guangdong is 658T, the minimum value is 423T, and the difference between the maximum and minimum values is 235T.
  • the maximum usage flow of the households in Tianjin is 132T, the minimum value is 60T, and the difference between the maximum and minimum values is 72T. Two regions The maximum and minimum values are greater than the predetermined threshold.
  • the to-be-screened data group generation unit saves the traffic used by the users in Guangdong and Tianjin to view the video using the Windows system as the next round of data to be filtered. And, as shown in step 203, after the data group to be filtered in the next round is saved by the data group generating unit to be filtered, the filtering path processing unit generates and saves a corresponding screening path.
  • the screening analysis system performs a third round of screening analysis.
  • the filtering dimension is browser, and the sub-items are 360 browser, Baidu browser and Google browser.
  • the target requirement determining unit determines that the target requirement in the current round of screening analysis is to find the maximum value of the user usage flow under the sub-item of the browser dimension, and calculate the minimum value, and the difference between the maximum value and the minimum value is greater than a predetermined threshold.
  • the predetermined threshold value in the analysis is determined by the predetermined threshold determining unit and the history database as three times the minimum value under each sub-item.
  • Windows users in Guangdong use 360 browser, Baidu browser and Google browser to watch video usage of 75T, 31T and 158T respectively.
  • Windows users in Tianjin use 360 browser, Baidu browser and Google browser.
  • the traffic used to watch the video is 12T, 5T and 23T respectively.
  • the maximum traffic usage of Windows users in Guangdong is 158T, the minimum value is 31T, and the difference between the maximum and minimum values is 127T, which is greater than the predetermined threshold of 92T.
  • the maximum value of the used flow is 23T, the minimum value is 5T, and the difference between the maximum and minimum values is 18T, which is greater than the predetermined threshold of 15T.
  • the Windows users in the two regions use the maximum and minimum traffic usage of each sub-item in the round screening analysis to be greater than the predetermined threshold. Therefore, Windows users in Guangdong use Google Chrome to watch video traffic and Windows users in Tianjin use Google Chrome to watch. The video traffic meets the target requirements.
  • the to-be-screened data group generation unit saves the traffic used by the Windows users of Guangdong and Tianjin to view the video under the Google browser as the next round of data to be filtered.
  • the screening path processing unit generates and saves a corresponding screening path.
  • the screening result is the data of the data to be filtered obtained in the third round of screening analysis, that is, the traffic of the Windows users in Guangdong and Tianjin watching the video under Google Chrome. Save the filter results in the history database to update the history database.
  • the screening path generated and saved by the screening path processing unit in the third round of screening analysis may be used as an entry for the combined query for querying the traffic usage of the video viewed by the user within the specific time.
  • the enterprise can obtain that the users in Guangdong and Tianjin use the Windows system to watch the video to generate the most traffic, and use the Google browser to watch the video under the Windows system. Generates the most traffic, and draws other corresponding conclusions to help companies make relevant decisions, for example, to avoid users in Windows and Tianjin using Windows systems from watching video during peak hours, causing congestion and scheduling more bandwidth for them. .
  • the target requirement in this embodiment may also be a requirement under other reference conditions, for example, the ranking of the data of each region is changed by two or more compared with the reference value in the history database.
  • the setting filter dimensions are: region, carrier, player, video ID, and viewing percentage.
  • the screening analysis unit obtains the video availability rate ranking in Beijing according to the target requirement, and changes more than two times in the past.
  • the data group to be filtered generates the data corresponding to Beijing as the next round of data to be filtered.
  • the data under the dimension item of China Mobile is selected for screening under the video ID dimension, and the screen is filtered by the region (Beijing)-operator (China Mobile)-video ID (Video 1 and Video 2). data.
  • the player dimension is selected for screening, and no data meeting the target requirements is found.
  • the ID (Video 1 and Video 2) filtered data is passed through the data set generating unit to be filtered as the next round of data to be filtered.
  • the embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, ie may be located in one Places, or they can be distributed to multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment. Those of ordinary skill in the art can understand without paying creative labor. And implemented.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Multimedia (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本发明提供一种大数据的筛选分析方法，包括多轮筛选分析，每一轮筛选分析包括：按照一个未选择的筛选维度对待筛选数据组中的数据进行筛选分析；将满足目标要求的、对应于筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组；其中，多轮筛选分析的轮数根据筛选维度的数量和目标要求来确定。本发明还提供了相应的筛选分析系统。本发明通过多轮筛选分析对数据进行逐步筛选，每轮筛选分析都将上轮的筛选结果作为本轮的待筛选数据组，使得每轮筛选分析都比上轮筛选分析的数据量小，与组合筛选相比，不会因数据量过大造成系统负担过大从而崩溃的问题，且目标要求根据待筛选数据组在该轮筛选分析的参考值设置，提高了筛选分析的准确度。

Description

大数据的筛选分析方法及系统 技术领域
本发明涉及数据分析领域，具体涉及一种大数据的筛选分析方法及系统。
背景技术
随着信息化的高速发展，大数据应运而生，为了弥补传统方法无法处理如此量大且非结构的大数据的缺陷，人们研究出了云计算，以云计算为基础的信息存储、分享和挖掘手段，可以便宜、有效地将这些大量、高速、多变化的终端大数据存储下来，然而如何对这些数据进行筛选分析，并且使用筛选结果从不同维度对企业决策进行指导已经成为热门话题。
现有技术中，对数据的筛选分析方法仅是对数据在某单一维度下进行展开分析，或者在多个维度下进行组合筛选。单一维度下的筛选缺陷在于如果数据信息点隐藏在多个筛选维度下，则很难被找到；组合筛选的缺陷在于确定某一维度子项以进行数据分析时，子项的选择很大程度取决于做出判断的人的经验，导致容易出现错误的判断情况。无论是单一维度的筛选方式或是组合维度的筛选方式，针对筛选过程中因选择了错误的筛选维度而无法得到最终的筛选结果时，均需要重新进行筛选，严重影响筛选效率。
例如，在视频领域，通常在操作平台上通过不同筛选维度的组合实现对目标信息的流量或者卡顿情况的监测分析，筛选维度包括：地域、城市、操作系统、浏览器、性别、年龄段等，现有技术的监测方法是根据先前经验在所有筛选维度中分别选取其子项对目标信息进行组合筛选分析，如果该目标信息恰好为问题信息点，则完成监测，否则重新选取筛选维度子项的其它排列组合进行筛选分析完成监测。该方法虽然能实现对视频流量、视频卡顿等信息的监测，但整个处理过程信息处理量大，导致处理器负担较大，处理效率低，不利于推广应用。并且，即使利用该方法找到了疑似问题的信息点，由于存在大量其他排列组合的可能，因此也很难确认该信息点就是最优的。
发明内容
本发明实施例提供一种大数据的筛选分析方法及系统，用以解决现有技术中对数据在多维度下只能进行组合筛选的缺陷，实现对数据的多轮筛选分析以得到更准确的筛选结果。
本发明实施例一方面提供一种大数据的筛选分析方法，包括多轮筛选分析，每一轮筛选分析包括：
按照一个未选择的筛选维度对待筛选数据组中的数据进行筛选分析；
将满足目标要求的、对应于所述筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组；
其中，所述多轮筛选分析的轮数根据筛选维度的数量和目标要求来确定。
另一方面本发明实施例提供一种大数据的筛选分析系统，配置以执行多轮筛选分析，所述系统包括：
筛选分析单元，配置以按照一个未选择的筛选维度对待筛选数据组中的数据进行筛选分析；
目标要求确定单元，配置以提供目标要求；
待筛选数据组生成单元，配置以将满足目标要求的、对应于所述筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组；
其中，所述多轮筛选分析的轮数根据筛选维度的数量和目标要求来确定。
本发明提供的筛选分析方法及系统，通过多个筛选维度对待处理数据进行逐步筛选，形成多轮筛选分析，每一轮筛选分析都是将上一轮的筛选结果作为本轮筛选分析待筛选数据组，使得每轮筛选分析都比上一轮筛选分析的数据量小，因此与现有技术一次性在多个筛选条件下进行组合筛选相比，不容易因数据量过大造成系统负担过大从而崩溃的问题，且每一轮筛选分析中要满足的目标要求均根据其待筛选数据组在该轮的筛选子项下的参考值设置，提高了筛选分析的准确度。
附图说明
为了更清楚地说明本发明实施例的技术方案，下面将对实施例描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提 下，还可以根据这些附图获得其他的附图。
图1是本发明一实施方式的筛选分析方法的流程图；
图2是本发明另一实施方式的筛选分析方法的流程图；
图3是本发明一实施方式的筛选分析系统的结构示意图；
图4为实施本发明实施例的筛选分析方法的服务器的结构示意图。
具体实施方式
为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
需要说明的是，在不冲突的情况下，本申请中的实施方式及实施方式中的特征可以相互组合。
本发明可用于众多通用或专用的计算系统环境或配置中。例如：个人计算机、服务器计算机、手持设备或便携式设备、平板型设备、多处理器系统、基于微处理器的系统、置顶盒、可编程的消费电子设备、网络PC、小型计算机、大型计算机、包括以上任何系统或设备的分布式计算环境等等。
本发明可以在由计算机执行的计算机可执行指令的一般上下文中描述，例如程序模块。一般地，程序模块包括执行特定任务或实现特定抽象数据类型的例程、程序、对象、组件、数据结构等等。也可以在分布式计算环境中实践本发明，在这些分布式计算环境中，由通过通信网络而被连接的远程处理设备来执行任务。在分布式计算环境中，程序模块可以位于包括存储设备在内的本地和远程计算机存储介质中。
最后，还需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”，不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括……”限定的要素，并不排除在包括所述要 素的过程、方法、物品或者设备中还存在另外的相同要素。
图1是本发明一实施方式的筛选分析方法的流程图。如图1所示，该筛选分析方法包括多轮筛选分析过程。其中，每一轮筛选分析包括：
S101：筛选分析服务器按照一个未选择的筛选维度对待筛选数据组中的数据进行筛选分析；
S102：筛选分析服务器将满足目标要求的、对应于该轮筛选分析中筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组。
该筛选分析方法中多轮筛选分析的轮数由筛选维度的数量和目标要求确定。
本发明实施例中筛选分析服务器可以预先对数据具有的属性进行设定，并把适配的属性设定为可筛选的属性，从而得到筛选维度。对视频领域来说，筛选维度，例如，可以包括：地域、城市、操作系统、浏览器、性别、年龄段等。其中，各维度下的维度子项是筛选维度的具体分类项，例如，筛选维度为地域的维度子项可以是地理位置上的分区(例如南方地区、北方地区)，可以是以居民小区为单位的分区，或者以商圈为单位的划分，还可以是行政区域为单位的划分(例如北京地区、上海地区等)。
目标要求是对待筛选数据进行筛选分析的根据，可以理解为需要筛选分析服务器得到的筛选结果，例如得到的数据值最大、最小、趋势最平滑等。通过筛选维度和目标要求，筛选分析服务器可以从待筛选数据组中得到想要的筛选结果。其中，筛选分析过程的轮数，即为了得到所需筛选结果而需要对数据进行几轮筛选分析，由筛选维度的数量和目标要求确定，例如，筛选分析过程的轮数不会超过筛选维度的数量；在筛选分析过程中，筛选分析服务器得到了满足目标要求的筛选结果，则筛选分析过程停止，筛选分析过程的轮数也随即确定。
本发明所示实施例的筛选分析方法中，筛选分析服务器通过多个筛选维度对数据进行多轮筛选分析得到筛选结果，每一轮筛选分析都是将上一轮的筛选结果作为本轮筛选分析待筛选数据组(除第一轮外)，使得每轮筛选分析都比上一轮筛选分析的数据量小，因此与现有技术一次性在多个筛选条件下进行组合筛选相比，本发明实施例所示筛选分析方法不容易因数据量过大造成系统负担过大从而崩溃的问题，且每一轮筛选分析中要满足的目标要求均 根据其待筛选数据组在该轮的筛选子项下的参考值设置，提高了筛选分析的准确度。
图2是本发明另一实施方式的筛选分析方法的流程图。如图2所示，该筛选分析方法包括多轮筛选分析过程。其中，每一轮筛选分析包括：
S201：筛选分析服务器按照一个未选择的筛选维度对待筛选数据组中的数据进行筛选分析；
S202：筛选分析服务器将满足目标要求的、对应于所述筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组；
S203：筛选分析服务器生成和保存相应的筛选路径。
该筛选分析方法中多轮筛选分析的轮数由筛选维度的数量和目标要求确定。
图2所示实施例的筛选分析方法，相对于图1所示方法，在S202将满足目标要求的、对应于所述筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组之后，还包括S203：筛选分析服务器生成和保存相应的筛选路径。
通过S203，在每一轮筛选分析过后保存其筛选路径，可以在以后查询该待处理数据本次的筛选结果时，将保存好的筛选路径作为组合查询的入口，通过一次筛选就得到同样的筛选结果，减少系统重复进行多轮筛选分析的负担。
图2所示实施例的筛选分析方法，当某一轮的筛选分析未得到满足目标要求的数据时，若不再重新选择筛选维度进行筛选分析，则表明之前的筛选路径有误，此时，还包括S204：筛选分析服务器撤回有误的筛选分析，删除撤回的筛选分析下已生成和保存的筛选路径。
在筛选分析过程中，如果发现某一轮的选择的维度子项有错误，筛选路径不正确，通过撤回该轮筛选分析并删除该筛选路径，使得多轮筛选分析中除去该轮筛选分析得到的数据成为下一轮的待筛选数据组，可以避免从最初始的数据重新选择删除了该轮维度子项的筛选维度或其子项进行筛选分析的麻烦。
作为图1或图2所示方法实施例的进一步优化，本发明实施例中的目标要求包括：待筛选数据组中的数据对应的数值最大、待筛选数据组中的数据对应的数值最小以及最大数值和最小数值之差的绝对值大于预定阈值；或各维度子项下数据对应的数值相对于参考值的波动范围大于预定范围。预定阈值、参考值和预定范围根据历史数据库中的历史数据来确定。
本发明实施例可以将系统存有的大量的历史结果数据作为参考，并以此设定阈值和范围，利用待筛选数据组中在维度子项下的最大值、最小值和预定阈值或参考值和预定范围进行筛选分析，且每次筛选分析得到的筛选结果均保存在历史数据库中，为以后的筛选分析作指导，历史数据库不断被越来越准确的数据扩充和更新，相对现有技术中根据个人经验做出的选择进行筛选分析来说准确度更高。
需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作合并，但是本领域技术人员应该知悉，本申请并不受所描述的动作顺序的限制，因为依据本申请，某些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本申请所必须的。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。
图3是本发明一实施方式的筛选分析系统的结构示意图。本发明所述的筛选分析方法可以基于本实施例中的筛选分析系统实施。如图3所示，该筛选分析系统包括筛选分析单元301、目标要求确定单元302和待筛选数据组生成单元303。
筛选分析单元301用于根据一个未选择的筛选维度对通过待筛选数据组生成单元303生成的待筛选数据组中的数据进行筛选分析。
目标要求确定单元302与筛选分析单元301连接，用于向筛选分析单元301提供目标要求，提供的目标要求包括：待筛选数据组中的数据对应的数值最大的要求，待筛选数据组中的数据对应的数值最小的要求，和最大数值 和最小数值之差的绝对值大于预定阈值的要求；或各维度子项下数据对应的数值相对于参考值的波动范围大于预定范围。
待筛选数据组生成单元303与筛选分析单元301连接，用于将满足目标要求确定单元302提供的目标要求的、对应于通过筛选分析单元301进行的该轮筛选分析中的筛选维度下的至少一个维度子项的数据保存为下一轮筛选分析的待筛选数据组。
本发明所示实施例的筛选分析系统中，筛选分析单元301通过多个筛选维度对数据进行多轮筛选分析得到筛选结果，每一轮筛选分析都是待筛选数据组生成单元303将上一轮的筛选结果作为本轮筛选分析待筛选数据组(除第一轮外)，使得每轮筛选分析都比上一轮筛选分析的数据量小，因此与现有技术一次性在多个筛选条件下进行组合筛选相比，本发明实施例所示筛选分析方法不容易因数据量过大造成系统负担过大从而崩溃的问题，且每一轮筛选分析中要满足的目标要求确定单元302提供的目标要求均根据其待筛选数据组在该轮的筛选子项下的参考值设置，提高了筛选分析的准确度。
本实施例中的筛选分析系统一个服务器或者服务器集群，其中每个单元可以是单独的服务器或者服务器集群，此时，上述单元之间的交互表现为各单元所对应的服务器或者服务器集群之间的交互，所述多个服务器或服务器集群共同构成本发明的筛选分析系统。
具体地，所述多个服务器或服务器集群共同构成本发明的筛选分析系统包括：
筛选分析服务器或服务器集群，用于根据一个未选择的筛选维度对通过待筛选数据组生成服务器或服务器集群生成的待筛选数据组中的数据进行筛选分析；
目标要求确定服务器或服务器集群，用于向筛选分析服务器或服务器集群提供目标要求，提供的目标要求可以包括：待筛选数据组中的数据对应的数值最大的要求，待筛选数据组中的数据对应的数值最小的要求，和最大数值和最小数值之差的绝对值大于预定阈值的要求；或各维度子项下数据对应的数值相对于参考值的波动范围大于预定范围；
待筛选数据组生成服务器或服务器集群，用于将满足目标要求确定服务器或服务器集群提供的目标要求的、对应于通过筛选分析服务器或服务器集 群进行的该轮筛选分析中的筛选维度下的至少一个维度子项的数据保存为下一轮筛选分析的待筛选数据组。
在一种替代实施例中，可以是上述多个单元中的几个单元共同组成一个服务器或者服务器集群。例如：筛选分析单元和待筛选数据组生成单元共同组成第一服务器或者第一服务器集群，目标要求确定单元构成第二服务器或者第二服务器集群。
此时，上述单元之间的交互表现为各第一服务器和第二服务器之间的交互或者第一服务器集群和第二服务器集群之间的交互，所述第一服务器和第二服务器或第一服务器集群和第二服务器集群共同构成本发明的筛选分析系统。
作为图3所示实施例系统的进一步优化，图3所示实施例中的筛选分析系统还可以包括与待筛选数据组生成单元303连接的筛选路径处理单元304，用于在将满足目标要求确定单元302提供的目标要求的、对应于筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组后，生成和保存相应的筛选路径。
本发明实施例中，筛选路径处理单元304在每一轮筛选分析过后保存其筛选路径，可以在以后查询该待处理数据本次的筛选结果时，将保存好的筛选路径作为组合查询的入口，通过一次筛选就得到同样的筛选结果，减少系统重复进行多轮筛选分析的负担。
本实施例中的筛选路径处理单元可以为一个服务器或者服务器集群。此时，筛选路径处理单元和图3所示实施例中的所有单元之间的交互表现为各单元所对应的服务器或者服务器集群之间的交互，所述多个服务器或者服务器集群共同构成本发明的筛选分析系统。
在一种替代实施例中，可以是上述多个单元中的几个单元共同组成一个服务器或者服务器集群。例如：筛选分析单元和待筛选数据组生成单元共同组成第一服务器或者第一服务器集群，目标要求确定单元构成第二服务器或者第二服务器集群，筛选路径处理单元构成第三服务器或者第三服务器集群。
此时，上述单元之间的交互表现为各第一服务器至第三服务器之间的交互或者第一服务器集群至第三服务器集群之间的交互，所述第一服务器至第 三服务器或第一服务器集群至第三服务器集群共同构成本发明的筛选分析系统。
作为图3所示实施例系统的进一步优化，本发明实施例中，筛选路径处理单元304还可以用于在每一轮筛选分析撤回后，删除所述撤回的筛选分析下已生成和保存的筛选路径。
在筛选分析过程中，如果发现某一轮的选择的维度子项有错误，筛选路径不正确，通过撤回该轮筛选分析并通过筛选路径处理单元304删除该筛选路径，使得多轮筛选分析中除去该轮筛选分析得到的数据成为下一轮的待筛选数据组，可以避免从最初始的数据重新选择删除了该轮维度子项的筛选维度或其子项进行筛选分析的麻烦。
作为图3所示实施例系统的进一步优化，本发明实施例的筛选分析系统还可以包括与目标确定单元302连接的预定阈值确定单元305和历史数据库306。预定阈值确定单元305用于根据历史数据库306中的历史数据来确定预定阈值、参考值和预定范围，历史数据库306能够根据所述多轮筛选分析后的筛选结果更新。其中，对视频领域来说，历史数据库中部分数据可以通过用户设备经网络上传得到。
本实施例中预定阈值确定单元和历史数据库分别可以是单独的服务器或者服务器集群。此时，预定阈值确定单元、历史数据库和前述实施例中的所有单元之间的交互表现为各单元所对应的服务器或者服务器集群之间的交互，所述多个服务器或者服务器集群共同构成本发明的筛选分析系统。
在一种替代实施例中，可以是上述多个单元中的几个单元共同组成一个服务器或者服务器集群。例如：筛选分析单元和待筛选数据组生成单元共同组成第一服务器或者第一服务器集群，目标要求确定单元、预定阈值确定单元和历史数据库共同构成第二服务器或者第二服务器集群，筛选路径处理单元构成第三服务器或者第三服务器集群。
此时，上述单元之间的交互表现为各第一服务器至第三服务器之间的交互或者第一服务器集群至第三服务器集群之间的交互，所述第一服务器至第三服务器或第一服务器集群至第三服务器集群共同构成本发明的筛选分析系 统。
本发明实施例中可以通过硬件处理器(hardware processor)来实现相关功能模块。
如图4所示，为实施本发明实施例的筛选分析服务器的结构示意图，本申请具体实施例并不对服务器400的具体实现做限定。如图4所示，该服务器400可以包括：
处理器(processor)410、通信接口(Communications Interface)420、存储器(memory)430、以及通信总线440。其中：
处理器410、通信接口420、以及存储器430通过通信总线440完成相互间的通信。
通信接口420，用于与比如客户端等的网元通信。
处理器410，用于执行程序432，具体可以执行上述方法实施例中的相关步骤。
具体地，程序432可以包括程序代码，所述程序代码包括计算机操作指令。
处理器410可能是一个中央处理器CPU，或者是特定集成电路ASIC(Application Specific Integrated Circuit)，或者是被配置成实施本申请实施例的一个或多个集成电路。
上述实施例中的服务器中：
存储器，用于存放计算机操作指令；
处理器，用于执行所述存储器存储的计算机操作指令，以执行：
按照一个未选择的筛选维度对待筛选数据组中的数据进行筛选分析；
将满足目标要求的、对应于所述筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组。
下面将以视频领域中查看用户的视频流量的使用情况为例对本发明做进一步的说明。
企业想要在服务平台上查看某特定时段内用户观看视频使用的流量以发现隐藏的信息时，先设置多个筛选维度，如地域、操作系统、浏览器等，其 中每个筛选条件下都有各自的维度子项，例如，地域包括北京、上海、天津、广东等中国的部分省份，操作系统包括Windows、Android、IOS系统，浏览器包括360浏览器、百度浏览器、谷歌浏览器。
筛选分析系统执行第一轮筛选分析，过程如下。
待筛选数据组生成单元将初始数据库中的数据即用户观看视频使用的流量作为待筛选数据组。随机选择一个筛选维度，例如地域，在该筛选维度下通过筛选分析单元进行筛选。目标要求确定单元确定该轮筛选分析中目标要求是寻找到地域维度的子项下用户使用流量的最大值和最小值，且最大值和最小值之差大于预定阈值，预定阈值由预定阈值确定单元和历史数据库确定为1000T。
通过筛选分析单元得到北京、上海、天津、广东等地的用户观看视频使用的流量：北京的用户使用了568T，上海的用户使用了642T，天津的用户使用了295T，广东的用户使用了1546T。由此得到最大值为广东1546T，最小值为天津295T，同时最大最小值之差为1251T，大于预定阈值1000T。维度子项广东和天津下的使用流量满足数据要求，因此待筛选数据组生成单元将广东和天津的使用流量保存为下一轮的待筛选数据组。并且，如步骤203所示，通过待筛选数据组生成单元将下一轮的待筛选数据组保存后，筛选路径处理单元生成和保存相应的筛选路径。
筛选分析系统执行第二轮筛选分析。
待筛选数据组已经变为天津、广东地区用户观看视频的流量。选择操作系统作为本轮的筛选维度，目标要求确定单元确定该轮筛选分析中目标要求是寻找到操作系统维度的子项下用户使用流量的最大值，同时计算最小值，且最大值和最小值之差大于预定阈值，本轮筛选分析中预定阈值由预定阈值确定单元和历史数据库确定为50T。
重复步骤202和步骤203：通过筛选分析单元得到广东地区的用户使用Windows、Android和IOS操作系统观看视频使用的流量分别为658T、423T和460T，天津地区的用户使用Windows、Android和IOS操作系统观看视频使用的流量分别是132T、95T和60T，由此得到广东地区的用户使用流量的最大值为658T，最小值为423T，最大最小值之差为235T；天津地区的户使用流量的最大值为132T，最小值为60T，最大最小值之差为72T。两个地区 的最大最小值均大于预定阈值，故广东地区下使用Windows系统的用户的流量和天津地区下使用Windows系统的用户的流量满足目标要求。因此待筛选数据组生成单元将广东和天津的用户在使用Windows系统下观看视频使用的流量保存为下一轮的待筛选数据组。并且，如步骤203所示，通过待筛选数据组生成单元将下一轮的待筛选数据组保存后，筛选路径处理单元生成和保存相应的筛选路径。
筛选分析系统执行第三轮筛选分析。
筛选维度为浏览器，子项为360浏览器、百度浏览器和谷歌浏览器。目标要求确定单元确定本轮筛选分析中的目标要求是寻找到浏览器维度的子项下用户使用流量的最大值，同时计算最小值，且最大值和最小值之差大于预定阈值，本轮筛选分析中预定阈值由预定阈值确定单元和历史数据库确定为各子项下最小数值的3倍数值。
通过筛选分析单元得到广东地区Windows用户使用360浏览器、百度浏览器和谷歌浏览器观看视频使用的流量分别为75T、31T和158T，天津地区Windows用户使用360浏览器、百度浏览器和谷歌浏览器观看视频使用的流量分别是12T、5T和23T，由此得到广东地区Windows用户使用流量的最大值为158T，最小值为31T，最大最小值之差为127T，大于预定阈值92T；天津地区Windows用户使用流量的最大值为23T，最小值为5T，最大最小值之差为18T，大于预定阈值15T。两地区的Windows用户在该轮筛选分析中各自的子项下使用流量的最大最小值均大于预定阈值，故广东地区Windows用户使用谷歌浏览器观看视频的流量和天津地区Windows用户使用谷歌浏览器观看视频的流量满足目标要求。此时待筛选数据组生成单元将广东和天津的Windows用户在谷歌浏览器下观看视频使用的流量保存为下一轮的待筛选数据组。并且，如步骤203所示，下一轮的待筛选数据组被保存后，筛选路径处理单元生成和保存相应的筛选路径。
通过判断得到所有筛选维度下的筛选分析均执行完毕，故筛选结果为第三轮筛选分析中得到待筛选数据组，即广东和天津的Windows用户在谷歌浏览器下观看视频使用的流量。将该筛选结果保存在历史数据库中以更新历史数据库。第三轮筛选分析中筛选路径处理单元生成和保存的筛选路径可以作为下次查询该特定时间内用户观看视频的流量使用情况的组合查询的入口。
通过硬件处理器和服务平台实现相关功能并将筛选结果显示出来后，企业可以得出广东地区和天津地区的用户使用Windows系统观看视频产生的流量最多，且在Windows系统下使用谷歌浏览器观看视频产生的流量最多，并由此得出其他相应的结论，以帮助企业的相关决策，例如为了避免广东地区和天津地区使用Windows系统的用户在高峰时期观看视频产生拥堵，为其调度更多的带宽。
本实施例中的目标要求也可以是其他参考条件下的要求，例如：各地区数据的排名与历史数据库中的参考值相比变化两位以上等。例如，查找某视频网站的视频可用率为何偏低时，设定筛选维度有：地域、运营商、播放器、视频ID、观看占比。先选择地域维度展开，筛选分析单元根据目标要求得到北京的视频可用率排名与过去相比变化了两位以上，待筛选数据组生成单元选择北京对应的数据作为下一轮待筛选数据组。再选择观看占比维度进行筛选，发现没有满足目标要求的数据，故重新选择运营商维度进行筛选。根据筛选分析单元选择中国移动这一维度子项下的数据进行视频ID维度下的筛选，得到经过地域(北京)——运营商(中国移动)——视频ID(视频1和视频2)筛选的数据。此时选择播放器维度进行筛选，未发现满足目标要求的数据，经分析知选择北京的筛选路径有误，通过筛选路径处理单元删除北京这一路径，得到经过运营商(中国移动)——视频ID(视频1和视频2)筛选的数据，并通过待筛选数据组生成单元作为下一轮的待筛选数据组。再次选择播放器维度，得到经过运营商(中国移动)——视频ID(视频1和视频2)——播放器(flash)筛选的数据，筛选分析完成。得到结论：在中国移动网络下，用flash打开的视频1和视频2的视频可用率太低，进而拉低了整个网站的视频可用率。找到拉低整个网站视频可用率的原因后，可以对其进行相应的修复，例如删除flash格式的视频1和视频2，或重新上传，以提升该网站的用户体验。
以上所描述的实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。本领域普通技术人员在不付出创造性的劳动的情况下，即可以理解 并实施。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到各实施方式可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件。基于这样的理解，上述技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如ROM/RAM、磁碟、光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行各个实施例或者实施例的某些部分所述的方法。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的精神和范围。

Claims (10)

  1. 一种大数据的筛选分析方法，包括多轮筛选分析，每一轮筛选分析包括：
    按照一个未选择的筛选维度对待筛选数据组中的数据进行筛选分析；
    将满足目标要求的、对应于所述筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组；
    其中，所述多轮筛选分析的轮数根据筛选维度的数量和目标要求来确定。
  2. 根据权利要求1所述的筛选分析方法，其中，在所述将满足目标要求的、对应于所述筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组后，生成和保存相应的筛选路径。
  3. 根据权利要求2所述的筛选分析方法，其中，每一轮筛选分析能够撤回，在撤回后，所述撤回的筛选分析下已生成和保存的筛选路径被删除。
  4. 根据权利要求1-3中任一项所述的筛选分析方法，其中，所述目标要求是所述待筛选数据组中的数据在各维度子项下对应的数值最大或最小，并且最大数值和最小数值之差的绝对值大于预定阈值；或
    各维度子项下数据对应的数值相对于参考值的波动范围大于预定范围。
  5. 根据权利要求4所述的筛选分析方法，其中，所述预定阈值、参考值和预定范围根据历史数据库中的历史数据来确定，并且所述历史数据库能够根据所述多轮筛选分析后的筛选结果更新。
  6. 一种大数据的筛选分析系统，配置以执行多轮筛选分析，所述系统包括：
    筛选分析单元，配置以按照一个未选择的筛选维度对待筛选数据组中的数据进行筛选分析；
    目标要求确定单元，配置以提供目标要求；
    待筛选数据组生成单元，配置以将满足目标要求的、对应于所述筛选维 度下的至少一个维度子项的数据保存为下一轮的待筛选数据组；
    其中，所述多轮筛选分析的轮数根据筛选维度的数量和目标要求来确定。
  7. 根据权利要求6所述的筛选分析系统，其中，还包括筛选路径处理单元，配置以
    在所述将满足目标要求的、对应于所述筛选维度下的至少一个维度子项的数据保存为下一轮的待筛选数据组后，生成和保存相应的筛选路径。
  8. 根据权利要求7所述的筛选分析系统，其中，所述筛选路径处理单元还配置以：
    在每一轮筛选分析撤回后，删除所述撤回的筛选分析下已生成和保存的筛选路径。
  9. 根据权利要求6-8中任一项所述的筛选分析系统，其中，所述目标要求确定单元提供有：
    所述待筛选数据组中的数据对应的数值最大的要求；
    所述待筛选数据组中的数据对应的数值最小的要求；和
    最大数值和最小数值之差的绝对值大于预定阈值的要求；或
    各维度子项下数据对应的数值相对于参考值的波动范围大于预定范围的要求。
  10. 根据权利要求9所述的筛选分析系统，其中，还包括：
    预定阈值确定单元和历史数据库，
    所述预定阈值确定单元配置以根据所述历史数据库中的历史数据来确定预定阈值、参考值和预定范围，
    所述历史数据库配置以根据所述多轮筛选分析后的筛选结果更新。
PCT/CN2016/083187 2015-11-13 2016-05-24 大数据的筛选分析方法及系统 Ceased WO2017080171A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/248,592 US20170139969A1 (en) 2015-11-13 2016-08-26 Method for filtering and analyzing big data, electronic device, and non-transitory computer-readable storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510779664.7 2015-11-13
CN201510779664.7A CN105893408A (zh) 2015-11-13 2015-11-13 大数据的筛选分析方法及系统

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/248,592 Continuation US20170139969A1 (en) 2015-11-13 2016-08-26 Method for filtering and analyzing big data, electronic device, and non-transitory computer-readable storage medium

Publications (1)

Publication Number Publication Date
WO2017080171A1 true WO2017080171A1 (zh) 2017-05-18

Family

ID=57001804

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/083187 Ceased WO2017080171A1 (zh) 2015-11-13 2016-05-24 大数据的筛选分析方法及系统

Country Status (2)

Country Link
CN (1) CN105893408A (zh)
WO (1) WO2017080171A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109189775A (zh) * 2018-09-27 2019-01-11 深圳中广核工程设计有限公司 一种工业监控平台海量数据处理系统及方法
CN109727146A (zh) * 2018-06-15 2019-05-07 中国平安人寿保险股份有限公司 准备金变化原因分析方法、装置、设备及可读存储介质
CN114139815A (zh) * 2021-12-02 2022-03-04 深圳市华傲数据技术有限公司 一种基于维度约束下的数据优化方法、设备及介质
CN115296399A (zh) * 2022-06-22 2022-11-04 云南电力技术有限责任公司 基于海量数据趋势变化的在线监测数据预警方法及系统
CN118094148A (zh) * 2024-02-19 2024-05-28 中国科学院空间应用工程与技术中心 一种航天工程大数据挖掘与分析方法及系统

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107391724A (zh) * 2017-08-01 2017-11-24 佛山市深研信息技术有限公司 一种大数据的筛选方法
CN110147384B (zh) * 2019-04-17 2023-06-20 平安科技（深圳）有限公司 数据查找模型建立方法、装置、计算机设备和存储介质
CN112347052B (zh) * 2020-11-04 2025-02-14 北京智创数字科技服务有限公司 一种文件匹配方法及相关装置
CN116680262A (zh) * 2023-06-13 2023-09-01 工业互联网创新中心（上海）有限公司 一种工业互联网大数据平台及数据处理方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102023977A (zh) * 2009-09-21 2011-04-20 陈俊 一种数据筛选方法、数据筛选系统及其应用
CN104216922A (zh) * 2013-06-05 2014-12-17 腾讯科技（深圳）有限公司 数据筛选的方法及装置
CN104965851A (zh) * 2015-04-28 2015-10-07 上海新储集成电路有限公司 一种分析数据的系统及方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102023977A (zh) * 2009-09-21 2011-04-20 陈俊 一种数据筛选方法、数据筛选系统及其应用
CN104216922A (zh) * 2013-06-05 2014-12-17 腾讯科技（深圳）有限公司 数据筛选的方法及装置
CN104965851A (zh) * 2015-04-28 2015-10-07 上海新储集成电路有限公司 一种分析数据的系统及方法

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109727146A (zh) * 2018-06-15 2019-05-07 中国平安人寿保险股份有限公司 准备金变化原因分析方法、装置、设备及可读存储介质
CN109727146B (zh) * 2018-06-15 2023-07-21 中国平安人寿保险股份有限公司 准备金变化原因分析方法、装置、设备及可读存储介质
CN109189775A (zh) * 2018-09-27 2019-01-11 深圳中广核工程设计有限公司 一种工业监控平台海量数据处理系统及方法
CN109189775B (zh) * 2018-09-27 2022-02-22 深圳中广核工程设计有限公司 一种工业监控平台海量数据处理系统及方法
CN114139815A (zh) * 2021-12-02 2022-03-04 深圳市华傲数据技术有限公司 一种基于维度约束下的数据优化方法、设备及介质
CN114139815B (zh) * 2021-12-02 2025-08-05 深圳市华傲数据技术有限公司 一种基于维度约束下的数据优化方法、设备及介质
CN115296399A (zh) * 2022-06-22 2022-11-04 云南电力技术有限责任公司 基于海量数据趋势变化的在线监测数据预警方法及系统
CN118094148A (zh) * 2024-02-19 2024-05-28 中国科学院空间应用工程与技术中心 一种航天工程大数据挖掘与分析方法及系统

Also Published As

Publication number Publication date
CN105893408A (zh) 2016-08-24

Similar Documents

Publication Publication Date Title
WO2017080171A1 (zh) 大数据的筛选分析方法及系统
US11386156B1 (en) Threshold establishment for key performance indicators derived from machine data
US12450503B2 (en) Automatically-generated labels for time series data and numerical lists to use in analytic and machine learning systems
CN111782965B (zh) 意图推荐方法、装置、设备及存储介质
US12130866B1 (en) Creating a correlation search
CN112000763B (zh) 兴趣点竞争关系确定方法、装置、设备和介质
WO2017071251A1 (zh) 信息推送方法和装置
CN111125344B (zh) 相关词推荐方法和装置
US10768794B2 (en) Truncated synchronization of data object instances
CN111241225B (zh) 常驻区域变更的判断方法、装置、设备及存储介质
CN105373619A (zh) 一种基于用户大数据的用户群体分析方法和系统
CN116048691A (zh) 资源展示方法、装置、电子设备及存储介质
WO2024160273A1 (zh) 数据处理方法、装置、设备及存储介质
CN110096501A (zh) 基于标签的数据搜索方法、装置、介质及电子设备
CN113807926A (zh) 推荐信息生成方法、装置、电子设备和计算机可读介质
CN112529646A (zh) 一种商品分类方法和装置
CN116756616A (zh) 数据处理方法、装置、计算机可读介质及电子设备
CN110019456A (zh) 数据导入方法、装置和系统
US10511556B2 (en) Bursty detection for message streams
CN114528493B (zh) 推荐方法、装置、电子设备以及存储介质
WO2024160255A1 (zh) 页面交互方法、装置、设备及存储介质
US20170139969A1 (en) Method for filtering and analyzing big data, electronic device, and non-transitory computer-readable storage medium
US9536199B1 (en) Recommendations based on device usage
CN114925108B (zh) 排序模型的确定方法、装置、设备以及存储介质
US20250158914A1 (en) Systems and methods for intelligently routing digital activity profile data in an automated scheduled cascade

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16863352

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16863352

Country of ref document: EP

Kind code of ref document: A1