TWI684147B - Cloud self-service analysis platform and analysis method thereof - Google Patents

Abstract

The present invention is a cloud self-service analysis platform and an analysis method thereof. The analysis method includes: searching the existing data extraction rules applicable to a data analysis task in a shared resource pool, wherein the shared resource pool stores multiple existing data extraction rules; analyzing whether there is a conflict between the data analysis task and the existing data extraction rules, so as to generate a final data extraction rule; generating an acquisition script according to the final data extraction rule to perform the required data collection; and allocating an analysis algorithm and the collected data to a computing resource pool for calculation according to the priority of the analysis task and generating the final analysis result.

Description

Cloud self-service analysis platform and its analysis method

The present invention relates to multi-person collaborative information sharing technology. In detail, it relates to a cloud self-service analysis platform and analysis method.

Generally speaking, the files of data analysis projects are mostly high-capacity, and a file often has many different types or attributes. When faced with the first data contact, data analysts often adopt trial and error methods to set up data capture Take the rules to get a meaningful outline of the data, for example: count the number of times the attribute value A appears every day, count the number of times a certain field value is greater than the set value every day, etc... On the contrary, if the data analyst is familiar with the file structure or masters the experience shared by the predecessors, the analysis task will become an easy task. Therefore, it is an effort of the public to achieve high-efficiency analysis through data classification and organization. In the current topic of self-service analysis in the cloud, repeating data is an important issue. In short, one data analyst establishes an analysis criterion, and another data analyst has different research attributes from it, but the aforementioned analysis criterion can still be Applicable (only a few differences), but the lack of the concept of multi-person collaborative sharing will lead to repeated data, that is, two data analysts each establish a set of analysis criteria, but the two are obviously similar and common, so the current cloud self-service analysis In the field, whether it is raw analysis data or derivative data generated through data extraction rules, the lack of information sharing mechanism often results in repeated storage of data and waste of computing resources, resulting in poor self-service analysis efficiency.

It can be seen from the above that if a multi-person collaborative information sharing technology can be found, in particular, it is applicable to the cloud self-service analysis platform, so that data analysts have the opportunity to refer to the analysis criteria established by the predecessors when performing data analysis, thereby improving Analyzing efficiency and avoiding waste of computing resources will become a technical problem that those skilled in the art are eager to solve.

The purpose of the present invention is to propose a mechanism and service to improve the efficiency of cloud self-service analysis based on multi-person collaborative information sharing, by referring to the data retrieval specifications shared by previous people, thereby improving analysis efficiency and avoiding waste of computing resources for duplicate data .

In order to achieve the above or other objectives, the present invention proposes a cloud self-service analysis platform, including: a shared resource pool, which stores multiple existing data retrieval rules; a data retrieval rule setting module, which is connected to the shared resource pool and used for Receive the data analysis task input from outside to search through the data analysis task to the shared resource pool to obtain the existing data extraction rules applicable to the data analysis task; the data extraction rule analysis module is connected to the The data retrieval rule setting module is used to analyze whether the data analysis task conflicts with the existing data retrieval rule obtained by the data retrieval rule setting module, so as to generate the final data retrieval rule accordingly; the data set The collection module is connected to the data retrieval rule analysis module and the shared resource pool, and the data collection module is used to generate a retrieval foot according to the final data retrieval rule This is used to collect the required data; and the analysis task scheduler is connected to the data collection module and used to allocate the analysis algorithm and the collected required data to the computing resource pool according to the priority of the analysis task Perform calculations to produce the final analysis results.

In the above system, it further includes a visual presentation module connected to the data acquisition rule analysis module and the shared resource pool, which is used to present the statistical distribution of data through graphs or tables.

In the above system, it further includes an analysis algorithm selection module connected to the visual presentation module, which is used for a user to select the analysis algorithm and set related parameters.

In an embodiment, the data collection module is to obtain the resource set from the shared resource pool as the required data when the required data exists in the shared resource pool.

In another embodiment, a warning message is generated when the data acquisition rule analysis module analyzes that the data analysis task conflicts with the existing data acquisition rule, and chooses to extend the existing data acquisition rule or create a new one One of the data retrieval rules of is used as the final data retrieval rule.

In yet another embodiment, when the data retrieval rule setting module does not obtain the existing data retrieval rule applicable to the data analysis task, the user himself creates a new data retrieval rule as the final Data retrieval rules.

The invention further proposes a cloud self-service analysis method, which includes the following steps: searching for the data suitable for the data analysis task based on the received data analysis task into a shared resource pool storing multiple existing data retrieval rules There are data retrieval rules; analyze whether there is a conflict between the data analysis task and the existing data retrieval rules obtained for the data analysis task to generate the final data retrieval rules accordingly; generate according to the final data retrieval rules Retrieve the script to collect the required data; and allocate the analysis algorithm and the collected required data to the computing resource pool for operation according to the priority of the analysis task to generate the final analysis result.

In the above method, collecting required data further includes obtaining a resource set from the shared resource pool as the required data when the required data exists in the shared resource pool.

In the above method, a warning message is generated when the data analysis task conflicts with the existing data retrieval rule, and one of the existing data retrieval rule or the creation of a new data retrieval rule is selected as the Final data retrieval rules.

In the above method, when the existing data retrieval rule applicable to the data analysis task is not obtained, the user creates a new data retrieval rule as the final data retrieval rule.

Compared with the prior art, the cloud self-service analysis platform and its analysis method proposed by the present invention, when receiving data analysis tasks proposed by users, recommend highly relevant rules from the shared resource pool through the data extraction rule setting module, and cooperate with the visualization The user can quickly grasp the data outline to improve the self-service analysis performance of the cloud. After the user selects or creates the data extraction rule, the data extraction rule analysis module will confirm whether it conflicts with the existing data extraction rule. Finally, the data collection module is responsible for preparing the required data. Among them, the data collection module will first compare the existing data set in the shared resource pool. It does not recapture duplicate data to save computing resources.

100‧‧‧ Cloud self-service analysis platform

101‧‧‧ shared resource pool

102‧‧‧Data extraction rule setting module

103‧‧‧Data extraction rule analysis module

104‧‧‧Data collection module

105‧‧‧Analysis task scheduler

106‧‧‧ computing resource pool

107‧‧‧Visualized presentation module

108‧‧‧Analysis algorithm selection module

200‧‧‧Data analysis task

401~411‧‧‧Flow

501~509‧‧‧Flow

601~609‧‧‧Flow

701~707‧‧‧Flow

S301~S304‧‧‧Step

Figure 1 is a system architecture diagram of the cloud self-service analysis platform of the present invention; Figure 2 is a system architecture diagram of a specific embodiment of the cloud self-service analysis platform of the present invention; Figure 3 is a step diagram of the cloud self-service analysis method of the present invention Figure 4 is an execution flow chart of a cloud self-service analysis method in an embodiment of the invention; Figure 5 is an execution flow chart of a data acquisition rule setting of the cloud self-service analysis method of the invention; Figure 6 is a cloud embodiment of the invention The execution flow chart of the data acquisition rule analysis of the self-service analysis method; and FIG. 7 is the execution flow chart of the data collection of the cloud self-service analysis method of the present invention.

The technical content of the present invention will be described below with specific specific implementation forms, and those skilled in the art can easily understand the advantages and effects of the present invention from the content disclosed in this specification. However, the present invention can also be implemented or applied by other specific embodiments.

Figure 1 illustrates the system architecture of the cloud self-service analysis platform of the present invention. As shown in the figure, the cloud self-service analysis platform 100 can be used by data analysis experts to submit data analysis tasks 200 to the cloud self-service analysis platform 100 through a workstation with a browser, by which the platform provides assistance in data collection rules to save computing resources Provide the required information, including the cloud self-service analysis platform 100 It includes a shared resource pool 101, a data retrieval rule setting module 102, a data retrieval rule analysis module 103, a data collection module 104, an analysis task scheduler 105, and a computing resource pool 106.

The shared resource pool 101 is used to store multiple existing data retrieval rules. The aforementioned existing data retrieval rules are various data retrieval rules previously set, so this case and the reference to the previous data retrieval rules are provided to save computing resources and avoid duplicate data storage.

The data extraction rule setting module 102 is connected to the shared resource pool 101 and is used to receive the externally input data analysis task 200, so as to search through the data analysis task 200 to the shared resource pool 101 to obtain the data suitable for the data analysis task 200 There are data retrieval rules. After receiving the data analysis task 200, the cloud self-service analysis platform 100 will drive the data extraction rule setting module 102 to query the database to determine whether the analysis file exists before the data extraction rule. If the file has been analyzed, then You can recommend the existing existing data retrieval rules, or make new data retrieval rules, users can set the rules directly through the interface, or the platform system recommends the applicable data retrieval rules according to the attribute type, that is to say When the data retrieval rule setting module 102 does not obtain the existing data retrieval rules applicable to the data analysis task 200, the user creates a new data retrieval rule as the final data retrieval rule.

The data retrieval rule analysis module 103 is connected to the data retrieval rule setting module 102 and is used to analyze whether there is a conflict between the data analysis task 200 and the existing data retrieval rule obtained by the data retrieval rule setting module 102. Based on this, the final data extraction rules are generated. Data extraction rule analysis module 103 will confirm whether the data analysis task 200 conflicts with the existing data extraction rules, and if there is a conflict, it will warn you, and the user can finally decide whether to add this data junction extraction rule, that is, when the data extraction rule is analyzed The module 103 analyzes the data analysis task 200 and the existing data retrieval rule conflicts and generates a warning message. At this time, the existing data retrieval rule or a new data retrieval rule will be selected as the final data retrieval Take the rules.

The data collection module 104 is connected to the data retrieval rule analysis module 103 and the shared resource pool 101. The data collection module 104 is used to generate a retrieval script according to the final data retrieval rule to collect the required data. The data collection module 104 generates an extraction script according to the input data extraction rules, and submits the script to the analysis task scheduler 105 for data acquisition tasks.

The analysis task scheduler 105 is connected to the data collection module 104 and is used to allocate the analysis algorithm and the collected data to the computing resource pool 106 for operation according to the priority of the analysis task to generate the final analysis result. When the data collection is completed, the analysis task scheduler 105 will be triggered. The analysis task scheduler 105 will allocate the analysis algorithm and data set to the computing resource pool 106 for operation according to the priority of the analysis task, and finally store the results to the share In the resource pool 101, in addition to storing the final analysis result in the shared resource pool 101, information such as the data set and data extraction rules generated in the previous step will also be stored together.

In an embodiment, if the required data exists in the shared resource pool 101, the data collection module 104 can obtain the resource from the shared resource pool 101 Set as the required information.

Through the above method, the data analysis expert can send the data analysis task 200 to the cloud self-service analysis platform 100. The cloud self-service analysis platform 100 will search for the applicable data extraction rules according to the task content to the data analysis expert for reference. The data analysis expert can use the existing The data retrieval rules can also create new data retrieval rules and determine whether there is a conflict. Finally, after selecting the data retrieval rules, the data collection module 104 collects the data, and the analysis task scheduler 105 considers the task priority order, analysis algorithm, etc. to schedule, and then to the computing resource pool 106 for calculation. Get the analysis result.

FIG. 2 illustrates a system architecture diagram of a specific embodiment of the cloud self-service analysis platform of the present invention. As shown in the figure, the shared resource pool 101, the data retrieval rule setting module 102, the data retrieval rule analysis module 103, the data collection module 104, the analysis task scheduler 105, and the computing resources of the cloud self-service analysis platform 100 The pool 106 is the same as described in FIG. 1, so it will not be described in detail. In this embodiment, the cloud self-service analysis platform 100 further includes a visual presentation module 107 and an analysis algorithm selection module 108.

The visual presentation module 107 is connected to the data extraction rule analysis module 103 and the shared resource pool 101, and can be used to present the statistical distribution of data through graphs or tables. Specifically, the visual presentation module 107 can present the statistical distribution of data through different graphs (such as pie charts, bar charts, etc.) and tables, so that the user can easily know what the content of the data extraction rule to be selected, the probability of sorting, etc. News.

The analysis algorithm selection module 108 is connected to the visual presentation module 107, and can be used for the user to select the analysis algorithm to be used and related settings parameter. Specifically, after determining the data extraction rules, the analysis algorithm selection module 108 is used to select the analysis algorithm (such as support vector machine, decision number, regression analysis, etc.), and set the relevant parameters, and then the selected analysis algorithm will The analysis task scheduler 105 schedules and performs calculations in the calculation resource pool 106.

In addition, in addition to the above-mentioned component modules, the cloud self-service analysis platform 100 also masters the functions of submitting/suspending/replying/deleting analysis tasks, and finally returns the results of the analysis tasks to the data analysis experts.

Figure 3 illustrates the steps of the cloud self-service analysis method of the present invention. As shown in the figure, in step S301, the existing data retrieval rules applicable to the data analysis task are searched for in the shared resource pool storing multiple existing data retrieval rules according to the received data analysis task. Specifically, when receiving a data analysis task, it will first search the shared resource pool for applicable existing data retrieval rules, and if any user can choose to apply it, thereby reducing the waste of computing resources.

In an embodiment, when the existing data retrieval rules applicable to the data analysis task are not obtained, the user creates a new data retrieval rule as the final data retrieval rule.

In step S302, it is analyzed whether there is a conflict between the data analysis task and the existing data extraction rules obtained for the data analysis task, so as to generate a final data extraction rule accordingly. In short, whether it is applying existing data acquisition rules or creating new data acquisition rules, it is necessary to analyze with existing data acquisition rules to determine whether there is a conflict, thereby avoiding different results in subsequent calculations.

In one embodiment, a warning message is generated when the data analysis task conflicts with the existing data retrieval rule, and one of the existing data retrieval rule or the creation of a new data retrieval rule is selected as The final data retrieval rules.

In step S303, an extraction script is generated according to the final data extraction rule to collect the required data. When the final data extraction rule is selected, data collection begins. In particular, when the required data is collected, if the required data exists in the shared resource pool, the resource set is obtained from the shared resource pool as the institute Information required. Finally, the collected data will be sent to the scheduler for calculation scheduling.

In step S304, the analysis algorithm and the collected required data are allocated to the calculation resource pool for calculation according to the analysis task priority order, so as to generate a final analysis result. The scheduler will allocate the selected analysis algorithm and the data set collected in the previous step to the calculation resource pool for calculation according to the analysis task priority order, so as to generate the final analysis result. In addition, the final analysis results of the computing resource pool and related data (data sets, data extraction rules, etc.) will be restored to the shared resource pool for reference and use in subsequent analysis.

FIG. 4 is an execution flowchart of a cloud self-service analysis method according to an embodiment of the present invention, which illustrates the core method flowchart of the present invention. As shown in the figure, in the process 401, the user selects and analyzes the data through the interface, and then starts to set the data extraction rules, and then the user can adopt the following different ways to set the rules, for example, in the process 402, select the platform system recommended data acquisition Rules, where the platform system recommends based on data attributes. In addition, in process 403, when there is no suitable recommendation rule, the user can establish Material retrieval rules.

After the data rule set is set, enter the process 404 and submit the data extraction rule to the data extraction rule analysis module for analysis. In the process 405, the data extraction rule analysis module determines whether there is conflicting data according to the similarity Retrieve rules, if there are conflict rules, then enter the process 406, the user can try to resolve the conflict rules according to the platform system prompt, if there are no conflict rules, then enter the process 407, the user can submit the data retrieval rule set to the data The collection module performs collection tasks.

In the process 408, the data collection module will first confirm whether there is a corresponding data set in the shared resource pool. If it exists, enter the process 409, directly obtain the data set from the shared resource pool, if not, enter the process 410, start Perform the data collection task, and finally enter the process 411, store the data collection in the shared resource pool, and complete the data preparation task.

FIG. 5 is an execution flowchart of data extraction rule setting of the cloud self-service analysis method of the present invention, that is, the operation of the data extraction rule setting module 102 in FIGS. 1 and 2 is explained. Users can upload analysis data (data analysis tasks) through the web interface, or choose the existing data on the cloud self-service analysis platform. At this time, the data extraction rule setting module can also recommend highly relevant data extraction rules for users. Let users create their own rules.

In the process 501, after selecting the data to be analyzed, the user interacts with the data extraction rule module, that is, according to the selected data file, query whether the corresponding data extraction rule already exists in the shared resource pool, if so, Directly select existing data retrieval rules. Next, in the process 502, no matter whether the previous step selects the existing data retrieval rule, any data field attribute (referred to as the target attribute ta ) can be specified in this process to recommend related rules.

In the process 503, first of all, according to the data field attribute recommendation data extraction, the synonym dictionary is used to filter out words that are similar to the target attribute name, together with the target attribute itself to form a candidate word set C , for example: the target attribute name is "home location"", synonymous dictionary found "address", "address", "location", "postal code" are synonymous words, this time candidate set C contains {"home location", "address", "address", "location","Postalcode"}. In the process 504, the synonyms of different data types of the target attribute are filtered, for example: the data type of the target attribute "home location" is "string", but the attributes of the synonymous word "location" are "integer" and data The type "character string" is different, so the word "position" is removed from the candidate word set C. At this time, the candidate word set C contains {"home location", "address", "address", "postal code"}}. Next, in the process 505, according to the candidate word set C filtered in the previous step, the filter attribute value fields are mutually exclusive and independent synonyms, as shown in Table 1 below:

In Table 1 above, the range of "postal code" and the target attribute "home address" are mutually exclusive and independent, so the "postal code" is removed from the candidate word set C , and the candidate word set only exists {"home location" , "Address", "Address"}. Finally, in the process 506, the recommendations are sorted according to the probability of intersection of synonymous attribute value ranges, as shown in Table 2 below:

In addition to the original target attribute "home address", in order of "address" and "home address", the process 506 will extract the data extraction rules corresponding to the synonyms from the shared resource pool and sort them by probability. Then, in the process 507, the user can select appropriate data extraction rules. When the user finds that there is no suitable rule, he can also create a data retrieval rule by process 508. The user can repeat the operation flow 502~508 to establish several rules for one or more target attributes. Finally, the process 509 sends the data extraction rules generated by the processes 507 and 508 to the data extraction rule analysis module, and completes the data extraction rule formulation.

FIG. 6 illustrates the execution flow chart of the data extraction rule analysis of the cloud self-service analysis method of the present invention, that is, the operation of the data extraction rule analysis module 103 in FIGS. 1 and 2, the main purpose of this module is to analyze whether there is Conflicting rules exist. After submitting the data retrieval rules, in the process 601, the user interacts with the data retrieval rule analysis module, calculates the distance (similarity) according to the data retrieval rules to be added and the existing rules, and the method is about to input the data retrieval Take rules to model. In the process 602, mainly through the space The vector space model (Vector Space Model) represents each attribute, and the similarity is calculated using the processes 603 and 604. For example, as shown in Table 3 below, suppose that there are three data extraction rules in the shared resource pool, and each rule contains four attributes that can be set conditions {"home address", "annual income", "sex", " "Building date"}, the data types are "string", "integer", "Brin", "date". The newly added rules require that the data retrieved must meet: "Home Address" appears on [Zhongzheng Road or Zhongshan Road], and "Annual Income" is greater than 500, and "Gender" is male, and "House Construction Date" is 2002 .

The process 603 will first calculate the distance between numbers, date, time, Bollinger and bit data type vectors, and calculate the distance between the rules to be added and the existing rules through the criteria. The criteria include (1) calculating the absolute value of the two values and then opening the root sign; (2) If the two rule operands are inconsistent, the above value will be enlarged, and the maximum value will be added here. If the operand consistent records meet the coverage conditions, for example: rule ">400" coverage rule ">500". After the process 603, the similarity between the existing rules and the rules to be added is shown in Table 4 below:

In the above table, the existing rule 3 operand is different from the newly added rule operand, so the vector distance between the two rules is increased. Then the process 604 calculates the distance between the character, string, enumeration and text data type vector, and calculates the distance between the rule to be added and the existing rule through the criteria. The criteria include (1) judging whether the two texts are the same, the same takes 0, the opposite Take 1; (2) If the two rule operands are inconsistent, add 1 to the value of the previous step, otherwise many records will meet the coverage conditions. After the process 604, the similarity between the existing rules and the rules to be added is shown in Table 5 below:

Finally, the process 605 sorts and displays according to the vector distance, and determines whether there is a conflicting data extraction rule through the process 606. When the distance is 0, it indicates that there are already data acquisition rules with the same set conditions. And when all the coverage conditions of a rule are established, it is also regarded as a conflict. For example, although the existing rule 1 is far away from the new rule, its conditions cover the setting of the new rule. When a conflicting rule is established, the process 607 will display a data retrieval rule conflict warning. The user can modify the conflicting data retrieval rule through the process 608. The operations include: adopting existing rules, modifying new rules, and deleting new rules. If there are no conflicting rules, then flow 609 is entered to complete the establishment of data extraction rules.

FIG. 7 illustrates the execution flow chart of data collection in the cloud self-service analysis method of the present invention, that is, the operation of the data collection module 104 in FIGS. 1 and 2. The main purpose of this module is to receive data collection rule combinations Afterwards, according to the rules set the actual to produce the corresponding data set. In the process 701, a combination of data extraction rules is received to generate a corresponding data set according to the rule setting. In the process 702, it is confirmed whether there is a data set corresponding to the data extraction rule in the resource pool, and if it exists, the process 703 directly from the sharing The resource pool obtains the data set without generating data repeatedly. If the data set does not exist in the shared resource pool, the process enters the process 704. The data collection module will generate a data retrieval script, which can be directly executed on the command line. Standard formats exist, such as xml, json, yml, etc. The main content includes (1) data source; (2) data retrieval rules; (3) data set name; (4) data set storage location; (5) priority weight The Meta information includes: creator, creation time, version, etc.

After the script is created, the flow enters the process 705, and the scheduler executes the data retrieval script in sequence to generate the data set according to the currently available computing resources and data retrieval script priority. Next, in the process 706, the generated data set and script-related information are stored in the shared resource pool, and finally the process enters the process 707, that is, the data set preparation task is completed.

The following is an implementation case to illustrate the mechanism and service of the present invention on how multi-person collaborative information sharing can improve the efficiency of cloud self-service analysis. Based on the establishment of the Customized Customer Journey platform described in this case, and the data set is supplemented by the information of the telecommunications industry. This platform is built for machine marketing for use by marketing departments, data analysts, etc. The purpose is to use this service to help The customer service and marketing departments enhance the customer service experience, for example: detecting that the customer has an intention to be off-grid or has received high-frequency competition simply, so as to provide a promotional plan or related redemption strategy and increase sales opportunities before the customer actually leaves the network. With the structure proposed in this case, data analysis experts can integrate different data sources and customize different event attributes, and share machine learning models and analysis data.

The description will be made with reference to Figs. 1, 5 and 7 below. It is assumed that the data to be analyzed has been prepared with a structured data set of original telecommunications users, and these original data come from different services and data sources (data channels). Examples of data sets are as follows: customer mobile phone dialing and receiving records, local dialing and receiving records, customer video rental records, customer complaint customer service data, customer fixed broadband lease data, 4G LTE network quality, etc. As shown in Table 6 below:

Each data set has a unique user identification attribute (unique identifier) for identifying customers who do not use the data source for user home ownership. Each data set has different attributes such as customer complaint customer service data with the "appeal reason" attribute, 4G LTE network quality with the "base station location" attribute, and customer received SMS data with the "text content length". The above-mentioned original data sets are prepared and stored in the shared resource pool 101 as structured data.

The data analysis task can be performed after the original data set is completed. The following data analysis experts are referred to as marketing personnel or users. The data analysis experts choose to perform data analysis and need to combine the attributes required for data analysis. These attributes are captured here as events The rule claims that these event extraction rules come from selected analysis data, and the data come from different data sources, refer to the data analysis task 200. As shown in FIG. 1, after the data collection rule setting module 102 selects the data set, and then refer to FIG. 5, such as process 501, the user interacts with the data retrieval rule setting module to select the event retrieval rules to be combined Data fields, for example, three different data sources are selected for the customer mobile phone dialing and receiving data set and the 4G LTE network quality data set and the customer complaint customer service data set. To create a new data extraction rule, the name is 4G mobile user. Net intention r1. Specify the data field through process 502, select the target attribute "marketing line" in the customer mobile phone dialing and receiving data set, and the system uses the synonym dictionary to filter the attributes with similar names in process 503, such as whether to accept competitive marketing calls and promotional calls , Marketing dedicated line, whether to actively dial competitive marketing number, marketing time, suspected marketing newsletter and other similar data retrieval rules. The system filters the data extraction rules of different data types through the process 504 and the process 505 excludes the range of values that are mutually exclusive and independent of the target attribute, as shown in Table 7 below:

The candidate word set is {"promotional phone", "marketing hotline", "suspected marketing newsletter number"}, and sorted and recommended according to the probability of intersection of attribute value fields through process 506. The candidate word value fields are shown in Table 8 below:

The recommended probability is calculated through the intersection of attribute value fields. The calculation method is shown in Equation 1 below, and the calculation of the attribute of the promotional call is shown in Equation 2 as an example.

Finally, you can get recommendations for sorting the probability of intersection of attributes. As shown in Table 9 below, the following data extraction rules for recommended data analysis experts are in order: marketing line, promotional phone, and suspected marketing newsletter. The recommended probability helps data analysis experts to quickly increase or decrease or select the appropriate data extraction rules.

Referring again to FIG. 5, the data analysis expert selects the appropriate data extraction rule under the user selection data extraction rule (flow 507), such as the selection in flow 506 recommendation: promotional call, suspected marketing newsletter, or in flow 502 Directly specify the existing data fields and create data rules, that is, enter the process 508, such as customer mobile phone dialing and receiving data set selection, such as call time, dialing group is the proportion of the external network, etc., in 4G LTE network quality The data set selects attributes such as connection quality score, frequent bad quality reconnections, etc., and the customer complaint customer service data set selects attributes such as complaint events and customer complaints. The above are different data sources or data extraction rules, and the same user is concatenated by a unique user identification attribute. The data analysis expert selects the data extraction rules in the process 507 or 508 and sets each attribute threshold (filtering value range), for example, the target group is the proportion of the external network greater than 50%, the connection quality is less than 50 points, or the number of customer complaints More than 3 times per month, etc., as shown in Table 10 below. After setting the data extraction rules and filter values, the new rule name 4G mobile user off-grid intention r1 and data extraction rules will be submitted, that is, process 509, and sent to the data extraction rule analysis module 103.

Returning to FIG. 1, the data extraction rule analysis module 103 expresses the data rule to be added as a vector space model, each dimension represents the attribute and the filter value range, and then refers to the flow 602 of FIG. 6, And whether it conflicts with the existence rule calculation, as shown in Table 11 below.

Attribute types can be divided into two categories. The first category is numeric, date, time, Bollinger, and bit. Based on the concept of vector similarity, calculate the distance between the rule to be added and the existing rule. Refer to process 603. The steps are as follows: ( 1) Calculate the squared value after calculating the difference between the two values; (2) If the two rules are inconsistent, add 1 to the squared value. The second type of attribute type is character, string, enumeration and text. Refer to the process 604, the steps of calculating the similarity are as follows: (1) determine whether the two texts are the same, take 0 for the same, and take 1 for the opposite; (2) if the two rules If the operands are inconsistent, add 1 to the value of the previous step, otherwise, many records will meet the coverage conditions. After the two types of attribute types are calculated, the similar distance matrix is obtained and the rule similarity is calculated:

At flow 605, the numerical sum of the flow 603 and flow 604 is sorted and displayed, and at flow 606, it is determined whether there are rules with high similarity and Conflict, when the distance is 0, it means that there are already data extraction rules with the same set conditions. If there is no conflict, then flow 609 is entered, and a new data rule and a deposit rule r1 are successfully created to the shared resource pool 101, and the stored information includes attribute values and information of the data analysis expert who created the rule. On the contrary, if there is a rule conflict, the process enters the process 607 and the process 608, the system issues a rule conflict warning and triggers the modification of the conflict data retrieval rules, and allows the data analysis expert to revise.

Referring to FIG. 7, in the process 701, after completing the rule building, the system receives the data extraction rule combination. In the process 702, the system starts to confirm whether there is a data set corresponding to the data extraction rule in the shared resource pool 101. If there is already a data set in the resource pool 101, enter the process 703. The system selects the value range according to the event rule to perform data extraction. After the data extraction is completed, it is stored in the shared resource pool 101, and enters the process 707 to complete the data set Preparation task, conversely, if there is no data set in the shared resource pool 101, enter the process 704, the data analysis expert can generate a data extraction script, then the process 705, according to the system resources available to the data analysis expert to set different data source extraction schedule Then, following the process 706, the system filters the value range according to each attribute to extract and store the data in the resource pool, and finally enters the process 707 to complete the data set preparation task.

Please refer to Figure 2 as well. After the data analysis experts have established the data rules, they can analyze the data through the graphical interface, that is, the visual presentation module 107 selects the original data set or the data generated by the data rules that meet the rules. Data analysis, and select the appropriate algorithm, refer to the analysis algorithm selection module 108, which can integrate commonly used learning models, such as support vector machines, Models such as decision trees, random forest classification, K-means clustering, neural-like, regression analysis, PCA analysis, frequent pattern exploration, etc. After selecting the model and data set, the data collection module 104 starts to retrieve the selected original data set or the data set that has met the rules from the shared resource pool 101. After the data retrieval process is successful, the data analysis expert schedules through the analysis task The device 105 sets the number of iterations of the training model, adjusts the model analysis parameters, thereby starting the training task schedule and submits the model to the computing resource pool 106, and starts training the model according to the computing resources available to the data analysis expert and generates results. The computing resource pool 106 stores information such as training data sets, user data of data analysis experts, selected models, training model parameters, and analysis results in the shared resource pool 101.

The platform described in this case has project management functions. Please refer to FIG. 1, in the visual presentation module 107, users of the same project can also retrieve data rules created by other users, original data sets, and data sets that have been prepared to meet the rules through this module. For example, user a generates data sets m1, m2, and m3 that have met the rules through data rules r1, r2, and r3. User b can select appropriate data for analysis in the visual presentation module 107, such as Select m2 and other original data corresponding to r2, the system will retrieve m2 and related data from the shared resource pool 101 to the computing resource pool 106 to submit the algorithm and allocate computing resources for the user b to use the system resource training model, or, use User b refers to the event rule created by user a and finds that r1 and r2 are similar to the event rule to be created. At this time, user b can retrieve copies of r1 and r2 and modify the attributes in the rule by themselves. After completion, the data extraction rule is analyzed The module 103 performs discrimination, creates r1' and r2' belonging to user b, and extracts data sets m1' and m2' that meet the event rules. For example, the above The user a establishes the off-grid intent rule r1, which includes attributes such as the acceptance of competitive marketing calls, talk time, the proportion of dialed groups for the external network, connection quality score, complaint incidents, number of customer complaints, and the corresponding value range, and If user b wants to create an event rule "customer loyalty", user b can refer to r1, modify and combine appropriate attributes, such as selecting monthly tariff plan attributes from customer tariff information, and setting the value range to be greater than 1300 yuan to join In r1, delete the call time attribute, modify and reduce the dial-up group to the proportion of the external network to 20%, as shown in Table 12 below, after user b completes the update and submits it to the data extraction rule analysis module 103 Judging, if it is confirmed that there is no conflict of other event rules, a new rule r2 can be created and a data set m2 that matches the event rule can be extracted, and the user a who belongs to the same project can also share the r2 and m2 created by the user b.

In addition, after user a creates the model, user b can also select the model created by user a and the selected model parameters in the visual presentation module 107, that is, the user can use the data extraction rule analysis module 103 Accelerate model optimization for collaborative work and processing The project management function also has an analysis data sharing function for indirect use of data by different users. For example, user a establishes a forecast mobile online and off-line to competitive model, and generates a predicted high probability of off-line to competitive customer data sets. The data set is already stored in the shared resource pool 101. When the next time user b wants to create a future fixed-line broadband off-network to remote transmission large-broadband model, select the predicted high probability off-network to in the visual presentation module 107 Competitive customer data sets can reduce the time it takes to directly analyze all customer groups, or compare the customer base and high probability of off-net to competitive customers. For the future fixed-line broadband off-network to long-distance large broadband model prediction results, there are Help reduce the time for analysts to prepare data and model rework.

The analysis results in this case can be presented visually, and the visual presentation module 107 can present common statistical charts, including bar charts, scatter charts, pie charts, bubble charts, and heat maps, to help users analyze large amounts of data, such as (1 ) Analyze with the original data. The data analysis experts can count the pie charts of users browsing a specific website and the bar graphs of users making/receiving specific calls from the original data; (2) or by the data sets that have met the rules According to statistics, the off-grid intent rules in the above embodiments observe the user’s future off-grid intent with a line chart, and data analysis experts can determine suitable promotional offers based on this information to increase customer recovery rates; ( 3) Or is it the result of the model, such as event positioning, customer information, calling base station and other event rules, training crowd movement prediction model, using map heat map method It presents the frequent mobile locations and number of customers in order to effectively provide a reference for expanding the location of the store and increase the marketing opportunities of customers.

As can be seen from the above, the present invention discloses a mechanism and service based on multi-person collaborative information sharing to improve the efficiency of cloud self-service analysis. The service mainly includes data retrieval rule setting, data retrieval rule analysis, and data collection. Support the core process of cloud self-service analysis operations, including (1) users submit files to be analyzed to the cloud self-service analysis platform, or select remote files; (2) generate training data sets; (3) select machine learning algorithms and Set the relevant parameters of the algorithm; (4) Submit the model training task; (5) The model training task runs in the cloud environment; (6) The training model is stored on the cloud platform or downloaded to the local end. Compared with the existing self-service analysis in model training, often because of the same data source, only the views of each expert field are different, and repeated uploading of data, the lack of effective information sharing mechanism, resulting in poor self-service analysis efficiency. In addition, different analysis applications, if they can learn from each other's strengths, will be a benefit. The present invention proposes to introduce the concept of shared resource pools in the cloud autonomous analysis system, combined with the data extraction rule setting module and the data extraction rule analysis module, In addition to filtering the existing rules, and recommending the relevant rules, the data collection module is responsible for the preparation of the required data, thereby improving the performance of self-service analysis. If the cloud self-service analysis platform already has a data set corresponding to the rule, it is not necessary Repeated preparation to save computing resources.

The above-mentioned embodiments only exemplarily illustrate the principle and efficacy of the present invention, and are not intended to limit the present invention. Anyone familiar with this skill can modify and change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the rights of the present invention should be applied for The profit range is listed.

100‧‧‧ Cloud self-service analysis platform

101‧‧‧ shared resource pool

102‧‧‧Data extraction rule setting module

103‧‧‧Data extraction rule analysis module

104‧‧‧Data collection module

105‧‧‧Analysis task scheduler

106‧‧‧ computing resource pool

200‧‧‧Data analysis task

Claims (10)

A cloud self-service analysis platform includes: a shared resource pool, which stores multiple existing data retrieval rules; a data retrieval rule setting module, which is connected to the shared resource pool and used to receive externally input data analysis tasks, to Search through the data analysis task to the shared resource pool to obtain the existing data retrieval rules applicable to the data analysis task; the data retrieval rule analysis module is connected to the data retrieval rule setting module and used for Analyze whether the new data retrieval rule of the data analysis task conflicts with the existing data retrieval rule obtained by the data retrieval rule setting module, so as to generate the final data retrieval rule accordingly; the data collection module Is connected to the data acquisition rule analysis module and the shared resource pool. The data collection module is used to generate an extraction script according to the final data acquisition rule to collect the required data; and the analysis task schedule The device is connected to the data collection module and is used to allocate the analysis algorithm and the collected required data to the computing resource pool for operation according to the priority of the analysis task to generate the final analysis result. The cloud self-service analysis platform as described in item 1 of the patent application scope further includes a visual presentation module connected to the data acquisition rule analysis module and the shared resource pool, which is used to present the statistical distribution of data through graphs or tables. The cloud self-service analysis platform as mentioned in item 2 of the patent scope, more The analysis algorithm selection module connected to the visual presentation module is used for the user to select the analysis algorithm and set related parameters. The cloud self-service analysis platform as described in item 1 of the patent application scope, wherein the data collection module is to obtain a resource set from the shared resource pool as the required when the required data exists in the shared resource pool data. The cloud self-service analysis platform as described in item 1 of the patent application scope, wherein, when the data extraction rule analysis module analyzes the new data extraction rule of the data analysis task and the existing data extraction rule conflicts A warning message is generated, and one of the existing data acquisition rules or the creation of the new data acquisition rules is selected as the final data acquisition rule. The cloud self-service analysis platform as described in item 1 of the patent application scope, wherein the data extraction rule setting module is created by the user when the existing data extraction rule suitable for the data analysis task is not obtained The new data retrieval rule is used as the final data retrieval rule. A cloud self-service analysis method includes the following steps: searching for the existing data extraction rules applicable to the data analysis task according to the received data analysis task to a shared resource pool storing multiple existing data extraction rules; Analyze whether there is a conflict between the new data extraction rules of the data analysis task and the acquired existing data extraction rules applicable to the data analysis task to generate the final data extraction rules accordingly; according to the final data extraction rules Generate a capture script to collect the required data; and According to the analysis task priority order, the analysis algorithm and the collected required data are allocated to the calculation resource pool for calculation to generate the final analysis result. The cloud self-service analysis method as described in item 7 of the patent application scope, wherein collecting required data further includes obtaining the resource set from the shared resource pool as the institute when the required data exists in the shared resource pool Information required. The cloud self-service analysis method as described in item 7 of the patent application scope, in which a warning message is generated when the new data retrieval rule of the data analysis task conflicts with the existing data retrieval rule, and the option to continue using the One of the existing data retrieval rules or the creation of the new data retrieval rule serves as the final data retrieval rule. The self-service analysis method in the cloud as described in item 7 of the patent application scope, in which, when the existing data extraction rules applicable to the data analysis task are not obtained, the user himself creates the new data extraction rules as The final data retrieval rules.

Images