CN117076521A - Operational data analysis methods and systems based on big data - Google Patents

Abstract

The invention relates to the technical field of data analysis, and specifically relates to an operational data analysis method and system based on big data, which can receive analysis targets input by users, automatically collect and store original data, perform data repair or data cleaning, and obtain secondary data. Integrate it into the preset standardized data model and perform standardization processing to obtain standardized data; use preset big data technology and machine learning algorithms to conduct data analysis and mining of standardized data to obtain data hidden associated information; according to the analysis goals, design And calculate key indicators, establish mathematical models and algorithms related to key indicators; use mathematical models and algorithms to conduct data analysis and obtain analysis results. The technical solution shown in the present invention uses machine learning technology to discover more associations and patterns from data and establish relevant models, thereby obtaining more accurate and intelligent analysis results.

Description

Operational data analysis methods and systems based on big data

Technical field

The present invention relates to the technical field of data analysis, and specifically relates to an operational data analysis method and system based on big data.

Background technique

Operational data analysis refers to in-depth and comprehensive analysis and insights into operational data to reveal the patterns, trends and correlations behind the data, helping enterprises to make more accurate decisions and optimize operations.

Existing operational data analysis includes data warehouse & business intelligence (DW/BI), data mining and statistical analysis, etc. Data warehouse and business intelligence: DW/BI technology mainly involves the ETL process of data extraction (Extraction), transformation (Transformation), and loading (Loading), integrating data from various data sources of the enterprise into a centralized data warehouse. Then, perform data analysis through multi-dimensional data models (such as star schema, snowflake model, etc.), and use OLAP (online analytical processing) technology to quickly generate various reports and query results. Data mining: Mainly based on statistics and machine learning algorithms, it uses methods such as pattern recognition, classification, clustering, and association rule mining to discover hidden patterns and associated information from large-scale data; commonly used algorithms include decision trees, neural networks, Support vector machines, cluster analysis, etc. Statistical analysis: Based on statistical theories and methods, infer overall characteristics, relationships or differences from sample data through parameter estimation, hypothesis testing, variance analysis, regression analysis, etc.

However, existing operational data analysis needs to face the following problems:

Data volume and speed issues: Traditional technologies are difficult to handle large-scale data and rapidly changing real-time data. When the amount of data is too large, traditional data warehouse and ETL technology cannot efficiently process and store it, resulting in performance degradation. At the same time, traditional statistical analysis and data mining methods require pre-defined models and algorithms and cannot quickly adapt to new data and scenarios. Data quality issues: Traditional technologies are often unable to handle data quality issues, such as data loss, duplication, inconsistency, etc., so data analysis results may be inaccurate or biased. Moreover, data warehouse and business intelligence technology have strong constraints on the structure and format of data and cannot handle unstructured and semi-structured data. Decision-making limitations: Traditional technology mainly relies on manual experience and set rule logic, and cannot fully utilize the potential of data. Traditional data mining and statistical analysis methods often rely too much on feature engineering and fine tuning, thereby ignoring the inherent information and correlation of the data.

Therefore, the current operational data analysis methods have certain limitations and mainly rely on manual experience and set rule logic, which cannot fully utilize the potential of the data.

Contents of the invention

In view of this, the purpose of the present invention is to provide an operational data analysis method and system based on big data to solve the existing operational data analysis methods, which have certain limitations and mainly rely on manual experience and set settings. The problem of rule logic and inability to fully utilize the potential of data.

According to a first aspect of the embodiment of the present invention, an operational data analysis method based on big data is provided, including:

Receive the analysis target input by the user, automatically collect and store the original data related to the analysis target;

Perform data repair or data cleaning on the original data to obtain secondary data;

Integrate the secondary data into a preset standardized data model, and perform standardization processing on the secondary data to obtain standardized data;

Use preset big data technology and machine learning algorithms to perform data analysis and mining on the standardized data to obtain data hidden associated information;

According to the analysis objectives, use the standardized data and the data hidden associated information to design and calculate key indicators, and establish mathematical models and algorithms related to the key indicators;

Using the mathematical model and algorithm, data analysis is performed on the standardized data and the data hidden associated information according to the analysis target, and analysis results are obtained.

Preferably, after obtaining the analysis results, it also includes:

Utilize data visualization technology to generate a data visualization interface based on the analysis results.

Preferably, the automatic collection of raw data related to the analysis target includes:

Deploy data collection agents on different data sources, and use data collection interfaces to collect original data related to the analysis target.

Preferably, the automatic collection and storage of raw data related to the analysis target includes:

The storage service provided by the cloud computing platform is used to store the collected original data related to the analysis target.

Preferably, the use of preset big data technology and machine learning algorithms to perform data analysis and mining on the standardized data includes:

Using the machine learning algorithm to perform feature extraction on the standardized data and the data hidden associated information, and converting the standardized data and the data hidden associated information into feature vectors;

Pattern recognition, classification, clustering and association rule mining are performed on the feature vectors to obtain data hidden association information.

Preferably, the established mathematical models and algorithms include:

One or more of a linear regression model, a cluster analysis model, a text sentiment analysis model, a random forest algorithm, or an association rule mining algorithm.

According to a second aspect of the embodiment of the present invention, an operational data analysis system based on big data is provided, including:

Data analysis module, used to receive analysis goals input by users;

A data collection module, used to automatically collect and store raw data related to the analysis target;

A data preprocessing module is used to perform data repair or data cleaning on the original data to obtain secondary data;

A data integration standardization module is used to integrate the secondary data into a preset standardized data model, and perform standardization processing on the secondary data to obtain standardized data;

The data analysis and mining module is used to perform data analysis and mining on the standardized data using preset big data technology and machine learning algorithms to obtain data hidden associated information;

An indicator calculation model establishment module is used to design and calculate key indicators based on the analysis target, using the standardized data and the data hidden associated information, and establish mathematical models and algorithms related to the key indicators;

The data analysis module is also configured to use the mathematical model and algorithm to perform data analysis on the standardized data and the data hidden associated information according to the analysis target, and obtain analysis results.

Preferably, the system further includes:

A cloud storage module is used to store the collected original data related to the analysis target using the storage service provided by the cloud computing platform.

Preferably, one of the data analysis module, the data acquisition module, the cloud storage module, the data preprocessing module, the data integration standardization module, the data analysis mining module and the indicator calculation model establishment module. Connections and data transmission are carried out through the services and communication mechanisms provided by the cloud computing platform.

The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

It can be understood that the technical solution provided by the present invention can receive the analysis target input by the user, automatically collect and store the original data, perform data repair or data cleaning, obtain secondary data, and integrate it into the preset standardized data model. And perform standardization processing to obtain standardized data; use preset big data technology and machine learning algorithms to conduct data analysis and mining of standardized data to obtain data hidden related information; design and calculate key indicators based on the analysis goals, and establish relationships with key indicators mathematical models and algorithms; use mathematical models and algorithms to conduct data analysis and obtain analysis results. The technical solution shown in the present invention can automatically clean and repair errors, losses and inconsistencies in data, thereby improving data quality. At the same time, the use of big data technology and machine learning algorithms can discover hidden patterns and associations, improving the precision and accuracy of analysis. It does not rely on preset rules and logic, and can flexibly handle diverse analysis needs and decision-making scenarios. Through machine learning technology, it can discover more associations and patterns from the data and establish relevant models to obtain more accurate and intelligent results. analysis results.

Compared with traditional technology, it has the advantages of improved data quality and analysis accuracy, efficient data processing and analysis, simplified operations and improved user experience, and flexible decision support. These advantages make this technical solution better able to support enterprises' data-driven decisions. , improve the decision-making effect.

It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the present invention.

Description of the drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

Figure 1 is a schematic diagram of the steps of an operational data analysis method based on big data according to an exemplary embodiment;

Figure 2 is a schematic diagram of building a digital intelligent operation data analysis index model according to an exemplary embodiment.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the invention. Rather, they are merely examples of apparatus and methods consistent with aspects of the invention as detailed in the appended claims.

Embodiment 1

Figure 1 is a schematic diagram of the steps of a big data-based operational data analysis method according to an exemplary embodiment. Referring to Figure 1, a big data-based operational data analysis method is provided, including:

Step S11: Receive the analysis target input by the user, automatically collect and store the original data related to the analysis target;

It should be noted that the automatic collection of raw data related to the analysis target includes:

Deploy data collection agents on different data sources, and use data collection interfaces to collect original data related to the analysis target.

In specific practice, raw data can be collected from various data sources (such as mobile device terminals, applications, social media, etc.), data collection agents can be deployed on different data sources, and device interfaces, application program interfaces, etc. can be used for Collect raw data.

It should be noted that the automatic collection and storage of raw data related to the analysis target includes:

The storage service provided by the cloud computing platform is used to store the collected original data related to the analysis target.

In specific practice, the storage services (such as object storage, database, etc.) provided by the cloud computing platform can be used to store data. Cloud storage devices can be built: using hardware resources such as servers, storage devices, and network devices, deployed in reliable data centers or computer rooms. At the same time, network architecture and bandwidth need to be planned to ensure fast transmission and access of data. Configure a distributed storage system. Develop data management strategies, including data classification, data archiving and data backup. Ensure data security and integrity, back up data to prevent data loss. Establish a monitoring and management system to track the performance, availability and other indicators of the storage system in real time.

Step S12: Perform data repair or data cleaning on the original data to obtain secondary data;

In specific practice, data repair or data cleaning can improve data quality while ensuring data consistency. Data cleaning includes operations such as removing duplicate data, filling in missing values, processing outliers, and converting data formats to improve the accuracy and reliability of data. Data cleaning technologies such as PySpark and Pandas can be used to clean the collected data. Data cleaning The conversion data format in can convert data from different data sources and data types according to the analysis goals, and standardize and format the data.

Step S13: Integrate the secondary data into the preset standardized data model, and perform standardization processing on the secondary data to obtain standardized data;

In specific practice, standardizing (or isomorphizing) secondary data can facilitate subsequent data analysis and processing. You can use ETL (data warehouse technology) tools such as Apache Airflow and Talend to integrate data from different data sources into a centralized data warehouse and perform data standardization to ensure data consistency and comparability.

Step S14: Use preset big data technology and machine learning algorithms to perform data analysis and mining on the standardized data to obtain data hidden associated information;

It should be noted that the machine learning algorithm can be used to perform feature extraction on the standardized data and the data hidden associated information, and convert the standardized data and the data hidden associated information into feature vectors; Carry out pattern recognition, classification, clustering and association rule mining to obtain data hidden association information. The data hiding associated information at least includes hidden patterns, associated information and anomaly information. Preferably, according to the specific analysis goals, appropriate machine learning algorithms and model decision trees, neural networks, etc. can be selected to train and optimize the feature vectors of the data to establish accurate and robust models for data analysis and mining.

Step S15: According to the analysis target, use the standardized data and the data hidden associated information to design and calculate key indicators, and establish mathematical models and algorithms related to the key indicators;

In specific practice, key indicators can be KPIs (key performance indicators), KRIs (key risk indicators), etc., and SQL language or programming language can be used to implement indicator calculation logic.

It should be noted that the mathematical models and algorithms established include:

One or more of a linear regression model, a cluster analysis model, a text sentiment analysis model, a random forest algorithm, or an association rule mining algorithm.

The linear regression model is used to analyze the linear relationship between independent variables and dependent variables, and perform prediction and inference. For example, in the field of e-commerce, a linear regression model can be used to analyze the relationship between advertising investment and sales, and predict appropriate advertising investment based on the model.

The random forest algorithm is an ensemble learning algorithm that has good prediction accuracy and robustness by building multiple decision tree models and merging the final prediction results. For example, in the field of financial risk control, the random forest algorithm can be used to assess the credit risk of loan applications.

The cluster analysis model is used to group data samples into similar clusters, and perform data classification and analysis based on the differences between clusters. For example, in marketing, cluster analysis models can be used to divide customers into different groups and develop corresponding marketing strategies for each group.

Association rule mining algorithms are used to discover relevant patterns and association rules in data sets to help explain the relationships between data. For example, in the retail industry, association rule mining algorithms can be used to discover combinations of items that are often purchased together for cross-selling and promotion.

Text sentiment analysis models are used to identify and analyze sentiment tendencies and sentiment polarities in text data. For example, in social media analysis, text sentiment analysis models can be used to evaluate users' emotional attitudes towards a certain product or brand to guide corresponding market strategies.

Step S16: Use the mathematical model and algorithm to perform data analysis on the standardized data and the data hidden associated information according to the analysis target, and obtain analysis results.

In specific practice, the analysis results are in-depth and comprehensive analysis and insights into operational data to reveal the patterns, trends and correlations behind the data, helping enterprises to make more accurate decisions and optimize operations.

Analysis results may include but are not limited to the following:

Data characteristics and trends: Analyze and describe the characteristics and trends of operational data, such as sales growth trends, changes in user behavior, etc.

Correlation and association rules: By analyzing the correlation and association rules between data, we can understand the mutual influence between variables, such as the relationship between marketing expenses and sales.

Forecasting and early warning: Use the established forecast model to predict future trends and results, and provide early warning based on outliers and rules.

Optimization strategies and suggestions: Based on the analysis results, we propose optimization strategies and suggestions to help enterprises make more accurate decisions and optimize operations.

To sum up, the analysis results of data analysis indicators based on big data and cloud computing are mainly obtained through the processing and analysis of massive data to reveal the patterns, trends and correlations behind the data, and provide more comprehensive and comprehensive solutions for enterprise decision-making. Backed by accurate data.

It should be noted that after obtaining the analysis results, it also includes:

Utilize data visualization technology to generate a data visualization interface based on the analysis results.

In specific practice, the analysis results can be displayed to users through visual tools (such as dashboards, reports, charts, etc.) to help users understand the data and analysis results, thereby making more accurate and scientific decisions. You can choose appropriate data visualization tools such as Tableau and Power BI to display analysis results to produce easy-to-understand and transparent visual reports and charts. Preferably, it can also include self-service analysis and interactive functions: through the interactive functions of charts, users can freely explore and analyze data to make more accurate and scientific decisions.

It can be understood that according to the technical solution shown in this embodiment, functions such as data collection, preprocessing, integration, analysis, index calculation, model establishment, and result display can be realized, and efficient, accurate, and visual data analysis and decision support can be provided.

Preferably, a digital intelligent operation data analysis index model based on big data and cloud computing can be constructed according to the above technical solution, thereby making the implementation of the method more convenient and effective. The hierarchical structure of the digital intelligent operation data analysis indicator model is shown in Figure 2. The elements of the digital intelligence operation data analysis indicator model are user behavior conversion touch points. It mainly dismantles the cycle and priority of business goals, and summarizes some commonly used combination forms from the perspectives of purpose, time, incentive methods, etc. The digital intelligent operation data analysis indicator model based on big data and cloud computing can make up for the limitations of traditional technology and provide more powerful, efficient, accurate and intelligent data analysis and decision support: Data scale scalability: based on big data and cloud computing technology that can process massive amounts of structured and unstructured data and support fast, real-time data analysis and decision-making response. Automated processing capabilities: With the help of machine learning and deep learning technology, automatic feature extraction, model training and optimization can be achieved, reducing the workload of manual participation. Elasticity and flexibility: The cloud computing platform provides elastic resource allocation and expansion and contraction capabilities, and can adapt to data analysis tasks of different sizes and needs. Real-time and practical: Able to process, analyze and learn data in real time, providing more accurate and practical decision support. At the same time, because it does not rely on preset models and rules, more hidden patterns and correlations can be discovered. At the same time, based on the AARRR funnel model, the digital intelligent operation data analysis indicator model explains the five indicators for achieving user growth, which can help companies better explain the principles of acquiring and maintaining customers.

Compared with the existing technology, the technical solution disclosed in this embodiment can have the following effects and advantages:

Improve data quality and analysis accuracy: Through data preprocessing and cleaning, errors, losses, and inconsistencies in data can be automatically cleaned and repaired, thereby improving data quality. At the same time, advanced analysis algorithms and models can be used to discover hidden patterns and correlations, improving the precision and accuracy of analysis.

Achieve efficient data processing and analysis: Using the parallel computing and distributed storage capabilities of the cloud computing platform, large-scale data can be quickly processed and real-time analysis can be achieved. In aspects such as data collection, data preprocessing, model training and indicator calculation, computing tasks can be processed and optimized in parallel to improve efficiency and performance.

Simplify operations and improve user experience: The digital intelligent operation data analysis indicator model software based on big data and cloud computing designs a user-friendly interactive interface and simplifies the user's operation process and data analysis process through visual tools and self-service analysis functions. Users can easily perform data query, indicator calculation and analysis display through simple operations, improving user experience and efficiency.

Provide flexible decision support: The digital intelligent operation data analysis indicator model software based on big data and cloud computing does not rely on preset rules and logic, and can flexibly handle diverse analysis needs and decision-making scenarios. Through machine learning and deep learning technology, we can learn and optimize models, discover more associations and patterns from data, and provide more accurate and intelligent decision support.

Resource saving and cost reduction: Utilizing the elasticity and automation features of the cloud computing platform, resources can be allocated, expanded and reduced based on actual needs, and payment can be made based on the actually used computing, storage, network and other resources, reducing the cost of hardware equipment and human resources. Input costs.

Can bring the following effects:

In-depth insights: By analyzing big data, you can gain an in-depth understanding of the characteristics, trends and patterns of operational data, reveal the patterns and correlations behind the data, and help companies better understand the market, users and competitive environment.

Accurate prediction: Based on the established model, future trends and results can be accurately predicted to help enterprises make reasonable planning and decisions. For example, through the analysis of sales data, future sales and demand changes can be predicted.

Optimized decision-making: Through data analysis and mining of operational data, enterprises can be provided with the basis for optimized decision-making. By understanding the correlations and patterns in the data, companies can formulate more reasonable marketing strategies, product pricing strategies, etc.

Discover opportunities and risks: By analyzing data, potential opportunities and risks can be discovered. For example, through the analysis of market data, emerging markets and potential customer groups can be identified, providing guidance for companies to expand their business.

Real-time monitoring and adjustment: Using cloud computing and real-time data processing, operational data can be monitored in real time, abnormal situations can be discovered in a timely manner, and rapid decision-making and adjustments can be made. This helps businesses stay agile and competitive in a highly competitive market.

In order to achieve these effects, the digital intelligent operation data analysis indicator model and product structure need to cooperate with each other, including the following aspects:

Data integration and processing: Through cloud computing and big data technology, scattered and heterogeneous data sources are integrated, standardized and cleaned to ensure the analyzability and availability of data.

Model development and application: Based on business needs, establish suitable data analysis models, including machine learning models, statistical analysis models, prediction models, etc. At the same time, the model is applied to actual data for analysis and prediction.

Results visualization and reporting: Through visualization technology, analysis results are presented to decision makers in the form of charts, reports, etc., providing intuitive data-driven decision support. This helps decision-makers quickly understand the analysis results and make appropriate decisions.

Real-time monitoring and feedback: With the help of cloud computing and real-time data processing platform, real-time monitoring and feedback of operational data are achieved. Discover existing problems and opportunities in a timely manner and adjust decisions and strategies in a timely manner.

In actual application scenarios, take the e-commerce platform as an example to illustrate:

Data collection and cleaning: E-commerce platforms collect user behavior data, including purchase records, browsing behavior, search history, etc. These data are generated through user interactions on the e-commerce platform, and are cleaned and preprocessed to ensure data accuracy and consistency.

Data storage and processing: The e-commerce platform stores the collected user behavior data in the big data storage system of the cloud computing platform to utilize distributed storage and computing capabilities for data processing. Through cloud computing platforms, large-scale user behavior data can be efficiently processed.

Model development and application: Based on user behavior data, e-commerce platforms can build classification models to identify different types of users, such as high-quality customers, sleeping customers and new customers. Models can be trained using machine learning algorithms and classified based on users’ purchase frequency, purchase amount, activity and other characteristics.

Result analysis and optimization strategies: Using the established classification model, the e-commerce platform can analyze and formulate optimization strategies for different types of users. For example, for high-quality customers, you can provide personalized preferential benefits to enhance their loyalty; for sleeping customers, you can push specific activities or coupons to awaken their interest in purchasing; for new customers, you can take new promotion measures And focus on improving retention rates.

Results visualization and reporting: In order to enable decision-makers to better understand the analysis results, e-commerce platforms can use data visualization technology to display the analysis results of different types of users in the form of charts and reports. These visual results include the number of different types of users, conversion rates, repurchase rates and other indicators, as well as optimization strategy suggestions for different types of users.

Embodiment 2

Provide an operational data analysis system based on big data, including:

Data analysis module, used to receive analysis goals input by users;

A data collection module, used to automatically collect and store raw data related to the analysis target;

A data preprocessing module is used to perform data repair or data cleaning on the original data to obtain secondary data;

A data integration standardization module is used to integrate the secondary data into a preset standardized data model, and perform standardization processing on the secondary data to obtain standardized data;

The data analysis and mining module is used to perform data analysis and mining on the standardized data using preset big data technology and machine learning algorithms to obtain data hidden associated information;

An indicator calculation model establishment module is used to design and calculate key indicators based on the analysis target, using the standardized data and the data hidden associated information, and establish mathematical models and algorithms related to the key indicators;

The data analysis module is also configured to use the mathematical model and algorithm to perform data analysis on the standardized data and the data hidden associated information according to the analysis target, and obtain analysis results.

It can be understood that the technical solution provided by this embodiment can receive the analysis target input by the user, automatically collect and store the original data, perform data repair or data cleaning, obtain secondary data, and integrate it into the preset standardized data model. , and carry out standardization processing to obtain standardized data; use preset big data technology and machine learning algorithms to conduct data analysis and mining of standardized data to obtain data hidden related information; design and calculate key indicators based on the analysis goals, and establish key indicators Relevant mathematical models and algorithms; use mathematical models and algorithms to conduct data analysis and obtain analysis results. The technical solution shown in the present invention can automatically clean and repair errors, losses and inconsistencies in data, thereby improving data quality. At the same time, the use of big data technology and machine learning algorithms can discover hidden patterns and associations, improving the precision and accuracy of analysis. It does not rely on preset rules and logic, and can flexibly handle diverse analysis needs and decision-making scenarios. Through machine learning technology, it can discover more associations and patterns from the data and establish relevant models to obtain more accurate and intelligent results. analysis results.

It should be noted that the system also includes:

A cloud storage module is used to store the collected original data related to the analysis target using the storage service provided by the cloud computing platform.

It is understandable that by taking advantage of the elasticity and automation features of the cloud computing platform, resources can be allocated and expanded or reduced according to actual needs, and payment can be made according to the actually used computing, storage, network and other resources, reducing the investment in hardware equipment and human resources. cost.

It should be noted that the data analysis module, the data acquisition module, the cloud storage module, the data preprocessing module, the data integration standardization module, the data analysis mining module and the indicator calculation model are established The modules are connected and data transmitted through the services and communication mechanisms provided by the cloud computing platform.

It can be understood that the same or similar parts in the above-mentioned embodiments can be referred to each other, and the content that is not described in detail in some embodiments can be referred to the same or similar content in other embodiments.

It should be noted that in the description of the present invention, the terms "first", "second", etc. are only used for description purposes and cannot be understood as indicating or implying relative importance. Furthermore, in the description of the present invention, unless otherwise stated, the meaning of "plurality" means at least two.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments, or portions of code that include one or more executable instructions for implementing the specified logical functions or steps of the process. , and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed out of the order shown or discussed, including in a substantially simultaneous manner or in the reverse order, depending on the functionality involved, which shall It should be understood by those skilled in the art to which embodiments of the present invention belong.

It should be understood that various parts of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if it is implemented in hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a logic gate circuit with a logic gate circuit for implementing a logic function on a data signal. Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

Those of ordinary skill in the art can understand that all or part of the steps involved in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The program can be stored in a computer-readable storage medium. When executed, one of the steps of the method embodiment or a combination thereof is included.

In addition, each functional unit in various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

The storage media mentioned above can be read-only memory, magnetic disks or optical disks, etc.

In the description of this specification, reference to the terms "one embodiment," "some embodiments," "an example," "specific examples," or "some examples" or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above-mentioned embodiments are illustrative and should not be construed as limitations of the present invention. Those of ordinary skill in the art can make modifications to the above-mentioned embodiments within the scope of the present invention. The embodiments are subject to changes, modifications, substitutions and variations.

Claims (9)

1. An operational data analysis method based on big data, comprising:

receiving an analysis target input by a user, and automatically collecting and storing original data related to the analysis target;

performing data restoration or data cleaning on the original data to obtain secondary data;

integrating the secondary data into a preset standardized data model, and carrying out standardized processing on the secondary data to obtain standardized data;

carrying out data analysis and mining on the standardized data by using a preset big data technology and a machine learning algorithm to obtain data hiding associated information;

according to the analysis target, utilizing the standardized data and the data hiding associated information to design and calculate key indexes, and establishing a mathematical model and algorithm related to the key indexes;

and carrying out data analysis on the standardized data and the data hiding associated information according to the analysis target by using the mathematical model and the algorithm to obtain an analysis result.

2. The method of claim 1, further comprising, after the obtaining the analysis result:

and generating a data visualization interface according to the analysis result by utilizing a data visualization technology.

3. The method of claim 1, wherein the automatically collecting raw data related to the analysis target comprises:

and deploying data acquisition agents on different data sources, and acquiring the original data related to the analysis target by using a data acquisition interface.

4. The method of claim 1, wherein the automatically collecting and storing raw data related to the analysis target comprises:

and storing the collected original data related to the analysis target by using a storage service provided by the cloud computing platform.

5. The method of claim 1, wherein the data analysis and mining of the standardized data using pre-set big data techniques and machine learning algorithms comprises:

extracting features of the standardized data and the data hiding associated information by using the machine learning algorithm, and converting the standardized data and the data hiding associated information into feature vectors;

and carrying out pattern recognition, classification, clustering and association rule mining on the feature vectors to obtain the data hiding association information.

6. The method of claim 1, wherein the mathematical model and algorithm established comprises:

one or more of a linear regression model, a cluster analysis model, a text emotion analysis model, a random forest algorithm, or an association rule mining algorithm.

7. An operational data analysis system based on big data, comprising:

the data analysis module is used for receiving an analysis target input by a user;

the data acquisition module is used for automatically acquiring and storing the original data related to the analysis target;

the data preprocessing module is used for carrying out data restoration or data cleaning on the original data to obtain secondary data;

the data integration standardization module is used for integrating the secondary data into a preset standardization data model and carrying out standardization processing on the secondary data to obtain standardization data;

the data analysis mining module is used for carrying out data analysis and mining on the standardized data by utilizing a preset big data technology and a machine learning algorithm to obtain data hiding associated information;

the index calculation model building module is used for designing and calculating key indexes by utilizing the standardized data and the data hiding associated information according to the analysis target, and building a mathematical model and algorithm related to the key indexes;

the data analysis module is further used for carrying out data analysis on the standardized data and the data hiding associated information according to the analysis target by utilizing the mathematical model and the algorithm to obtain an analysis result.

8. The system of claim 7, further comprising:

and the cloud storage module is used for storing the collected original data related to the analysis target by using a storage service provided by the cloud computing platform.

9. The system of claim 8, wherein the system further comprises a controller configured to control the controller,

the data analysis module, the data acquisition module, the cloud storage module, the data preprocessing module, the data integration standardization module, the data analysis mining module and the index calculation model building module are connected and data transmission is carried out through a service and communication mechanism provided by a cloud calculation platform.