CN111708774B - An industrial analysis system based on big data - Google Patents

Abstract

The invention relates to an industry analysis system based on big data, which comprises an industry development related database module, a data analysis model module, a data base platform module and a user side interface module; the industry development related database module is used for storing certain industry development related data resources; the data analysis model module is used for storing the data analysis model; the data base platform module is simultaneously connected with the industry development related database module and the data analysis model module and is used for calling related data from the industry development related database module according to an analysis target, then calling a related model from the data analysis model module and inputting the related data into the related model; the data analysis model module is also used for outputting an analysis result after the relevant data is input into the relevant model; the user terminal interface module is connected with the data analysis model module and used for displaying the analysis result. The industrial analysis system can provide data support for industrial development and make decisions based on data analysis.

Description

An industrial analysis system based on big data

technical field

The invention belongs to the technical field of industrial analysis systems, and relates to an industrial analysis system based on big data.

Background technique

As an important factor of production, intangible assets and social wealth in the information society, data resources have become the basic strategic resources of the country. Through in-depth analysis of data, it is possible to summarize experience, discover laws, predict trends, and assist decision-making. Economic development has entered a new normal, and there will be many new situations and new problems. There is an urgent need for scientific research and judgment, timely response, and the preparation and preparation of tools in the policy toolbox. This puts forward higher requirements for traditional industrial monitoring and scheduling methods.

Government decision-making urgently needs to grasp information related to industrial development, including information from macro to micro levels, so as to provide quantitative decision-making basis for the formulation and adjustment of policies such as industrial transformation and upgrading, investment promotion, and enterprise development support, so as to achieve more accurate and effective development. The supporting work guides and promotes the rapid, healthy and orderly development of Shanghai's big data industry. The development of enterprises urgently needs to grasp the dynamics of industrial development and market development.

At present, the promotion and research of industrial development research is carried out in a relatively decentralized manner, and the research is mainly carried out on each factor separately, or based on expert experience, which is difficult to meet the needs of a certain industry development in rapid progress. Through the establishment of an industrial big data analysis platform, this problem can be solved, and government departments can grasp the status quo of industrial development, provide effective assistance in making decisions, and effectively promote industrial development.

Contents of the invention

In order to overcome the deficiencies of the above-mentioned technologies, the purpose of the present invention is to provide an industrial analysis system based on big data, which can analyze relevant factors affecting the development of the industry, sum up experience, discover laws, predict trends, and assist decision-making, providing industry development with Data support, making decisions based on data analysis rather than just expert experience.

In order to achieve the above object, the technical scheme adopted in the present invention is as follows:

An industrial analysis system based on big data, including an industrial development-related database module, a data analysis model module, a data foundation platform module, and a user interface module;

The industrial development-related database module is used to store data resources related to an industry development; the industrial development-related database module forms highly available data assets through data governance, provides support for the data basic platform module and data analysis model module, and supports data query and business analysis needs;

The technical realization of the database module related to industrial development mainly relies on the following technologies:

(1) A new domestic database that can support real-time big data processing, which has the following functions:

(1.1) Large-scale parallel processing: the storage is built on HDFS, and the intermediate results of the calculation are placed in memory;

(1.2) Columnar storage 2.0: enhanced metadata, dictionary encoded data, and automatic data sorting;

(1.3) Dynamic data allocation: broadcast small data tables, dynamic redistribution of large table hash values, Join localization, pipelines;

(1.4) Memory computing: vectorized processing, dynamic compilation based on the underlying virtual machine (LLVM);

(1.5) OLTP/OLAP dual analysis engine: Compatible with OLTP and OLAP data analysis and processing requirements;

(2) Data interface system and man-machine interface system, including:

(2.1) Machine-type data interface and collection and processing system, including:

(2.1.1) Infrastructure operating environment data engine processing subsystem, capable of adapting and configuring MIB information, etc.;

(2.1.2) Data engine processing subsystems such as SYSLOG;

(2.1.3) Big data industry and related enterprise information collection and processing subsystems;

(2.2) Business data interface system, business comprehensive data interface should support multiple interface formats and human-computer interaction mixed mode, to adapt to multi-source cold and hot data access processing, etc., business data interface format types expansion support and update management In response to big data related business data from different sources, it can support interfaces in various formats such as RestAPI/SNMP/SYSLOG/files, support real-time, timing, condition triggering and other update frequency modes, and can be configured, and relevant interface protocol specifications and standard customization Listed below:

(2.2.1) SYSLOG interface, Internet of Things, machine equipment data standard industrial interface specifications, including system logs of server hosts, general network equipment, and special security equipment should be sent to the security management and control system through the SYSLOG protocol, and the security management and control system passes the agreed SYSLOG Message format, parsing log information, and standardizing, analyzing, and correlating log information;

(2.2.2) SNMP interface, the system logs of IoT devices, server hosts, general network devices, and special security devices should implement the basic SNMP protocol MIB library, and the security management and control system collects the basic information of the devices and the operating performance of the devices through the SNMP protocol. Protocol and receiving device SNMP Trap event notification information;

(2.2.3) REST interface, the REST interface protocol is an implementation of WebService (Web service), which is mainly used to realize the interface between systems. In the mobile police platform, in addition to using general SNMP and SYSLOG on the equipment management level In addition to protocols, scenarios such as inter-software systems and policy configuration delivery should be implemented using the REST interface protocol. In general, the request-response mode should be used to implement monitoring information reporting, policy and instruction delivery, etc. The request and response communication protocol should use HTTP1 .1 over SSL/TLS, the request response mode should use HTTP as the interface implementation protocol, the interface parameters and return results all use JSON objects, the data packet size does not exceed 5MB, and the message push mode can be used when the reverse network is unreachable To implement the delivery of policy instructions, the communication protocol should adopt HTTP2.0 WEBSOCKET over SSL/TLS, and HTTP should be used as the interface implementation protocol. The interface parameters and return results should use JSON objects, and the size of the data packet should not exceed 5MB;

(2.2.4) Application system log interface. The application system log interface uses a standard SDK interface to send terminal application/business domain log data. In order to ensure collection performance, the application interface supports cluster deployment. The amount and the size of the concurrent number, select the number of nodes in the cluster;

(2.3) Human-machine interface system services, including:

(2.3.1) The front-end human-machine interface system establishes an active service interactive data collection human-machine interface for tens of thousands of enterprises in the industry;

(2.3.2) Security platform data exchange pre-service platform, establish a security data communication platform for the industry analysis platform, etc.;

(2.3.3) Enterprise-related data inventory and combing services, interface negotiation and customization services;

The data analysis model module is used to store the data analysis model;

The data base platform module is connected with the industrial development-related database module and the data analysis model module at the same time, and is used to retrieve relevant data from the industrial development-related database module according to the analysis target (matching the analysis target with the relevant data to achieve the smooth retrieval is done For example, if the analysis target is to analyze the enterprises established in the past five years, then the data of enterprises whose establishment time is greater than 2015 will be called from the database; "Enterprise data of the "core enterprise" label), retrieve relevant models from the data analysis model module (it is a known technology to match the analysis target with the relevant model to achieve smooth retrieval, for example, the analysis target is "classification according to the industrial chain", Then in the analysis module, according to the model name, manually select the "industrial chain model" to call), and input the relevant data into the relevant model; the data basic platform module provides a strong platform support for the system construction, shortens the development cycle, and reduces the risk of system construction , to improve performance and stability; data maintenance, including data completion and update; data query, wizard search and custom reports are available, and query results can be filtered, sorted, summarized upward, drilled down, Simple calculations, conditional formatting, etc., support the export and printing of results; for visual display, a variety of intelligent visualization algorithms can be built in to achieve multi-dimensional and multi-terminal three-dimensional presentation of massive data;

The technical implementation of the data base platform module relies on a visual big data management, analysis and display platform, the functions are as follows:

(a) Metadata-based data management: build a set of standardized, unified and common big data resources for users;

(b) Conventional multidimensional analysis + open modeling: provide users with multidimensional analysis and modeling analysis based on R language;

(c) Compatible with traditional and big data storage: Compatible with relational DB, Hadoop, NoSQL and other storage types;

(d) Visual dynamic chart analysis: provide users with a drag-and-drop chart display method that conforms to the operating habits of Chinese people;

The data analysis model module is also used to output the analysis results after the relevant data is input into the relevant model;

The user-end interface module is connected with the data analysis model module to display the analysis results; the user-end module includes key applications such as industrial economic operation, industrial management, and productivity layout. Each business application includes three parts, namely status monitoring and special analysis and trend analysis;

The technical realization of the user interface module mainly relies on the following technologies:

1)HTML

HTML is the core technology required to build a web interface; it is a tag-based language used to describe the structure of documents displayed by browsers;

2) CSS (Cascading Style Sheets)

In web applications, it is used to specify how HTML content is rendered on the screen;

3) JavaScript

Object-oriented JavaScript, supporting object-oriented, imperative and declarative at the same time;

4) hyperlink

Hyperlinks in web applications usually contain pre-set request parameters. These data items do not need to be input by the user, but are inserted by the server into the target URL of the hyperlink clicked by the user. Big Data Industry Analysis System The project reserve library, friendship links, etc. are all developed in this way;

5) Form

receive user input;

6) Document Object Model DOM

7) Ajax (Asynchronous JavaScript and XML, asynchronous JavaScript and XML)

AJAX is the art of exchanging data with a server and updating parts of a web page without reloading the entire page;

Big data industrial systems use Ajax, some user actions will be handled by client-side script code, and the whole page does not need to be reloaded; instead, the script will execute the request "in the background" and usually receive a smaller response for dynamic Update part of the user interface;

The core technology XML Http Request used by Ajax, after a certain degree of standard integration, this technology has now been transformed into a local JavaScript object, through which client scripts can make "background" requests without window-level navigation events;

8) JSON (JavaScript Object Notation, JavaScript Object Notation)

Big data industry systems use JSON to store and exchange text information, which is a lightweight text data exchange format;

Ajax applications often use JSON to replace the XML format originally used for data transmission. JSON is smaller, faster, and easier to parse than XML;

Usually, if the user performs an operation, the client-side JavaScript will use XML Http Request to transmit the operation to the server, and the server will return a lightweight response containing data in JSON format, and then the client-side script will process the data and Update the user interface accordingly;

9) Same Origin Policy

The core of the big data industry map system: prevent content from different sources from interfering with each other, and only allow content from the same source to interact;

Content received from one site can read and modify other content received from that site, but not access content received from other sites, and if the Same Origin Policy is not used, then when an unsuspecting user browses When a malicious website is accessed, script code running on that website will be able to access the data and functionality of any other website that the user is visiting at the same time;

The user terminal interface module provides the user result viewing function, and the user can view the industrial economic operation, industrial management, productivity layout, etc. to provide status monitoring, thematic analysis and trend research and judgment data analysis results.

As a preferred solution:

As mentioned above in an industry analysis system based on big data, the data resources related to the development of a certain industry include industry data, enterprise data, regional data and technical data related to the development of a certain industry.

As mentioned above, an industry analysis system based on big data, the data analysis model includes an industry classification model, an industry chain model and an enterprise performance evaluation model; the industry classification model is used to determine the industry classification of the enterprise according to the business scope of the enterprise to be classified ;The industrial chain model is used to determine the position of the enterprise in the industrial chain through the weighted scoring method, according to the scores of the four categories of "resources", "technology", "application" and "industrial support" according to the data of the enterprises to be classified , that is, the category in the industrial chain; the enterprise performance evaluation model is used based on the data related to the enterprise's comprehensive ability, research and development ability, industry influence, sustainable operation ability and data application ability (that is, the enterprise's registered capital, listing status, business income , R&D investment and other data) After scoring the enterprise's comprehensive ability, R&D ability, industry influence, sustainable operation ability and data application ability, calculate the weighted total score to get the enterprise performance.

As mentioned above, an industry analysis system based on big data has a relatively simple structure of the industry classification model. It adopts the decision tree algorithm. The decision tree classification is in line with the thinking of human classification. Many different questions will be raised during the decision tree classification. Judging the sample A certain feature of the model, and then combine all the judgment results to give the category of the sample. The establishment process of the industry classification model is as follows:

(1) start;

(2) Create a tree with the enterprise data set (complete set of samples) as the root node;

(3) Create a node;

(4) Determine whether the enterprise data set (complete set of samples) is empty, if yes, return to the previous node, and end; otherwise, enter the next step;

(5) Determine whether the current node data set (sample subset) belongs to the same attribute, if so, record it as a leaf node and mark it as class C, and then end; otherwise, enter the next step;

(6) Determine whether the candidate attribute set is empty, if yes, record it as the class C with the largest number of samples in S, and end; otherwise, enter the next step;

(7) Calculate the information gain rate of each enterprise condition attribute in the set;

(8) Select the largest information gain in the candidate set as the segmentation attribute of the current node;

(9) Determine the enterprise data set (sample subset) according to the value of the segmentation attribute, and establish corresponding branches;

(10) continuously recursively run the function to the enterprise data set (sample subset), and return to step (2);

The workflow of the industry classification model is as follows:

(1) start;

(2) Input the business scope data of the enterprise to be classified;

(3) Through the decision tree algorithm, the industry classification of enterprises;

(4) END.

In the industrial analysis system based on big data mentioned above, the industry chain division standards of enterprises are relatively vague, and a company may exist in multiple positions in the industry chain at the same time. Therefore, a single industry chain division method cannot meet actual needs. In order to accurately divide the position of the industrial chain of enterprises, the present invention adopts a weighted scoring method, wherein the classification algorithm adopts the random forest model, the scoring method adopts the exponential weighted average method, and the data set dimensions used in the research of the big data industrial chain model are more , The amount of data is large, the random forest algorithm can effectively solve these problems, and its advantages are mainly reflected in:

(1) On many current data sets, it has great advantages over other algorithms and performs well;

(2) It can handle very high-dimensional data and does not require feature selection;

(3) The training speed is fast, and it is easy to make a parallel method (the trees are independent of each other during training);

(4) During the training process, the interaction between data of different dimensions can be detected;

(5) When dealing with unbalanced data sets, errors can be balanced;

(6) If a large part of the features are missing, the accuracy can still be maintained;

The workflow of the industrial chain model is as follows:

(1) start;

(2) Data set acquisition: obtain the business information, intellectual property rights, and historical data of enterprise operations as the original data set, and construct the classification indicators of the big data industry chain, which are: "resources", "technology", "application" and " industry support";

(3) Data labeling: label the original data set according to the classification index of the big data industry chain to mark the category of the enterprise in the industry chain;

(4) Data preprocessing: perform data matching and outlier removal operations on the data in the original data set;

(5) Data set division: divide the data in the original data set into a training set and a test set according to a ratio of 3:1;

(6) Build a random forest: apply the traditional random forest algorithm on the training set to build a random forest for predicting the position of the enterprise in the industrial chain;

(7) Random forest model training: use the data in the training set to train the random forest model of N decision trees, N is an integer greater than 1, and each decision tree randomly extracts enterprise data from the training set for training, using gain entropy To select the appropriate attribute nodes, each tree randomly selects samples and attribute features from the training set to generate their own nodes until all decision trees classify the samples they have sampled;

(8) Model evaluation and correction: Input the test set into the trained random forest model for classification, count the classification results and actual results, and calculate the prediction accuracy rate. Both the classification results and the actual results are the categories of enterprises in the industrial chain , when the prediction accuracy rate is less than the set value (the set value can be set according to actual needs), calculate the classification result obtained by each decision tree and calculate its AUC value, based on the AUC value extracted from the current random forest model A set of relatively high-precision decision trees is clustered according to similarity and divided into different clusters. Finally, a set of high-precision decision trees is selected from different clusters to iteratively update the existing random forest model; when the prediction is accurate If the rate is greater than or equal to the set value, the random forest model will not be updated;

(9) Obtain the data of the enterprises to be classified;

(10) Perform data matching and outlier removal operations on the data;

(11) Input the data into the trained random forest model, and output the classification result;

(12) END.

As mentioned above, an industrial analysis system based on big data, the process of the enterprise performance evaluation model is as follows:

(1) start;

(2) Score the enterprise's comprehensive ability, research and development ability, industry influence, sustainable operation ability and data application ability;

Comprehensive ability score a=a1+a2+a3, when the registered capital>50000, a1=100; when the registered capital<100, a1=45; when 100≤registered capital≤50000, a1=registered capital ^1/8 × 25; when the company is a listed company, a2 is randomly selected in the interval [90,100]; when the enterprise is an unlisted company, a2 is randomly selected in the interval [50,60]; when the main business income>50000, a3=100; when the main business income <100, a3=45; when 100≤main business income≤50000, a3=main business income1 ^/8 ×25;

R&D capability score b=b1+b2+b3+b4, when R&D funds>10000, b1=100; when R&D funds<10, b1=45; when 10≤R&D funds≤10000, b1=R&D funds1 ^{/ 8} × 31; when the number of R&D personnel > 1000, b2 = 100; when the number of R&D personnel < 5, b2 = 50; when 5 ≤ the number of R&D personnel ≤ 1000, b2 = ^1/8 × 42 of the number of R&D personnel; when the number of patents > 50 , b3=100; when the number of patents<2, b3=45; when 2≤the number of patents≤50, b3= ^1/4 ×38 of the number of patents; when the number of soft deals ≥5, b4=100; When the number of moves = 4, b4 = 90; when the number of soft moves = 3, b4 = 80; when the number of soft moves = 2, b4 = 70; when the number of soft moves = 1, b4 = 60; =0, b4=50;

Industry influence score c = c1 + c2, when the company is a listed company, c1 = 60 + main business income ^1/8 × 10; when the company is a non-listed company, c1 = 40 + main business income ^1/8 ×10; when big data business revenue >20,000, c2=100; when big data business revenue <50, c2=45; when 50≤big data business revenue≤20,000, c2=big data business revenue ^1/8 × 29;

Sustainability score d=d1+d2+d3, d1=market share×0.5+economic benefits×0.25+main business income×0.25; when the sum of venture capital and foreign capital utilization>10000, d2=100; when risk When the sum of investment and utilization of foreign capital <1, d2=30; when 1≤the sum of venture capital and utilization of foreign capital≤10,000, d2=(venture investment + utilization of foreign capital) ^1/8 ×32; when operating profit>10,000, d3=100; when operating profit<1, d3=30; when 1≤operating profit≤10000, d3=operating profit ^1/8 ×32;

Data application ability score e=e1+e2+e3, when data soft assets>2000, e1=100; when data soft assets<10, e1=45; when 10≤data soft assets≤2000, e1=data soft assets Assets ^1/8 × 38; when data hard assets > 2000, e2 = 100; when data hard assets < 10, e2 = 45; when 10≤data hard assets≤2000, e2=data hard assets ^1/8 × 38; when the turnover of data products > 10000, e3 = 100; when the turnover of data products < 10, e3 = 45; when 10≤ turnover of data products ≤ 10000, e3 = turnover of data products ^1/8 × 32 ;

Among them, the unit of registered capital, main business income, research and development funds, big data business income, economic benefits, venture capital, utilization of foreign capital, operating profit, data soft assets, data hard assets and data product turnover is RMB 10,000; The unit of the number of people is person, the unit of the number of patents and the number of soft works is unit, and the unit of market share is % (calculated based on the sum of similar enterprises in the current data set);

(3) Calculate the weighted total score g to obtain enterprise performance, g=w1×a+w2×b+w3×c+w4×d+w5×e, w1=0.3, w2=0.25, w3=0.1, w4=0.15, w5=0.2;

(4) END.

As mentioned above, an industrial analysis system based on big data, the data basic platform module provides big data storage, calculation, analysis, visualization, daily operation and maintenance service support, and the data basic platform includes a data exchange platform, a big data platform, and an information release platform , GIS geographic information platform, basic component platform, report analysis platform and system management platform.

Beneficial effect:

The present invention builds an industrial analysis system based on big data, gathers relevant economic data of a certain industry together, conducts timely monitoring and scheduling on this basis, discovers problems in time, warns in time, and responds in time to maintain the stable operation of the economy , on this basis, combined with the changes in massive historical data, by adjusting the relevant factor variables, the development trend of a certain industry is predicted, and finally provides a quantitative basis for the government to scientifically formulate industrial development support policies.

Description of drawings

Figure 1 is a framework diagram of an industrial analysis system based on big data;

Fig. 2 is the working flowchart of request response mode;

Fig. 3 is the working flowchart of message push mode;

Figure 4 is a schematic structural diagram of the industry classification model;

Fig. 5 is the establishment flowchart of industry classification model;

Fig. 6 and Fig. 7 are the work flowchart of industrial chain model;

Figure 8 is a schematic structural diagram of an enterprise performance evaluation model.

Detailed ways

The present invention will be further described below in combination with specific embodiments. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the teachings of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

An industrial analysis system based on big data, as shown in Figure 1, consists of an industrial development-related database module, a data analysis model module, a data foundation platform module, and a user interface module.

The database module related to industrial development is used to store data resources related to the development of an industry, including industry data, enterprise data, regional data and technical data related to the development of a certain industry; the database module related to industrial development forms highly available data assets through data governance, Provide support for the data base platform module and data analysis model module, and support the needs of data query and business analysis;

The technical realization of the database module related to industrial development mainly relies on the following technologies:

(1) A new domestic database that can support real-time big data processing, which has the following functions:

(1.1) Large-scale parallel processing: the storage is built on HDFS, and the intermediate results of the calculation are placed in memory;

(1.2) Columnar storage 2.0: enhanced metadata, dictionary encoded data, and automatic data sorting;

(1.3) Dynamic data allocation: broadcast small data tables, dynamic redistribution of large table hash values, Join localization, pipelines;

(1.4) Memory computing: vectorized processing, dynamic compilation based on the underlying virtual machine (LLVM);

(1.5) OLTP/OLAP dual analysis engine: Compatible with OLTP and OLAP data analysis and processing requirements;

(2) Data interface system and man-machine interface system, including:

(2.1) Machine-type data interface and collection and processing system, including:

(2.1.1) Infrastructure operating environment data engine processing subsystem, capable of adapting and configuring MIB information, etc.;

(2.1.2) Data engine processing subsystems such as SYSLOG;

(2.1.3) Big data industry and related enterprise information collection and processing subsystems;

(2.2) Business data interface system, business comprehensive data interface should support multiple interface formats and human-computer interaction mixed mode, to adapt to multi-source cold and hot data access processing, etc., business data interface format types expansion support and update management In response to big data related business data from different sources, it can support interfaces in various formats such as RestAPI/SNMP/SYSLOG/files, support real-time, timing, condition triggering and other update frequency modes, and can be configured, and relevant interface protocol specifications and standard customization Listed below:

(2.2.1) SYSLOG interface, Internet of Things, machine equipment data standard industrial interface specifications, including system logs of server hosts, general network equipment, and special security equipment should be sent to the security management and control system through the SYSLOG protocol, and the security management and control system passes the agreed SYSLOG Message format, parsing log information, and standardizing, analyzing, and correlating log information;

(2.2.2) SNMP interface, the system logs of IoT devices, server hosts, general network devices, and special security devices should implement the basic SNMP protocol MIB library, and the security management and control system collects the basic information of the devices and the operating performance of the devices through the SNMP protocol. Protocol and receiving device SNMP Trap event notification information;

(2.2.3) REST interface, the REST interface protocol is an implementation of WebService (Web service), which is mainly used to realize the interface between systems. In the mobile police platform, in addition to using general SNMP and SYSLOG on the equipment management level In addition to protocols, scenarios such as inter-software systems and policy configuration delivery should be implemented using the REST interface protocol. In general, the request-response mode should be used to implement monitoring information reporting, policy and instruction delivery, etc. The workflow of the request-response mode is shown in the figure As shown in 2, the request and response communication protocol should use HTTP1.1 over SSL/TLS, the request response mode should use HTTP as the interface implementation protocol, the interface parameters and return results should use JSON objects, and the data packet size should not exceed 5MB. In the case of unreachable, the message push mode can be used to implement the delivery of policy instructions. The specific process is shown in Figure 3. The communication protocol should use HTTP2.0 WEBSOCKET over SSL/TLS, and HTTP should be used as the interface to implement the protocol. Interface parameters , The returned results all use JSON objects, and the data packet size does not exceed 5MB;

(2.2.4) Application system log interface. The application system log interface uses a standard SDK interface to send terminal application/business domain log data. In order to ensure collection performance, the application interface supports cluster deployment. The amount and the size of the concurrent number, select the number of nodes in the cluster;

(2.3) Human-machine interface system services, including:

(2.3.1) The front-end human-machine interface system establishes an active service interactive data collection human-machine interface for tens of thousands of enterprises in the industry;

(2.3.2) Security platform data exchange pre-service platform, establish a security data communication platform for the industry analysis platform, etc.;

(2.3.3) Enterprise-related data inventory and combing services, interface negotiation and customization services.

The data analysis model module is used to store data analysis models; the data analysis models include industry classification models, industry chain models and enterprise performance evaluation models;

The industry classification model is used to determine the industry classification of the enterprise according to the business scope of the enterprise to be classified; the structure of the industry classification model is relatively simple, and the decision tree algorithm is used. The decision tree classification is in line with the thinking of human classification. For different problems, judge a certain feature of the sample, and then combine all the judgment results to give the category of the sample. The structure of the industry classification model is shown in Figure 4. The establishment process of the industry classification model (see Figure 5) is as follows:

(1) start;

(2) Create a tree with the enterprise data set (complete set of samples) as the root node;

(3) Create a node;

(4) Determine whether the enterprise data set (complete set of samples) is empty, if yes, return to the previous node, and end; otherwise, enter the next step;

(5) Determine whether the current node data set (sample subset) belongs to the same attribute, if so, record it as a leaf node and mark it as class C, and then end; otherwise, enter the next step;

(6) Determine whether the candidate attribute set is empty, if yes, record it as the class C with the largest number of samples in S, and end; otherwise, enter the next step;

(7) Calculate the information gain rate of each enterprise condition attribute in the set;

(8) Select the largest information gain in the candidate set as the segmentation attribute of the current node;

(9) Determine the enterprise data set (sample subset) according to the value of the segmentation attribute, and establish corresponding branches;

(10) continuously recursively run the function to the enterprise data set (sample subset), and return to step (2);

The workflow of the industry classification model is as follows:

(1) start;

(2) Input the business scope data of the enterprise to be classified;

(3) Through the decision tree algorithm, the industry classification of enterprises;

(4) end;

The industrial chain model is used to determine the position of the enterprise in the industrial chain through the weighted scoring method, according to the data of the enterprises to be classified, and the scores in the four categories of "resources", "technology", "applications" and "industrial support". That is, the categories in the industrial chain; the division standard of the industrial chain of enterprises is relatively vague, and an enterprise may exist in multiple positions in the industrial chain at the same time. Therefore, a single industrial chain division method cannot meet the actual needs. To divide the position of the industrial chain, the present invention adopts a weighted scoring method, wherein the classification algorithm adopts the random forest model, and the scoring method adopts the exponential weighted average method. The data sets used in the big data industrial chain model research have more dimensions and a larger amount of data. The random forest algorithm can effectively solve these problems, and its advantages are mainly reflected in:

(1) On many current data sets, it has great advantages over other algorithms and performs well;

(2) It can handle very high-dimensional data and does not require feature selection;

(3) The training speed is fast, and it is easy to make a parallel method (the trees are independent of each other during training);

(4) During the training process, the interaction between data of different dimensions can be detected;

(5) When dealing with unbalanced data sets, errors can be balanced;

(6) If a large part of the features are missing, the accuracy can still be maintained;

The workflow of the industrial chain model is as follows:

(1) start;

(2) Data set acquisition: obtain the business information, intellectual property rights, and historical data of enterprise operations as the original data set, and construct the classification indicators of the big data industry chain, which are: "resources", "technology", "application" and " industry support;

(3) Data labeling: label the original data set according to the classification index of the big data industry chain to mark the category of the enterprise in the industry chain;

(4) Data preprocessing: perform data matching and outlier removal operations on the data in the original data set;

(5) Data set division: divide the data in the original data set into a training set and a test set according to a ratio of 3:1;

(6) Build a random forest: apply the traditional random forest algorithm on the training set to build a random forest for predicting the position of the enterprise in the industrial chain;

(7) Random forest model training: use the data in the training set to train the random forest model of N decision trees, N is an integer greater than 1, and each decision tree randomly extracts enterprise data from the training set for training, using gain entropy To select the appropriate attribute nodes, each tree randomly selects samples and attribute features from the training set to generate their own nodes until all decision trees classify the samples they have sampled;

(8) Model evaluation and correction: Input the test set into the trained random forest model for classification, count the classification results and actual results, and calculate the prediction accuracy rate. Both the classification results and the actual results are the categories of enterprises in the industrial chain , when the prediction accuracy rate is less than the set value (the set value can be set according to actual needs), calculate the classification result obtained by each decision tree and calculate its AUC value, based on the AUC value extracted from the current random forest model A set of relatively high-precision decision trees is clustered according to similarity and divided into different clusters. Finally, a set of high-precision decision trees is selected from different clusters to iteratively update the existing random forest model; when the prediction is accurate If the rate is greater than or equal to the set value, the random forest model will not be updated;

(9) Obtain the data of the enterprises to be classified;

(10) Perform data matching and outlier removal operations on the data;

(11) Input the data into the trained random forest model, and output the classification result;

(12) end;

Steps (1)-(8) correspond to Figure 6, and steps (9)-(12) correspond to Figure 7;

The enterprise performance evaluation model is used to comprehensively evaluate the enterprise based on the data related to the enterprise's comprehensive ability, research and development ability, industry influence, continuous operation ability and data application ability (that is, the enterprise's registered capital, listing status, business income, research and development investment, etc.) After scoring capabilities, research and development capabilities, industry influence, sustainable operation capabilities, and data application capabilities, the weighted total score is calculated to obtain enterprise performance; the structure of the enterprise performance evaluation model is shown in Figure 8, and the process of the enterprise performance evaluation model is as follows:

(1) start;

(2) Score the enterprise's comprehensive ability, research and development ability, industry influence, sustainable operation ability and data application ability;

Comprehensive ability score a=a1+a2+a3, when the registered capital>50000, a1=100; when the registered capital<100, a1=45; when 100≤registered capital≤50000, a1=registered capital ^1/8 × 25; when the company is a listed company, a2 is randomly selected in the interval [90,100]; when the enterprise is an unlisted company, a2 is randomly selected in the interval [50,60]; when the main business income>50000, a3=100; when the main business income <100, a3=45; when 100≤main business income≤50000, a3=main business income1 ^/8 ×25;

R&D capability score b=b1+b2+b3+b4, when R&D funds>10000, b1=100; when R&D funds<10, b1=45; when 10≤R&D funds≤10000, b1=R&D funds1 ^{/ 8} × 31; when the number of R&D personnel > 1000, b2 = 100; when the number of R&D personnel < 5, b2 = 50; when 5 ≤ the number of R&D personnel ≤ 1000, b2 = ^1/8 × 42 of the number of R&D personnel; when the number of patents > 50 , b3=100; when the number of patents<2, b3=45; when 2≤the number of patents≤50, b3= ^1/4 ×38 of the number of patents; when the number of soft deals ≥5, b4=100; When the number of moves = 4, b4 = 90; when the number of soft moves = 3, b4 = 80; when the number of soft moves = 2, b4 = 70; when the number of soft moves = 1, b4 = 60; =0, b4=50;

Industry influence score c = c1 + c2, when the company is a listed company, c1 = 60 + main business income ^1/8 × 10; when the company is a non-listed company, c1 = 40 + main business income ^1/8 ×10; when big data business revenue >20,000, c2=100; when big data business revenue <50, c2=45; when 50≤big data business revenue≤20,000, c2=big data business revenue ^1/8 × 29;

Sustainability score d=d1+d2+d3, d1=market share×0.5+economic benefits×0.25+main business income×0.25; when the sum of venture capital and foreign capital utilization>10000, d2=100; when risk When the sum of investment and utilization of foreign capital <1, d2=30; when 1≤the sum of venture capital and utilization of foreign capital≤10,000, d2=(venture investment + utilization of foreign capital) ^1/8 ×32; when operating profit>10,000, d3=100; when operating profit<1, d3=30; when 1≤operating profit≤10000, d3=operating profit ^1/8 ×32;

Data application ability score e=e1+e2+e3, when data soft assets>2000, e1=100; when data soft assets<10, e1=45; when 10≤data soft assets≤2000, e1=data soft assets Assets ^1/8 × 38; when data hard assets > 2000, e2 = 100; when data hard assets < 10, e2 = 45; when 10≤data hard assets≤2000, e2=data hard assets ^1/8 × 38; when the turnover of data products > 10000, e3 = 100; when the turnover of data products < 10, e3 = 45; when 10≤ turnover of data products ≤ 10000, e3 = turnover of data products ^1/8 × 32 ;

Among them, the unit of registered capital, main business income, research and development funds, big data business income, economic benefits, venture capital, utilization of foreign capital, operating profit, data soft assets, data hard assets and data product turnover is RMB 10,000; The unit of the number of people is person, the unit of the number of patents and the number of soft works is unit, and the unit of market share is % (calculated based on the sum of similar enterprises in the current data set);

(3) Calculate the weighted total score g to obtain enterprise performance, g=w1×a+w2×b+w3×c+w4×d+w5×e, w1=0.3, w2=0.25, w3=0.1, w4=0.15, w5=0.2;

(4) END.

The data base platform module is connected with the industrial development-related database module and the data analysis model module at the same time, and is used to retrieve relevant data from the industrial development-related database module according to the analysis target (matching the analysis target with the relevant data to achieve the smooth retrieval is done For example, if the analysis target is to analyze the enterprises established in the past five years, then the data of enterprises whose establishment time is greater than 2015 will be called from the database; "Enterprise data of the "core enterprise" label), retrieve relevant models from the data analysis model module (it is a known technology to match the analysis target with the relevant model to achieve smooth retrieval, for example, the analysis target is "classification according to the industrial chain", Then in the analysis module, according to the model name, manually select the "industrial chain model" to call), and input the relevant data into the relevant model; the data basic platform module provides a strong platform support for the system construction, shortens the development cycle, and reduces the risk of system construction , to improve performance and stability; data maintenance, including data completion and update; data query, wizard search and custom reports are available, and query results can be filtered, sorted, summarized upward, drilled down, Simple calculations, conditional formatting, etc., support the export and printing of results; for visual display, a variety of intelligent visualization algorithms can be built in to achieve multi-dimensional and multi-terminal three-dimensional presentation of massive data;

The basic data platform module provides big data storage, calculation, analysis, visualization, and daily operation and maintenance service support. The basic data platform includes data exchange platform, big data platform, information release platform, GIS geographic information platform, basic component platform, report analysis platform and System management platform;

The technical implementation of the data base platform module relies on a visual big data management, analysis and display platform, the functions are as follows:

(a) Metadata-based data management: build a set of standardized, unified and common big data resources for users;

(b) Conventional multidimensional analysis + open modeling: provide users with multidimensional analysis and modeling analysis based on R language;

(c) Compatible with traditional and big data storage: Compatible with relational DB, Hadoop, NoSQL and other storage types;

(d) Visual dynamic chart analysis: provide users with a drag-and-drop chart display method that conforms to the operating habits of Chinese people.

The data analysis model module is also used to output analysis results after the relevant data is input into the relevant model.

The user-end interface module is connected with the data analysis model module to display the analysis results; the user-end module includes key applications such as industrial economic operation, industrial management, and productivity layout. Each business application includes three parts, namely status monitoring and special analysis and trend analysis;

The technical realization of the user interface module mainly relies on the following technologies:

1)HTML

HTML is the core technology required to build a web interface; it is a tag-based language used to describe the structure of documents displayed by browsers;

2) CSS (Cascading Style Sheets)

In web applications, it is used to specify how HTML content is rendered on the screen;

3) JavaScript

Object-oriented JavaScript, supporting object-oriented, imperative and declarative at the same time;

4) hyperlink

Hyperlinks in web applications usually contain pre-set request parameters. These data items do not need to be input by the user, but are inserted by the server into the target URL of the hyperlink clicked by the user. Big Data Industry Analysis System The project reserve library, friendship links, etc. are all developed in this way;

5) Form

receive user input;

6) Document Object Model DOM

7) Ajax (Asynchronous JavaScript and XML, asynchronous JavaScript and XML)

AJAX is the art of exchanging data with a server and updating parts of a web page without reloading the entire page;

Big data industrial systems use Ajax, some user actions will be handled by client-side script code, and the whole page does not need to be reloaded; instead, the script will execute the request "in the background" and usually receive a smaller response for dynamic Update part of the user interface;

The core technology XML Http Request used by Ajax, after a certain degree of standard integration, this technology has now been transformed into a local JavaScript object, through which client scripts can make "background" requests without window-level navigation events;

8) JSON (JavaScript Object Notation, JavaScript Object Notation)

Big data industry systems use JSON to store and exchange text information, which is a lightweight text data exchange format;

Ajax applications often use JSON to replace the XML format originally used for data transmission. JSON is smaller, faster, and easier to parse than XML;

Usually, if the user performs an operation, the client-side JavaScript will use XML Http Request to transmit the operation to the server, and the server will return a lightweight response containing data in JSON format, and then the client-side script will process the data and Update the user interface accordingly;

9) Same Origin Policy

The core of the big data industry map system: prevent content from different sources from interfering with each other, and only allow content from the same source to interact;

Content received from one site can read and modify other content received from that site, but not access content received from other sites, and if the Same Origin Policy is not used, then when an unsuspecting user browses When a malicious website is accessed, script code running on that website will be able to access the data and functionality of any other website that the user is visiting at the same time;

The user terminal interface module provides the user result viewing function, and the user can view the industrial economic operation, industrial management, productivity layout, etc. to provide status monitoring, thematic analysis and trend research and judgment data analysis results.

Claims (3)

1. An industrial analysis system based on big data, characterized in that it includes an industrial development-related database module, a data analysis model module, a data foundation platform module and a user interface module; The industry development related database module is used to store data resources related to an industry development; The data analysis model module is used to store the data analysis model; the data analysis model includes the industry classification model, the industry chain model and the enterprise performance evaluation model; the industry classification model is used to determine the industry classification of the enterprise according to the business scope of the enterprise to be classified; the industry chain The model is used to determine the position of the enterprise in the industrial chain through the weighted scoring method, according to the data of the enterprise to be classified, in the four categories of "resources", "technology", "application" and "industrial support". The category in the industrial chain; the enterprise performance evaluation model is used to evaluate the enterprise's comprehensive ability, research and development ability, industry influence, and sustainable operation ability according to the data related to the enterprise's comprehensive ability, research and development ability, industry influence, continuous operation ability and data application ability. After scoring with the data application ability, calculate the weighted total score to get the enterprise performance; The establishment process of the industry classification model is as follows: (1) start; (2) Create a tree with the enterprise dataset as the root node; (3) Create a node; (4) Determine whether the enterprise data set is empty, if yes, return to the previous node, and end; otherwise, enter the next step; (5) Determine whether the current node data set belongs to the same attribute, if so, record it as a leaf node and mark it as class C, and end; otherwise, go to the next step; (6) Determine whether the candidate attribute set is empty, if it is, record it as the class C with the largest number of samples in S, and end; otherwise, go to the next step; (7) Calculate the information gain rate of each enterprise condition attribute in the set; (8) Select the largest information gain in the candidate set as the segmentation attribute of the current node; (9) Determine the enterprise data set according to the value of the segmentation attribute, and establish the corresponding branch; (10) Continuously recursively run the function on the enterprise data set, and return to step (2); The workflow of the industry classification model is as follows: (1) start; (2) Input the business scope data of the enterprise to be classified; (3) Industry classification of enterprises through decision tree algorithm; (4) end; The workflow of the industrial chain model is as follows: (1) start; (2) Acquisition of data sets: Obtain the business information, intellectual property rights, and historical data of enterprise operations as the original data sets, and construct the classification indicators of the big data industry chain, which are: "resources", "technology", "applications" and " industry support; (3) Data labeling: mark the original data set according to the classification index of the big data industry chain to mark the category of the enterprise in the industry chain; (4) Data preprocessing: perform data matching and outlier removal operations on the data in the original data set; (5) Data set division: divide the data in the original data set into training set and test set according to the ratio of 3:1; (6) Build a random forest: apply the traditional random forest algorithm on the training set to build a random forest for predicting the position of the enterprise in the industrial chain; (7) Random forest model training: use the data in the training set to train the random forest model of N decision trees, N is an integer greater than 1, and each decision tree randomly extracts enterprise data from the training set for training, using gain entropy To select the appropriate attribute nodes, each tree randomly selects samples and attribute features from the training set to generate their own nodes until all decision trees classify the samples they have sampled; (8) Model evaluation and correction: Input the test set into the trained random forest model for classification, count the classification results and actual results, and calculate the prediction accuracy rate. Both the classification results and the actual results are the categories of enterprises in the industrial chain , when the prediction accuracy rate is less than the set value, calculate the classification result obtained by each decision tree and calculate its AUC value, and extract a relatively high-precision decision tree set from the current random forest model based on the AUC value, and then based on the similarity Cluster it, divide it into different clusters, and finally select a high-precision decision tree set from different clusters to iteratively update the existing random forest model; when the prediction accuracy is greater than or equal to the set value, the random forest model will not be renew; (9) Obtain the data of the enterprises to be classified; (10) Perform data matching and outlier removal operations on the data; (11) Input the data into the trained random forest model, and output the classification result; (12) end; The process of enterprise performance evaluation model is as follows: (1) start; (2) Score the enterprise's comprehensive ability, research and development ability, industry influence, sustainable operation ability and data application ability; Comprehensive ability score a=a1+a2+a3, when the registered capital>50000, a1=100; when the registered capital<100, a1=45; when 100≤registered capital≤50000, a1=registered capital ^1/8 × 25; when the company is a listed company, a2 is randomly selected in the interval [90,100]; when the enterprise is an unlisted company, a2 is randomly selected in the interval [50, 60]; when the main business income>50000, a3=100; when the main business income<100, a3=45; when 100≤main business income≤50000, a3=main business income1 ^/8 ×25; R&D capability score b=b1+b2+b3+b4, when R&D funds>10000, b1=100; when R&D funds<10, b1=45; when 10≤R&D funds≤10000, b1=R&D funds1 ^{/ 8} × 31; when the number of R&D personnel > 1000, b2 = 100; when the number of R&D personnel < 5, b2 = 50; when 5 ≤ the number of R&D personnel ≤ 1000, b2 = the number of R&D personnel ^1/8 × 42; when the number of patents > 50 , b3=100; when the number of patents<2, b3=45; when 2≤the number of patents≤50, b3= ^1/4 ×38 of the number of patents; when the number of soft deals ≥5, b4=100; When the number of soft moves=4, b4=90; when the number of soft moves=3, b4=80; when the number of soft moves=2, b4=70; when the number of soft moves=1, b4=60; when the number of soft moves=1, b4=60; =0, b4=50; Industry influence score c=c1+c2, when the company is a listed company, c1=60+main business income ^1/8 ×10; when the company is a non-listed company, c1=40+main business income ^1/8 ×10; when big data business revenue >20,000, c2=100; when big data business revenue <50, c2=45; when 50≤big data business revenue≤20,000, c2=big data business revenue ^1/8 × 29; Continuous operation ability score d=d1+d2+d3, d1=market share×0.5+economic benefits×0.25+main business income×0.25; when the sum of venture capital and foreign capital utilization>10000, d2=100; when the risk When the sum of investment and utilization of foreign capital <1, d2=30; when 1≤the sum of venture capital and utilization of foreign capital≤10,000, d2=(venture investment + utilization of foreign capital) ^1/8 ×32; when operating profit>10,000, d3=100; when operating profit<1, d3=30; when 1≤operating profit≤10000, d3=operating profit ^1/8 ×32; Data application ability score e=e1+e2+e3, when data soft assets>2000, e1=100; when data soft assets<10, e1=45; when 10≤data soft assets≤2000, e1=data soft assets Assets ^1/8 × 38; when data hard assets > 2000, e2 = 100; when data hard assets < 10, e2 = 45; when 10 ≤ data hard assets ≤ 2000, e2 = data hard assets ^1/8 × 38; when the turnover of data products is >10,000, e3=100; when the turnover of data products is <10, e3=45; when 10≤the turnover of data products≤10,000, e3= ^1/8 of the turnover of data products ×32 ; Among them, the unit of registered capital, main business income, research and development funds, big data business income, economic benefits, venture capital, utilization of foreign capital, operating profit, data soft assets, data hard assets and data product turnover is RMB 10,000; The unit of the number of people is person, the unit of the number of patents and the number of soft works is unit, and the unit of market share is %; (3) Calculate the weighted total score g to get the enterprise performance, g=w1×a+w2×b+w3×c+w4×d+w5×e, w1=0.3, w2=0.25, w3=0.1, w4=0.15, w5=0.2; (4) end; The data base platform module is connected with the industrial development-related database module and the data analysis model module at the same time, and is used to retrieve relevant data from the industrial development-related database module according to the analysis objectives, and then transfer the relevant models from the data analysis model module, and transfer the relevant data input into the relevant model; The data analysis model module is also used to output the analysis results after the relevant data is input into the relevant model; The user terminal interface module is connected with the data analysis model module for displaying analysis results. 2. An industry analysis system based on big data according to claim 1, wherein the data resources related to the development of a certain industry include industry data, enterprise data, regional data and technical data related to the development of a certain industry. 3. A kind of industrial analysis system based on big data according to claim 1, characterized in that, the basic data platform includes a data exchange platform, a big data platform, an information release platform, a GIS geographic information platform, a basic component platform, and a report analysis platform platform and system management platform.

Images