CN110909017A

CN110909017A - Data analysis method and system

Info

Publication number: CN110909017A
Application number: CN201911095118.6A
Authority: CN
Inventors: 李张继
Original assignee: Suning Financial Technology Nanjing Co Ltd
Current assignee: Suning Financial Technology Nanjing Co Ltd
Priority date: 2019-11-11
Filing date: 2019-11-11
Publication date: 2020-03-24
Anticipated expiration: 2039-11-11
Also published as: CN110909017B

Abstract

The application relates to a data analysis method and system, wherein the method comprises the following steps: acquiring an analysis chart, and analyzing the analysis chart; splicing the analyzed information into an execution script, and sending the execution script to a computing engine so that the computing engine performs data analysis according to the execution script; and receiving a calculation result returned by the calculation engine, and performing visualization processing on the calculation result according to the analysis chart. The scheme of the application can be realized on the basis of the browser without using a professional tool, so that a user does not need to master professional technology; the required data analysis content can be freely constructed according to the requirement, the data analysis result can be obtained in real time, and the requirements of convenient and fast analysis are met.

Description

Data analysis method and system

Technical Field

The application relates to the technical field of business intelligence, in particular to a data analysis method and device.

Background

In the field of traditional BI (Business Intelligence), technicians generally use professional BI software to construct relevant reports, the construction period is long, the overall consumption of construction and adjustment is high, the feedback period of the construction result is long, and the first-line demand cannot be quickly responded.

With the development of services, the breadth and depth of the services are increased; based on the corresponding service scenario, the content of the demand for data analysis is also increasing; meanwhile, more and more adjustment appeal is provided for the established data analysis content. The data analysis is more and more demanding and flexible, the requirement for the construction efficiency is more and more high, how to quickly and effectively meet the analysis demand of the mass data of the business side becomes a difficult point of enterprises.

In the related art, the efficiency of building a data analysis report through the traditional BI technology is low, the requirement on professional technology is high, and the business requirement cannot be quickly responded. However, data analysis products based on Web browsers generally have weak support for data volume, simple functions, and even impossible to obtain what you see is what you get for analysis of large data volume.

Disclosure of Invention

To overcome, at least to some extent, the problems in the related art, the present application provides a data analysis method and system.

According to a first aspect of embodiments of the present application, there is provided a data analysis method, including:

acquiring an analysis chart, and analyzing the analysis chart;

splicing the analyzed information into an execution script, and sending the execution script to a computing engine so that the computing engine performs data analysis according to the execution script;

and receiving a calculation result returned by the calculation engine, and performing visualization processing on the calculation result according to the analysis chart.

Further, the analyzing the analysis chart includes:

and analyzing field information, dimension information and index information of data analysis to be performed from the analysis chart.

Further, the field information includes: field name, field length, field type;

the dimension information is the dimension which needs to be analyzed and associated in the data analysis process;

the index information is a calculation mode for calculating and processing data lines in the data analysis process.

Further, the assembling the parsed information into an execution script includes:

performing semantic analysis and splitting on the calculation result;

assembling basic field information according to the field information;

assembling any multi-level aggregation according to the dimension information;

and assembling various operation contents based on any multi-level aggregation according to the index information.

Further, the visualizing the calculation result according to the analysis chart includes:

analyzing the calculation result into field information, dimension information and calculation result information;

matching the analyzed field information with the field information in the analysis chart;

integrating the analyzed calculation result information into the analysis chart according to the matching result;

and outputting the integrated analysis chart.

According to a second aspect of embodiments of the present application, there is provided a data analysis system, including:

the chart analysis module is used for acquiring an analysis chart and analyzing the analysis chart;

the script assembly module is used for assembling the analyzed information into an execution script and sending the execution script to the computing engine;

the computing engine is used for carrying out data analysis according to the execution script;

and the result analysis module is used for receiving the calculation result returned by the calculation engine and carrying out visualization processing on the calculation result according to the analysis chart.

Further, the analyzing the analysis chart by the chart analyzing module specifically includes:

Further, the field information includes: field name, field length, field type;

Further, the script assembling module assembles the analyzed information into an execution script, and specifically includes:

performing semantic analysis and splitting on the calculation result;

assembling basic field information according to the field information;

assembling any multi-level aggregation according to the dimension information;

Further, the result analyzing module performs visualization processing on the calculation result according to the analysis chart, and specifically includes:

and outputting the integrated analysis chart.

The technical scheme provided by the embodiment of the application has the following beneficial effects:

the scheme of the application can be realized on the basis of the browser without using a professional tool, so that a user does not need to master professional technology; the required data analysis content can be freely constructed according to the requirement, the data analysis result can be obtained in real time, and the requirements of convenient and fast analysis are met.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

FIG. 1 is a flow chart illustrating a method of data analysis in accordance with an exemplary embodiment.

FIG. 2 is a schematic diagram illustrating an analysis chart according to an exemplary embodiment.

FIG. 3 is a flow diagram illustrating assembly of a script in accordance with an illustrative embodiment.

FIG. 4 is a flow diagram illustrating a result parsing in accordance with an example embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of methods and systems consistent with certain aspects of the present application, as detailed in the appended claims.

FIG. 1 is a flow chart illustrating a method of data analysis in accordance with an exemplary embodiment. The method comprises the following steps:

step S1: acquiring an analysis chart, and analyzing the analysis chart;

step S2: splicing the analyzed information into an execution script, and sending the execution script to a computing engine so that the computing engine performs data analysis according to the execution script;

step S3: and receiving a calculation result returned by the calculation engine, and performing visualization processing on the calculation result according to the analysis chart.

The scheme of the application is based on a common Web browser, and provides a data analysis function for performing what you see is what you get on mass data, namely, an initially designed chart is the same as a chart of a final analysis result. The system mainly comprises a chart design module, a chart analysis module, a script assembly module, a calculation engine and a result analysis module.

A user can directly access a page of the visual chart design module through a common Web browser, data analysis content required by the page design of the chart design module is designed, then the chart analysis module translates the data analysis content designed by the user, and further the assembly of corresponding query scripts is carried out, then a calculation result is queried through a calculation engine, and is analyzed through the result analysis module and finally presented in the browser at the user side in a visual mode.

The following describes the scheme of the present application in an expanded manner with reference to a specific application scenario.

Some preparatory work is required before the embodiments of the present application can be implemented. The method is characterized in that model information needing data analysis is prepared in advance, and the prepared content mainly comprises all field information of the model, including several core contents of field names, field types and field lengths. After the preparation is completed, the subsequent implementation steps can be performed.

1. And (4) visual chart design. And (4) adopting a chart design module to carry out man-machine interaction and receiving chart contents designed by a user. And providing user operation by using the visual interface obtained in the visible mode to design a chart. After the design is completed, the system stores the design content for use in the next stage.

According to the scheme, the self-service visual design page is adopted, the corresponding data analysis requirements of the user can be completed in a mouse dragging mode, and professional technical knowledge is not needed. It should be noted that the operation mode of mouse dragging is an existing human-computer interaction technology, and there are existing open source solutions, such as Vue front end frames, which can be directly used.

Referring to fig. 2, when a user designs a chart using a chart design module, the field information of the model in the preparation work is used, and therefore, the field information used in the chart is known. Through a chart design module, one or more fields can be directly designated to be used as dimensions or indexes, and if the fields are used as the indexes, corresponding calculation modes including maximum values, minimum values, summations, counts, average values and the like can be set. For any field, a specific conditional constraint may be set, for example, equal to/containing/greater than a certain value, and the constraint is determined by the type information of the field. For various charts, setting auxiliary information such as result sorting and the like is also included. The final content of the design forms a data structure body which is transmitted to the chart analysis module.

When the scheme of the application is implemented, fields such as gender, user type, account name and age are provided on a page of the browser for the user to operate and select. In the program, all metadata information is recorded in these fields.

For example, the user needs to design a form with a header of "gender, user type, account number, age". The user only needs to select the fields of 'gender, user type, account name and age', and the account is set to be the number to be counted, and the age needs to find out the maximum value and the minimum value respectively.

2. And the chart analysis module analyzes the user designed chart content stored in the last stage.

In some embodiments, the parsing the analysis graph includes:

The chart analysis module decomposes the data structure formed by the chart design module, and analyzes the content by combining with the pre-stored data model to obtain the content to be analyzed, and the method comprises the following steps: field information, condition information, dimension information, index information, and the like.

The field information is field information to be analyzed, and includes key information such as a field name, a field length, and a field type.

The condition information, i.e., the constraint condition set in the previous stage, is used to filter the field information.

Dimension information, which is the dimension that needs to be analyzed in the data analysis process, provides aggregation mainly based on the dimension.

The index information is a calculation mode for calculating and processing data lines in the data analysis process. Namely, according to a certain calculation rule, the calculation modes of calculation processing are carried out on the corresponding fields, such as summation, counting, maximum value, minimum value and the like.

The scheme supports data analysis freely in a multi-dimensional and multi-index calculation mode, and results can be directly presented in a form and a graph mode.

Referring to the embodiment as described above, it is analyzed that 5 fields are used, one for each of gender, user type, and account name, and two for age. The index needs to be processed for 1 account name and 2 ages set by the user, and the rest fields are automatically used as dimensions. The condition information is also independently set in the first step, and the step can be analyzed after the setting.

3. And the script assembling module performs script assembling by using the information obtained by the parser.

In some embodiments, assembling the parsed information into an execution script includes:

performing semantic analysis and splitting on the calculation result;

assembling basic field information according to the field information;

assembling any multi-level aggregation according to the dimension information;

The script assembly module can further process the content analyzed by the chart analysis module and assemble the analyzed content into an execution script required by the next calculation engine. The ElasticSearch cluster can be used as a computing engine, and the operable script of the ElasticSearch can be freely assembled and formed as required through a script assembly module. The method mainly comprises the assembly of basic field information, the assembly of any multi-level aggregation and the assembly of various operation contents based on any multi-level aggregation.

Referring to fig. 3, a process flow of the script assembly module is shown, which mainly includes:

receiving the content which is analyzed;

the content is assembled into a script that can run in an ElasticSearch.

More specifically, the script assembling module performs script assembling and splicing according to the set dimension information, index information (including calculation mode information) and condition information and in combination with the field information of the model. Taking the assembly standard SQL as an example, dimension information is assembled into content in group by, index information is assembled into content in select, and condition information is assembled into content in where. Taking the elastic search as an example, the dimension information is assembled into contents in aggregations (aggregation in the elastic search is hierarchically nested, so the multi-dimension aggregation information described above is also multi-level aggregation information here), the index information is mainly assembled into contents of child nodes in aggregations (such as max, count, etc.), and the condition information is assembled into contents in query.

According to the embodiment, the adaptive computing engine generates a script according to the parsed content, for example, Json splicing is performed by using an elastic search, and the final result set uses a unique identification code to identify the field, which can be understood as an alias of the field. For example, two ages in the example are identified as age _1 and age _ 2.

4. And sending the assembled script to a calculation engine for calculation.

The computing engine can use the ElasticSearch cluster and use the execution script generated by the script assembly module to directly obtain the needed computing result. The scheme of the application guarantees the response speed of the Web application through the real-time computing capacity of the cluster.

The scheme can realize rapid calculation support of mass data, under the condition of hundred million-level data volume, multidimensional combination analysis (within 10 dimensionalities), response time is controlled at the second level, and the performance is not weaker than that of traditional BI software.

5. And the result analysis module analyzes the calculation result information.

In some embodiments, the visualizing the calculation result according to the analysis chart includes:

and outputting the integrated analysis chart.

And the result analysis module intelligently analyzes the complex JSON of the ElasticSearch cluster calculation result, and respectively analyzes the field information, the dimension information, the aggregation information, the calculation result information and the like of the response according to the content designed by the user. Finally, the related information is presented in a user interface, and the user use is provided in a chart visualization mode.

Referring to fig. 4, a processing flow of the result parsing module is shown, which mainly includes:

receiving JSON (Java service object) completed by the ElasticSearch calculation;

the JSON content is dynamically analyzed, and all common field information, dimension information, aggregation information and calculation result information can be analyzed;

and displaying the result to the user in a visualized content.

More specifically, the result parsing module needs to perform two aspects of parsing: one aspect is the parsing of the result data; another aspect is the correspondence resolution of data to content in the earliest diagram design module.

The parsing of the resulting data includes several main parts: basic field information, dimension information and index information are matched through self-defining setting (such as field alias) of fields in the previous steps and the matching of the stages, and result data can be correlated, namely analysis basic field information, multi-level aggregation information and operation result information are obtained by taking elastic search as an example. It should be noted that, in the scheme, a field unique identifier in a result set is specified through a determined encoding rule, and association matching can be performed through the identifier; the result set has the unique identification of the field for correlation matching, and the analyzed 'field information, dimension information and index information' are the same as the corresponding information in the previous step.

And analyzing the corresponding relation of the content of the chart design module, namely, after the relevant field information in the result information is in one-to-one correspondence or mapping with the field information used in the chart analysis module and the script assembly module, so that the relevant field information in the calculation result can be translated and obtained and finally corresponds to the content initially set in the chart design module by the user. And finally, feeding back the data information to the chart design module and presenting the data information to the user.

The data information mainly refers to result information of query and calculation, for example, if a field is summed, the data information in the result is a value obtained by accumulating and summing the field.

The data information feedback is carried out through interaction of the front-end interface and the back-end interface. For example, the interface may be a common http request interface, and the parsing may be performed according to a well-agreed message format. Taking a table as an example, specifically: the returned message mainly consists of two parts: one part is metadata information, namely, a table header of the table is described, and the fields are also distinguished and identified by the unique identification codes described above; the other part is specific table data. After receiving the message content, the chart design module analyzes the two parts of content respectively and associates the corresponding content, so that the final content can be displayed.

Referring to the foregoing embodiments, according to the field unique identifier (field alias), the result data corresponding to the corresponding field can be obtained; with the data, the form can be filled with the data and finally presented to the user with the desired content. For example, the original Chinese name of the field is "name", the English name is "name", and the display result also has the field. If the field name customized by the user is the 'user name', the program generates a unique identification code according to a certain rule (for example, the currently used rule is to count the same used fields, the field name + the number is assembled), for example, a new field identification 'name _ 123' is generated, a result set is specified to give corresponding result data according to the 'name _ 123', and thus, the data in the result, the field metadata information, the field name set by the user and other contents can be uniformly associated, mapped and matched.

By adopting the scheme, special client software does not need to be installed, and the operation can be performed by directly using a common Web browser, which is more convenient and faster than the traditional BI software; the business personnel can freely and quickly build the data analysis content required by the business personnel according to the requirements of the business personnel without professional technology by a graphical interface; and the data analysis result can be obtained in real time, and the requirements of convenient and fast analysis are met.

The present application further provides the following embodiments:

a data analysis system, comprising:

In some embodiments, the analyzing the analysis chart by the chart analyzing module specifically includes:

In some embodiments, the field information comprises: field name, field length, field type;

In some embodiments, the assembling the parsed information into the execution script by the script assembling module specifically includes:

performing semantic analysis and splitting on the calculation result;

assembling basic field information according to the field information;

assembling any multi-level aggregation according to the dimension information;

In some embodiments, the visualizing the calculation result by the result analysis module according to the analysis chart specifically includes:

and outputting the integrated analysis chart.

With regard to the system in the above embodiment, the specific steps in which the respective modules perform operations have been described in detail in the embodiment related to the method, and are not described in detail herein.

It is understood that the same or similar parts in the above embodiments may be mutually referred to, and the same or similar parts in other embodiments may be referred to for the content which is not described in detail in some embodiments.

It should be noted that, in the description of the present application, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Further, in the description of the present application, the meaning of "a plurality" means at least two unless otherwise specified.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.

It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.

In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc.

In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.

Claims

1. A method of data analysis, comprising:

acquiring an analysis chart, and analyzing the analysis chart;

2. The method of claim 1, wherein the parsing the analysis graph comprises:

3. The method of claim 2, wherein the field information comprises: field name, field length, field type;

4. The method of claim 2, wherein assembling the parsed information into an execution script comprises:

performing semantic analysis and splitting on the calculation result;

assembling basic field information according to the field information;

assembling any multi-level aggregation according to the dimension information;

5. The method according to any one of claims 1 to 4, wherein the visualizing the calculation based on the analysis chart comprises:

and outputting the integrated analysis chart.

6. A data analysis system, comprising:

7. The system of claim 6, wherein the graph analysis module analyzes the analysis graph, and specifically comprises:

8. The system of claim 7, wherein the field information comprises: field name, field length, field type;

9. The system according to claim 7, wherein the script assembling module assembles the parsed information into the execution script, and specifically includes:

performing semantic analysis and splitting on the calculation result;

assembling basic field information according to the field information;

assembling any multi-level aggregation according to the dimension information;

10. The system according to any one of claims 6 to 9, wherein the result analysis module performs visualization processing on the calculation result according to the analysis chart, specifically comprising:

and outputting the integrated analysis chart.