WO2023061454A1

WO2023061454A1 - Visualization processing system and method for unstructured data, and storage medium and terminal

Info

Publication number: WO2023061454A1
Application number: PCT/CN2022/125188
Authority: WO
Inventors: 王永旭; 陈良旭; 周锋; 陈�光; 田丰
Original assignee: 广州广电运通金融电子股份有限公司
Priority date: 2021-10-14
Filing date: 2022-10-13
Publication date: 2023-04-20
Also published as: CN113934820A

Abstract

A visualization processing system and method for unstructured data, and a storage medium and a terminal, which belong to the technical field of data visualization processing. The system comprises a data visualization processing module, a data engine module and a data source module, which are in two-way communication, wherein the data visualization processing module comprises an application template library, a chart library, a condition constructor and a drawing component, and is used for receiving and rendering data results, which are from the data engine module and are subjected to unified analysis and processing, so as to realize data presentation meeting diversified requirements. In the method, by means of a unified data processing solution, the problem of it being impossible to directly visualize unstructured data in the past is solved, the problem of an application being required to be redeployed after a visualized chart is generated is also solved, and service personnel can easily realize visualization analysis of mass data.

Description

Visual processing system, method, storage medium and terminal of unstructured data

This disclosure claims the priority of the Chinese patent application with the application number 202111197952.3 and the title of the invention "system, method, storage medium and terminal for visual processing of unstructured data" submitted to the China Patent Office on October 14, 2021, all of which The contents are incorporated by reference in this disclosure.

technical field

The disclosure belongs to the technical field of data visualization processing, and in particular relates to a system, method, storage medium and terminal for visual processing of unstructured data.

Background technique

The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art or prior art.

Data visualization is a scientific and technological research on the visual representation of data. Among them, the visual representation of this data is defined as information extracted in a certain summary form, including various attributes and variables of the corresponding information unit. The technology allows the visual interpretation of data through representation, modeling and display of volumes, surfaces, properties and animations using graphics, image processing, computer vision and user interfaces. Data visualization is mainly aimed at conveying and communicating information clearly and effectively by means of graphics

Data visualization is widely used in the field of Internet big data operations, through the function of mining data and simply summarizing the mined data for visual display. However, in existing data processing, for example, the patent application number 201810299437.8 discloses a data visualization display method and system, which lacks unified processing of structured and unstructured data, and unstructured data cannot be directly visualized At the same time, the application needs to be redeployed after the visualization chart is generated, which leads to difficult data processing, complex data visualization process, long implementation cycle and high cost.

Contents of the invention

(1) Technical problems to be solved

The technical problem to be solved in the present disclosure is that unstructured data cannot be directly visualized in the prior art.

(2) Technical solution

In order to overcome the deficiencies of the prior art, the purpose of the present disclosure is to provide a visual processing system, method, storage medium and terminal for unstructured data, which can solve the above-mentioned problems.

Design principle:

Overall scheme: In order to solve the above-mentioned problems, the overall design scheme of the present application is as follows.

A visual processing system for unstructured data, the visual processing system includes a two-way communication data visualization processing module, a data engine module and a data source module, wherein the data visualization processing module includes an application template library, a chart library, a condition The constructor and drawing component are used to receive and render the data results from the unified analysis and processing of the data engine module, so as to realize the data display of diverse needs.

The present disclosure also provides a method for visualizing unstructured data, the method comprising:

S1. Establish the analysis dimension, and establish the index analysis dimension based on the actual visualization needs;

S2. Select the chart, and select the chart type based on the analysis dimension;

S3. Select the data source, select the data source to be analyzed from the data source list, and classify the selected data source fields by dimensions and indicators;

S4. Construct drawing conditions, set axis dimensions and measurements, and filter conditions;

S5. Unstructured data processing, submit the selected conditions to the data engine module for unified analysis and processing and return the results, and then render the results and the set page style attributes in real time;

S6. Typesetting layout, performing rasterized layout and style adjustment in the front-end drawing area;

S7. Generate an application, generate an application preview page with the data after the typesetting layout is completed, save it in the database and generate an independent application project in the background and assign an unused port number, and add it to the application list to generate an application in a deployable state;

S8. Deploy the application, compile and package the generated visualization application into a microservice image, and push it to the image server;

S9. Running/viewing the application, running the application image, generating an application URL, and accessing the application through the URL.

Further, in step S1, four analysis dimensions are determined: connection, distribution, comparison, and composition. Among them, connection refers to the relationship between data, comparison refers to the ranking of data series, distribution refers to the frequency of items, and composition concerns each part. percentage of the whole.

Further, the data types of the data sources in step S3 include CSV/Excel files, structured data sources, and/or unstructured data sources, and the selected data source fields are classified according to dimensions and indicators, and the dimensions are used as "X Coordinate/Dimension Axis" selects the parameters, and uses the indicator as the "Y Coordinate/Measurement Axis" statistical parameter.

The present disclosure also provides a computer-readable storage medium on which computer instructions are stored, and the steps of the aforementioned method are executed when the computer instructions are run.

The present disclosure also provides a terminal, including a memory and a processor, the memory stores computer instructions that can run on the processor, and the processor executes the steps of the foregoing method when running the computer instructions.

(3) Beneficial effects

Compared with the prior art, the beneficial effect of the present disclosure lies in: the unified data processing scheme of the present disclosure solves the problem that unstructured data cannot be directly visualized in the past, and at the same time solves the problem of needing to redeploy the application after generating the visualized chart, allowing business Personnel can also easily implement visual analysis of massive data.

Description of drawings

FIG. 1 is a schematic diagram of a visualization processing system of the present disclosure;

Fig. 2 is a flowchart of the disclosed method;

Figure 3 is an interface diagram of establishing the analysis dimension, that is, creating a new application;

Figure 4 is a visual application editing interface;

Figure 5 is an interface diagram for style adjustment or setting;

Figure 6 is a diagram of the application deployment interface;

FIG. 7 is a diagram of the application running start interface;

Fig. 8 is an interface diagram of the application running result.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments It is a part of embodiments of the present disclosure, but not all embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present disclosure.

The flowchart is used in this specification to illustrate the operations performed by the system according to the embodiment of this specification. It should be understood that the preceding or following operations are not necessarily performed in the exact order. Instead, various steps may be processed in reverse order or simultaneously. At the same time, other operations can be added to these procedures, or a certain step or steps can be removed from these procedures.

first embodiment

A visual processing system for unstructured data, see FIG. 1 , the visual processing system includes a data visual processing module with two-way communication, a data engine module and a data source module.

Wherein, the data visualization processing module includes an application template library, a chart library, a condition builder and a drawing component, which are used to receive and render the data results from the unified analysis and processing of the data engine module, so as to realize the data display of diverse requirements.

The system provides one-stop self-service visual data analysis and presentation. From platform data source access to upper-level application self-service analysis data and interactive presentation, all can be easily and efficiently completed in the method. At the same time, it solves the problem of excessive reliance on SQL operations for visualization in the past, zero code, mouse drag and drop operation interaction, so that business personnel can easily realize the visual analysis of massive data.

second embodiment

A visualization method for unstructured data, see Fig. 2-Fig. 8, the method specifically includes the following steps.

S1. Establish the analysis dimension, and establish the indicator analysis dimension based on the actual visualization needs.

In one example, the analysis dimension identifies four as connection, distribution, comparison, and composition.

Contact: the relationship between data;

Comparison: the ranking of data series, which can show the order of things;

Distribution: The frequency of items, mainly concerned with how many items are included in each value range;

Composition: Focus on what percentage each part contributes to the whole.

This step is the creation of the application. See Figure 3. You can choose a blank template or a preset application template, and fill in application-related information (including application name, developer, canvas scale, application description, and application logo).

Figure 3 Marking Explanation: 11. Template list; 12 is the application basic information filling area.

After establishing the analysis dimensions, the types of charts we can choose can be basically determined, and the chart selection is based on the above dimensions.

Icon types mainly include:

Contact class: scatter diagram, bubble diagram, radar diagram;

Comparison categories: histogram, bar chart, line chart;

Distribution category: map, scatter plot, bubble chart;

Constituent categories: pie chart, area chart, funnel chart.

Specifically, refer to the visual application editing interface in FIG. 4 , select a chart type from the chart library according to requirements, and drag the chart to the chart drawing area.

Accompanying drawing 4 includes label explanation:

21. Application title;

22. Toolbar;

23. Operation button;

24. Graphical list;

25. Chart work area;

26. Data and style tab page;

27. Data source selection area;

28. Condition screening area;

29. Dimension and measure settings.

S3. Select a data source, select a data source to be analyzed from the data source list, and classify the fields of the selected data source by dimensions and indicators. And the dimension is used as the "X coordinate/dimension axis" selection parameter, and the indicator is used as the "Y coordinate/measurement axis" statistical parameter.

Wherein, the data type of the data source includes a CSV/Excel file, a structured data source, and/or an unstructured data source.

Among them, dimension: usually refers to a field that cannot be aggregated, and is often used for classification.

Metrics: Also known as metrics, are fields that can be measured and aggregated or used for data manipulation.

S4. Construct drawing conditions, set axis dimensions and measures, and filter conditions. Specifically include:

S41. Setting dimensions and measures:

X coordinate axis, select the dimension field to be analyzed from the dimension list;

Y-coordinate axis, select the indicator field to be counted from the indicator list for aggregation operation, and the multi-dimensional chart supports selecting multiple indicators for statistics;

Among them, the numerical type - can carry out (summation, average value, maximum value, minimum value, count) aggregation statistics;

Non-numeric type - counting statistics can be performed.

S42. Conditional filtering, selecting a conditional field, and identifying a field type.

Field description:

Character type: supports multiple selection, and obtains the value list through group by.

Numerical type: interval value, take the minimum value of the field as the start value, and the maximum value as the end value

Date type: interval value, take the minimum date in the field as the start date, and the maximum date as the end date.

S5. Unstructured data processing, submitting the selected conditions to the data engine module for unified analysis and processing and returning the results, and then rendering the results and the set page style attributes in real time; specifically includes the following steps.

S51. After setting the dimension and index, submit the construction condition to the data processing engine module in JSON format, and request the parameter format.

S52. The data processing engine module receives the request parameters, and converts the parameters into SQL.

Conversion rules:

Select;

traverse measures;

Function name (indicator field 1), function name (indicator field 2), ...;

from table name;

group by dimension field.

S53, after completing the SQL conversion, submit through x-pack sql;

POST /_xpack /sql? format=json

{

"query": converted sql

}

S54. After the query, the result is returned to the front end, and the corresponding graph is rendered in real time according to the result data.

S6. Typesetting layout, performing rasterized layout and style adjustment in the front-end drawing area; specifically including:

S61. Grid layout: place the chart at the desired position in the drawing area by dragging and dropping;

S62, style adjustment, refer to the style setting interface of Fig. 5, specifically as follows:

apply a background color adjustment;

Chart background color adjustment;

Define the title, font size, color, upper and lower spacing, and placement;

Chart style, color scheme selection, line size/spacing;

Hover prompt;

X-axis, Y-axis angle adjustment, etc.

Through this step, you can drag the chart to any layout in the drawing area, set the application and chart colors, define titles, fonts, positions and other styles.

S7. Generate an application, generate an application preview page with the completed data layout, store it in the database, generate an independent application project in the background and assign an unused port number, add it to the application list, and generate an application in a deployable state.

Through steps 1-6, the production of the application and chart is completed. Then the application can be generated with one click, and an independent visual application project will be generated on the server side, and an unused port number will be automatically assigned to the application as the port number for subsequent application access. At this time, the application status becomes deployable .

Specifically include: popping up the large-screen application browsing page, generating html and saving it in the database; generating an application, generating an independent application project in the background and assigning an unused port number, and adding it to the application list.

S8. Deploy the application, see FIG. 6 , compile and package the generated visualization application into a microservice image, and push it to the image server.

S9. Run/view the application, see FIG. 7, run the application image, click the run button, generate the application URL, and access the application through the URL, see FIG. 8, display the running result, and can view and disable it.

URL format: http://IP:port.

It can be seen that the method supports multiple types of data sources. In addition to traditional Excel, CSV, and relational databases, it also supports non-relational database Elasticsearch. The drawing components are rich, easy to operate, multi-dimensional exploration, and drawing can be realized by dragging and dropping. The application template library is supported. After users select the corresponding template according to the business scenario, they only need to replace the prefabricated chart data source to render the chart without drawing, which greatly reduces the difficulty and time cost of user visualization development. After editing, you can directly preview the application on a large screen, generate the application with one click, deploy the application, and publish the application, realizing the entire life cycle from creating a visual application to publishing the application.

Key points of the program:

1. Convert unstructured data into unified query SQL for data extraction;

2. Generate and deploy the visualization chart with one click.

Through unified data processing, the solution solves the problem that unstructured data cannot be directly visualized in the past, and at the same time solves the problem of needing to redeploy applications after generating visualization charts, allowing business personnel to easily implement visual analysis of massive data.

third embodiment

The present disclosure also provides a computer-readable storage medium on which computer instructions are stored, and the steps of the aforementioned method are executed when the computer instructions are run. Wherein, for the method, please refer to the detailed introduction in the foregoing part, and details will not be repeated here.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the computer-readable medium includes a permanent Both non-permanent and non-permanent, removable and non-removable media can be implemented by any method or technology for information storage. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

The computer program codes required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python etc., conventional procedural programming languages such as C language, VisualBasic, Fortran2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages such as Python, Ruby and Groovy, or other programming languages. The program code may run entirely on the user's computer, or as a stand-alone software package, or run partly on the user's computer and partly on a remote computer, or entirely on the remote computer or processing device. In the latter case, the remote computer can be connected to the user computer through any form of network, such as a local area network (LAN) or wide area network (WAN), or to an external computer (such as through the Internet), or in a cloud computing environment, or as a service Use software as a service (SaaS).

Fourth embodiment

The present disclosure also provides a terminal, including a memory and a processor, the memory stores data sources and computer instructions that can run on the processor, and the processor executes the aforementioned method when running the computer instructions step. Wherein, for the method, please refer to the detailed introduction in the foregoing part, and details will not be repeated here.

It should be noted that different embodiments may have different beneficial effects. In different embodiments, the possible beneficial effects may be any one or a combination of the above, or any other possible beneficial effects.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Industrial Applicability

The unstructured data visualization processing system provided by this disclosure can solve the problem that unstructured data cannot be directly visualized, and also solve the problem of redeploying applications after generating visualization charts, so that business personnel can easily realize massive data visualization analysis, with strong industrial applicability.

Claims

A visual processing system for unstructured data, characterized in that: the visual processing system includes a two-way communication data visualization processing module, a data engine module and a data source module, wherein the data visualization processing module includes an application template library, Chart library, condition builder and drawing components are used to receive and render the data results from the unified analysis and processing of the data engine module, so as to realize the data display of diverse needs.
A visualization method for unstructured data, characterized in that the method comprises:

S1. Establish the analysis dimension, and establish the index analysis dimension based on the actual visualization needs;

S2. Select the chart, and select the chart type based on the analysis dimension;

S3. Select the data source, select the data source to be analyzed from the data source list, and classify the selected data source fields by dimensions and indicators;

S4. Construct drawing conditions, set axis dimensions and measurements, and filter conditions;

S5. Unstructured data processing, submit the selected conditions to the data engine module for unified analysis and processing and return the results, and then render the results and the set page style attributes in real time;

S6. Typesetting layout, performing rasterized layout and style adjustment in the front-end drawing area;

S7. Generate an application, generate an application preview page with the data after the typesetting layout is completed, save it to the database and generate an independent application project in the background and assign an unused port number, and add it to the application list to generate an application in a deployable state;

S8. Deploy the application, compile and package the generated visualization application into a microservice image, and push it to the image server;

S9. Running/viewing the application, running the application image, generating an application URL, and accessing the application through the URL.
The visualization method according to claim 2, characterized in that: in step S1, four dimensions of analysis are determined: connection, distribution, comparison, and composition, wherein the connection is the relationship between data, the comparison is the ranking of the data series, and the distribution is The frequency profile of the items constitutes the percentage of the whole that focuses on each part.
The visualization method according to claim 2, characterized in that: the data type of the data source in step S3 includes CSV/Excel file, structured data source, and/or unstructured data source, and the selected data source field is pressed Dimensions and indicators are classified, and the dimension is used as the "X coordinate/dimension axis" selection parameter, and the indicator is used as the "Y coordinate/measurement axis" statistical parameter.
The visualization method according to claim 2, wherein step S4 comprises:

S41. Setting dimensions and measures: X axis, select the dimension field to be analyzed from the dimension list; Y axis, select the index field to be counted from the index list to perform aggregation operation, and the multidimensional chart supports selecting multiple indexes for statistics;

S42. Conditional filtering, selecting a conditional field, and identifying a field type.
The visualization method according to claim 2, wherein step S5 comprises:

S51. After setting the dimension and index, submit the construction condition to the data processing engine module in JSON format, and request the parameter format;

S52. The data processing engine module receives the request parameter, and performs SQL conversion on the parameter;

S53, after completing the SQL conversion, submit through x-pack sql;

S54. After the query, the result is returned to the front end, and the corresponding graph is rendered in real time according to the result data.
The visualization method according to claim 2, wherein step S6 comprises:

S61. Grid layout: place the chart at the desired position in the drawing area by dragging and dropping;

S62. Style adjustment: including application background color adjustment, chart background color adjustment, title definition, chart style, floating prompt, and X-axis and Y-axis angle adjustment.
The visualization method according to claim 2, characterized in that: in step S9, the URL format is http://IP:port.
A computer-readable storage medium, on which computer instructions are stored, wherein the steps of the method according to any one of claims 2-8 are executed when the computer instructions are executed.
A terminal, comprising a memory and a processor, wherein the memory stores computer instructions that can be run on the processor, and is characterized in that: when the processor runs the computer instructions, any one of claims 2-8 is executed. steps of the method described in the item.