CN112486975A - Method for automatically visualizing data based on big data - Google Patents
Method for automatically visualizing data based on big data Download PDFInfo
- Publication number
- CN112486975A CN112486975A CN202011456010.8A CN202011456010A CN112486975A CN 112486975 A CN112486975 A CN 112486975A CN 202011456010 A CN202011456010 A CN 202011456010A CN 112486975 A CN112486975 A CN 112486975A
- Authority
- CN
- China
- Prior art keywords
- data
- web page
- noise reduction
- visual
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 230000000007 visual effect Effects 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 20
- 238000004458 analytical method Methods 0.000 claims abstract description 13
- 230000009467 reduction Effects 0.000 claims description 13
- 238000011161 development Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 238000012800 visualization Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 4
- 238000012360 testing method Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 claims description 2
- 238000000605 extraction Methods 0.000 claims description 2
- 238000001914 filtration Methods 0.000 claims description 2
- 238000009499 grossing Methods 0.000 claims description 2
- 238000010606 normalization Methods 0.000 claims description 2
- 238000012216 screening Methods 0.000 claims description 2
- 230000003993 interaction Effects 0.000 abstract description 5
- 238000007405 data analysis Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 abstract description 2
- 230000002452 interceptive effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013079 data visualisation Methods 0.000 description 2
- 238000001035 drying Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
- G06F16/212—Schema design and management with details for data modelling support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method for automatically visualizing data based on big data, which uses a visualized rule configuration interface on the basis of traditional attack scene modeling, can preview and alarm after the rule takes effect, improves the visibility and interactivity of the scene modeling, realizes the attack scene modeling of visual interaction, is convenient for the safety analysis of big data, fully utilizes an open-source streaming processing framework, an event processing model and a visual interaction operation feedback model, and enables the flow of data analysis to form a complete closed loop and good interaction between man and machine by the visual configuration of the scene rule and the feedback of the rule after the rule acts on the data, thereby finally forming a good visual interaction flow of the scene modeling and having good application prospect.
Description
Technical Field
The invention particularly relates to a method for automatically visualizing data based on big data.
Background
With the explosive increase of data volume of various industries at present, aiming at mass data at least taking PB as a unit, when the operation is carried out in a manual mode, because visual operation is difficult to carry out, a plurality of errors are generated; in order to reduce various errors in the manual operation process of mass data, a big data visualization technology is applied. However, the traditional big data visualization technology still needs a great amount of manual operation when analyzing the data types; in order to reduce the complexity caused by manual operation on mass data, a method which can intelligently analyze the mass data and automatically visualize according to the service is urgently needed.
Disclosure of Invention
The present invention aims to provide a method for automatically visualizing data based on big data, which can solve the above problems well, in view of the shortcomings of the prior art.
In order to meet the requirements, the technical scheme adopted by the invention is as follows: a method for automatically visualizing data based on big data is provided, which comprises the following steps:
s1: acquiring or inputting original data, integrating a large-scale data source, storing the large-scale data source in a distributed database, preprocessing and storing the original data, and preprocessing to obtain accurate initial data;
s2: sample data for analysis is extracted from a large-scale data source through configuration engine interface configuration parameters;
s3: drying the sample data, eliminating irrelevant data and obtaining an analysis sample;
s4: carrying out visual matching processing on the obtained sample data;
s5: mapping is carried out, data set establishment is carried out on the data processed in the step S2, and numerical data are converted into geometric data to complete data modeling;
s6: drawing and designing a chart, selecting the type of the chart according to the requirement of a business data presentation mode, matching the display numerical value of the chart to be presented, and drawing the chart by using a drawing engine of a visual class library;
s7: visual presentation, which is integrated through page layout, customization of local charts, configuration of data sources and data sets and a uniform interface for acquiring data from a big data platform;
s8: and displaying the data source to be presented at the front end of the Web page, thereby realizing the configuration and presentation of the automatic visual analysis page of the big data platform.
The method for automatically visualizing the data based on the big data has the following advantages:
on the basis of traditional attack scene modeling, a visual rule configuration interface is used, warning can be previewed after a rule takes effect, visibility and interactivity of scene modeling are improved, visual interactive attack scene modeling is achieved, large data safety analysis is facilitated, an open-source streaming processing framework, an event processing model and a visual interactive operation feedback model are fully utilized, a complete closed loop is formed in a data analysis process through visual configuration of scene rules and feedback after the rule acts on data, good interaction is formed between a human machine and the computer, a good visual interactive process of scene modeling is formed finally, and the application prospect is good.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 schematically shows a flow diagram of a method for big data based automatic visualization according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described in further detail with reference to the accompanying drawings and specific embodiments.
In the following description, references to "one embodiment," "an embodiment," "one example," "an example," etc., indicate that the embodiment or example so described may include a particular feature, structure, characteristic, property, element, or limitation, but every embodiment or example does not necessarily include the particular feature, structure, characteristic, property, element, or limitation. Moreover, repeated use of the phrase "in accordance with an embodiment of the present application" although it may possibly refer to the same embodiment, does not necessarily refer to the same embodiment.
Certain features that are well known to those skilled in the art have been omitted from the following description for the sake of simplicity.
According to an embodiment of the present application, there is provided a method for automatically visualizing data based on big data, as shown in fig. 1, including the following steps:
s1: acquiring or inputting original data, integrating a large-scale data source, storing the large-scale data source in a distributed database, preprocessing and storing the original data, and preprocessing to obtain accurate initial data;
s2: sample data for analysis is extracted from a large-scale data source through configuration engine interface configuration parameters;
s3: drying the sample data, eliminating irrelevant data and obtaining an analysis sample;
s4: carrying out visual matching processing on the obtained sample data;
s5: mapping is carried out, data set establishment is carried out on the data processed in the step S2, and numerical data are converted into geometric data to complete data modeling;
s6: drawing and designing a chart, selecting the type of the chart according to the requirement of a business data presentation mode, matching the display numerical value of the chart to be presented, and drawing the chart by using a drawing engine of a visual class library;
s7: visual presentation, which is integrated through page layout, customization of local charts, configuration of data sources and data sets and a uniform interface for acquiring data from a big data platform;
s8: and displaying the data source to be presented at the front end of the Web page, thereby realizing the configuration and presentation of the automatic visual analysis page of the big data platform.
According to an embodiment of the present application, in step S2 of the method for automatically visualizing big data based on the big data, the obtained initial data is subjected to visualization matching processing, including filtering processing, smoothing processing, normalization processing, geometric transformation, linear transformation, and feature detection and extraction.
According to an embodiment of the present application, the method for automatically visualizing big data based data further comprises the following steps: loading the extracted data into a distributed file system (HDFS); and converting the data in the HDFS according to the received service rule to obtain a processing result. The processing results are derived from the HDFS and loaded into a relational database.
According to an embodiment of the present application, the step S3 of automatically visualizing the big data-based data: carrying out noise reduction operation on the sample data, eliminating irrelevant data and obtaining an analysis sample, wherein the steps of carrying out noise reduction are as follows:
receiving a noise reduction request, acquiring data to be subjected to noise reduction, and acquiring a corresponding feature combination according to the noise reduction request;
establishing a data noise reduction comparison model according to the feature combination;
calculating the discrimination parameters of the feature combinations;
screening the discrimination of the feature combinations by using a preset initial discrimination threshold value to obtain feature combinations corresponding to the discrimination meeting the preset requirements;
generating an initial feature combination according to the feature combination corresponding to the discrimination meeting the preset requirement;
extracting available feature combinations from the initial feature combinations according to preset evaluation indexes; and denoising the initial data according to the available feature combination, and deleting the noise data in the initial data to obtain available data.
According to an embodiment of the application, the step of displaying the data source to be presented at the front end of the Web page in the method for automatically visualizing the data based on the big data specifically includes: receiving user operation; generating a Web page code according to user operation, and analyzing the Web page code to generate a Web page; and converting the Web page into a picture.
According to an embodiment of the application, the method for automatically visualizing the big data based data further comprises the following steps of Web page detection: establishing a communication channel between a data line between the development equipment and the tested equipment; the development equipment sends a test instruction to the tested equipment through the communication channel, and the test instruction indicates the tested equipment to acquire target data information of a target Web page; and the development equipment acquires target data information of the target Web page returned by the tested equipment.
The above-mentioned embodiments only show some embodiments of the present invention, and the description thereof is more specific and detailed, but should not be construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the claims.
Claims (6)
1. A method for automatically visualizing data based on big data is characterized by comprising the following steps:
s1: acquiring or inputting original data, integrating a large-scale data source, storing the large-scale data source in a distributed database, preprocessing and storing the original data, and preprocessing to obtain accurate initial data;
s2: sample data for analysis is extracted from a large-scale data source through configuration engine interface configuration parameters;
s3: carrying out noise reduction operation on the sample data, eliminating irrelevant data and obtaining an analysis sample;
step S3: carrying out noise reduction operation on the sample data, eliminating irrelevant data and obtaining an analysis sample, wherein the steps of carrying out noise reduction are as follows:
receiving a noise reduction request, acquiring data to be subjected to noise reduction, and acquiring a corresponding feature combination according to the noise reduction request;
establishing a data noise reduction comparison model according to the feature combination;
calculating the discrimination parameters of the feature combinations;
screening the discrimination of the feature combinations by using a preset initial discrimination threshold value to obtain feature combinations corresponding to the discrimination meeting the preset requirements;
generating an initial feature combination according to the feature combination corresponding to the discrimination meeting the preset requirement;
extracting available feature combinations from the initial feature combinations according to preset evaluation indexes; and denoising the initial data according to the available feature combination, and deleting the noise data in the initial data to obtain available data.
S4: carrying out visual matching processing on the obtained sample data;
s5: mapping is carried out, data set establishment is carried out on the data processed in the step S2, and numerical data are converted into geometric data to complete data modeling;
s6: drawing and designing a chart, selecting the type of the chart according to the requirement of a business data presentation mode, matching the display numerical value of the chart to be presented, and drawing the chart by using a drawing engine of a visual class library;
s7: visual presentation, which is integrated through page layout, customization of local charts, configuration of data sources and data sets and a uniform interface for acquiring data from a big data platform;
s8: and displaying the data source to be presented at the front end of the Web page, thereby realizing the configuration and presentation of the automatic visual analysis page of the big data platform.
2. The method for big data based automatic visualization of claim 1, wherein: and step S2, performing visualization matching processing on the obtained initial data, wherein the visualization matching processing comprises data filtering processing, smoothing processing, normalization processing, geometric transformation, linear transformation, and feature detection and extraction.
3. The method for automatically visualizing big data based on claim 1, further comprising the steps of: loading the extracted data into a distributed file system (HDFS); and converting the data in the HDFS according to the received service rule to obtain a processing result.
4. The method for big data based automatic visualization of claim 3, wherein: the processing results are derived from the HDFS and loaded into a relational database.
5. The method for automatically visualizing big data according to claim 1, wherein the step of displaying the data source to be presented at the front end of the Web page specifically comprises: receiving user operation; generating a Web page code according to user operation, and analyzing the Web page code to generate a Web page; and converting the Web page into a picture.
6. The method for automatically visualizing big data based on the claim 1, further comprising the step of Web page detection: establishing a communication channel between a data line between the development equipment and the tested equipment; the development equipment sends a test instruction to the tested equipment through the communication channel, and the test instruction indicates the tested equipment to acquire target data information of a target Web page; and the development equipment acquires target data information of the target Web page returned by the tested equipment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011456010.8A CN112486975A (en) | 2020-12-10 | 2020-12-10 | Method for automatically visualizing data based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011456010.8A CN112486975A (en) | 2020-12-10 | 2020-12-10 | Method for automatically visualizing data based on big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112486975A true CN112486975A (en) | 2021-03-12 |
Family
ID=74917632
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011456010.8A Pending CN112486975A (en) | 2020-12-10 | 2020-12-10 | Method for automatically visualizing data based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112486975A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109471853A (en) * | 2018-09-18 | 2019-03-15 | 平安科技（深圳）有限公司 | Data noise reduction, device, computer equipment and storage medium |
| CN111444230A (en) * | 2019-01-17 | 2020-07-24 | 苏州黑牛新媒体有限公司 | Data visualization analysis method based on big data platform |
| CN111444103A (en) * | 2020-03-31 | 2020-07-24 | 腾讯音乐娱乐科技（深圳）有限公司 | Automatic testing method for Web page and related equipment |
| CN111597010A (en) * | 2020-05-27 | 2020-08-28 | 北京智美智学科技有限公司 | Method and device for generating pictures of Web pages, printing equipment and recording medium |
-
2020
- 2020-12-10 CN CN202011456010.8A patent/CN112486975A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109471853A (en) * | 2018-09-18 | 2019-03-15 | 平安科技（深圳）有限公司 | Data noise reduction, device, computer equipment and storage medium |
| CN111444230A (en) * | 2019-01-17 | 2020-07-24 | 苏州黑牛新媒体有限公司 | Data visualization analysis method based on big data platform |
| CN111444103A (en) * | 2020-03-31 | 2020-07-24 | 腾讯音乐娱乐科技（深圳）有限公司 | Automatic testing method for Web page and related equipment |
| CN111597010A (en) * | 2020-05-27 | 2020-08-28 | 北京智美智学科技有限公司 | Method and device for generating pictures of Web pages, printing equipment and recording medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112966139B (en) | Data processing method, device, electronic equipment and computer storage medium | |
| CN108073760A (en) | For obtaining the method and system that analysis model writes knowledge | |
| Bakaev et al. | Auto-extraction and integration of metrics for web user interfaces | |
| CN110489593B (en) | Topic processing method and device for video, electronic equipment and storage medium | |
| CN114912533B (en) | State monitoring system and monitoring method applied to transformer | |
| CN114090582A (en) | Method, apparatus, device, storage medium and program product for generating domain model | |
| CN111444230A (en) | Data visualization analysis method based on big data platform | |
| CN117875293A (en) | Method for generating service form template in quick digitization mode | |
| CN112698897A (en) | Method and system for arranging visual big data operator | |
| CN117420998A (en) | Client UI interaction component generation method, device, terminal and medium | |
| CN112486975A (en) | Method for automatically visualizing data based on big data | |
| CN116225522A (en) | Method and device for generating software prototype, electronic equipment and storage medium | |
| CN115018473A (en) | Service processing method, device, storage medium and equipment | |
| CN113688134B (en) | Visual variable management method, system and equipment based on multidimensional data | |
| CN115238662A (en) | Bidding file rapid editing method and system | |
| CN112085636B (en) | Urban functional shrinkage analysis method, device and storage medium | |
| US20210026997A1 (en) | Method for creating knowledge representation model for product | |
| CN109558418B (en) | Method for automatically identifying information | |
| CN110019828B (en) | Knowledge graph-based reference implementation verification method and system | |
| JP2008009819A (en) | Security diagnostic system | |
| CN113536701A (en) | CFD simulation model automatic setting method, system, computer equipment and storage medium | |
| US11928123B2 (en) | Systems and methods for network explainability | |
| CN114863450B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN110764853B (en) | Web interface display method between multiple electronic medical records and single document defects | |
| CN114841136A (en) | Method, device and equipment for editing inspection report |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210312 |
|
| RJ01 | Rejection of invention patent application after publication |