CN117033365B - Visual analysis graph-based UI data processing method and system - Google Patents
Visual analysis graph-based UI data processing method and system Download PDFInfo
- Publication number
- CN117033365B CN117033365B CN202311277055.2A CN202311277055A CN117033365B CN 117033365 B CN117033365 B CN 117033365B CN 202311277055 A CN202311277055 A CN 202311277055A CN 117033365 B CN117033365 B CN 117033365B
- Authority
- CN
- China
- Prior art keywords
- data
- graph structure
- structure data
- graph
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 80
- 238000004458 analytical method Methods 0.000 title claims abstract description 59
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 238000000547 structure data Methods 0.000 claims abstract description 100
- 230000003993 interaction Effects 0.000 claims abstract description 73
- 238000000034 method Methods 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 22
- 238000007781 pre-processing Methods 0.000 claims abstract description 20
- 238000013461 design Methods 0.000 claims description 26
- 238000004140 cleaning Methods 0.000 claims description 12
- 238000001914 filtration Methods 0.000 claims description 9
- 230000009471 action Effects 0.000 claims description 7
- 238000005457 optimization Methods 0.000 claims description 7
- 230000002452 interceptive effect Effects 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 6
- 238000007418 data mining Methods 0.000 claims description 5
- 238000013523 data management Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000000926 separation method Methods 0.000 abstract description 5
- 230000006978 adaptation Effects 0.000 abstract description 3
- 230000000694 effects Effects 0.000 description 16
- 230000006870 function Effects 0.000 description 6
- 238000007405 data analysis Methods 0.000 description 5
- 238000012800 visualization Methods 0.000 description 5
- 230000006399 behavior Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000003012 network analysis Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000019771 cognition Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000015572 biosynthetic process Effects 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000005484 gravity Effects 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2237—Vectors, bitmaps or matrices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a UI data processing method and a system based on a visual analysis chart, which relate to the field of data processing, wherein the UI data processing method comprises the following steps: s1, constructing a distributed gateway based on Node multi-process clusters, reading a local database, and acting for remote data service; s2, building a Hadoop cluster, performing man-machine interaction preprocessing operation of visual analysis on data of a local database, and obtaining graph structure data; s3, constructing a distributed storage cluster, optimizing and inquiring, merging, sorting and storing the graph structure data by using a document database, and accessing the graph structure data based on a copy set; s4, preprocessing the graph structure data based on MapReduce. The method uses a strategy of front-end and back-end separation, and improves the multi-terminal adaptation capability of the display layer of the visual analysis system.
Description
Technical Field
The invention relates to the field of data processing, in particular to a visual analysis graph-based UI data processing method and system.
Background
In recent years, the rapid development of information technology in various fields has led to an increase in the data volume, which has led to the acquisition and storage of data becoming simpler than ever due to advances in technology. However, this results in increased difficulty in data analysis, and if the data can be analyzed and utilized, the underlying value in the data can be mined, thereby helping enterprises make more intelligent decisions, and visual analysis is one of the analysis modes.
Currently, visual analysis is an effective way to analyze and understand a wide variety of data sets of varying structures. The human brain generally has sharper insight into the graph than the number, and information which is difficult to be insight into by conventional statistics can be obtained more easily through the graph. The graphics in visual space are more helpful to the human brain to recognize and discover patterns underlying them than the numbers in the table. In recent years, both traditional and network media have tended to use graphical reports, information graphs, and visualization applications to report news and convey important information graphs to the public.
The visual analysis is a brand new research direction generated by cross fusion of a plurality of research fields such as information visualization, man-machine interaction, cognitive science, data mining, information theory, decision theory and the like. The traditional visual research work mainly focuses on the aspects of visual layout algorithm, fusion of data mining and visual analysis, design and implementation of a small-scale data set-oriented visual analysis system and the like. Most of the traditional visual analysis systems are limited by the processing capacity of a single device, automatic data import cannot be realized, and the analyzed data set is very limited. In the big data age, the construction of visual analysis systems is facing unprecedented challenges.
For the problems in the related art, no effective solution has been proposed at present.
Disclosure of Invention
Aiming at the problems in the related art, the invention provides a UI data processing method and a system based on a visual analysis chart, so as to overcome the technical problems in the prior art.
For this purpose, the invention adopts the following specific technical scheme:
according to an aspect of the present invention, there is provided a UI data processing method based on a visual analysis chart, the UI data processing method including the steps of:
s1, constructing a distributed gateway based on Node multi-process clusters, reading a local database, and acting for remote data service;
s2, building a Hadoop cluster, performing man-machine interaction preprocessing operation of visual analysis on data of a local database, and obtaining graph structure data;
s3, constructing a distributed storage cluster, optimizing and inquiring, merging, sorting and storing the graph structure data by using a document database, and accessing the graph structure data based on a copy set;
s4, preprocessing the graph structure data based on MapReduce.
Further, the Node-based multi-process cluster constructs a distributed gateway, reads a local database, and proxies remote data services, which comprises the following steps:
s11, adding an intermediate layer between the front end module and the back end module as a gateway;
s12, taking a dotted line frame in the front-end module as an intermediate gateway layer, and providing a model proxy service of data;
and S13, providing data service by utilizing a back-end module, and enabling the front-end to acquire and read the local database and render the acquired data into the page.
Further, the building of the Hadoop cluster includes the following steps:
java software is installed, hadoop users are created, and Hadoop release packages are downloaded and installed;
logging in Hadoop, and executing the operation of the Hadoop cluster by using SSH;
and constructing different Hadoop clusters aiming at different analysis scenes and data set scales with different sizes by adopting the same configuration file for the main machine and the working machine.
Further, the step of constructing a Hadoop cluster, performing man-machine interaction preprocessing operation of visual analysis on data of a local database, and obtaining graph structure data includes the following steps:
s21, constructing a data scene of a local database by utilizing a Hadoop cluster, and visually displaying the data at a mobile terminal;
s22, providing an operation interface and a display interface for the user side according to the allocation authority of the administrator;
s23, adopting letter window mode design to respectively arrange data of a presentation local database of preview and information input operation;
s24, the observation interface adopts a main window design mode, a screen area is used for displaying visual graphics, and a button function area is arranged on one side conforming to the operation behavior habit of a user;
s25, the user side processes the data of the local database in an interactive mode to obtain graph structure data;
the interaction mode comprises gesture interaction, voice interaction and action interaction;
according to the gesture interaction, visual feedback is given to an operator by detecting touch and positioning by using an absolute coordinate system, so that the user experience on interaction control is enhanced, and man-machine interaction becomes simpler, more convenient and more natural;
the microphone and the sound equipment of the mobile equipment provide a hardware basis for voice interaction, and the voice interaction is used for facilitating information input and output operation of a user by combining various technologies of voice recognition, synthesis and understanding;
the action interaction realizes the action interaction function through detecting the user behaviors of the gravity sensor and the triaxial gyroscope.
Further, the user side processes the data of the local database in an interactive manner to obtain the graph structure data, which comprises the following steps:
s251, a user side opens a system operation interface and enters a data reading interface;
s252, clicking to display data reading and reading history, and rapidly acquiring once processed data through a local database;
s253, uniformly processing the data in various formats into structured data for preview display;
s254, selecting the type of the visual graph and determining the mapping relation from abstract data to visual graph attributes on the basis of preview display;
s255, converting the data into corresponding visual graphics according to the system setting information and the visual graphics configuration information, and displaying and interacting the visual graphics.
Furthermore, the building of the distributed storage cluster, the optimization query, data merging, data sorting and storage of the graph structure data by using the document database, and the access of the graph structure data based on the copy set comprise the following steps:
s31, constructing a distributed storage cluster, creating a main node and a plurality of backup nodes, and providing data management for node equipment;
s31, creating a graph structure data catalog for a server, and selecting a server port to add a name to equipment;
s32, creating a document type database in the equipment, providing a data slicing rule, positioning the position of the bitmap structure data, and processing the data;
s33, storing and reading the graph structure data through the copy set.
Further, the preprocessing of the graph structure data based on the MapReduce implementation includes the following steps:
s41, performing filtering and cleaning operations of the graph structure data and a specific data mining algorithm on the graph structure data by using MapReduce;
s42, filtering and cleaning the Map structure data through a Map stage;
s43, finishing the restoration of the user access sequence in the Reduce stage;
wherein, the formula of cleaning is:
in the formula, weight [ i ]]Weight of the i-th graph structure data, R 1 And R is R 2 Is the data in the graph structure data; is Valid (i) Is the value of the instance variable in the ith graph structure data; similar field [ i ]]Is the value of a similar variable in the i-th graph structure data.
Further, the Map stage for filtering and cleaning the graph structure data comprises the following steps:
s421, starting Map, and inputting a row of records in a log file;
s422, judging whether session is effective, for example judging whether the session is a crawler record, if not, returning to the step S431, and if yes, continuing to operate;
s423, acquiring sessionId as a Key, using the rest fields as Value, and outputting Key/Value.
Further, the reduction of the user access sequence completed in the Reduce stage comprises the following steps:
s421, starting with a single session as input;
s422, ordering all records in the session according to time, and adding the records into a set according to time sequence;
s423, traversing the collection, and judging the validity of each record;
s424, identifying the initial node of the session, adding the initial node to the head of the path sequence queue, sequentially identifying the subsequent nodes of the session, and adding the subsequent nodes to the path sequence queue;
s425, taking sessionId as a key, taking a path sequence queue as a value, and outputting the key/value.
According to another aspect of the present invention, there is also provided a UI data processing system based on a visual analysis chart, the system including: the system comprises a data reading module, a man-machine interaction module, a graph structure data optimizing module and a graph structure data processing module;
the data reading module is connected with the graph structure data optimizing module through the man-machine interaction module, and the graph structure data optimizing module is connected with the graph structure data processing module;
the data reading module constructs a distributed gateway based on Node multiprocess clusters, reads a local database and proxies remote data service;
the man-machine interaction module builds a Hadoop cluster, performs man-machine interaction preprocessing operation of visual analysis on data of a local database, and obtains graph structure data;
the graph structure data optimizing module is used for constructing a distributed storage cluster, optimizing and inquiring, merging, sorting and storing the graph structure data by utilizing a document database, and accessing the graph structure data based on a copy set;
and the graph structure data processing module is used for preprocessing graph structure data based on MapReduce.
The beneficial effects of the invention are as follows:
1. the invention uses a strategy of front-end and back-end separation, which makes it possible to perform visual analysis tasks at multiple terminals, which requires that the presentation layer of the visual system be able to adapt to perform visual analysis tasks on devices with different sized screens. Therefore, future work will further optimize the front-end and back-end separation architecture of the system, allow the system to integrate the multi-source heterogeneous service ends, provide a unified data access interface for the multi-terminal interface, and promote the multi-terminal adaptation capability of the presentation layer of the visual analysis system.
2. The invention is oriented to different analysis fields and provides layout scheme customizing capability which is easier to realize. Because the system is only built with four common visual layout methods at present, the analyzed emphasis is different in different analysis fields, such as social network analysis, biological network analysis and the like, the future optimization goal is to add visual layout methods for more analysis fields, and provide a unified programming interface for users to define layout components more meeting requirements.
3. The invention further enriches the interaction modes of the system, for the visual analysis system, the interaction is the connection point of the visualization and the data analysis, the interaction capability of the system directly influences the data analysis and exploration capability of the visual analysis system, and the future optimization target is to provide richer interaction means and allow a user to perform simpler and more advanced interaction methods.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a UI data processing method based on a visual analysis chart according to an embodiment of the present invention.
Detailed Description
For the purpose of further illustrating the various embodiments, the present invention provides the accompanying drawings, which are a part of the disclosure of the present invention, and which are mainly used to illustrate the embodiments and, together with the description, serve to explain the principles of the embodiments, and with reference to these descriptions, one skilled in the art will recognize other possible implementations and advantages of the present invention, wherein elements are not drawn to scale, and like reference numerals are generally used to designate like elements.
According to the embodiment of the invention, a UI data processing method and system based on a visual analysis chart are provided.
The present invention will be further described with reference to the accompanying drawings and detailed description, as shown in fig. 1, a UI data processing method based on a visual analysis chart according to an embodiment of the present invention, the UI data processing method including the steps of:
s1, constructing a distributed gateway based on Node multi-process clusters, reading a local database, and acting for remote data service;
in one embodiment, the Node-based multi-process cluster constructs a distributed gateway, reads a local database, and proxies a remote data service, comprising the steps of:
s11, adding an intermediate layer between the front end module and the back end module as a gateway;
s12, taking a dotted line frame in the front-end module as an intermediate gateway layer, and providing a model proxy service of data;
and S13, providing data service by utilizing a back-end module, and enabling the front-end to acquire and read the local database and render the acquired data into the page.
S2, building a Hadoop cluster, performing man-machine interaction preprocessing operation of visual analysis on data of a local database, and obtaining graph structure data;
in one embodiment, the building a Hadoop cluster comprises the steps of:
java software is installed, hadoop users are created, and Hadoop release packages are downloaded and installed;
logging in Hadoop, and executing the operation of the Hadoop cluster by using SSH;
and constructing different Hadoop clusters aiming at different analysis scenes and data set scales with different sizes by adopting the same configuration file for the main machine and the working machine.
In one embodiment, the building a Hadoop cluster, performing a preprocessing operation of man-machine interaction for visual analysis on data of a local database, and obtaining graph structure data includes the following steps:
s21, constructing a data scene of a local database by utilizing a Hadoop cluster, and visually displaying the data at a mobile terminal;
s22, providing an operation interface and a display interface for the user side according to the allocation authority of the administrator;
s23, adopting letter window mode design to respectively arrange data of a presentation local database of preview and information input operation;
s24, the observation interface adopts a main window design mode, a screen area is used for displaying visual graphics, and a button function area is arranged on one side conforming to the operation behavior habit of a user;
s25, the user side processes the data of the local database in an interactive mode to obtain graph structure data;
the interaction mode comprises gesture interaction, voice interaction and action interaction;
according to the gesture interaction, visual feedback is given to an operator by detecting touch and positioning by using an absolute coordinate system, so that the user experience on interaction control is enhanced, and man-machine interaction becomes simpler, more convenient and more natural;
the microphone and the sound equipment of the mobile equipment provide a hardware basis for voice interaction, and the voice interaction is used for facilitating information input and output operation of a user by combining various technologies of voice recognition, synthesis and understanding;
the action interaction realizes the action interaction function through detecting the user behaviors of the gravity sensor and the triaxial gyroscope.
In one embodiment, the processing, by the user side, the data of the local database in an interactive manner, to obtain the graph structure data includes the following steps:
s251, a user side opens a system operation interface and enters a data reading interface;
s252, clicking to display data reading and reading history, and rapidly acquiring once processed data through a local database;
s253, uniformly processing the data in various formats into structured data for preview display;
s254, selecting the type of the visual graph and determining the mapping relation from abstract data to visual graph attributes on the basis of preview display;
s255, converting the data into corresponding visual graphics according to the system setting information and the visual graphics configuration information, and displaying and interacting the visual graphics.
In addition, in the specific application, in man-machine interaction and mutual design, in order to update the display interface of the living image and the content thereof, feedback interaction operation and response thereof, abundant visual guidance and hearing and touch effects are generally used in the interface design and the interaction design, and the use principle is as follows:
1. considering the use environment of the application, not all effects are suitable for any environment, and when the effects are designed, the necessity and the selectivity of the effect presentation are determined by fully considering the specific situation when the effects are presented, so that the effects are flexibly used.
2. The application of the effect takes the object with the effect as the center, the specific situation and the requirement of the acting object of the effect should be considered, and different effects are applicable to different objects and meet different requirements.
3. Fitting the real life. The effect designed in the man-machine interaction process is as vivid and visual as possible and is close to the actual situation, so that the effect accords with the cognition and common knowledge of the user, the resonance of the user is more easily aroused, and the understanding of the user on the interface and interaction is deepened.
4. Consistency: similar to interface design and interaction design, the effect design should be kept consistent so that the user can understand and master the information expressed by the effect more quickly and the cognition of the user is not confused.
5. The system has the advantages that the system is unique, attractive and attractive, the attention of a user is more easily attracted, the user is enthusiasm, the user is excited, the system is a fundamental guiding principle of man-machine interaction design, platform differences, aesthetic cognition, common sense specifications and the like have different degrees of influence on the man-machine interaction design, the system is reasonable and suitable as a fundamental foundation of the man-machine interaction design, the main purposes of serving the user, realizing functions and presenting a user-friendly interface are man-machine interaction design, and the user-friendly interface is the interface design, the interaction design or the effect design and remembers the unprecedented theory at any time.
In summary, the man-machine interaction design should follow the principles of ease of use, rationality, normalization, consistency, aesthetics, coordination, etc.
Ease of use: the usability comprises accurate and easily understood interface content, comfortable and available interaction mode and excellent man-machine interaction usability design, so that the application is easier to understand, learn and operate and is attractive.
Rationality: all man-machine interaction designs are reasonable and proper, and various principles are balanced within a certain range and a certain rule to select the optimal scheme.
Normalization. The man-machine interaction design is carried out on the basis of the existing standard specification.
Consistency: for common sense and unified design, the consistency principle should be observed, so that the user can understand the conventional knowledge without deviation, and the user can be guaranteed to be tightly fused with the actual situation.
Beautiful and coordinated: the excellent man-machine interaction design scheme has the important effect of improving user experience besides the error-free input and output of information and the safe and stable realization of functions.
S3, constructing a distributed storage cluster, optimizing and inquiring, merging, sorting and storing the graph structure data by using a document database, and accessing the graph structure data based on a copy set;
in one embodiment, the building a distributed storage cluster, performing optimized query, data merging, data sorting and storage on the graph structure data by using a document database, and accessing the graph structure data based on a copy set includes the following steps:
s31, constructing a distributed storage cluster, creating a main node and a plurality of backup nodes, and providing data management for node equipment;
s31, creating a graph structure data catalog for a server, and selecting a server port to add a name to equipment;
s32, creating a document type database in the equipment, providing a data slicing rule, positioning the position of the bitmap structure data, and processing the data;
s33, storing and reading the graph structure data through the copy set.
In addition, the distributed database can realize the characteristics of high safety, high performance, high availability and the like when being applied to specific applications. For the visual analysis system, the storage layers are separated, so that a user can adopt a flexible storage scheme to interface with various databases when in actual implementation. NoSQL is taken as an effective storage scheme of plate structure data, and great convenience is provided for storage and query of graph structure data, so that the section introduces a cluster configuration scheme of a storage layer by taking a document database MongoDB as an example. MongoDB is a document type database, takes BSON documents as units of data storage, and has the advantages of cross-platform, flexibility, simplicity, easy expansion and the like.
MongoDB, when building clusters, needs to provide data slicing rules that will be recorded in MongoDB for locating the position of data in the replicas. MongoDB's clusters typically employ a replica set policy, which is a master-slave cluster with fail-over functionality. The most obvious difference compared to a common master-slave cluster is that the replica set does not set a fixed master node. Based on the copy set strategy, the cluster can select one main node, and when the main node cannot work, the main node is automatically changed into other nodes.
In one embodiment, the preprocessing of the graph structure data based on the MapReduce implementation includes the following steps:
s41, performing filtering and cleaning operations of the graph structure data and a specific data mining algorithm on the graph structure data by using MapReduce;
s42, filtering and cleaning the Map structure data through a Map stage;
s43, finishing the restoration of the user access sequence in the Reduce stage;
wherein, the formula of cleaning is:
in the formula, weight [ i ]]Weight of the i-th graph structure data, R 1 And R is R 2 Is the data in the graph structure data; is Valid (i) Is the value of the instance variable in the ith graph structure data; similar field [ i ]]Is the value of a similar variable in the i-th graph structure data.
S4, preprocessing the graph structure data based on MapReduce.
In one embodiment, the filtering and cleaning of the Map structural data through the Map stage includes the following steps:
s421, starting Map, and inputting a row of records in a log file;
s422, judging whether session is effective, for example judging whether the session is a crawler record, if not, returning to the step S431, and if yes, continuing to operate;
s423, acquiring sessionId as a Key, using the rest fields as Value, and outputting Key/Value.
In one embodiment, the reduction of the user access sequence completed in the Reduce phase includes the steps of:
s421, starting with a single session as input;
s422, ordering all records in the session according to time, and adding the records into a set according to time sequence;
s423, traversing the collection, and judging the validity of each record;
s424, identifying the initial node of the session, adding the initial node to the head of the path sequence queue, sequentially identifying the subsequent nodes of the session, and adding the subsequent nodes to the path sequence queue;
s425, taking sessionId as a key, taking a path sequence queue as a value, and outputting the key/value.
There is also provided, in accordance with another embodiment of the present invention, a UI data processing system based on a visual analysis graph, the system including: the system comprises a data reading module, a man-machine interaction module, a graph structure data optimizing module and a graph structure data processing module;
the data reading module is connected with the graph structure data optimizing module through the man-machine interaction module, and the graph structure data optimizing module is connected with the graph structure data processing module;
the data reading module constructs a distributed gateway based on Node multiprocess clusters, reads a local database and proxies remote data service;
the man-machine interaction module builds a Hadoop cluster, performs man-machine interaction preprocessing operation of visual analysis on data of a local database, and obtains graph structure data;
the graph structure data optimizing module is used for constructing a distributed storage cluster, optimizing and inquiring, merging, sorting and storing the graph structure data by utilizing a document database, and accessing the graph structure data based on a copy set;
and the graph structure data processing module is used for preprocessing graph structure data based on MapReduce.
In summary, by means of the above technical solution of the present invention, the present invention uses a front-end and back-end separation strategy, which makes it possible to perform visual analysis tasks on multiple terminals, and this requires that the presentation layer of the visual system be able to adapt to perform visual analysis tasks on devices with different size screens. Therefore, future work will further optimize the front-end and back-end separation architecture of the system, allow the system to integrate the multi-source heterogeneous service ends, provide a unified data access interface for the multi-terminal interface, and promote the multi-terminal adaptation capability of the presentation layer of the visual analysis system. The invention is oriented to different analysis fields and provides layout scheme customizing capability which is easier to realize. Because the system is only built with four common visual layout methods at present, the analyzed emphasis is different in different analysis fields, such as social network analysis, biological network analysis and the like, the future optimization goal is to increase the visual layout methods facing more analysis fields and provide a unified programming interface for users to define layout components more meeting requirements; the invention further enriches the interaction modes of the system, for the visual analysis system, the interaction is the connection point of the visualization and the data analysis, the interaction capability of the system directly influences the data analysis and exploration capability of the visual analysis system, and the future optimization target is to provide richer interaction means so as to allow a user to perform simpler and more advanced interaction methods; the invention improves the data display capability of the system. Because of the limitation of a user screen, the current visual analysis system is limited in data point display capacity and cannot respond to simultaneous display of tens of thousands of data points quickly. The future optimization objective is to further enhance the preloading capability of the data, turn the scaling operation of the user onto the preloading and restoring of the data, and further optimize the visualization through the caching system.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.
Claims (7)
1. A visual analysis graph-based UI data processing method, characterized in that the UI data processing method comprises the steps of:
s1, constructing a distributed gateway based on Node multi-process clusters, reading a local database, and acting for remote data service;
s2, building a Hadoop cluster, performing man-machine interaction preprocessing operation of visual analysis on data of a local database, and obtaining graph structure data;
s3, constructing a distributed storage cluster, optimizing and inquiring, merging, sorting and storing the graph structure data by using a document database, and accessing the graph structure data based on a copy set;
s4, preprocessing the graph structure data based on MapReduce;
the Node-based multi-process cluster builds a distributed gateway, reads a local database and proxies remote data service, and comprises the following steps:
s11, adding an intermediate layer between the front end module and the back end module as a gateway;
s12, taking a dotted line frame in the front-end module as an intermediate gateway layer, and providing a model proxy service of data;
s13, providing data service by utilizing a back-end module, and enabling a front end to acquire and read a local database and render the acquired data into a page;
the Hadoop cluster is built, man-machine interaction preprocessing operation of visual analysis is carried out on data of a local database, and graph structure data are obtained, wherein the method comprises the following steps:
s21, constructing a data scene of a local database by utilizing a Hadoop cluster, and visually displaying the data at a mobile terminal;
s22, providing an operation interface and a display interface for the user side according to the allocation authority of the administrator;
s23, adopting letter window mode design to respectively arrange data of a presentation local database of preview and information input operation;
s24, the observation interface adopts a main window design mode, a screen area is used for displaying visual graphics, and a button function area is arranged on one side conforming to the operation behavior habit of a user;
s25, the user side processes the data of the local database in an interactive mode to obtain graph structure data;
the interaction mode comprises gesture interaction, voice interaction and action interaction;
the map structure data preprocessing based on MapReduce comprises the following steps:
s41, performing filtering, cleaning operation and data mining algorithm on the graph structure data by using MapReduce;
s42, filtering and cleaning the Map structure data through a Map stage;
s43, finishing the restoration of the user access sequence in the Reduce stage;
wherein, the formula of cleaning is:
in the formula, weight [ i ]]Weight of the i-th graph structure data, R 1 And R is R 2 Is the data in the graph structure data; is Valid (i) Is the value of the instance variable in the ith graph structure data; similar field [ i ]]Is the value of a similar variable in the i-th graph structure data.
2. The visual analysis graph-based UI data processing method according to claim 1, wherein the building the Hadoop cluster comprises the following steps:
java software is installed, hadoop users are created, and Hadoop release packages are downloaded and installed;
logging in Hadoop, and executing the operation of the Hadoop cluster by using SSH;
and constructing different Hadoop clusters aiming at different analysis scenes and data set scales with different sizes by adopting the same configuration file for the main machine and the working machine.
3. The UI data processing method based on the visual analysis chart according to claim 2, wherein the user side processes the data of the local database in an interactive manner to obtain the chart structure data, and the method comprises the following steps:
s251, a user side opens a system operation interface and enters a data reading interface;
s252, clicking to display data reading and reading history, and rapidly acquiring once processed data through a local database;
s253, uniformly processing the data in various formats into structured data for preview display;
s254, selecting the type of the visual graph and determining the mapping relation from abstract data to visual graph attributes on the basis of preview display;
s255, converting the data into corresponding visual graphics according to the system setting information and the visual graphics configuration information, and displaying and interacting the visual graphics.
4. The UI data processing method based on the visual analysis graph according to claim 1, wherein the building a distributed storage cluster, performing optimization query, data merging, data sorting and storage on the graph structure data by using a document database, and accessing the graph structure data based on a copy set comprises the following steps:
s31, constructing a distributed storage cluster, creating a main node and a plurality of backup nodes, and providing data management for node equipment;
s31, creating a graph structure data catalog for a server, and selecting a server port to add a name to equipment;
s32, creating a document type database in the equipment, providing a data slicing rule, positioning the position of the bitmap structure data, and processing the data;
s33, storing and reading the graph structure data through the copy set.
5. The visual analysis graph-based UI data processing method according to claim 4, wherein the Map stage filtering and cleaning the graph structure data comprises the following steps:
s421, starting Map, and inputting a row of records in a log file;
s422, judging whether session is effective or not, otherwise returning to the step S431, and continuing to operate if the session is available;
s423, acquiring sessionId as a Key, using the rest fields as Value, and outputting Key/Value.
6. The visual analysis graph-based UI data processing method according to claim 5, wherein the reducing the user access sequence is completed in the Reduce stage comprises the steps of:
s421, starting with a single session as input;
s422, ordering all records in the session according to time, and adding the records into a set according to time sequence;
s423, traversing the collection, and judging the validity of each record;
s424, identifying the initial node of the session, adding the initial node to the head of the path sequence queue, sequentially identifying the subsequent nodes of the session, and adding the subsequent nodes to the path sequence queue;
s425, taking sessionId as a key, taking a path sequence queue as a value, and outputting the key/value.
7. A visual analysis graph-based UI data processing system for implementing the visual analysis graph-based UI data processing method of any one of claims 1 to 6, the system comprising: the system comprises a data reading module, a man-machine interaction module, a graph structure data optimizing module and a graph structure data processing module;
the data reading module is connected with the graph structure data optimizing module through the man-machine interaction module, and the graph structure data optimizing module is connected with the graph structure data processing module;
the data reading module constructs a distributed gateway based on Node multiprocess clusters, reads a local database and proxies remote data service;
the man-machine interaction module builds a Hadoop cluster, performs man-machine interaction preprocessing operation of visual analysis on data of a local database, and obtains graph structure data;
the graph structure data optimizing module is used for constructing a distributed storage cluster, optimizing and inquiring, merging, sorting and storing the graph structure data by utilizing a document database, and accessing the graph structure data based on a copy set;
and the graph structure data processing module is used for preprocessing graph structure data based on MapReduce.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311277055.2A CN117033365B (en) | 2023-10-07 | 2023-10-07 | Visual analysis graph-based UI data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311277055.2A CN117033365B (en) | 2023-10-07 | 2023-10-07 | Visual analysis graph-based UI data processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117033365A CN117033365A (en) | 2023-11-10 |
| CN117033365B true CN117033365B (en) | 2023-12-22 |
Family
ID=88641335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311277055.2A Active CN117033365B (en) | 2023-10-07 | 2023-10-07 | Visual analysis graph-based UI data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117033365B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104951529A (en) * | 2015-06-16 | 2015-09-30 | 焦点科技股份有限公司 | Interactive analyzing method for website logs |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8972899B2 (en) * | 2009-02-10 | 2015-03-03 | Ayasdi, Inc. | Systems and methods for visualization of data analysis |
-
2023
- 2023-10-07 CN CN202311277055.2A patent/CN117033365B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104951529A (en) * | 2015-06-16 | 2015-09-30 | 焦点科技股份有限公司 | Interactive analyzing method for website logs |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117033365A (en) | 2023-11-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10346229B2 (en) | System for decomposing events from managed infrastructures | |
| CN102880709B (en) | Data warehouse management system and data warehouse management method | |
| US20180096078A1 (en) | Visualizing convolutional neural networks | |
| EP2369480A2 (en) | Mashup infrastructure with learning mechanism | |
| KR20180087279A (en) | NETWORK TOPOLOGY Adaptive Data Visualization Method, Apparatus, Apparatus and Storage Medium | |
| WO2014179332A1 (en) | System for decomposing events from managed infrastructures | |
| US10114676B2 (en) | Building multimodal collaborative dialogs with task frames | |
| US20220116291A1 (en) | Networking data analysis in a visual spreadsheet | |
| CN114244595B (en) | Authority information acquisition method and device, computer equipment and storage medium | |
| US20180239821A1 (en) | Converting graphical data-visualizations into sonified output | |
| WO2024012221A1 (en) | Method for creating virtual service shared pool on basis of formatted data stream file base template | |
| CN103413041B (en) | A kind of collaborative perception system constituting method based on complex product development object | |
| EP3405946A1 (en) | Configurable generic language understanding models | |
| CN112329184A (en) | Network architecture configuration information generation method and device, storage medium and electronic equipment | |
| CN113742366A (en) | Data processing method and device, computer equipment and storage medium | |
| US20170060663A1 (en) | Methods for decomposing events from managed infrastructures | |
| CN112350872B (en) | Network architecture configuration information generation method and device, storage medium and electronic equipment | |
| CN117033365B (en) | Visual analysis graph-based UI data processing method and system | |
| US11080291B2 (en) | Downloading visualization data between computer systems | |
| WO2020207252A1 (en) | Data storage method and device, storage medium, and electronic apparatus | |
| CN112783779A (en) | Test case generation method and device, electronic equipment and storage medium | |
| US11159364B2 (en) | System in communication with a managed infrastructure | |
| US10901577B2 (en) | Methods and systems for input suggestion | |
| CN114816408A (en) | Information processing method and device | |
| CN115203172A (en) | Model construction method, model data subscription method, model construction device, model data subscription device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |