CN117668278A

CN117668278A - Multi-mode data chart visualization method, storage medium and electronic device

Info

Publication number: CN117668278A
Application number: CN202311846256.XA
Authority: CN
Inventors: 陈御; 郑文庭
Original assignee: Jiangxi Qiushi Higher Research Institute; Zhejiang University ZJU
Current assignee: Jiangxi Qiushi Higher Research Institute; Zhejiang University ZJU
Priority date: 2023-12-29
Filing date: 2023-12-29
Publication date: 2024-03-08

Abstract

The invention discloses a multi-modal data chart visualization method, which comprises a two-dimensional multi-modal data chart displayed in a two-dimensional screen space, a data chart displayed in a three-dimensional stereo space in the form of a bulletin board, namely, the two-dimensional screen space chart is understood to be a chart attached to a certain fixed plane specific position in the three-dimensional space, and the multi-modal data chart visualization method based on real-time interaction seamless switching of the two-dimensional screen space and the three-dimensional stereo space chart in the form of the chart; the method comprises the steps of obtaining multi-modal data of a three-dimensional model, calculating a chart position in a three-dimensional space, judging a shielding relation, automatically adjusting shielding, converting a visual angle, dynamically interpolating paths and the like, and realizing visual display and analysis of a dynamic multi-modal data chart in a two-dimensional screen space and a three-dimensional space scene based on GPU texture data. The method solves the problems of single display mode, shielding of the chart position, unnatural viewing angle conversion and the like of the conventional chart, and improves the interactivity and practicality of data visualization.

Description

Multi-mode data chart visualization method, storage medium and electronic device

Technical Field

The present invention relates to a data visualization technology, and in particular, to a real-time interactive switching analysis method, a storage medium, and an electronic device for a multi-modal data chart in a two-dimensional screen space and a three-dimensional space scene.

Background

Of the various human sensory organs, the human eye has the strongest signal processing capacity. Vision is the most important channel for information acquisition, and more than 50% of human brain functions are used for visual perception, including decoding visual information and thinking visual symbols, etc. However, human vision works with little memory and often relies on external assistance in the cognitive process. Pictures, graphics, etc. have been the most important tool in the human cognitive process for the past two hundred years. Visualization is a cognitive view of things that humans produce during the cognitive process, enhancing cognitive understanding and facilitating knowledge exchange. Visualization techniques have been used for a long time, and during the middle century, people began to use geomagnetic charts containing contours, arrow charts and astronomical charts representing the prevailing wind direction at sea. Visual analytics have been developed to address the analysis challenges of massive, high-dimensional, multi-source, and dynamic data. This is the origin of the development of data visualization. Challenges come from huge, high-order, multi-element sources and polymorphisms of data volume, and more importantly, dynamics of data acquisition, noise and contradiction of data content, heterogeneity and heterogeneity of data relationship and the like.

Wherein the data visualization is effected by visual awareness, i.e. from the view of the object to the acquisition of knowledge. For complex and large-scale data, the existing statistical analysis or data mining method is often simplification and abstraction of the data, the real structure of the data set is hidden, and the data visualization can restore or even enhance the global structure and specific details in the data. If the data visualization is regarded as an artistic creation process, the data visualization needs to be balanced truly, well and gracefully, so that information, knowledge and ideas contained in the data are effectively mined, propagated and communicated, and the balance between design and functions is realized. In this sense, data visualization shows a widely known effect.

Multimodal data refers to data that consists of multiple modes or sources of data. For example, a three-dimensional model may contain data for multiple modes of shape, color, texture, etc. Multimodal data schematics are a method of visualizing multimodal data that can be presented in a form of a schema that enables one to understand and analyze the data from multiple angles.

In the current data visualization technology, a multi-modal data chart is commonly displayed in a two-dimensional screen space. However, this manner of presentation has some limitations such as inability to vividly present the spatial structure of the three-dimensional model, inability to view data from multiple angles, and the like. To overcome these problems, researchers have begun to attempt to incorporate multimodal data charts into three-dimensional space scenes for presentation. For example: the multi-mode data synchronous visualization system provided by the patent application of publication No. CN113079411A is characterized in that when an electroencephalogram signal is acquired, synchronously recorded induction video and facial expression video of a subject are overlapped under the waveform of the electroencephalogram, recorded eye gazing positions are marked on the recorded video presenting an induction image screen and displayed in a circle mode with a fixed size, so that experimenters can detect the eye movement condition of the subject and the current corresponding state of the subject by observing the marked shape and video image information which are easier to notice; the visual analysis method for multi-mode data based on sketch interaction provided by the patent application with publication number of CN108710628A divides an original data set into a plurality of visual data structures, and matches the visual data structures with visual forms; according to the mapping relation from the original data set to the visual data structure to the visual form, decomposing the original data set into a plurality of information side surfaces with association, wherein each information side surface is presented in one view through one visual form, and generating a multi-view associated view by combining layout information selected by a user; and identifying sketch symbols drawn by a user when performing sketch circle selection operation on the multi-view associated view according to target analysis requirements, further analyzing meanings of sketch gestures formed by the sketch symbols, and generating a new view according to the meanings of the sketch gestures.

The existing display method for integrating the multi-mode data chart into the three-dimensional space scene often needs to manually adjust the position and the direction of the chart, so that the operation is complicated, and the position and the direction of the chart are difficult to be always in the optimal state. In addition, when the number of charts is large, a problem of shading between charts may occur. At the same time, when the viewing angle changes, the direction of the chart may need to be manually adjusted to accommodate the new viewing angle, which undoubtedly increases the complexity of the operation. Existing methods typically focus only on the presentation of two-dimensional screen space or the presentation of three-dimensional space scenes, lacking sufficient interactivity and practicality.

Therefore, exploring a method for automatically and interactively switching a multi-mode data chart in a two-dimensional screen space and a three-dimensional space scene and solving the problems of chart position shielding, unnatural viewing angle conversion and the like has become an important challenge to be solved in the current data visualization technology.

Disclosure of Invention

The invention aims to provide a multi-mode data chart visualization method based on real-time interaction seamless switching of two three-dimensional spaces, a two-dimensional multi-mode data chart displayed in a two-dimensional screen space, a data chart displayed in a three-dimensional space in the form of a bulletin board, namely, a two-dimensional screen space chart is understood to be a chart attached to a specific position of a certain fixed plane in the three-dimensional space, and a multi-mode data chart visualization method based on real-time interaction seamless switching of the two-dimensional screen space and the three-dimensional space chart in the form of the chart.

The invention provides a two-dimensional space multi-mode data chart visualization method based on bidirectional real-time interaction, which enables a user to realize real-time switching and display of charts between a two-dimensional screen space and a three-dimensional space scene. The method is suitable for multi-mode data charts, such as various data attributes of shapes, colors, textures and the like, and the dimension and efficiency of data analysis are improved. The invention adopts the following technical scheme:

a multi-mode data chart visualization method based on real-time interaction seamless switching of two three-dimensional spaces, a two-dimensional multi-mode data chart displayed in a two-dimensional screen space, a data chart displayed in a three-dimensional space in the form of a bulletin board, namely, the two-dimensional screen space chart is understood to be a chart attached to a certain fixed plane specific position in the three-dimensional space, and the multi-mode data chart visualization method based on real-time interaction seamless switching of the two-dimensional screen space and the three-dimensional space chart in the form of the chart;

in the real-time interaction process, aiming at the visual display of switching the two-dimensional screen space chart to the three-dimensional space and the visual display of switching the three-dimensional space chart to the two-dimensional screen space, S1-S6 steps are executed:

S1, in a real-time interaction process, a multi-mode data chart is obtained, a display form of the chart is selected through real-time interaction with a user, and a corresponding two-dimensional and three-dimensional space real-time interaction seamless switching mode is selected according to the designated display form;

s2, acquiring the current position of the chart and the placement position of the target chart, firstly acquiring the initial position in the three-dimensional space of the chart currently being displayed and recording, then acquiring the placement position of the target chart and recording, and acquiring the placement position of the target chart, wherein the method comprises the steps of calculating the possible placement position of the target chart in the corresponding space according to the two-dimensional or three-dimensional space characteristics of the target chart, determining whether bounding boxes of the chart position intersect with other chart bounding boxes to mutually block after calculating the possible placement position of the target chart, and automatically adjusting the position and the chart orientation to avoid blocking;

s3, collecting and integrating multi-modal data, selecting a proper chart type to display the data according to the characteristics and the visual requirements of the multi-modal data, calling a rendering engine, drawing a dynamic multi-modal data chart into GPU texture data, and updating pixel data in the GPU texture in real time according to the change of the data;

S4, calculating a translation and rotation dynamic interpolation path from the current position of the chart to the final placed target position of the chart according to the initial position of the chart and the placement position of the target chart obtained in the step S2, and calculating the path by using a plurality of effective interpolation methods so as to realize a smooth and natural switching process;

s5, drawing a dynamic multi-mode data chart in a three-dimensional space based on GPU texture data updated in real time and dynamic path interpolation data in the multi-mode data chart switching process;

and S6, after the multi-mode data chart target position is reached, the chart position is static, and the chart content is still refreshed dynamically.

On the basis of adopting the technical scheme, the invention can also adopt the following further technical schemes or use the further technical schemes in combination:

the step S2 is to acquire the initial position of the chart and the placement position of the target chart, firstly acquire the initial position of the chart currently being displayed and record the initial position, then acquire the placement position of the target chart, and the method to acquire the placement position of the target chart includes calculating the possible placement position of the target chart in the corresponding space according to the two-dimensional or three-dimensional space characteristics of the target chart, and after calculating the possible placement position of the target chart, determining whether bounding boxes of the chart position intersect with other bounding boxes of the chart to block each other, automatically adjusting the position and the chart orientation, and avoiding the blocking; the method specifically comprises the following steps:

S2.1, acquiring the current position of the chart, generating the current three-dimensional space position of the chart through the space characteristics of the chart selected by the user and the position of the chart in the current space in real-time interaction, and recording the current three-dimensional space position for subsequent use;

s2.2, acquiring and recording the placement position of the target chart, wherein the method for acquiring the placement position of the target chart comprises the steps of calculating the possible placement position of the target chart in the corresponding space according to the two-dimensional or three-dimensional space characteristics of the target chart, determining whether bounding boxes of the chart position intersect with other chart bounding boxes to mutually block the bounding boxes after calculating the possible placement position of the target chart, automatically adjusting the position and the chart orientation, and avoiding the blocking, and comprises the following steps:

s2.2.1, calculating the possible placement positions of the target chart in the corresponding space according to the two-dimensional or three-dimensional space characteristics of the target chart;

s2.2.2, whether bounding boxes of chart positions are intersected with other chart bounding boxes to be mutually blocked is determined by using various methods, wherein one specific method is to detect whether collision occurs between convex polygons (such as AABB, OBB, spheres and the like) by using a separation theorem. The main idea is that if there is a separation axis such that on the projection of this axis the projections of the two bounding boxes do not overlap, then the two bounding boxes are separated in 3D space, without intersecting or occluding;

S2.2.3 if there is an occlusion, automatically adjusting the chart position until there is no occlusion, using a simulated annealing algorithm for occlusion adjustment, the simulated annealing being a probabilistic search algorithm for solving the global optimization problem. The main idea is derived from the simulation of the solid annealing process, namely, after the solid is heated, particles randomly move and then slowly cool, and the particles gradually tend to be in an ordered state;

s2.2.4 the change of the orientation of the object placement of the chart is automatically performed according to the conversion of the view angle in the three-dimensional space, and the view angle conversion and the change of the orientation of the object placement are realized through quaternion and camera control by using a method integrating the view angle and the chart shielding relation of the camera, wherein the quaternion is a mathematical concept which expands complex numbers and can be used for representing the rotation in the three-dimensional space. The quaternion can avoid the gimbal lock problem, provide a more stable rotation expression, and can smoothly transition the change between two rotations by interpolation, compared to the euler angle.

Preferably, if there is an occlusion, the step S2.2.3 automatically adjusts the chart position until there is no occlusion, and uses a simulated annealing algorithm to perform occlusion adjustment, including the steps of:

(a) Initializing: the initial temperature, the cooling coefficient (typically less than 1 and close to 1) and the final temperature are set. Randomly selecting a reasonable initial chart position, and calculating the shielding degree as initial cost;

(b) The iterative process: the following steps are repeatedly performed during the process of reducing the temperature from the initial temperature to the end temperature:

(ba) perturbation: randomly selecting a new location, which may be a random disturbance at the current location;

(bb) accepting the new location if the new location is not occluded or occlusion is reduced; otherwise, the new position is accepted according to a certain probability, wherein the probability is determined by the current temperature and the shielding degree of the new position and the old position;

(bc) cooling: updating the temperature, typically by multiplying by a factor less than 1, to simulate a cooling process;

(c) Ending the iteration: when the temperature drops below the end, the iteration is ended. Returning the current solution as the final solution.

Preferably, the step S2.2.4 automatically performs the change of the orientation of the object placement of the chart according to the conversion of the view angle in the three-dimensional space, and uses a method that integrates the view angle of the camera chart and the shielding relation of the chart to implement the conversion of the view angle and the change of the orientation of the object placement through quaternion and camera control, and includes the following steps:

(a) Initializing a scene and objects: firstly, acquiring the current world space positions of a camera and a target chart in a three-dimensional space scene, as well as the view angle of the camera and the orientation of the target chart;

(b) Calculating a rotation quaternion: and calculating a rotation quaternion of the target chart under the new view angle according to the view angle of the camera and the orientation of the target chart. Multiple rotation operations may be combined using quaternion multiplication;

(c) Interpolation transition: to achieve a smooth viewing angle transition, spherical linear interpolation may be used to interpolate between the current quaternion and the target quaternion. The interpolation process may set the transition time and speed as desired. The method calculates the quaternion dot product, determines the interpolation direction, calculates the interpolation angle, and applies the quaternion scheme for adjusting the interpolation effect according to time to realize the smooth transition of the angle based on time:

(d) Updating object orientation: and applying the quaternion obtained by interpolation to the rotation transformation of the object, and updating the orientation of the object in real time.

In step S4, according to the starting position of the graph and the placement position of the target graph obtained in step S2, a translational and rotational dynamic interpolation path from the current position of the graph to the final placement target position of the graph is calculated, and a plurality of effective interpolation methods are used to calculate the path to realize a smooth and natural switching process, including the following steps:

S4.1, defining the starting and ending positions of the chart: and defining the starting position and the ending position of the chart according to the starting position and the target chart placing position of the chart acquired in the step S2. These two positions may be represented as two-dimensional or three-dimensional vectors;

s4.2, defining a control point: before calculating the spline shape, the control point group needs to be defined, the spline shape is determined by a group of control points, at least four control points (including a starting point and an ending point) are determined, but more control points are generated by random sampling to realize more complex curve shapes;

s4.3, calculating a spline curve equation: given a set of control points and a parameter t (ranging from 0 to 1), spline curve equations can be calculated using spline calculation schemes, such as one of the following formulas for calculating any point on a spline using the Catmull-Rom spline interpolation method: c (t) =0.5 [ (2×p1) +(-p0+p2) ×t+ (2×p0-5.P1+4.P2-P3) t++2(-P) 0+3P 1-3P 2+P3) t 3. Wherein P0, P1, P2 and P3 are four adjacent control points, t is a parameter, and C (t) is the position of a certain point on the Catmull-Rom spline curve calculated according to the parameter t;

s4.4, calculating translation and rotation: calculating translation and rotation of the graph by using spline curves, wherein the translation is directly determined by points on the curves, the rotation is determined by calculating tangents of the curves, the tangential direction is the direction of rotation, and the tangents are calculated by derivatives of the spline curves;

S4.5, dynamic interpolation: finally, the position and rotation of the graph are dynamically changed by changing the value of the parameter t, and when t is changed from 0 to 1, the graph moves from the starting position to the ending position along the spline curve calculated by us, and the corresponding rotation is performed.

Preferably, the specific steps of the step S5 include:

s5.1, preparing rendering parameters: and according to the dynamic path interpolation data calculated in the previous step, a translation vector and a rotation vector are included so as to realize translation and rotation effects of the chart in a three-dimensional space.

S5.2, configuring camera parameters and view settings in the three-dimensional rendering environment so as to observe the dynamic multi-mode data chart under proper angles and viewing distances;

s5.3, GPU texture data of the rendered dynamic multi-mode data chart are applied to a three-dimensional chart model, mapped to a three-dimensional plane model, and texture coordinates are associated with model vertex information;

s5.4, applying dynamic path interpolation data for the three-dimensional chart model according to the translation vector and the rotation vector which are obtained through previous calculation, and generating a transformation matrix of the new three-dimensional chart model;

and S5.5, drawing a dynamic multi-mode data chart into a three-dimensional space according to the camera, the view and the dynamic transformation setting.

According to a second aspect of the object of the present invention, there is provided a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above-described multi-modal data graph visualization method based on two-dimensional space real-time interactive seamless switching.

According to a second aspect of the object of the present invention, the electronic device comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the steps of the multi-mode data chart visualization method based on the real-time interactive seamless switching of two three-dimensional spaces are realized when the processor executes the program.

The invention has the beneficial effects that: the invention is oriented to the display requirements of complex multi-mode data association and structure and the limitations of the prior art, introduces comprehensive technologies such as a space acceleration structure, a parallel computing technology, a computing cache technology, GPU computing and the like, and a mechanism for automatically computing the placement position of a chart in a two-dimensional screen space and a three-dimensional space, automatically processes a chart shielding relation mechanism, smooth conversion of a visual angle in the three-dimensional space and an orientation change mechanism of target placement, and improves the utilization rate and interactivity of a visual space. The invention also introduces a plurality of spline interpolation techniques to calculate the translation and rotation dynamic interpolation paths, realizes smooth switching from the two-dimensional screen space to the three-dimensional space, improves the visual experience of data visualization, and ensures that the real-time interactive experience of users is better. The method improves the space utilization efficiency and the interactivity, and solves the limitation of the traditional two-dimensional chart when multi-mode data are displayed, thereby realizing more visual and efficient data visualization effect and optimizing user experience. Experimental test results show that the invention can realize the visual display of seamless smooth switching of two-dimensional space and three-dimensional space on hundreds of multi-modal data charts on a common PC machine with medium performance, the running frame rate of the system reaches more than 60 frames per second, the real-time is completely reached, and the smooth interaction of users is met in performance. The invention improves the efficiency and user experience of the real-time interaction of the multi-mode data chart, and can be conveniently integrated into any mature and complete drawing engine, and the invention has remarkable application potential.

Drawings

FIG. 1 is a schematic flow chart of the method of the present invention;

FIG. 2 is a diagram showing the visual display effect of a multi-modal data graph in a three-dimensional space according to an embodiment of the present invention;

FIG. 3 is a visual display effect diagram of a multi-mode data chart in seamless smooth switching of two three-dimensional spaces in an embodiment of the invention;

fig. 4 is a visual display effect diagram of a multi-modal data chart in a two-dimensional space according to an embodiment of the present invention.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those described herein, and therefore the present invention is not limited to the specific embodiments disclosed below.

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative and intended to explain the present invention and should not be construed as limiting the invention.

It should be noted that all directional indicators (such as up, down, left, right, front, and rear) in the embodiments of the present invention are merely used to explain the relative positional relationship, movement conditions, etc. between the components in a specific posture (as shown in the drawings), and if the specific posture is changed, the directional indicators are changed accordingly.

The flow of the multi-mode data chart visualization method based on the two-dimensional real-time interactive seamless switching of the embodiment is shown in fig. 1, and the implementation method specifically comprises the following 6 steps, including four steps of visual calculation of each frame of picture in the real-time interactive process and two preparation steps before the four steps:

(1) Defining a data source: determining a source of multimodal data; at this stage, various sources that provide data for the model need to be determined and the data processed.

This embodiment supports determining the source of the applicable multimodal data, at which stage various sources of data for the model need to be determined. This includes, but is not limited to, the following types: real-time sensor data, such as various types of sensor data for temperature sensors, pressure sensors, humidity sensors, and the like. Database information, such as relational databases, non-relational databases, etc., may provide data such as historical data, equipment information, operating parameters, etc. The image and video data can be used for image data of tasks such as artificial intelligent recognition, computer vision and the like.

Meanwhile, the real-time performance of the data sources can be guaranteed, and the data acquisition system guarantees the time consistency among the data sources through accurate clock synchronization, a low-delay communication network and other modes. And adopting a proper data transmission protocol, and selecting the proper data transmission protocol according to the stability, transmission speed and other factors of the communication environment so as to ensure the instantaneity in the data transmission process. The reliability of a data source is ensured, data preprocessing is performed, and the operations of cleaning, denoising, interpolation and the like are performed on the original data by adopting a corresponding preprocessing mode aiming at different types of data, so that invalid and abnormal data in the data are removed. And correcting and checking data, namely performing validity check on the collected data, and performing data correction and checking on possibly occurring abnormal values or error values to ensure the accuracy of a data source. And the data backup and redundancy ensure that key data can still be obtained when a certain data source fails through backup and redundancy strategies, so that the reliability of the whole system is maintained.

(2) Design data visual display scheme: to present the multimodal data diagram, a corresponding data visualization component and style is designed. Factors to be considered include data type, chart category, color, layout, typesetting, and the like. And visual elements are reasonably designed, so that user experience and data analysis effects are improved.

In this embodiment, the multi-mode data chart will select a corresponding chart type, such as a line chart, a bar chart, a pie chart, a scatter chart, a thermodynamic diagram, etc., according to the data characteristics and the display requirements. Colors are reasonably applied, different colors are adopted to represent different categories or trends, and the data readability is improved; meanwhile, the visual transmission standard is followed, color difference misleading is avoided, and a colorless color difference scheme is used. The design is clear and reasonable, so that the balance among the chart, the label and other elements is kept, and the chart, the label and other elements are easy to distinguish and read; and the spatial distribution is adjusted according to the characteristics of the data chart, so that proper margin and spacing are ensured. Typesetting details such as fonts, font sizes, line spacing and the like are kept consistent, so that the visual effect and the reading comfort level are improved; regarding the hierarchical relationship, proper typesetting techniques such as thickening and tilting are used. According to the requirements, animation, dynamic effect or real-time refreshing can be introduced, so that the data visualization is more active; meanwhile, friendly interactive interfaces such as functions of chart screening, zooming, highlighting and the like are designed, so that a user can conduct deep analysis through operation. And different equipment and terminal types are considered, so that adaptability and responsive display of the visual component in various scenes are realized, and user experience is improved. And the performance factors such as rendering speed, loading delay and the like are focused, proper measures are taken to optimize the performance of the visual component, and the waiting time of a user is reduced.

In the subsequent real-time interaction process, the steps S1-S6 are needed to be executed for the visual calculation of each frame of picture:

s1, acquiring a multi-mode data chart in a real-time interaction process, selecting a display form of the chart through real-time interaction with a user, and selecting a corresponding two-dimensional and three-dimensional space real-time interaction seamless switching mode according to the designated display form.

In the embodiment, in the real-time interaction process, the multi-mode data chart of the three-dimensional model is dynamically collected and optimized by capturing various operations of a user and according to the interaction mode supported by equipment, and the multi-mode data chart is presented to the user in real time in a high-quality visual component and style so as to provide deep data analysis, information transmission and real-time feedback.

S1.1, capturing user interaction operation: during the real-time interaction, various operations of the user, such as clicking, dragging, zooming, selecting, etc., are recorded. Capturing data may be used to analyze user behavior, enhance data visualization effects, or trigger other functions;

s1.2, updating the model and the data chart in real time: according to the interactive operation of the user, updating the multi-mode data of the operated object in the three-dimensional model, ensuring the real-time performance of data transmission and processing, and enabling the user to obtain feedback and results in time in the interactive process;

S1.2, selecting a corresponding two-dimensional space real-time interactive seamless switching mode according to a chart display form selected by a user in real time and a display form of a designated target chart, for example, when the user currently selects a chart of a two-dimensional screen space and sends out a command for switching the display space, the system can select a mode of seamlessly switching from the two-dimensional screen space to the three-dimensional space and execute switching operation on the target chart;

s2, acquiring the current position of the chart and the placement position of the target chart; firstly, acquiring a starting position and record in a three-dimensional space of a chart currently being displayed, and then acquiring a placement position and record of a target chart; the method for obtaining the placement position of the target chart comprises the steps of calculating the possible placement position of the target chart in a corresponding space according to the two-dimensional or three-dimensional space characteristics of the target chart, determining whether bounding boxes of the chart position intersect with other chart bounding boxes to be blocked or not after calculating the possible placement position of the target chart, automatically adjusting the position and the chart orientation, and avoiding blocking; the main steps include:

s2.2.2, whether bounding boxes of the chart positions intersect with other chart bounding boxes to be blocked or not is determined by combining a plurality of methods, and a specific method adopted in the embodiment is to detect whether collision occurs between convex polygons (such as AABB, OBB, spheres, etc.) by using a separation theorem. The main idea is that if there is a separation axis such that on the projection of this axis the projections of the two bounding boxes do not overlap, then the two bounding boxes are separated in 3D space, without intersecting or occluding; the specific process of step S2.2.2 can be divided into the following sub-steps:

(a) Determining all potential separation axes, and calculating 15 potential separation axes when processing the OBB;

(b) Calculating projections for each potential separation axis, firstly calculating projections of central points of A and B on the axis, and secondly calculating half expansion range of A and B on the axis;

(c) The overlap is checked and for each separation axis, it is checked whether the projections of the two OBBs overlap. If one separation axis exists, so that the absolute value of the projection difference value is larger than the sum of the expansion ranges of the two separation axes, the projections on the separation axes are not overlapped, and the two OBBs do not collide in the 3D space;

(d) Judging whether the projections are intersected or not, and if the projections on all the separation axes are overlapped, intersecting the two OBBs;

(e) If the length of the separation axis vector obtained in calculating the outlier is close to zero (e.g., the length is less than some threshold), then the separation axis is ignored because the separation axis at this time may be approximately parallel. To check if the length of the separation axis vector is close to zero, it can be compared to the square length of the threshold to avoid the evolution calculation.

S2.2.3 if there is an occlusion, automatically adjusting the chart position until there is no occlusion, using a simulated annealing algorithm for occlusion adjustment, the simulated annealing being a probabilistic search algorithm for solving the global optimization problem. The main idea is derived from the simulation of the solid annealing process, namely, after the solid is heated, particles randomly move and then slowly cool, and the particles gradually tend to be in an ordered state; the specific process of step S2.2.3 can be divided into the following sub-steps:

(a) Initializing: an initial temperature t_initial, a cooling coefficient cooling_factor (typically less than 1 and close to 1), and a final temperature t_final are set. Randomly selecting a reasonable initial chart position current_solution, and calculating the shielding degree of the current_solution as an initial cost current_cost;

(b) The iterative process: the following steps are repeatedly executed in the process of reducing the temperature from T_initial to T_final:

(ba) perturbation: randomly selecting a new position new_solution, which can be a random disturbance at the current position current_solution;

(bb) calculating a cost change, determining whether to new the solution and updating the solution and the cost: the occlusion degree new_cost of the new position new_solution is calculated. Calculating a cost change delta_cost = new_cost-current_cost, accepting the new solution if delta_cost < = 0 (i.e., the new solution is better than or equal to the current solution); otherwise, accepting the new solution according to a certain probability. The probability of acceptance is acceptance_prob=exp (-delta_cost/T), where exp represents an exponential function. A random number random_number (evenly distributed between 0 and 1) may be used to determine whether to accept a new solution: if random_number < accept_prob then accept new solution, otherwise keep current solution, if accept new solution then update current_solution = new_solution and current_cost = new_cost;

(bc) cooling: updating the temperature t=t cooling_factor to simulate the cooling process;

(c) Ending the iteration: when the temperature falls below t_final, the iteration is ended. Returning the current solution as the final solution.

S2.2.4, automatically changing the orientation of the placement of the chart object according to the conversion of the view angle in the three-dimensional space; the method for integrating the view angle and the graph shielding relation of the camera graph is used for realizing view angle conversion and orientation change of target placement through quaternion and camera control, and the quaternion is an expanded complex mathematical concept and can be used for representing rotation in a three-dimensional space. Compared with Euler angles, the quaternion can avoid the problem of universal joint locking, provide more stable rotation expression, and can smoothly transition the change between two rotations through interpolation; the specific process of step S2.2.4 can be divided into the following sub-steps:

(c) Interpolation transition: smooth viewing angle conversion between quaternions is achieved by a number of methods, one of which uses spherical linear interpolation to interpolate between the current quaternion and the target quaternion. The interpolation process may set the transition time and speed as desired. Calculating a quaternion dot product, determining an interpolation direction, calculating an interpolation angle, and applying a quaternion scheme for adjusting an interpolation effect according to time to realize time-based angle smooth transition;

S3, calling a rendering engine according to the acquired multi-modal data chart content, drawing a dynamic multi-modal data chart into GPU texture data, and updating pixel data in GPU textures in real time according to the change of the data; the specific process of step S3 may be divided into the following sub-steps:

s3.1, mapping data to chart elements: for a selected chart type (e.g., line graph, bar graph, pie graph, etc.), the multimodal data is mapped onto individual elements of the chart. The method comprises the steps of determining the corresponding relation and numerical range of visual elements such as coordinate axes, colors, labels and the like;

S3.2, creating GPU textures: a block of texture space is created in the GPU for storing pixel data of the chart. Setting a texture size and a color format matched with the chart according to the size, the precision and the color depth of the chart;

s3.3, rendering the chart to texture: and calling a rendering engine to render the pixel data of the chart into the GPU texture. Creating a vertex buffer area according to vertex data of the chart, uploading the vertex buffer area to the GPU, setting a rendering pipeline state, setting a proper mixing mode for a rendering process, associating the created GPU texture with the chart, calling a rendering engine, and rendering pixel data of the chart into the GPU texture;

and S3.4, updating pixel data in the GPU texture in real time according to the change of the data so as to reflect the dynamic change of the data. Monitoring the change of the multi-mode data in real time through event monitoring, remapping and locally rendering chart elements when the data change, uploading the changed chart pixel data to the GPU texture in real time, and reflecting the dynamic change of the data;

s4, calculating a translation and rotation dynamic interpolation path from the current position of the chart to the final placed target position of the chart according to the initial position of the chart and the placement position of the target chart obtained in the step S3, and calculating the path by using a plurality of effective interpolation methods so as to realize a smooth and natural switching process;

In this embodiment, step S4 calculates the path using various spline interpolation methods. One such method is the Catmull-Rom spline interpolation method, which is a smooth interpolation method that computes a smooth curve from a set of control points. In the present invention, the control points may be a start position, a target position, and an intermediate position of the graph. Through a spline interpolation method, translation and rotation paths of the graph in the switching process can be calculated to realize a smooth and natural switching process, and the specific process can be divided into the following sub-steps:

s4.1, defining a start and end position: the start position startPos (e.g., two-dimensional screen space) and the end position endPos (e.g., specific position in three-dimensional space) of the chart are defined according to the start position and the target chart placement position of the chart acquired in step S2. These two positions may be represented as two-dimensional or three-dimensional vectors;

s4.2, defining a control point: before calculating the spline shape, the control points are defined, the spline shape is determined by a group of control points, at least four control points (including a starting point and an ending point) are determined, but more control points are generally randomly sampled and generated to realize a more complex curve shape, and when the control points are determined, not only the total number of the control points is required to be selected, but also the specific position of each control point is required to be determined;

S4.3, calculating a spline curve equation: given a set of control points and a parameter t (ranging from 0 to 1), spline curve equations are calculated using spline calculation schemes, such as the Catmull-Rom spline interpolation method in this embodiment, the formula for calculating any point on a spline is as follows: c (t) =0.5 [ (2×p1) +(-p0+p2) ×t+ (2×p0-5.P1+4.P2-P3) t++2(-P) 0+3P 1-3P 2+P3) t 3. Wherein P0, P1, P2 and P3 are four adjacent control points, t is a parameter, and C (t) is the position of a certain point on the Catmull-Rom spline curve calculated according to the parameter t;

s4.5, dynamic interpolation: finally, the position and rotation of the graph are dynamically changed by changing the value of the parameter t, and when t is changed from 0 to 1, the graph is moved from the starting position to the ending position along the calculated spline curve while the corresponding rotation is performed.

In this embodiment, step S5 is based on the computation performance of the Nvidia RTX 2080GPU and the graphics rendering engine of the UE5 to achieve the correct multi-modal data chart coloring and drawing effect, and the specific process may be divided into the following sub-steps:

s5.1, preparing rendering parameters: the dynamic path interpolation data (comprising translation vectors and rotation vectors) calculated in the previous step are used for realizing translation and rotation effects of the chart in the three-dimensional space.

S5.2, camera and view are set: camera parameters (e.g., camera position, viewing angle, etc.) and view settings (e.g., clipping plane, viewing cone, etc.) in a three-dimensional rendering environment are configured to view a dynamic multimodal data diagram at an appropriate angle and viewing distance.

S5.3, creating a three-dimensional chart model: and (3) applying the GPU texture data of the dynamic multi-mode data chart rendered in the step (S5) to a three-dimensional chart model, mapping the GPU texture data to a three-dimensional plane model, and associating texture coordinates with model vertex information.

S5.4, performing dynamic transformation: and applying dynamic path interpolation data to the three-dimensional chart model according to the translation vector and the rotation vector which are calculated previously, and generating a transformation matrix of the new three-dimensional chart model.

S5.5, rendering a dynamic multi-modal data chart: the dynamic multi-modal data graph is drawn into three-dimensional space according to the camera, view, and dynamic transformation settings. The rendering sequence is followed in the rendering process to prevent the shielding problem, and meanwhile, parameters such as transparency, color and the like are properly set to achieve better visual effect.

S6.1, monitoring target positions: detecting when the dynamic multi-mode data chart reaches a target position, namely the translation vector and the rotation vector reach preset end point parameters. By judging whether the interpolation parameter t' is close to or equal to 1 (indicating the end position) or not.

S6.2, fixing the chart position and rotation: after the multi-modal data map is determined to have reached the target position, its position and rotation are set to a stationary state. I.e. stopping the interpolation calculation on the path, keeping its position and rotation values stable.

S6.3, continuously rendering and updating chart contents: although the chart position and rotation are stationary, the GPU texture data still needs to be updated in real-time in order to dynamically update the chart contents. And calling a rendering engine to continuously redraw the chart, and drawing the chart into the GPU texture by new real-time data.

S6.4, optimizing performance: in order to ensure superior performance in dynamic refreshing of chart content, appropriate optimization of data and images may be considered. For example, for frequently changing parts, efficient algorithms are used for processing; for image rendering, mipmapping or LOD techniques may be used for optimization.

S6.5, interaction and feedback: user interaction is provided so that the user can continue to explore and analyze when the chart is stationary. For example, allowing the user to zoom, scroll, or display additional data in the chart, etc. Meanwhile, the user operation is responded in real time, and the chart content is updated, so that the chart feedback is accurate.

The embodiment can achieve the visual display visual frame rate of seamless smooth switching of two three-dimensional spaces with more than 60 frames per second through testing.

According to a second aspect of the object of the present invention, there is provided a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps S1-S6 of the above-described multi-modal data graph visualization method based on two-dimensional space real-time interactive seamless switching.

The computer readable storage medium may include: any entity or device capable of carrying computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth.

According to a second aspect of the object of the present invention, an electronic device according to the present invention comprises a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps S1-S6 of the multi-modal data graph visualization method based on two-dimensional space real-time interactive seamless switching are realized when the processor executes the program.

The processor may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The memory may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The memory may also be an external storage device of the computer device, such as a plug-in hard disk provided on the computer device, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like.

The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims

1. A method for visualizing a multi-modal data graph, comprising the steps of:

s2, acquiring the current position of the chart and the placement position of the target chart, firstly acquiring the initial position and recording in the three-dimensional space of the chart currently being displayed, then acquiring the placement position and recording of the target chart, and the method for acquiring the placement position of the target chart comprises the steps of calculating the possible placement position of the target chart in the corresponding space according to the two-dimensional or three-dimensional space characteristics of the target chart, determining whether bounding boxes of the chart position intersect with other chart bounding boxes to shield each other, automatically adjusting the position and the chart orientation, and avoiding shielding;

s4, calculating a translation and rotation dynamic interpolation path from the current position of the chart to the final placed target position of the chart according to the initial position of the chart and the placement position of the target chart obtained in the step S2;

2. The method for visualizing a multi-modal data chart according to claim 1, wherein in step S2, the current position of the chart is obtained, and the current three-dimensional spatial position of the chart is generated and recorded by the spatial characteristics of the chart selected by the user and the position of the chart in the current space in real-time interaction.

3. The method for visualizing a multimodal data chart as in claim 2, wherein said step S2 comprises the specific steps of:

s2.2.1, calculating the possible placement position of the target chart in the corresponding space according to the two-dimensional or three-dimensional space characteristics of the target chart;

s2.2.2, a method for detecting whether collision occurs between convex polygons by using a separation theorem is used, and judging whether bounding boxes of chart positions intersect with other chart bounding boxes to shield each other;

s2.2.3, if the shielding exists, automatically adjusting the position of the chart until the shielding does not exist, and performing shielding adjustment by using a simulated annealing algorithm;

s2.2.4, automatically changing the direction of the object placement of the chart according to the conversion of the view angle in the three-dimensional space, and specifically, using a method for integrating the view angle and the chart shielding relation of the chart of the camera, realizing the conversion of the view angle and the direction change of the object placement through quaternion and camera control.

4. A multi-modal data chart visualization method as described in claim 3 wherein said step S2.2.3 of using a simulated annealing algorithm to effect occlusion adjustment comprises the steps of:

(a) Setting an initial temperature, a cooling coefficient and a termination temperature, randomly selecting a reasonable initial chart position, and calculating the shielding degree as initial cost;

(b) The following steps are repeatedly performed during the process of reducing the temperature from the initial temperature to the end temperature:

(ba) randomly selecting a new location, which may be a random disturbance at the current location;

(bb) accepting the new location if the new location is not occluded or occlusion is reduced; otherwise, accepting the new position according to the probability, wherein the probability is determined by the current temperature and the shielding degree of the new position and the old position;

(bc) updating the temperature by multiplying a factor less than 1 to simulate a cooling process;

(c) When the temperature drops below the end, the iteration is ended.

5. A multi-modal data chart visualization method as claimed in claim 3 wherein said step S2.2.4 automatically changes the orientation of the chart object placement based on the transition of the viewing angle in three-dimensional space, comprising the steps of:

(a) Initializing a scene and objects: the current world space positions of the camera and the target chart in the three-dimensional space scene, as well as the visual angle of the camera and the orientation of the target chart need to be acquired;

(b) Calculating a rotation quaternion: according to the visual angle of the camera and the direction of the target chart, calculating a rotation quaternion of the target chart under the new visual angle;

(c) Interpolation transition: interpolating between the current quaternion and the target quaternion using spherical linear interpolation;

6. The method for visualizing a multi-modal data chart according to claim 1, wherein said step S3 specifically comprises:

s3.1, mapping data to chart elements: mapping the multimodal data onto individual elements of the chart for the selected chart type;

s3.2, creating GPU textures: creating a block of texture space in the GPU for storing pixel data of the chart; setting a texture size and a color format matched with the chart according to the size, the precision and the color depth of the chart;

s3.3, rendering the chart to texture: invoking a rendering engine to render the pixel data of the chart into the GPU texture; creating a vertex buffer area according to vertex data of a chart, uploading the vertex buffer area to a GPU, setting a rendering pipeline state, setting a proper mixing mode for a rendering process, associating the created GPU texture with the chart, calling a rendering engine, and rendering pixel data of the chart into the GPU texture;

and S3.4, updating pixel data in the GPU texture in real time according to the change of the data so as to reflect the dynamic change of the data.

7. The method for visualizing a multi-modal data chart according to claim 1, wherein the step S4 is performed by using a plurality of spline interpolation methods to calculate the path, and specifically comprises:

s4.1, defining the starting and ending positions of the chart: defining a starting position and an ending position of the chart according to the starting position of the chart and the target chart placing position obtained in the step S2;

s4.2, defining a control point: before calculating the spline shape, defining a control point group, determining the spline shape by a group of control points, and determining at least four control points;

s4.3, calculating a spline curve equation: given a group of control points and a parameter t, calculating a spline curve equation by using a Catmull-Rom spline interpolation method;

s4.5, dynamic interpolation: the position and rotation of the graph are dynamically changed by changing the value of the parameter t, and when t is changed from 0 to 1, the graph moves from the starting position to the ending position along the spline curve calculated by us, and the corresponding rotation is performed.

8. The method for visualizing a multi-modal data chart according to claim 1, wherein the specific step of step S5 comprises:

s5.1, preparing rendering parameters: according to the dynamic path interpolation data calculated in the previous step, a translation vector and a rotation vector are included so as to realize translation and rotation dynamic effects of the chart in a three-dimensional space;

9. A computer readable storage medium having stored thereon a computer program, which when executed by a processor performs the steps of the multi-modal data chart visualization method of any of claims 1 to 8.

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor performs the steps of the multi-modal data chart visualization method of any one of claims 1 to 8 when the program is executed by the processor.