CN115457220A - Simulator multi-screen view simulation method based on dynamic viewpoint - Google Patents

Simulator multi-screen view simulation method based on dynamic viewpoint Download PDF

Info

Publication number
CN115457220A
CN115457220A CN202211407435.9A CN202211407435A CN115457220A CN 115457220 A CN115457220 A CN 115457220A CN 202211407435 A CN202211407435 A CN 202211407435A CN 115457220 A CN115457220 A CN 115457220A
Authority
CN
China
Prior art keywords
screen
coordinate system
head
point cloud
formula
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202211407435.9A
Other languages
Chinese (zh)
Other versions
CN115457220B (en
Inventor
朱震宇
李兴德
孙靖
刘伟伟
魏世博
徐建军
韦洋
祝小康
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Yutianzhiyun Simulation Technology Co ltd
Army Engineering University of PLA
Original Assignee
Nanjing Yutianzhiyun Simulation Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Yutianzhiyun Simulation Technology Co ltd filed Critical Nanjing Yutianzhiyun Simulation Technology Co ltd
Priority to CN202211407435.9A priority Critical patent/CN115457220B/en
Publication of CN115457220A publication Critical patent/CN115457220A/en
Application granted granted Critical
Publication of CN115457220B publication Critical patent/CN115457220B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Geometry (AREA)
  • Architecture (AREA)
  • Processing Or Creating Images (AREA)

Abstract

The invention discloses a simulator multi-screen view simulation method based on a dynamic viewpoint, which comprises the steps of firstly respectively arranging Kinect sensors at two sides of a multi-screen to collect head original point cloud data of a driving learner, and carrying out conditional filtering and denoising treatment on the collected original point cloud data to obtain processed point cloud data; the head pose estimation method used by the invention can track the head in real time, can simultaneously detect the head and estimate the pose parameters of the head, can update the position and the angle of a screen virtual viewpoint in real time through the change of the head pose of a driving student, realizes a highly realistic virtual scene, is particularly sensitive to noise by aiming at a dynamic viewpoint system, enhances the stability of projection parameters by using Accela filtering, and carries out smooth processing on a motion curve of the head parameters, can avoid the influence of jitter on the content of the screen view in the process of head motion, and is suitable for wide popularization and use.

Description

Simulator multi-screen view simulation method based on dynamic viewpoint
Technical Field
The invention relates to the technical field of view simulation splicing, in particular to a simulator multi-screen view simulation method based on dynamic viewpoints.
Background
Aircraft simulators have appeared late in the simulator field, but over time, the first aircraft simulator was developed in the early 60's of the last century. Since then, a large amount of manpower and material resources are input in each country, and the development and the rapid development of the aircraft simulator are promoted. The countries such as English and American carry out power modeling on the aircraft at the earliest, the popularization and the application of computer networks increase a new thought for analog simulation, and great progress is made in the aspect of high-precision motion calculation. The appearance of virtual technology provides better visual function for the simulator for the operation personnel have the sensation of being personally on the scene, can reach the same training effect in real scene, avoid unnecessary casualties simultaneously. Nowadays, aircraft simulators evolved from early mechanical simulation models to advanced simulation training devices now with 360 ° spherical view, six-degree-of-freedom simulation functions, and a combination of mechanical and electronic technologies. The vision system is used for simulating the scene outside the window of the aircraft cockpit in real time, providing visual information for the pilot and creating a real flight environment.
At present, most of the existing analog simulation systems adopt a perspective projection mode of a fixed viewpoint, and once the geometric size and the placing position of a display screen are determined, all parameters of a projection matrix are kept unchanged in the simulation process; the virtual visual scene displayed by the projection screen can not be dynamically updated along with the change of the eyes or the viewpoint of the flight trainee relative to the position of the cockpit, and the imaging content and the imaging effect have larger difference compared with the effect of observing the external environment of the trainee through the cockpit window under the real condition; therefore, it is necessary to design a simulator multi-screen view simulation method based on dynamic viewpoints.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, and better and effectively solves the problems that the prior analog simulation system mostly adopts a perspective projection mode with a fixed viewpoint, and once the geometric size and the placing position of a display screen are determined, all parameters of a projection matrix are kept unchanged in the simulation process; the method for simulating the multi-screen visual scene of the simulator based on the dynamic viewpoints solves the problems that the virtual visual scene displayed by a projection screen cannot be dynamically updated along with the change of the eyes or the viewpoint of a flight student relative to the position of a cockpit, and the imaging content and the imaging effect are different from the effect of observing the external environment by the student through the cockpit window under the real condition.
In order to achieve the purpose, the invention adopts the technical scheme that:
the simulator multi-screen visual simulation method based on the dynamic viewpoint comprises the following steps,
step (A), respectively placing a Kinect sensor at two sides of a multi-screen to acquire head original point cloud data of a driving student, and performing conditional filtering and denoising processing on the acquired original point cloud data to obtain processed point cloud data;
step (B), adopting a three-surface target algorithm to carry out external reference rotation matrix on the Kinect sensor
Figure 236035DEST_PATH_IMAGE001
And translation vector
Figure 418754DEST_PATH_IMAGE002
Calibrating and unifying the processed point cloud data to a global coordinate system to complete the fusion of the processed point cloud data of the two Kinect sensors;
step (C), based on the fusion of the processed point cloud data, adopting
Figure 667333DEST_PATH_IMAGE003
The algorithm obtains head pose parameters in real time, and completes the detection of the face area of the driving learner and the estimation of the head pose to obtain an object model of the virtual three-dimensional scene;
step (D), mapping the obtained virtual three-dimensional scene object model to a screen coordinate system for visualization through visual transformation, projective transformation, perspective division and viewport transformation;
and (E) respectively calculating a perspective matrix from the virtual viewpoint to each screen in a 3D space according to the actual placement position and the size of each screen, enhancing the stability of the projection parameters based on Accela filtering, and smoothing the motion curve of the head parameters to complete the multi-screen view simulation work of the simulator.
Preferably, in the step (A), a Kinect sensor is respectively arranged at two sides of a multi-screen to collect the head original point cloud data of the driving student, and the collected original point cloud data is subjected to conditional filtering and denoising to obtain processed point cloud data, and the specific steps are as follows,
step (A1) of performing conditional filtering on the acquired original point cloud data and using a conditional filtering algorithm to perform conditional filtering on the point cloud data
Figure 403820DEST_PATH_IMAGE004
A shaft,
Figure 421455DEST_PATH_IMAGE005
Shaft and
Figure 509497DEST_PATH_IMAGE006
the axes are processed to filter out useless information and background of each axis,
Figure 370005DEST_PATH_IMAGE007
filtering of the shaft as shown in equation (1),
Figure 772168DEST_PATH_IMAGE008
wherein the content of the first and second substances,
Figure 706626DEST_PATH_IMAGE009
，
Figure 778618DEST_PATH_IMAGE010
representing a conditionally filtered set of point cloud data,
Figure 1789DEST_PATH_IMAGE011
representing an input raw point cloud data set;
step (A2), denoising the acquired original point cloud data, using a bilateral filtering algorithm as shown in formula (2),
Figure 4380DEST_PATH_IMAGE012
wherein, the first and the second end of the pipe are connected with each other,
Figure 855661DEST_PATH_IMAGE013
representing the point cloud characteristic information before bilateral filtering,
Figure 754347DEST_PATH_IMAGE014
representing the point cloud characteristic information acquired by the bilateral filtering algorithm,
Figure 261552DEST_PATH_IMAGE015
the overall weight parameter is represented by a value,
Figure 615304DEST_PATH_IMAGE016
representing the characteristic information of a normal vector before point cloud bilateral filtering; and the overall weight parameter
Figure 524354DEST_PATH_IMAGE017
The specific steps of the definition are as follows,
a step (A21) of,
Figure 328362DEST_PATH_IMAGE018
is defined as shown in the formula (3),
Figure 385180DEST_PATH_IMAGE019
wherein the content of the first and second substances,
Figure 932836DEST_PATH_IMAGE020
the representation feature holds a weight calculation function,
Figure 696392DEST_PATH_IMAGE021
represents a smooth filtering weight calculation function,
Figure 546668DEST_PATH_IMAGE022
indicating points
Figure 762886DEST_PATH_IMAGE023
Is/are as follows
Figure 848653DEST_PATH_IMAGE024
The number of the neighboring values is,
Figure 794613DEST_PATH_IMAGE025
indicating points
Figure 2740DEST_PATH_IMAGE026
To any point in its neighborhood
Figure 643937DEST_PATH_IMAGE027
The distance of the vector of (a) to the target,
Figure 330133DEST_PATH_IMAGE028
indicating points
Figure 143981DEST_PATH_IMAGE029
Normal vector feature information in a neighboring region;
step (A22), in the formula (3)
Figure 460693DEST_PATH_IMAGE030
And
Figure 651503DEST_PATH_IMAGE031
as shown in equation (4) and equation (5), respectively,
Figure 469286DEST_PATH_IMAGE032
Figure 734045DEST_PATH_IMAGE033
wherein the content of the first and second substances,
Figure 283975DEST_PATH_IMAGE034
and
Figure 40710DEST_PATH_IMAGE035
indicating points
Figure 271971DEST_PATH_IMAGE036
Gaussian filter coefficients of the tangent plane.
Preferably, in the step (B), a three-plane target algorithm is adopted to carry out the external reference rotation matrix on the Kinect sensor
Figure 719133DEST_PATH_IMAGE037
And translation vector
Figure 502281DEST_PATH_IMAGE038
Calibrating and unifying the processed point cloud data to be completeThe local coordinate system completes the fusion of the point cloud data processed by the two Kinect sensors, and the specific steps are as follows,
step (B1) of setting a vector
Figure 605366DEST_PATH_IMAGE039
As vectors in a three-dimensional coordinate system, i.e.
Figure 437056DEST_PATH_IMAGE040
，
Figure 82932DEST_PATH_IMAGE041
Are points of a three-dimensional coordinate system, and
Figure 646769DEST_PATH_IMAGE042
in a coordinate system
Figure 299467DEST_PATH_IMAGE043
And
Figure 997164DEST_PATH_IMAGE044
are respectively defined as
Figure 356602DEST_PATH_IMAGE045
And
Figure 419236DEST_PATH_IMAGE046
to, for
Figure 106700DEST_PATH_IMAGE047
And
Figure 749034DEST_PATH_IMAGE048
carrying out normalization processing to obtain unit vector
Figure 25294DEST_PATH_IMAGE049
And
Figure 321147DEST_PATH_IMAGE050
and unit vector
Figure 682858DEST_PATH_IMAGE051
And
Figure 863303DEST_PATH_IMAGE052
as shown in the formula (6),
Figure 804190DEST_PATH_IMAGE053
wherein the content of the first and second substances,
Figure 411889DEST_PATH_IMAGE054
representing an external reference rotation matrix;
a step (B2) of setting a rotation vector
Figure 260896DEST_PATH_IMAGE055
Formed matrix
Figure 838508DEST_PATH_IMAGE056
And matrix are
Figure 89361DEST_PATH_IMAGE057
Can make
Figure 867961DEST_PATH_IMAGE058
Matrix of
Figure 17314DEST_PATH_IMAGE059
As shown in the formula (7),
Figure 336300DEST_PATH_IMAGE060
wherein the content of the first and second substances,
Figure 379342DEST_PATH_IMAGE061
representing a third order identity matrix;
step (B3) of using
Figure 453477DEST_PATH_IMAGE062
Group of vectors is in
Figure 11498DEST_PATH_IMAGE063
And
Figure 71857DEST_PATH_IMAGE064
unit vector in coordinate system
Figure 110352DEST_PATH_IMAGE065
And
Figure 27492DEST_PATH_IMAGE066
can solve the rotation matrix
Figure 276071DEST_PATH_IMAGE067
Is provided with
Figure 936859DEST_PATH_IMAGE068
，
Figure 79128DEST_PATH_IMAGE069
Then rotation matrix
Figure 636011DEST_PATH_IMAGE070
The obtained process is shown as formula (8), formula (9) and formula (10),
Figure 981673DEST_PATH_IMAGE071
Figure 446152DEST_PATH_IMAGE072
Figure 52714DEST_PATH_IMAGE073
wherein the content of the first and second substances,
Figure 905132DEST_PATH_IMAGE074
，
Figure 925041DEST_PATH_IMAGE075
which is indicative of the vector of rotation of the,
Figure 865315DEST_PATH_IMAGE076
represent
Figure 654280DEST_PATH_IMAGE077
，
Figure 425402DEST_PATH_IMAGE078
Represent
Figure 604711DEST_PATH_IMAGE079
；
Step (B4) of setting a three-dimensional space point D at
Figure 410993DEST_PATH_IMAGE080
And
Figure 647939DEST_PATH_IMAGE081
are respectively defined as
Figure 186368DEST_PATH_IMAGE082
And
Figure 180868DEST_PATH_IMAGE083
then, then
Figure 603891DEST_PATH_IMAGE084
Is reused
Figure 305130DEST_PATH_IMAGE085
Point averaging, translation vector
Figure 76777DEST_PATH_IMAGE086
As shown in equation (11),
Figure 886470DEST_PATH_IMAGE087
wherein the content of the first and second substances,
Figure 972238DEST_PATH_IMAGE088
representing translation vectors，
Figure 590301DEST_PATH_IMAGE089
Representing a series of coordinate points obtained under the Kinect1 sensor,
Figure 877057DEST_PATH_IMAGE090
a series of coordinate points obtained under the Kinect2 sensor are represented.
Preferably, step (C) is based on the fusion of the processed point cloud data, and
Figure 518254DEST_PATH_IMAGE091
the algorithm obtains the head pose parameters in real time, completes the face area detection and the head pose estimation of the driving trainees, obtains an object model of the virtual three-dimensional scene, and comprises the following specific steps,
step (C1), a loss function is established, the loss function is used for reflecting the error between the prediction result of the model for the sample and the actual label of the sample, and the total loss function of the model is
Figure 204450DEST_PATH_IMAGE092
、
Figure 4916DEST_PATH_IMAGE093
And
Figure 321628DEST_PATH_IMAGE094
the sum of these three hierarchical feature map loss functions, the total loss function of the model is shown in equation (12),
Figure 512438DEST_PATH_IMAGE095
wherein the content of the first and second substances,
Figure 815374DEST_PATH_IMAGE096
the function of the total loss is expressed as,
Figure 611292DEST_PATH_IMAGE097
、
Figure 161222DEST_PATH_IMAGE098
and
Figure 901645DEST_PATH_IMAGE099
respectively represent
Figure 929644DEST_PATH_IMAGE100
、
Figure 580068DEST_PATH_IMAGE101
And
Figure 134457DEST_PATH_IMAGE102
loss functions for the three hierarchical layers;
step (C2), the loss function of each layer contains the position deviation for calculating the face area
Figure 299859DEST_PATH_IMAGE103
Part for calculating classification errors
Figure 334811DEST_PATH_IMAGE104
Part and method for determining whether a target object is contained in a face region
Figure 698796DEST_PATH_IMAGE105
In part, the objective function of each layer is shown in equation (13),
Figure 590529DEST_PATH_IMAGE106
wherein, the first and the second end of the pipe are connected with each other,
Figure 446489DEST_PATH_IMAGE107
and
Figure 629340DEST_PATH_IMAGE108
the Sigmoid is used as an activation function, and corresponding results are converted into probability values;
step (C3), each part of each layer of function adopts cross entropy as a loss function, as shown in formula (14),
Figure 51094DEST_PATH_IMAGE109
wherein the content of the first and second substances,
Figure 51411DEST_PATH_IMAGE110
the value of the loss function of the output is expressed,
Figure 253722DEST_PATH_IMAGE111
the actual label corresponding to the sample is represented,
Figure 692794DEST_PATH_IMAGE112
indicating the corresponding prediction result of the sample.
Preferably, in step (D), the obtained virtual three-dimensional scene object model is mapped to a screen coordinate system for visualization through view transformation, projection transformation, perspective division and viewport transformation, wherein the view transformation is to convert a world coordinate system into a camera coordinate system, the projection transformation is to map three-dimensional coordinates into two-dimensional coordinates, and the perspective division is to map three-dimensional coordinates into two-dimensional coordinates
Figure 906738DEST_PATH_IMAGE113
The component becomes 1 and the viewport transformation is to convert the processed coordinates to screen coordinate system space.
Preferably, step (E) is to calculate a perspective matrix from the virtual viewpoint to each screen in the 3D space according to the actual placement position and size of each screen, enhance the stability of the projection parameters based on the Accela filter, and smooth the head parameter motion curve to complete the multi-screen view simulation of the simulator, which comprises the following specific steps,
step (E1), a fixed viewpoint mode is adopted, a head coordinate system and a screen coordinate system are mapped into the same world coordinate system, a perspective matrix is obtained by calculating a transformation matrix of the head coordinate system and the screen coordinate system, and the splicing work of multiple screens is completed;
step (E2), based on fixed viewpoint multi-screen splicing, calculating a perspective matrix from each frame of virtual viewpoint to each screen in real time according to a dynamic viewpoint technology;
and (E3) enhancing the stability of the projection parameters based on Accela filtering, and smoothing the motion curve of the head parameters.
Preferably, the specific steps of step (E1) are as follows,
step (E11), a calculation formula of the perspective matrix is constructed, as shown in formula (15),
Figure 218901DEST_PATH_IMAGE114
wherein the content of the first and second substances,
Figure 580612DEST_PATH_IMAGE115
representing the distance of the virtual viewpoint to the near clipping plane,
Figure 229900DEST_PATH_IMAGE116
representing the distance from the virtual viewpoint to the far clipping plane;
and (E12) solving the constructed perspective matrix, wherein the concrete steps are as follows,
step (E121) of obtaining the center coordinates of each screen
Figure 954142DEST_PATH_IMAGE117
And
Figure 358579DEST_PATH_IMAGE118
coordinates with screen vertex
Figure 145269DEST_PATH_IMAGE119
And
Figure 739192DEST_PATH_IMAGE120
then, the sub normal vector, the tangent vector and the normal vector of each screen are obtained through the center coordinate and the screen vertex coordinate, and then normalization processing is carried out in sequence to obtain
Figure 990045DEST_PATH_IMAGE121
、
Figure 503066DEST_PATH_IMAGE122
And
Figure 839370DEST_PATH_IMAGE123
thereby obtaining a rotation matrix
Figure 486252DEST_PATH_IMAGE124
And translation vector
Figure 529294DEST_PATH_IMAGE125
As shown in the formula (16) and the formula (17),
Figure 275533DEST_PATH_IMAGE126
Figure 909252DEST_PATH_IMAGE127
step (E122) of rotating the matrix according to the rotation matrix
Figure 969612DEST_PATH_IMAGE128
And translation vector
Figure 929478DEST_PATH_IMAGE129
Finding out an apparent transformation matrix
Figure 174514DEST_PATH_IMAGE130
As shown in the formula (18),
Figure 423093DEST_PATH_IMAGE131
step (E123) of setting the distance from the far and near cutting planes to the viewpoint to be
Figure 83882DEST_PATH_IMAGE132
Then calculates the scaling factor of perspective projection
Figure 711303DEST_PATH_IMAGE133
As shown in the formula (19) and the formula (20),
Figure 268186DEST_PATH_IMAGE134
Figure 800799DEST_PATH_IMAGE135
step (E124) of calculating the boundary conditions of the view frustum
Figure 62016DEST_PATH_IMAGE136
And
Figure 199736DEST_PATH_IMAGE137
then will be
Figure 724259DEST_PATH_IMAGE138
And
Figure 291637DEST_PATH_IMAGE139
substituting into equation (15), as shown in equation (21),
Figure 763070DEST_PATH_IMAGE140
。
preferably, the specific steps of step (E2) are as follows,
step (E21), a world coordinate system, a screen coordinate system, a Kinect camera coordinate system and a head coordinate system are constructed, the specific steps are as follows,
step (E21), kinect camera coordinate system
Figure 552034DEST_PATH_IMAGE141
And
Figure 309775DEST_PATH_IMAGE142
unifying by adopting an external calibration algorithm, mapping one Kinect acquisition information to another Kinect camera coordinate system space, and simultaneously converting the Kinect camera coordinate system to a world coordinate systemThe mapping is also done by external scaling algorithm and solved
Figure 754663DEST_PATH_IMAGE143
And
Figure 639573DEST_PATH_IMAGE144
；
step (E22), each screen coordinate system
Figure 283044DEST_PATH_IMAGE145
Unifying to the screen central coordinate system by the position relation and the screen size
Figure 87052DEST_PATH_IMAGE146
And obtain
Figure 143870DEST_PATH_IMAGE147
And
Figure 753843DEST_PATH_IMAGE148
and screen central coordinate system
Figure 455082DEST_PATH_IMAGE149
To world coordinate system
Figure 226729DEST_PATH_IMAGE150
Is done by manual measurement and is solved
Figure 784225DEST_PATH_IMAGE151
And
Figure 869993DEST_PATH_IMAGE152
wherein errors caused by manual measurement can be corrected by setting an effective compensation value by a program;
a step (E23) of,
Figure 488056DEST_PATH_IMAGE153
to
Figure 758500DEST_PATH_IMAGE154
There is only a translation transformation, so that the compensation vector is defined as
Figure 399697DEST_PATH_IMAGE155
And the translation vector part of the target conversion matrix is returned to zero to obtain
Figure 351473DEST_PATH_IMAGE156
And
Figure 637092DEST_PATH_IMAGE157
then the head coordinate system can be obtained through the motion posture of the head
Figure 219383DEST_PATH_IMAGE158
Relative to the camera coordinate system
Figure 410193DEST_PATH_IMAGE159
As shown in equation (22),
Figure 227976DEST_PATH_IMAGE160
wherein the content of the first and second substances,
Figure 758314DEST_PATH_IMAGE161
and
Figure 308244DEST_PATH_IMAGE162
respectively representing corresponding rotation matrixes and translation vectors;
step (E24) of calculating a transformation matrix from each screen coordinate system to the head coordinate system
Figure 533820DEST_PATH_IMAGE163
As shown in the formula (23),
Figure 30661DEST_PATH_IMAGE164
。
preferably, the Accela algorithm in step (E3) is to use the header parameters
Figure 477823DEST_PATH_IMAGE165
Splitting into position parts
Figure 260971DEST_PATH_IMAGE166
And a rotating part
Figure 629635DEST_PATH_IMAGE167
The method comprises the following specific steps of,
step (E31), constructing a noise filtering function, as shown in formula (24),
Figure 461325DEST_PATH_IMAGE168
wherein the content of the first and second substances,
Figure 576043DEST_PATH_IMAGE169
to represent
Figure 467775DEST_PATH_IMAGE170
Any of the independent variables in (a);
Figure 58157DEST_PATH_IMAGE171
indicating a noise threshold corresponding to the position part and lower than
Figure 755854DEST_PATH_IMAGE172
The disturbance noise of (2) is ignored;
Figure 177608DEST_PATH_IMAGE173
representing a smoothing coefficient corresponding to the position part;
step (E32), the noise threshold can restrain the tiny noise on the single channel, but can not filter the jitter generated by the superposition of the noise, and the position noise restraining factor is set
Figure 177925DEST_PATH_IMAGE174
And
Figure 128039DEST_PATH_IMAGE175
respectively shown as formula (25 And as shown in equation (26),
Figure 301532DEST_PATH_IMAGE176
。
the invention has the beneficial effects that:
(1) According to the invention, the Kinect sensors are arranged on two sides of the screen to acquire the point cloud data of the head of the driving student, and the point cloud data of the head of the driving student is obtained through point cloud fusion, so that the problem of self-shielding of the head of the driving student is avoided, and meanwhile, the background and useless information are filtered by using conditional filtering.
(2) The head pose estimation method used by the invention can track the head in real time, can simultaneously detect the head and estimate the pose parameters of the head, and then dynamically updates the visual scene to a multi-screen display unit in real time by using a multi-screen splicing technology.
(3) According to the multi-screen visual scene splicing based on the dynamic viewpoint, on the basis of fixed viewpoint splicing, the position and the angle of the virtual viewpoint are updated in real time by using the obtained head pose parameters through global coordinate system modeling, and then the position and the angle of the screen virtual viewpoint can be updated in real time through the head pose change of a driver by using a screen splicing scheme, so that a virtual scene with high telepresence is realized.
(4) The invention aims at the fact that a dynamic viewpoint system is particularly sensitive to noise, accela filtering is used for enhancing the stability of projection parameters, and smoothing of a head parameter motion curve is carried out, so that the influence of jitter generated in the head motion process on the screen visual content can be avoided.
Drawings
FIG. 1 is an overall flow diagram of the present invention;
FIG. 2 is a schematic representation of the present invention
Figure 781055DEST_PATH_IMAGE177
And (4) an algorithm network schematic diagram.
Detailed Description
The invention will be further described with reference to the accompanying drawings.
As shown in fig. 1, the multi-screen view simulation method of the simulator based on dynamic viewpoint according to the present invention includes the following steps,
step (A), respectively arranging a Kinect sensor at two sides of a multi-screen to collect head original point cloud data of a driving student, and carrying out conditional filtering and denoising processing on the collected original point cloud data to obtain processed point cloud data,
the method comprises the steps that original point cloud data collected by a Kinect sensor has a lot of noise and useless information, the useless information can be well filtered by using a conditional filtering algorithm for the point cloud data, but part of noise exists, and therefore a bilateral filtering algorithm is used again; the bilateral filtering algorithm can achieve the effect of noise reduction and smoothing on the edge of the picture, meanwhile, the bilateral filtering algorithm can also keep the edge information of the picture, and the Euclidean distance between the current point and the adjacent point and the local geometric information of the current point in the adjacent area of the current point can be considered when distinguishing the noise point from the outlier.
Step (A1) of performing conditional filtering on the acquired original point cloud data and using a conditional filtering algorithm to perform conditional filtering on the point cloud data
Figure 76907DEST_PATH_IMAGE178
A shaft,
Figure 438618DEST_PATH_IMAGE005
Shaft and
Figure 353484DEST_PATH_IMAGE006
the axes are processed to filter out useless information and background of each axis,
Figure 828459DEST_PATH_IMAGE179
the filtering of the axis is shown in equation (1),
Figure 232896DEST_PATH_IMAGE008
wherein, the first and the second end of the pipe are connected with each other,
Figure 754007DEST_PATH_IMAGE009
，
Figure 862777DEST_PATH_IMAGE010
representing a conditionally filtered set of point cloud data,
Figure 113630DEST_PATH_IMAGE011
representing an input raw point cloud data set;
step (A2), denoising the acquired original point cloud data, using a bilateral filtering algorithm as shown in formula (2),
Figure 892230DEST_PATH_IMAGE012
wherein the content of the first and second substances,
Figure 776004DEST_PATH_IMAGE013
representing the characteristic information of the point cloud before bilateral filtering,
Figure 94990DEST_PATH_IMAGE014
representing the point cloud characteristic information obtained by bilateral filtering algorithm,
Figure 138032DEST_PATH_IMAGE015
the overall weight parameter is represented by a weight value,
Figure 212167DEST_PATH_IMAGE016
representing the characteristic information of a normal vector before point cloud bilateral filtering; and the overall weight parameter
Figure 35767DEST_PATH_IMAGE017
The specific steps of the definition are as follows,
a step (A21) of,
Figure 830547DEST_PATH_IMAGE018
is defined as shown in the formula (3),
Figure 134621DEST_PATH_IMAGE180
wherein the content of the first and second substances,
Figure 51761DEST_PATH_IMAGE020
the representation feature holds a weight calculation function,
Figure 34761DEST_PATH_IMAGE021
represents a smooth filtering weight calculation function,
Figure 23445DEST_PATH_IMAGE022
indicating points
Figure 103397DEST_PATH_IMAGE023
Is/are as follows
Figure 129122DEST_PATH_IMAGE024
The number of the neighboring values is,
Figure 661734DEST_PATH_IMAGE025
indicating points
Figure 264229DEST_PATH_IMAGE026
To any point in its neighborhood
Figure 136370DEST_PATH_IMAGE027
The distance of the vector of (a) to the target,
Figure 395313DEST_PATH_IMAGE028
indicating points
Figure 493851DEST_PATH_IMAGE029
Normal vector feature information in a neighboring region;
step (A22), in the formula (3)
Figure 699704DEST_PATH_IMAGE030
And
Figure 816565DEST_PATH_IMAGE031
as shown in equation (4) and equation (5) respectively,
Figure 246409DEST_PATH_IMAGE032
Figure 691297DEST_PATH_IMAGE033
wherein the content of the first and second substances,
Figure 497579DEST_PATH_IMAGE034
and
Figure 219678DEST_PATH_IMAGE035
indicating points
Figure 23686DEST_PATH_IMAGE036
Gaussian filter coefficients of the tangent plane.
Step (B), adopting a three-surface target algorithm to carry out external reference rotation matrix on the Kinect sensor
Figure 752608DEST_PATH_IMAGE037
And translation vector
Figure 690477DEST_PATH_IMAGE181
Calibrating and unifying the processed point cloud data to a global coordinate system to complete the fusion of the processed point cloud data of the two Kinect sensors,
step (B1) of setting a vector
Figure 126137DEST_PATH_IMAGE039
As vectors in a three-dimensional coordinate system, i.e.
Figure 163363DEST_PATH_IMAGE040
，
Figure 458210DEST_PATH_IMAGE041
Are points of a three-dimensional coordinate system, and
Figure 543977DEST_PATH_IMAGE042
in a coordinate system
Figure 162040DEST_PATH_IMAGE043
And
Figure 698064DEST_PATH_IMAGE044
are respectively defined as
Figure 339261DEST_PATH_IMAGE045
And
Figure 291036DEST_PATH_IMAGE046
to, for
Figure 839305DEST_PATH_IMAGE047
And
Figure 156017DEST_PATH_IMAGE048
normalization processing is carried out to obtain unit vector
Figure 346827DEST_PATH_IMAGE049
And
Figure 164610DEST_PATH_IMAGE050
and unit vector
Figure 694948DEST_PATH_IMAGE051
And
Figure 979299DEST_PATH_IMAGE052
as shown in the formula (6),
Figure 736034DEST_PATH_IMAGE053
wherein the content of the first and second substances,
Figure 967295DEST_PATH_IMAGE054
representing an external reference rotation matrix;
a step (B2) of setting a rotation vector
Figure 742353DEST_PATH_IMAGE055
Formed matrix
Figure 463184DEST_PATH_IMAGE056
And matrix of
Figure 300690DEST_PATH_IMAGE057
Can make
Figure 211009DEST_PATH_IMAGE058
Matrix of
Figure 778256DEST_PATH_IMAGE059
As shown in the formula (7), the,
Figure 607672DEST_PATH_IMAGE182
wherein, the first and the second end of the pipe are connected with each other,
Figure 994791DEST_PATH_IMAGE061
representing a third order identity matrix;
step (B3) of using
Figure 958068DEST_PATH_IMAGE062
Group of vectors is in
Figure 317505DEST_PATH_IMAGE063
And
Figure 380139DEST_PATH_IMAGE064
unit vector in coordinate system
Figure 333182DEST_PATH_IMAGE065
And
Figure 444358DEST_PATH_IMAGE066
can solve the rotation matrix
Figure 720618DEST_PATH_IMAGE067
Is provided with
Figure 16471DEST_PATH_IMAGE068
，
Figure 581444DEST_PATH_IMAGE069
Then rotation matrix
Figure 558627DEST_PATH_IMAGE070
The obtained process is shown as formula (8), formula (9) and formula (10),
Figure 765093DEST_PATH_IMAGE071
Figure 107213DEST_PATH_IMAGE072
Figure 956220DEST_PATH_IMAGE073
wherein the content of the first and second substances,
Figure 64991DEST_PATH_IMAGE074
，
Figure 987947DEST_PATH_IMAGE075
which is indicative of the vector of rotation of the,
Figure 563285DEST_PATH_IMAGE076
represent
Figure 978217DEST_PATH_IMAGE077
，
Figure 234886DEST_PATH_IMAGE078
To represent
Figure 668141DEST_PATH_IMAGE079
；
Step (B4) of setting a three-dimensional space point D at
Figure 414380DEST_PATH_IMAGE080
And
Figure 175663DEST_PATH_IMAGE081
are respectively defined as
Figure 111389DEST_PATH_IMAGE082
And
Figure 71255DEST_PATH_IMAGE083
then, then
Figure 191658DEST_PATH_IMAGE084
Is reused
Figure 236974DEST_PATH_IMAGE085
Point averaging, translation vector
Figure 225659DEST_PATH_IMAGE086
As shown in equation (11),
Figure 243293DEST_PATH_IMAGE087
（11）
wherein, the first and the second end of the pipe are connected with each other,
Figure 331335DEST_PATH_IMAGE088
which represents the translation vector(s) of the image,
Figure 942576DEST_PATH_IMAGE089
representing a series of coordinate points obtained under the Kinect1 sensor,
Figure 344738DEST_PATH_IMAGE090
a series of coordinate points obtained under the Kinect2 sensor are indicated.
As shown in FIG. 2, step (C), based on the fusion of the processed point cloud data, adopts
Figure 279196DEST_PATH_IMAGE183
The algorithm obtains the head pose parameters in real time, completes the face area detection and the head pose estimation of the driving trainees, obtains an object model of the virtual three-dimensional scene,the specific steps are as follows,
wherein adopt
Figure 866036DEST_PATH_IMAGE091
The algorithm is used for acquiring the head pose parameters in real time, and meanwhile, the algorithm can finish head detection and head pose estimation at the same time;
Figure 823627DEST_PATH_IMAGE184
basically completely consistent with the yolov4 processing process, but a front view BEV is used for replacing an RGB image input by the yolov4 network before input; in order to enable the head pose algorithm to predict the target parameters, the processed three-dimensional point cloud data is programmatically converted into a frontal view BEV. In order to make the algorithm suitable for face region detection and head pose estimation, each anchor frame of the model corresponds to an output containing 9 values, wherein 1 represents whether the anchor frame is a positive sample, 2 represents the number of parameters of the position of the anchor frame, 3 represents the offset of the center point of the anchor frame to the nose of the center of the head, and 3 represents the information of the rotation angle of the head.
Step (C1), a loss function is established, the loss function is used for reflecting the error between the prediction result of the model for the sample and the actual label of the sample, and the total loss function of the model is
Figure 826218DEST_PATH_IMAGE092
、
Figure 690882DEST_PATH_IMAGE093
And
Figure 323988DEST_PATH_IMAGE094
the sum of these three hierarchical feature map loss functions, the total loss function of the model is shown in equation (12),
Figure 831193DEST_PATH_IMAGE095
wherein the content of the first and second substances,
Figure 434213DEST_PATH_IMAGE096
the function of the total loss is expressed as,
Figure 546525DEST_PATH_IMAGE097
、
Figure 147271DEST_PATH_IMAGE098
and
Figure 954821DEST_PATH_IMAGE099
respectively represent
Figure 299215DEST_PATH_IMAGE100
、
Figure 266034DEST_PATH_IMAGE101
And
Figure 365577DEST_PATH_IMAGE102
loss functions for the three hierarchical layers;
step (C2), the loss function of each layer contains the position deviation for calculating the face area
Figure 581794DEST_PATH_IMAGE103
Part for calculating classification errors
Figure 933141DEST_PATH_IMAGE104
Part and method for determining whether a target object is contained in a face region
Figure 364254DEST_PATH_IMAGE105
In part, the objective function of each layer is shown in equation (13),
Figure 572381DEST_PATH_IMAGE106
wherein the content of the first and second substances,
Figure 479157DEST_PATH_IMAGE107
and
Figure 227670DEST_PATH_IMAGE108
the Sigmoid is used as an activation function, and corresponding results are converted into probability values;
step (C3), each part of each layer of function adopts cross entropy as a loss function, as shown in formula (14),
Figure 169082DEST_PATH_IMAGE185
wherein the content of the first and second substances,
Figure 548110DEST_PATH_IMAGE110
the value of the loss function of the output is expressed,
Figure 551970DEST_PATH_IMAGE111
the actual label corresponding to the sample is represented,
Figure 41857DEST_PATH_IMAGE112
indicating the corresponding prediction result of the sample.
And (D) mapping the obtained virtual three-dimensional scene object model to a screen coordinate system for visualization through visual transformation, projection transformation, perspective division and view port transformation, wherein the visual transformation is to convert a world coordinate system into a camera coordinate system, the projection transformation is to map three-dimensional coordinates into two-dimensional coordinates, and the perspective division is to map the three-dimensional coordinates into two-dimensional coordinates
Figure 572195DEST_PATH_IMAGE113
The component becomes 1 and the viewport transform is a transformation of the processed coordinates into the screen coordinate system space.
The projection transformation is a key step, and maps a three-dimensional coordinate into a two-dimensional coordinate in an orthogonal projection mode and a perspective projection mode; the orthogonal projection uses a projection mode of a rectangular observation body, does not scale an object according to the distance between the object and a virtual viewpoint, and is equivalent to omitting
Figure 184442DEST_PATH_IMAGE186
The axis information directly projects the three-dimensional object to a two-dimensional plane; perspective projection adoptsThe projection mode of the viewing cone observation body simulates the mode of eyes observing the world, the three-dimensional object is projected onto a two-dimensional plane according to the rule of large and small distances, and the three-dimensional scene except the viewing cone body cannot be projected.
Step (E), respectively calculating a perspective matrix from the virtual viewpoint to each screen in a 3D space according to the actual placing position and the size of each screen, enhancing the stability of the projection parameters based on Accela filtering, and smoothing the motion curve of the head parameters to complete the multi-screen view simulation work of the simulator, which comprises the following specific steps,
the screen display unit of the aircraft simulator is formed by an upper liquid crystal screen and a lower liquid crystal screen in an arc shape, and the high-degree-of-field-feeling large-field-angle virtual visual environment can be realized according with the design of human engineering; the multi-screen view splicing technology respectively calculates a projection matrix from a virtual scene viewpoint to each screen in a 3D space according to the actual placement position and the size of each liquid crystal screen. Because the edges of the screens are jointed with each other during actual placement, the whole picture in the virtual scene can be spliced according to the picture projected to each screen by the calculated perspective matrix. When the head of the driving learner rotates, the head position and the head posture are calculated by the method and used for updating the parameters of the virtual viewpoint, and then the perspective matrix from the virtual viewpoint to each screen is recalculated, so that the multi-screen view splicing scheme under the dynamic viewpoint is realized.
Step (E1), a head coordinate system and a screen coordinate system are mapped into the same world coordinate system by adopting a fixed viewpoint mode, and a perspective matrix is obtained by calculating a transformation matrix of the head coordinate system and the screen coordinate system to finish the splicing work of multiple screens, which comprises the following specific steps,
step (E11), a calculation formula of the perspective matrix is constructed, as shown in formula (15),
Figure 862548DEST_PATH_IMAGE187
wherein, the first and the second end of the pipe are connected with each other,
Figure 359389DEST_PATH_IMAGE115
representing the distance of the virtual viewpoint to the near clipping plane,
Figure 640108DEST_PATH_IMAGE116
representing the distance from the virtual viewpoint to the far clipping plane;
and (E12) solving the constructed perspective matrix, wherein the specific steps are as follows,
step (E121) of finding the center coordinates of each screen
Figure 95360DEST_PATH_IMAGE117
And
Figure 464024DEST_PATH_IMAGE118
coordinates with screen vertex
Figure 358031DEST_PATH_IMAGE119
And
Figure 925278DEST_PATH_IMAGE120
then, the sub normal vector, the tangent vector and the normal vector of each screen are obtained through the center coordinate and the screen vertex coordinate, and then normalization processing is carried out in sequence to obtain
Figure 489115DEST_PATH_IMAGE121
、
Figure 220442DEST_PATH_IMAGE122
And
Figure 855822DEST_PATH_IMAGE123
thereby obtaining a rotation matrix
Figure 949680DEST_PATH_IMAGE124
And translation vector
Figure 340210DEST_PATH_IMAGE125
As shown in the formula (16) and the formula (17),
Figure 214625DEST_PATH_IMAGE126
Figure 856959DEST_PATH_IMAGE127
step (E122) of rotating the matrix according to the rotation matrix
Figure 946269DEST_PATH_IMAGE128
And translation vector
Figure 179804DEST_PATH_IMAGE129
To find out the apparent transformation matrix
Figure 479199DEST_PATH_IMAGE130
As shown in the formula (18),
Figure 784278DEST_PATH_IMAGE131
step (E123) of setting distances from the far and near cutting planes to the viewpoint to be
Figure 915045DEST_PATH_IMAGE132
Then, the scaling factor of perspective projection is calculated
Figure 522744DEST_PATH_IMAGE133
As shown in the formula (19) and the formula (20),
Figure 184801DEST_PATH_IMAGE134
Figure 700096DEST_PATH_IMAGE135
step (E124) of calculating the boundary condition of the view frustum
Figure 154211DEST_PATH_IMAGE136
And
Figure 791865DEST_PATH_IMAGE137
then will be
Figure 128169DEST_PATH_IMAGE138
And
Figure 384838DEST_PATH_IMAGE139
substituting into equation (15), as shown in equation (21),
Figure 565896DEST_PATH_IMAGE188
。
step (E2), based on fixed viewpoint multi-screen splicing, calculating a perspective matrix from each frame of virtual viewpoint to each screen in real time according to a dynamic viewpoint technology, and specifically comprises the following steps;
the multi-screen splicing can be completed in a fixed viewpoint mode by mapping a head coordinate system and a screen coordinate system to the same world coordinate system in advance and calculating a transformation matrix of the head coordinate system and the screen coordinate system to obtain a perspective matrix; therefore, the multi-screen view splicing based on the dynamic viewpoint technology needs to calculate the perspective matrix from the virtual viewpoint to each screen in real time every frame.
Step (E21), a world coordinate system, a screen coordinate system, a Kinect camera coordinate system and a head coordinate system are constructed, the specific steps are as follows,
step (E21), kinect camera coordinate system
Figure 312135DEST_PATH_IMAGE141
And
Figure 807838DEST_PATH_IMAGE142
unifying by adopting an external calibration algorithm, mapping one Kinect acquisition information to another Kinect camera coordinate system space, simultaneously completing the mapping from the Kinect camera coordinate system to a world coordinate system by adopting the external calibration algorithm, and solving
Figure 258411DEST_PATH_IMAGE143
And
Figure 218277DEST_PATH_IMAGE144
；
step (E22), each screen coordinate system
Figure 73101DEST_PATH_IMAGE145
Unified to screen central coordinate system by mutual position relation and screen size
Figure 462625DEST_PATH_IMAGE146
And obtain
Figure 123413DEST_PATH_IMAGE147
And
Figure 875469DEST_PATH_IMAGE148
and screen central coordinate system
Figure 556986DEST_PATH_IMAGE149
To world coordinate system
Figure 89598DEST_PATH_IMAGE150
Is done by manual measurement and finds
Figure 757340DEST_PATH_IMAGE151
And
Figure 504847DEST_PATH_IMAGE152
wherein, the error caused by manual measurement can be corrected by setting an effective compensation value by a program;
a step (E23) of,
Figure 232632DEST_PATH_IMAGE153
to
Figure 314857DEST_PATH_IMAGE154
There is only a translation transformation, so that the compensation vector is defined as
Figure 520711DEST_PATH_IMAGE155
And return the translation vector part of the destination transformation matrix to zeroTo find out
Figure 309675DEST_PATH_IMAGE156
And
Figure 83727DEST_PATH_IMAGE157
then, a head coordinate system can be obtained through the motion posture of the head
Figure 263036DEST_PATH_IMAGE158
Relative to the camera coordinate system
Figure 69318DEST_PATH_IMAGE159
As shown in equation (22),
Figure 306264DEST_PATH_IMAGE160
wherein, the first and the second end of the pipe are connected with each other,
Figure 844693DEST_PATH_IMAGE161
and
Figure 839194DEST_PATH_IMAGE162
respectively representing corresponding rotation matrixes and translation vectors;
step (E24) of calculating a transformation matrix from each screen coordinate system to the head coordinate system
Figure 259286DEST_PATH_IMAGE163
As shown in the formula (23),
Figure 960526DEST_PATH_IMAGE189
。
and (E3) enhancing the stability of the projection parameters based on Accela filtering, and smoothing the motion curve of the head parameters, wherein the Accela algorithm is to use the head parameters
Figure 732173DEST_PATH_IMAGE190
Splitting into position parts
Figure 541866DEST_PATH_IMAGE191
And a rotating part
Figure 627634DEST_PATH_IMAGE167
The method comprises the following specific steps of,
the dynamic viewpoint system is very sensitive to noise, the head parameters comprise six independent variables, and even if a driving student keeps the head still, the noise influences one independent variable to enable the screen to generate a shaking sense; for real-time systems, multi-frame per second calculation is required to fit a nonlinear continuous model in time, and reasonable smooth filtering is required to reduce the perception of human eyes to the frequency and enhance the comfort.
Step (E31), constructing a noise filtering function, as shown in formula (24),
Figure 245697DEST_PATH_IMAGE168
wherein, the first and the second end of the pipe are connected with each other,
Figure 532453DEST_PATH_IMAGE169
to represent
Figure 173650DEST_PATH_IMAGE170
Any of the independent variables in (a);
Figure 859846DEST_PATH_IMAGE171
representing a noise threshold corresponding to a portion of the location, and below
Figure 925891DEST_PATH_IMAGE172
The disturbance noise of (2) is ignored;
Figure 977023DEST_PATH_IMAGE173
representing a smoothing coefficient corresponding to the position part;
step (E32), the noise threshold can restrain the tiny noise on the single channel, but can not filter the jitter generated by the superposition of the noise, and the position noise restraining factor is set
Figure 167833DEST_PATH_IMAGE174
And
Figure 736349DEST_PATH_IMAGE175
as shown in equation (25) and equation (26), respectively,
Figure 266688DEST_PATH_IMAGE192
。
in order to verify the validity and effectiveness of the method according to the invention, a specific embodiment of the invention is described below,
the screen display unit of the embodiment adopts eight 3D liquid crystal televisions as the projection screen, and generates a full 3D virtual simulation environment in a mode of splicing multiple screen views. The 4 televisions correspond to a front window of the cockpit, are vertically arranged at 135 degrees between every two televisions and are integrally inclined forward by 30 degrees; the rest 4 televisions correspond to the windows under the feet of the cockpit, are vertically arranged by 180 degrees between every two televisions and form an included angle of 40 degrees with the horizontal plane. Two are adopted
Figure 816618DEST_PATH_IMAGE193
The cameras are arranged right above the front television pairwise junction positions, and the optical axis faces the head position of a flying student in a normal sitting posture to complete dynamic detection of the viewpoint of the student. Each liquid crystal display screen is driven by a graphic computer, and the computers finish the synchronization of data such as control data, viewpoint parameters, simulation entity pose and the like through an open source CIGI distributed network protocol.
Figure 557041DEST_PATH_IMAGE194
The algorithm needs to be trained in advance, and the training data adopted is from
Figure 788302DEST_PATH_IMAGE195
A database. The database provided 42 sets of Kinect collected head depth data, 32000 total, from 26 men, 10 women and 6 women, respectivelyThe wearer of the glasses. And acquiring a depth image and a color image by each frame of Kinect, and labeling the pose parameters by using a Faceshift technology in the later stage. Faceshift is given in terms of depth image per frame
Figure 235464DEST_PATH_IMAGE196
As tag information. As shown in table 1, the recognition results of four consecutive image sequences in the data set are given.
Figure 769344DEST_PATH_IMAGE197
By comparing the output of the head pose estimation algorithm with the actual tag, the algorithm can be found to have good accuracy, and the real-time tracking and the output of the head pose parameters of the frame can be realized.
This real-time example realizes that aircraft simulator multiple screen views concatenation scheme based on eight high definition LCD screens from top to bottom to select four different scenes such as residential area, offshore platform, open-air and airport, show multiple screen views concatenation effect from the inboard external angle respectively, the multiple screen concatenation scheme of this case can carry out fine concatenation with virtual views, provide the big angle of vision virtual environment of higher telepresence, bring better reality sense and the sense of immersing for the driving student, thereby reach better training effect.
Because the actual system is easy to mix sensor noise and algorithm random noise, such as involuntary movement of the head, mechanical shaking of a cockpit base and recognition errors of a head posture estimation calculation method, the screen visual contents can generate shaking feeling, and the training effect is influenced. The projection parameter stability enhancement algorithm based on Accela filtering is used for restraining the projection parameter stability enhancement algorithm, experimental analysis is carried out on the algorithm, the smoothing effect of the algorithm is good, and noise fluctuation of data can be processed in real time.
The visual display system designed by the scheme is composed of 8 high-resolution liquid crystal displays, the 8 displays are arranged in four-up-four-down-four mode, and the angles of the 4 displays on the upper surface and the distance between the displays and a driving student can be adjusted through the screen adjusting structureThe distance to the driving learner can also be adjusted as follows. The visual display system of the simulator splices the images together by utilizing a multi-screen splicing technology, and the horizontal visual angle of 8 liquid crystal displays is larger than that of the
Figure 138009DEST_PATH_IMAGE198
Vertical angle of view greater than
Figure 969698DEST_PATH_IMAGE199
And a wide visual field range is provided for the driving trainees. Meanwhile, the liquid crystal display screen can provide high resolution, brightness and contrast, so that the visual display system has the characteristics of large visual field, high brightness, high contrast, high resolution and the like, and provides a set of continuous and complete extrawindow scenes with high fidelity for training personnel. Scheme based on many screen concatenations also can carry out fine concatenation with virtual scene when realizing low price to provide the virtual scene of big visual angle of high telepresence, bring better sense of immersing for the driving student, thereby make the training effect reach better.
In summary, the simulator multi-screen view simulation method based on the dynamic viewpoint of the invention firstly utilizes a Kinect sensor non-contact mode to collect point cloud data of a head of a driving student, estimates head pose information of the student in real time, simultaneously adopts an arrangement mode of two Kinects for solving the self-shielding problem of the head, then uses a head pose estimation calculation method to simultaneously detect the head and estimate the pose of the head, and realizes real-time tracking, and then on the basis of a multi-screen view splicing scheme based on a fixed viewpoint, the multi-screen view splicing scheme based on the dynamic viewpoint is realized by modeling a global coordinate system and updating the position and the angle of a virtual scene viewpoint in real time by the obtained head pose parameters, and the curve difference value smoothing algorithm Accela filtering based on experience estimation is provided for jitter information possibly occurring in the head motion process.
The foregoing illustrates and describes the principles, general features, and advantages of the present invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are given by way of illustration of the principles of the present invention, but that various changes and modifications may be made without departing from the spirit and scope of the invention, and such changes and modifications are within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.

Claims (9)

1. The simulator multi-screen view simulation method based on the dynamic viewpoint is characterized in that: comprises the following steps of (a) carrying out,
step (A), respectively placing a Kinect sensor at two sides of a multi-screen to acquire head original point cloud data of a driving student, and performing conditional filtering and denoising processing on the acquired original point cloud data to obtain processed point cloud data;
step (B), adopting a three-surface target algorithm to carry out external reference rotation matrix on the Kinect sensor
Figure 2880DEST_PATH_IMAGE001
And translation vector
Figure 654441DEST_PATH_IMAGE002
Calibrating and unifying the processed point cloud data to a global coordinate system to complete the fusion of the processed point cloud data of the two Kinect sensors;
step (C), based on the fusion of the processed point cloud data, adopting
Figure 27654DEST_PATH_IMAGE003
The algorithm obtains head pose parameters in real time, and completes the detection of the face area of the driving learner and the estimation of the head pose to obtain an object model of the virtual three-dimensional scene;
mapping the obtained virtual three-dimensional scene object model to a screen coordinate system for visualization through visual transformation, projection transformation, perspective division and viewport transformation;
and (E) respectively calculating a perspective matrix from the virtual viewpoint to each screen in a 3D space according to the actual placement position and the size of each screen, enhancing the stability of projection parameters based on Accela filtering, and smoothing the motion curve of the head parameters to complete the multi-screen view simulation work of the simulator.
2. The multi-screen view simulation method of the simulator based on the dynamic viewpoint as claimed in claim 1, wherein: step (A), respectively arranging a Kinect sensor at two sides of a multi-screen to collect head original point cloud data of a driving student, and carrying out conditional filtering and denoising processing on the collected original point cloud data to obtain processed point cloud data,
step (A1) of performing conditional filtering on the collected original point cloud data and using a conditional filtering algorithm to perform conditional filtering on the point cloud data
Figure 891705DEST_PATH_IMAGE004
A shaft,
Figure 299552DEST_PATH_IMAGE005
Shaft and
Figure 590856DEST_PATH_IMAGE006
the axes are processed to filter out useless information and background of each axis,
Figure 952830DEST_PATH_IMAGE007
filtering of the shaft as shown in equation (1),
Figure 620572DEST_PATH_IMAGE008
wherein the content of the first and second substances,
Figure 617346DEST_PATH_IMAGE009
，
Figure 938606DEST_PATH_IMAGE010
representing a conditionally filtered collection of point cloud data,
Figure 161777DEST_PATH_IMAGE011
representing an input raw point cloud data set;
step (A2), carrying out denoising processing on the collected original point cloud data, using a bilateral filtering algorithm as shown in formula (2),
Figure 492265DEST_PATH_IMAGE012
wherein, the first and the second end of the pipe are connected with each other,
Figure 484491DEST_PATH_IMAGE013
representing the characteristic information of the point cloud before bilateral filtering,
Figure 12205DEST_PATH_IMAGE014
representing the point cloud characteristic information acquired by the bilateral filtering algorithm,
Figure 457093DEST_PATH_IMAGE015
the overall weight parameter is represented by a weight value,
Figure 591271DEST_PATH_IMAGE016
representing feature information of a normal vector before point cloud bilateral filtering; and the overall weight parameter
Figure 703584DEST_PATH_IMAGE017
The specific steps of the definition are as follows,
a step (A21) of,
Figure 773171DEST_PATH_IMAGE018
is defined as shown in the formula (3),
Figure 361147DEST_PATH_IMAGE019
wherein, the first and the second end of the pipe are connected with each other,
Figure 908803DEST_PATH_IMAGE020
the representation feature holds a weight calculation function,
Figure 256DEST_PATH_IMAGE021
represents a smooth filtering weight calculation function,
Figure 506324DEST_PATH_IMAGE022
indicating points
Figure 191383DEST_PATH_IMAGE023
Is/are as follows
Figure 808309DEST_PATH_IMAGE024
The number of the adjacent neighbor values is,
Figure 255733DEST_PATH_IMAGE025
indicating points
Figure 932702DEST_PATH_IMAGE026
To any point in its neighborhood
Figure 105058DEST_PATH_IMAGE027
The vector distance of (a) is greater than (b),
Figure 384729DEST_PATH_IMAGE028
indicating points
Figure 326140DEST_PATH_IMAGE029
Normal vector feature information in a neighboring region;
step (A22), in the formula (3)
Figure 174011DEST_PATH_IMAGE030
And
Figure 692717DEST_PATH_IMAGE031
respectively as formula (4) and formula (5)As shown in the drawings, the first and second,
Figure 651445DEST_PATH_IMAGE032
Figure 447363DEST_PATH_IMAGE033
wherein, the first and the second end of the pipe are connected with each other,
Figure 325189DEST_PATH_IMAGE034
and
Figure 472137DEST_PATH_IMAGE035
indicating points
Figure 234556DEST_PATH_IMAGE036
Gaussian filter coefficients of the tangent plane.
3. The multi-screen view simulation method of the simulator based on the dynamic viewpoint as claimed in claim 2, wherein: step (B), adopting a three-surface target algorithm to carry out external reference rotation matrix on the Kinect sensor
Figure 508150DEST_PATH_IMAGE037
And translation vector
Figure 432243DEST_PATH_IMAGE038
Calibrating and unifying the processed point cloud data to a global coordinate system to complete the fusion of the processed point cloud data of the two Kinect sensors, which comprises the following steps,
step (B1) of setting a vector
Figure 66487DEST_PATH_IMAGE039
As vectors in a three-dimensional coordinate system, i.e.
Figure 491652DEST_PATH_IMAGE040
，
Figure 262162DEST_PATH_IMAGE041
Being points of a three-dimensional coordinate system, and
Figure 357157DEST_PATH_IMAGE042
in a coordinate system
Figure 603330DEST_PATH_IMAGE043
And
Figure 707553DEST_PATH_IMAGE044
are respectively defined as
Figure 66990DEST_PATH_IMAGE045
And
Figure 723099DEST_PATH_IMAGE046
to, for
Figure 800777DEST_PATH_IMAGE047
And
Figure 708690DEST_PATH_IMAGE048
normalization processing is carried out to obtain unit vector
Figure 79890DEST_PATH_IMAGE049
And
Figure 782267DEST_PATH_IMAGE050
and unit vector
Figure 612820DEST_PATH_IMAGE051
And
Figure 917899DEST_PATH_IMAGE052
as shown in the formula (6),
Figure 517508DEST_PATH_IMAGE053
wherein, the first and the second end of the pipe are connected with each other,
Figure 390786DEST_PATH_IMAGE054
representing an external reference rotation matrix;
a step (B2) of setting a rotation vector
Figure 708635DEST_PATH_IMAGE055
Formed matrix
Figure 551826DEST_PATH_IMAGE056
And matrix of
Figure 271520DEST_PATH_IMAGE057
Can make
Figure 315700DEST_PATH_IMAGE058
Matrix of
Figure 245478DEST_PATH_IMAGE059
As shown in the formula (7),
Figure 33306DEST_PATH_IMAGE060
wherein, the first and the second end of the pipe are connected with each other,
Figure 341927DEST_PATH_IMAGE061
representing a third order identity matrix;
step (B3) of using
Figure 10255DEST_PATH_IMAGE062
Group of vectors is in
Figure 37117DEST_PATH_IMAGE063
And
Figure 628635DEST_PATH_IMAGE064
unit vector in coordinate system
Figure 916397DEST_PATH_IMAGE065
And
Figure 302379DEST_PATH_IMAGE066
can obtain a rotation matrix
Figure 82116DEST_PATH_IMAGE067
Is provided with
Figure 211746DEST_PATH_IMAGE068
，
Figure 619594DEST_PATH_IMAGE069
Then rotation matrix
Figure 176477DEST_PATH_IMAGE070
The obtained process is shown as formula (8), formula (9) and formula (10),
Figure 177931DEST_PATH_IMAGE071
Figure 235886DEST_PATH_IMAGE072
Figure 108027DEST_PATH_IMAGE073
wherein, the first and the second end of the pipe are connected with each other,
Figure 101391DEST_PATH_IMAGE074
，
Figure 950661DEST_PATH_IMAGE075
which is indicative of the vector of rotation of the,
Figure 422093DEST_PATH_IMAGE076
to represent
Figure 679899DEST_PATH_IMAGE077
，
Figure 703219DEST_PATH_IMAGE078
To represent
Figure 413686DEST_PATH_IMAGE079
；
Step (B4) of setting a three-dimensional space point D at
Figure 688809DEST_PATH_IMAGE080
And
Figure 191335DEST_PATH_IMAGE081
are respectively defined as
Figure 995343DEST_PATH_IMAGE082
And
Figure 458685DEST_PATH_IMAGE083
then, then
Figure 396554DEST_PATH_IMAGE084
Is reused
Figure 628952DEST_PATH_IMAGE085
Point averaging, translation vector
Figure 603862DEST_PATH_IMAGE086
As shown in equation (11),
Figure 177669DEST_PATH_IMAGE087
wherein the content of the first and second substances,
Figure 794595DEST_PATH_IMAGE088
a translation vector is represented that represents the translation vector,
Figure 615921DEST_PATH_IMAGE089
representing a series of coordinate points obtained under the Kinect1 sensor,
Figure 417524DEST_PATH_IMAGE090
a series of coordinate points obtained under the Kinect2 sensor are represented.
4. The multi-screen view simulation method of the simulator based on the dynamic viewpoint as claimed in claim 3, wherein: step (C), based on the fusion of the processed point cloud data, adopting
Figure 589879DEST_PATH_IMAGE091
The algorithm obtains the head pose parameters in real time, completes the face area detection and the head pose estimation of the driving trainees, obtains an object model of the virtual three-dimensional scene, and comprises the following specific steps,
step (C1), a loss function is established, the loss function is used for reflecting the error between the prediction result of the model for the sample and the actual label of the sample, and the total loss function of the model is
Figure 744917DEST_PATH_IMAGE092
、
Figure 951907DEST_PATH_IMAGE093
And
Figure 393253DEST_PATH_IMAGE094
the sum of these three hierarchical feature map loss functions, the total loss function of the model is shown in equation (12),
Figure 52904DEST_PATH_IMAGE095
wherein the content of the first and second substances,
Figure 11633DEST_PATH_IMAGE096
the function of the total loss is expressed as,
Figure 197764DEST_PATH_IMAGE097
、
Figure 950956DEST_PATH_IMAGE098
and
Figure 97904DEST_PATH_IMAGE099
respectively represent
Figure 220843DEST_PATH_IMAGE100
、
Figure 136846DEST_PATH_IMAGE101
And
Figure 326519DEST_PATH_IMAGE102
loss functions for the three hierarchical layers;
step (C2), the loss function of each layer contains the position deviation for calculating the face area
Figure 819817DEST_PATH_IMAGE103
Part for calculating classification errors
Figure 120348DEST_PATH_IMAGE104
Part and method for judging whether a target object is contained in a face region
Figure 890858DEST_PATH_IMAGE105
In part, the objective function of each layer is shown in equation (13),
Figure 110487DEST_PATH_IMAGE106
wherein, the first and the second end of the pipe are connected with each other,
Figure 232027DEST_PATH_IMAGE107
and
Figure 70670DEST_PATH_IMAGE108
the Sigmoid is used as an activation function, and corresponding results are converted into probability values;
step (C3), each part of each layer of function adopts cross entropy as a loss function, as shown in formula (14),
Figure 961266DEST_PATH_IMAGE109
wherein, the first and the second end of the pipe are connected with each other,
Figure 617375DEST_PATH_IMAGE110
the value of the loss function of the output is expressed,
Figure 695052DEST_PATH_IMAGE111
the actual label corresponding to the specimen is represented,
Figure 602965DEST_PATH_IMAGE112
indicating the corresponding prediction result of the sample.
5. The multi-screen view simulation method of the simulator based on the dynamic viewpoint as claimed in claim 4, wherein: and (D) mapping the obtained virtual three-dimensional scene object model to a screen coordinate system for visualization through visual transformation, projection transformation, perspective division and view port transformation, wherein the visual transformation is to convert a world coordinate system into a camera coordinate system, the projection transformation is to map three-dimensional coordinates into two-dimensional coordinates, and the perspective division is to map the three-dimensional coordinates into two-dimensional coordinates
Figure 977096DEST_PATH_IMAGE113
The component becomes 1 and the viewport transformation is to convert the processed coordinates to screen coordinate system space.
6. The multi-screen view simulation method of a simulator based on dynamic viewpoints of claim 5, wherein: step (E), respectively calculating a perspective matrix from a virtual viewpoint to each screen in a 3D space according to the actual placement position and the size of each screen, enhancing the stability of projection parameters based on Accela filtering, and smoothing the motion curve of head parameters to complete the multi-screen view simulation of the simulator, wherein the specific steps are as follows,
step (E1), a fixed viewpoint mode is adopted, a head coordinate system and a screen coordinate system are mapped into the same world coordinate system, and a perspective matrix is obtained by calculating transformation matrixes of the head coordinate system and the screen coordinate system, so that the multi-screen splicing work is completed;
step (E2), based on fixed viewpoint multi-screen splicing, calculating a perspective matrix from each frame of virtual viewpoint to each screen in real time according to a dynamic viewpoint technology;
and (E3) enhancing the stability of the projection parameters based on Accela filtering, and smoothing the motion curve of the head parameters.
7. The multi-screen view simulation method of a simulator based on dynamic viewpoints of claim 6, wherein: the specific steps of step (E1) are as follows,
step (E11), a calculation formula of the perspective matrix is constructed, as shown in formula (15),
Figure 679473DEST_PATH_IMAGE114
wherein the content of the first and second substances,
Figure 244446DEST_PATH_IMAGE115
representing virtual viewpoint to sanctionsThe distance between the plane of the scissors is,
Figure 549525DEST_PATH_IMAGE116
representing the distance from the virtual viewpoint to the far clipping plane;
and (E12) solving the constructed perspective matrix, wherein the specific steps are as follows,
step (E121) of obtaining the center coordinates of each screen
Figure 414713DEST_PATH_IMAGE117
And
Figure 287991DEST_PATH_IMAGE118
coordinates with screen vertex
Figure 605840DEST_PATH_IMAGE119
And
Figure 449031DEST_PATH_IMAGE120
then, the sub-normal vector, the tangent vector and the normal vector of each screen are obtained through the center coordinate and the screen vertex coordinate, and then normalization processing is sequentially carried out to obtain
Figure 168726DEST_PATH_IMAGE121
、
Figure 212905DEST_PATH_IMAGE122
And
Figure 877105DEST_PATH_IMAGE123
thereby obtaining a rotation matrix
Figure 664932DEST_PATH_IMAGE124
And translation vector
Figure 599652DEST_PATH_IMAGE125
As shown in the formula (16) and the formula (17),
Figure 814733DEST_PATH_IMAGE126
Figure 841595DEST_PATH_IMAGE127
step (E122) of rotating the matrix according to the rotation matrix
Figure 292168DEST_PATH_IMAGE128
And translation vector
Figure 455296DEST_PATH_IMAGE129
Finding out an apparent transformation matrix
Figure 841278DEST_PATH_IMAGE130
As shown in the formula (18),
Figure 480069DEST_PATH_IMAGE131
step (E123) of setting the distance from the far and near cutting planes to the viewpoint to be
Figure 609699DEST_PATH_IMAGE132
Then, the scaling factor of perspective projection is calculated
Figure 751968DEST_PATH_IMAGE133
As shown in the formula (19) and the formula (20),
Figure 574430DEST_PATH_IMAGE134
Figure 575884DEST_PATH_IMAGE135
step (E124) of calculating the boundary conditions of the view frustum
Figure 601216DEST_PATH_IMAGE136
And
Figure 4515DEST_PATH_IMAGE137
then will be
Figure 732300DEST_PATH_IMAGE138
And
Figure 80105DEST_PATH_IMAGE139
substituting into equation (15), as shown in equation (21),
Figure 817116DEST_PATH_IMAGE140
。
8. the multi-screen view simulation method of a simulator based on dynamic viewpoints of claim 6, wherein: the specific steps of step (E2) are as follows,
step (E21), a world coordinate system, a screen coordinate system, a Kinect camera coordinate system and a head coordinate system are constructed, the specific steps are as follows,
step (E21), kinect camera coordinate system
Figure 74923DEST_PATH_IMAGE141
And
Figure 832663DEST_PATH_IMAGE142
unifying by adopting an external calibration algorithm, mapping the Kinect acquisition information to the space of the coordinate system of another Kinect camera, simultaneously completing the mapping from the coordinate system of the Kinect camera to the world coordinate system by adopting the external calibration algorithm, and solving
Figure 808709DEST_PATH_IMAGE143
And
Figure 83833DEST_PATH_IMAGE144
；
step (E22), each screen coordinate system
Figure 196145DEST_PATH_IMAGE145
Unified to screen central coordinate system by mutual position relation and screen size
Figure 390366DEST_PATH_IMAGE146
And find out
Figure 588129DEST_PATH_IMAGE147
And
Figure 666944DEST_PATH_IMAGE148
and screen central coordinate system
Figure 525441DEST_PATH_IMAGE149
To world coordinate system
Figure 500350DEST_PATH_IMAGE150
Is done by manual measurement and is solved
Figure 310043DEST_PATH_IMAGE151
And
Figure 661390DEST_PATH_IMAGE152
wherein errors caused by manual measurement can be corrected by setting an effective compensation value by a program;
a step (E23) of,
Figure 748295DEST_PATH_IMAGE153
to
Figure 753160DEST_PATH_IMAGE154
There is only translation transformation, so that the vector is compensatedIs defined as
Figure 925515DEST_PATH_IMAGE155
And the translation vector part of the target conversion matrix is returned to zero to obtain
Figure 470766DEST_PATH_IMAGE156
And
Figure 146598DEST_PATH_IMAGE157
then the head coordinate system can be obtained through the motion posture of the head
Figure 994468DEST_PATH_IMAGE158
Relative to the camera coordinate system
Figure 740271DEST_PATH_IMAGE159
As shown in equation (22),
Figure 558054DEST_PATH_IMAGE160
wherein the content of the first and second substances,
Figure 619551DEST_PATH_IMAGE161
and
Figure 638322DEST_PATH_IMAGE162
respectively representing corresponding rotation matrixes and translation vectors;
step (E24) of calculating a transformation matrix from each screen coordinate system to the head coordinate system
Figure 378745DEST_PATH_IMAGE163
As shown in the formula (23),
Figure 141165DEST_PATH_IMAGE164
。
9. the multi-screen view simulation method of a simulator based on dynamic viewpoints of claim 6, wherein: in the step (E3), the Accela algorithm is to use the head parameters
Figure 57168DEST_PATH_IMAGE165
Splitting into position parts
Figure 105896DEST_PATH_IMAGE166
And a rotating part
Figure 740140DEST_PATH_IMAGE167
The method comprises the following specific steps of,
step (E31), constructing a noise filtering function, as shown in formula (24),
Figure 40671DEST_PATH_IMAGE168
wherein, the first and the second end of the pipe are connected with each other,
Figure 811181DEST_PATH_IMAGE169
to represent
Figure 797854DEST_PATH_IMAGE170
Any of the independent variables in (a);
Figure 388235DEST_PATH_IMAGE171
indicating a noise threshold corresponding to the position part and lower than
Figure 492457DEST_PATH_IMAGE172
The disturbance noise of (2) is ignored;
Figure 507687DEST_PATH_IMAGE173
a smoothing coefficient indicating a correspondence of the position part;
step (E32), the noise threshold can suppress the smallness on a single channelBut not filtering the jitter generated by the noise superposition, setting the position noise suppression factor
Figure 773583DEST_PATH_IMAGE174
And
Figure 116839DEST_PATH_IMAGE175
as shown in equation (25) and equation (26), respectively,
Figure 883807DEST_PATH_IMAGE176
。
CN202211407435.9A 2022-11-10 2022-11-10 Simulator multi-screen visual simulation method based on dynamic viewpoint Active CN115457220B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211407435.9A CN115457220B (en) 2022-11-10 2022-11-10 Simulator multi-screen visual simulation method based on dynamic viewpoint

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211407435.9A CN115457220B (en) 2022-11-10 2022-11-10 Simulator multi-screen visual simulation method based on dynamic viewpoint

Publications (2)

Publication Number Publication Date
CN115457220A true CN115457220A (en) 2022-12-09
CN115457220B CN115457220B (en) 2023-04-07

Family

ID=84295850

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211407435.9A Active CN115457220B (en) 2022-11-10 2022-11-10 Simulator multi-screen visual simulation method based on dynamic viewpoint

Country Status (1)

Country Link
CN (1) CN115457220B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105959665A (en) * 2016-05-05 2016-09-21 清华大学深圳研究生院 Panoramic 3D video generation method for virtual reality equipment
CN106778710A (en) * 2017-02-17 2017-05-31 吉林大学 A kind of flight simulator dynamic view system based on kinect sensors

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105959665A (en) * 2016-05-05 2016-09-21 清华大学深圳研究生院 Panoramic 3D video generation method for virtual reality equipment
CN106778710A (en) * 2017-02-17 2017-05-31 吉林大学 A kind of flight simulator dynamic view system based on kinect sensors

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
DONGLIN ZHU等: "Object Detection in Complex Road Scenarios:Improved YOLOv4-Tiny Algorithm", 《2021 2ND INFORMATION COMMUNICATION TECHNOLOGIES CONFERENCE》 *
王浩雪等: "基于改进YOLOv4的航拍图像多目标检测方法", 《电光与控制》 *

Also Published As

Publication number Publication date
CN115457220B (en) 2023-04-07

Similar Documents

Publication Publication Date Title
CN109003325B (en) Three-dimensional reconstruction method, medium, device and computing equipment
EP3057066B1 (en) Generation of three-dimensional imagery from a two-dimensional image using a depth map
Wei et al. Fisheye video correction
US10489956B2 (en) Robust attribute transfer for character animation
CN112509151A (en) Method for generating sense of reality of virtual object in teaching scene
US20140015924A1 (en) Rapid 3D Modeling
CN111968165B (en) Dynamic human body three-dimensional model complement method, device, equipment and medium
CN108171673A (en) Image processing method, device, vehicle-mounted head-up-display system and vehicle
CN103559737A (en) Object panorama modeling method
CN114785996A (en) Virtual reality parallax correction
CN102184572B (en) 3-D graphic method of cutting out, rendering method and its graphic processing facility
CN113366491B (en) Eyeball tracking method, device and storage medium
CN107145224B (en) Human eye sight tracking and device based on three-dimensional sphere Taylor expansion
US20090085910A1 (en) Method and apparatus for creating a composite image
CN111062869A (en) Curved screen-oriented multi-channel correction splicing method
CN111275731A (en) Projection type real object interactive desktop system and method for middle school experiment
CN111260765B (en) Dynamic three-dimensional reconstruction method for microsurgery field
CN112802208B (en) Three-dimensional visualization method and device in terminal building
CN113129451B (en) Holographic three-dimensional image space quantitative projection method based on binocular vision positioning
Yuan et al. Presim: A 3d photo-realistic environment simulator for visual ai
Zhai et al. Image real-time augmented reality technology based on spatial color and depth consistency
CN111915739A (en) Real-time three-dimensional panoramic information interactive information system
CN115457220B (en) Simulator multi-screen visual simulation method based on dynamic viewpoint
CN114494582A (en) Three-dimensional model dynamic updating method based on visual perception
CN110097644B (en) Expression migration method, device and system based on mixed reality and processor

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20240124

Address after: 210019 14th floor, building 01, 68 Aoti street, Jianye District, Nanjing City, Jiangsu Province

Patentee after: Nanjing yutianzhiyun Simulation Technology Co.,Ltd.

Country or region after: China

Patentee after: ARMY ENGINEERING University OF THE PEOPLE'S LIBERATION ARMY OF CHINA

Address before: 210000 14th floor, building 01, 68 Aoti street, Jianye District, Nanjing City, Jiangsu Province

Patentee before: Nanjing yutianzhiyun Simulation Technology Co.,Ltd.

Country or region before: China