KR102616646B1 - Realtime dynamic image warping system for screen based glasses-free VR and its verification method - Google Patents

Abstract

The present invention relates to a glasses-free immersive virtual reality image registration system applicable to all screens and a method for verifying the image registration. More specifically, the present invention relates to the geometric angle of three-dimensional content and images projected on the screen projection surface in real time based on the user's location. This relates to a real-time image matching system for screen-based glasses-free VR that tracks and matches three-dimensional images projected on the screen surface to obtain the same visual effect as looking at the real world from a viewing angle, and a method for verifying the image matching.

Description

Real-time image warping system for screen-based glasses-free VR and its verification method {Realtime dynamic image warping system for screen based glasses-free VR and its verification method}

The present invention relates to a glasses-free immersive virtual reality image registration system applicable to all screens and a method for verifying the image registration. More specifically, the present invention relates to the geometric angle of three-dimensional content and images projected on the screen projection surface in real time based on the user's location. This relates to a real-time image matching system for screen-based glasses-free VR that tracks and matches three-dimensional images projected on the screen surface to obtain the same visual effect as looking at the real world from a viewing angle, and a method for verifying the image matching.

When visualization technologies such as virtual reality, augmented reality, and complex reality are applied to metaverse platforms, civilian and military simulations, play and exhibition facilities, factory automation, and smart farms, equipment in the form of Head Mounted Display (HMD) is usually used. there is. However, HMD-type equipment has problems such as inconvenience in wearing, safety accidents due to visual blocking from the outside world, increased intraocular pressure due to distant images and close focus on display devices, perspective effects that are different from the real world, and inconvenience in long-term use. There are many problems.

For this reason, even if high costs are applied, there is an urgent need for technology that maximizes the sense of immersion in virtual reality by tracking the location of the experiencer in real time in a screen-based three-dimensional simulation or exhibition content system that does not use an HMD.

Accordingly, the present inventor developed a glasses-free type virtual reality implementation system.

In the prior invention (Patent No. 10-2245223), which is the basis of the present invention, a virtual reality implementation technology for displaying multi-layered beam projection content on an irregular and unstructured projection surface was presented. The invention compares an image obtained by projecting beam projection content on a three-dimensional geometric model with an image obtained by projecting the beam projection content on the projection surface, and then corrects the projected image using a center of gravity-based position change correction algorithm. . As shown in Figures 5a to 5c, the center of gravity for vertices A, B, and C becomes point P, and when the position of the vertex is changed through image warping, the center of gravity P' created by the new vertices A', B', and C' It is a technology that allows content to be displayed without visual distortion by using a warping algorithm in which the area (w) distribution is transformed into w' and the image corresponding to the area is transformed.

However, this technology has limitations in realizing a more immersive virtual reality as it changes dynamically in real time according to the user's location change.

To overcome these limitations, the present invention is based on the above warping algorithm, but applies a warping algorithm that is modified in real time. Image warping is a beam projection design that determines the projection range and position of the beam projector according to the physical design of the video screen, and a technology that corrects the image of video content to match the viewing angle to the physical shape of the video screen. By tracking the location or viewing angle of the experiencer (or user), the experiencer can experience a more natural and immersive virtual reality, and the image matching algorithm is mounted directly on the three-dimensional content and processed to improve image delay delivery and presentation performance during the content development process. All limitations issues have been resolved.

Accordingly, the present invention was conceived to solve the above-mentioned problems, and the geometric angle of the three-dimensional content and image projected on the screen projection surface is tracked and matched in real time based on the user's location to adjust the viewing angle of the three-dimensional image projected on the screen surface. The purpose is to provide a real-time image matching system for screen-based glasses-free VR that achieves the same visual effect as looking at the real world and a verification room for the image matching.

To achieve this purpose, the real-time image matching system for screen-based glasses-free VR according to the present invention is a device equipped with a memory and microprocessor, and includes an image mapping unit 102, a storage unit 103, and a location tracking unit ( 104), an image processing device 100 including a real-time image correction unit 105, an image output unit 106, and a control unit 101, and a beam projector for projecting the image output from the image output unit 106. As the same device, an image display device 110, a hyperspectral location tracking device 120 in which the location tracking unit 104 tracks the user's location in real time in order to transmit real-time location information of the user to the control unit 101, and the image The output unit 106 reflects the image correction information of the real-time image correction unit 105 into the image content and finally outputs the video content, thereby including an image screen 130 that implements virtual reality.

In addition, the real-time image registration verification method for screen-based glasses-free VR includes the steps of setting a reference point for image correction through defining the user starting position, measuring and displaying the viewing angle distribution (S10), and running the computer 100 after setting the reference point. After projecting the content, an initialization image correction step (S20) of matching the viewing angle distribution grid displayed in the content with the corresponding viewing angle marker displayed on the screen through the image correction function. After the initialization correction is completed, the hyperspectral positioning device 120 A step of operating and measuring the changed position by tracking the position where the user moves from the initial position to a random position (S30) and measuring the viewing angle distribution of the position change point according to the moving position (S40), measurement point and content It is characterized by including a viewing angle distribution error confirmation step (S50), measurement completion and result confirmation step (S60).

The present invention tracks the real-time location change of the user through the development of a real-time image registration test module and image correction software, and provides a system that provides the user with a viewing angle and image similar to the real world even if the user is free to move the location within the screen environment. Because users can always experience images that match their viewing angle without being distorted regardless of changes in their position, this creates a more dramatic visual effect in fields such as stand-alone shooting simulators and experience simulators where it is difficult or unwanted to use an HMD. can be provided.

In addition, the present invention can be expanded to curved and spherical screen-based in addition to the flat screen-based presented as an example, and to secure commercial feasibility through advancement and effectiveness testing, so that more immersive content can be created in various public-private-military simulation video systems, experience centers, exhibition halls, etc. system can be provided.

Figure 1 is a configuration diagram of a glasses-free VR system according to the present invention.
Figure 2 is a block diagram of the image processing device of Figure 1
Figure 3 is a configuration diagram of an embodiment of real-time image registration of the glasses-free VR system according to the present invention.
Figure 4 shows the correlation between user vision and projection.
Figure 5 is an example of image distortion phenomenon according to change in user viewpoint.
Figure 6 shows an example of content of the glasses-free VR system according to the present invention.
Figure 7 is a flowchart of a real-time image matching method for screen-based glasses-free VR according to the present invention.
Figure 8 shows content projection and initial image correction of the glasses-free VR system according to the present invention.
Figure 9 is a correction completed image of the glasses-free VR system according to the present invention.
Figure 10 shows user location tracking using a hyperspectral location tracking device.
Figure 11 shows the measurement stage after the software is paused.
Figure 12 is a user location image before correction
Figure 13 is a user location image after correction

The realization of the present invention involves complex factors such as the physical design of the video screen, the optical design of the video projection device, the viewing angle design of the three-dimensional content, real-time image matching and edge-blending processing, location tracking of the experiencer, and delayed delivery processing. Consideration of design and system processing is required. In addition, the invention is achieved through image correction based on a non-parametic structure. Non-parametric image correction minimizes the impact on other areas by linking the surrounding coordinates around the point to be corrected in the entire image. It is an algorithm that can be corrected. The present invention redistributes the angular position on the image in real time according to changes in the user's position to always provide the same viewing angle no matter where the user goes.

This will be explained in detail through drawings.

Figure 1 is a block diagram of a glasses-free VR system according to the present invention, and Figure 2 is a block diagram of the image processing device of Figure 1.

The present invention includes an image processing device 100, an image display device 110, a hyperspectral location tracking device 120, and an image screen 130.

There is no limitation in using the image processing device 100 as an image processing device of the present invention as long as it is a device equipped with a memory and a microprocessor.

The image display device 110 is a beam projector-like device that projects images and may be provided in plural numbers (111, 112, 113...n).

The hyperspectral location tracking device 120 is a device that tracks the user's location. When using conventional devices that use stereo cameras, IMUs, ultrasonic sensors, etc., there are limitations due to slow detection response speed or accumulation of position tracking errors, so it is recommended to use Lidar sensors. desirable. Lidar can secure the precision and rapid response speed of the user's real-time location tracking.

There are various types of video screens 130, such as flat, curved, and spherical. For typical virtual reality implementation, spherical screens are most suitable, and spherical screens are also the best for immersion.

2, the image processing device 100 in the glasses-free VR system according to the present invention will be described.

The image processing device 100 includes an image mapping unit 102, a storage unit 103, a location tracking unit 104, a real-time image correction unit 105, an image output unit 106, and a control unit 101.

The image mapping unit 102 is an image correction module that performs initial correction to match the image to the physical shape of the screen through a warping function. This correction information becomes the reference point setting information for initialization.

The storage unit 103 functions to store correction information acquired through the image mapping unit 102 and update information changed as needed, such as changing the user's reference position or physically changing the screen.

The location tracking unit 104 processes and transmits the user's real-time location information obtained from the lidar sensor 120 to the control unit 101 and performs location information delivery, sensitivity adjustment, and location correction.

The real-time image correction unit 105 adjusts the warping information stored in the storage unit 103 according to the user's location changed according to the location information obtained and processed from the location tracking unit 104 to match the real-time location of the user. It performs the role of correction with warping information. This is temporary data, and when stopped being used, it returns to S30, which is the initial position warping data stored in the storage unit 103.

The image output unit 106 is an area that performs final output (rendering) by reflecting the image correction information of the real-time image correction unit 105 to the image content.

The control unit 101 operates the operation interface of the image mapping unit 102, data management of the storage unit 103, data reception management of the location tracking unit 104, real-time matching processing of the real-time image correction unit 105, and image output unit ( 106) It performs the role of integrated management of content video output.

Through Figure 3, the real-time image matching algorithm of the glasses-free VR system according to the present invention is explained.

Looking at the entire system of the present invention organically, the image processing device 100 is a computer equipped with a memory and a microprocessor, and processes the user's location information tracked through the LiDAR sensor 120 and produces an image suitable for the location. Correction is processed in real time. The image is reflected on the image content mounted on the computer 100 and projected on the screen 130 through the beam projector 110, which is an image display device, through which the user always receives a constantly corrected image regardless of real-time location changes. You will experience a sense of immersion.

With the improvement of computer hardware performance, software technology has been developed to process image registration by hooking video output signals directly from the graphics card of the content computer, so the burden of video delay delivery has disappeared, but image processing is directly connected to the content. There is still a problem of deteriorating image display performance (frame rate) because it consumes computer performance resources. The present invention applies Dynamic Image Warping while managing image presentation performance during the content development process by directly loading and processing the Dynamic Image Warping algorithm on three-dimensional content, thereby solving both the problems of image delay delivery and limitations in presentation performance.

In other words, it can be confirmed that the warping is automatically readjusted according to the user's real-time location change, resulting in a viewing angle distribution suitable for the user's perspective. Through this, the user always has a viewing angle distribution suitable for him/her at any location, resulting in an effect similar to wearing an HMD device. At this time, since the image is affected by the shape and angle of the screen on which it is projected, a formula according to the screen shape is applied as shown in Table 1 above. The algorithm in Table 1 above is presented because a spherical screen is most suitable for implementing typical virtual reality.

Through Figure 4, user vision and three-dimensional content visual processing will be explained.

In a screen projection situation, the size of the screen, that is, the image projection range and angle, is determined according to the field of view determined from the user's eyepoint location and the distance from the screen.

For this reason, when the viewing angle is fixed, as the distance between the viewpoint and the screen increases, the screen must become larger, and as the distance approaches, the screen must become smaller, and the projection range and virtual presentation angle of the projected content change accordingly.

Figure 5 (a) (b) shows how image distortion occurs when the projection angle does not change according to the change in the distance between the actual viewpoint position and the screen.

As shown in the drawing, even if the image projected on the screen is evenly arranged and projected according to the user's viewing angle, as the user's position changes and the physical viewing angle of the screen expands, the projected image expands and the image changes in proportion to the expanded viewing angle. Distortion occurs where the image is concentrated and refracted in the center, which is different from the angle viewed in the real world.

Figure 6(a) shows an example of content according to the present invention, and Figure 6(b) shows the final calculated content.

Figure 6(a) is a real-time 3D content development screen for verifying the present invention. A virtual camera corresponding to the user's viewpoint and a virtual sphere centered around it are provided, and the angle of the virtual sphere can be confirmed from the virtual camera's viewpoint. The grid and angle numbers at 5-degree intervals are displayed as text. Figure 6(b) is a content execution screen, showing the angle distribution for each viewpoint as seen from the virtual camera viewpoint (same as the user's viewpoint).

As an example of the present invention, the real-time image registration module system was performed on a flat screen basis under single image output to check accuracy and make quick corrections.

When applying a flat screen to ensure precision in measuring the viewing angle angle change, a high-resolution laser light source beam projector with high brightness of 5000 lumens and 1920 A lens ratio of 0.81:1 projection ratio was applied. In order to secure the precision and quick response speed of the user's real-time location tracking, the LiDAR sensor can be used to track the location at a maximum of 9000Hz. However, when too much data is detected, the content software cannot accept the performance, so it was adjusted to 90Hz. . The image correction solution is based on a non-parametic structure, and an algorithm is used that allows correction while minimizing the impact on other areas by linking the surrounding coordinates to the point to be corrected in the entire image.

It is a computer-mounted content for conducting tests. It is arbitrary image content that expresses a three-dimensional space, and includes a viewing angle distribution grid output function that can check the viewing angle distribution according to the user's eye point change.

The test content was produced based on a game engine (Unity3D) and consisted of real-time dynamic video. In addition, based on the location information transmitted from the hyperspectral location tracking device (Lidar), image correction control points (Warping Control points) are arranged in real time to suit the viewing angle distribution that changes according to the user's location change (X, Y plane coordinates). It is equipped with the following functions.

Figure 7 explains the real-time image registration verification method for screen-based glasses-free VR according to the present invention.

The present invention includes a reference point setting step for image correction (S10), an initial image correction step (S20), a position tracking and position change measurement step (S30), a viewing angle distribution measurement step at the location change point (S40), and an error checking step between distributions (S50). ), measurement completion and result confirmation step (S60).

The reference point of image correction is the step of confirming the center point of the screen and defining the user's location in order to set the user's starting point (eye point). A goniometer is installed at the user's starting position at the user's eye level, measures the viewing angle at 5-degree intervals, and displays a marker to visually confirm the angle.

Figure 8 shows the content projection and initial image correction work of the glasses-free VR system according to the present invention. Through this, these markers are set as reference points for initial image correction and angle change by defining the user starting position, measuring and displaying the viewing angle distribution (S10).

After setting the reference point, the computer 100 is run, the content is projected, and an initial image correction step is performed to match the viewing angle distribution grid displayed in the content with the corresponding viewing angle marker displayed on the screen through the image correction function (S20).

Figure 9 shows an image of the initialization and correction completion of the glasses-free VR system according to the present invention. At this stage, the image correction is the initial initialization correction and does not need to be applied thereafter. If the correction operation is performed normally, the viewing angle grid of the projected image is evenly distributed and appears close to a square shape when viewed from the initial setting position.

When the initialization correction is completed, the hyperspectral location tracking device is operated and the user moves from the initial location to a random location to track the location to measure the changed location (S30).

Figure 10 shows the process of performing a user location tracking process using a hyperspectral location tracking device. To facilitate measuring the change in user position, move as far away from the initial position as possible within the screen range.

After moving, a goniometer must be installed and the viewing angle distribution must be measured again at that location, so the moved location is indicated. At this time, the location tracking function is stopped and the content is temporarily stopped, and the moving position is displayed. Figure 11 shows the state in which the software is paused and measured.

In the step (S40) of measuring the viewing angle distribution of the position change point according to the moving position, image registration information (Warping), which is automatically changed by the image processing device 100 after the user moves, is measured at the user's changed position. This is the step to verify whether it matches the actual viewing angle information.

When the user's movement and change location is tracked with the LiDAR sensor 120 and the location data is transmitted to the verification content, image correction is automatically applied. At this time, the content is paused to maintain the correction state and a goniometer is placed at the user's changed location. Install it to measure the actual angle of the viewing angle grid information displayed in the content. Through this measurement, the closer the display angle of the content viewing angle grid is to the actual measurement angle, the more images of situations similar to the actual situation are provided to the user.

This step is a necessary procedure to ensure that the real-time image correction algorithm operates normally, that image correction is applied appropriately to the user's changed position, and that the angle displayed in the content is similar to the actual measured angle.

The measurement point and content viewing angle distribution error confirmation step (S50) is a step to increase reliability.

In theory, the actual viewing angle distribution measured at the changed position should match the grid angle displayed in the content, but in reality, errors may occur due to various variables such as the physical uniformity of the screen, position changes that occur during installation of the goniometer, and eye level changes. .

However, since minor errors are ignored and operated in the existing commonly used fixed-point image correction, the minor errors due to the variables specified above are judged to be no abnormalities in operation.

By setting up a routine that returns to S30, you can further secure the reliability of the measurement values. In other words, reliability can be further increased by measuring the error several times and obtaining average data. When applying to actual content, the measurement step may not be performed, and in the case of tolerance, it is appropriate to view it as within 1 degree. Since the installation of the goniometer at the changed location is done manually, minor positional errors that occur during installation should be ignored. This is because subtle angle changes can occur depending on the user's neck movements, changes in posture, etc. In addition, the angle error caused by this is within a range of 1 degree, so it is difficult to distinguish through visual judgment alone without precise measurement.

[Table 2] above shows the error value compared to the actual angle displayed on the goniometer when the display angle (measurement viewing angle) of the grid displayed on the content is measured with a goniometer. For example, when the X: 15°, Y: 0° points of the grid displayed in the content are measured with a goniometer,

Measurement completion and result confirmation stage (S60)

Through the measurement results and error analysis in [Table 2], the change in viewing angle according to the user's location was confirmed, in line with the expected effect. However, as mentioned earlier, an error of approximately 0.1 degrees occurred due to the positioning error that occurred between the installation of the goniometer and the slight physical curvature of the screen surface.

Figure 12 is an image of the user's position before correction, and the user is looking at the screen in the same diagonal direction. However, in the image before correction, it can be observed that the viewing angle is stretched according to the direction of the screen and the vanishing point, and the image visible in the content is also oriented in the direction of disappearance. It is confirmed that it is sagging.

Figure 13 is an image of the user's location after correction. It can be seen that the viewing angle of the image after correction is evenly distributed according to the user's perspective regardless of the vanishing point, and the same effect as looking at an object in the real world is displayed and the content image is not distorted. You can confirm that it is not.

100: image processing device 101: control unit
102: Image mapping unit 103: Storage unit
104: Location tracking unit 105: Image correction unit
106: video output unit 110: video display device
120: Hyperspectral location tracking device 130: Video screen

Claims (8)

It is a device equipped with memory and microprocessor and includes an image mapping unit 102, a storage unit 103, a location tracking unit 104, a real-time image correction unit 105, an image output unit 106, and a control unit 101. An image processing device 100 including; An image display device 110 as a device such as a beam projector that projects the image output from the image output unit 106; The location tracking unit 104 transmits the user's real-time location information from the control unit 101 to the user. A hyperspectral location tracking device 120 that tracks the location in real time; And an image screen 130 for implementing virtual reality by reflecting the image correction information of the real-time image correction unit 105 in the image output unit 106 to the image content and finally outputting it. In the image registration system including,
The image mapping unit 102 defines the user's location by checking the center point of the screen to set the user's starting point (eye point) as a reference point for image correction, and uses the warping function as an image correction module to display the image on the screen. The first correction is performed according to the physical shape of and this correction information becomes the reference point setting information for initialization.
The real-time image correction unit 105 adjusts the warping information stored in the storage unit 103 according to the user's location changed according to the location information obtained and processed from the location tracking unit 104 to match the real-time location of the user. It performs the role of correction with warping information, and the warping is automatically readjusted according to the user's real-time location change, adjusting the screen size or image projection range according to the field of view determined from the changed user's location and the distance to the screen. The angle is determined as in the equation below,

Based on the location information transmitted from the hyperspectral location tracking device 120, image correction control points are arranged in real time to suit the viewing angle distribution that changes according to the user's location change (X, Y plane coordinates). A real-time image matching system for screen-based glasses-free VR, characterized by including the function delete According to clause 1,
The location tracking unit 104 processes and transmits the user's real-time location information obtained from the lidar sensor 120 by the control unit 101, and is characterized in that it transmits location information, adjusts sensitivity, and performs location correction. Real-time image matching system for screen-based glasses-free VR delete According to clause 1,
When applying a flat screen to ensure precision in measuring the viewing angle angle change, a high-resolution laser light source beam projector with high brightness of 5000 lumens and 1920 A real-time image registration system for screen-based glasses-free VR, characterized by applying a lens ratio of 0.81:1 projection ratio. According to clause 1,
A real-time image registration system for screen-based glasses-free VR that uses a lidar sensor to track the user's real-time location and ensures fast response speed by tracking the location at up to 9000Hz and adjusting it to 90Hz. Setting a reference point for image correction by measuring and displaying the viewing angle distribution at 5-degree intervals from the user's starting position (S10);
After setting the reference point, running the computer 100 and projecting the content, an initialization image correction step (S20) of matching the viewing angle distribution grid displayed on the content with the corresponding viewing angle marker displayed on the screen through an image correction function;
After the initialization correction is completed, operating the hyperspectral location tracking device 120 and measuring the changed location by tracking the location where the user moves from the initial location to a random location (S30);
When the changed location data is transmitted to the verification content, image correction is automatically applied. At this time, the content is paused to maintain the correction state, and a goniometer is installed at the user's changed location to measure the actual angle of the viewing angle grid information displayed in the content. , measuring the viewing angle distribution of the position change point according to the moving position (S40);
Setting up a routine to return to S30 with the viewing angle tolerance within 1 degree at the changed position, and checking the measurement point and content viewing angle distribution error step (S50);
Real-time image registration verification method for screen-based glasses-free VR, comprising a measurement completion and result confirmation step (S60). According to clause 7,
Real-time image registration verification method for screen-based glasses-free VR, characterized by performing steps to further ensure reliability of measured values by setting a routine returning to S30