KR20230164990A - Method and apparatus for visualizing glass-free multiview autostereoscopic three-dimensional display - Google Patents

Abstract

The glasses-free multi-view stereoscopic display visualization device acquires plenoptic image data for at least one video from a plenoptic imaging device, and generates plenoptic spatial division images for each frame using the plenoptic image data for each video, Using the frame-by-frame plenoptic spatial division image for each video, a multi-view image corresponding to the number of viewpoints that can be provided by the glasses-free multi-view stereoscopic display device is generated for each frame for each video, and then each video is divided into frames corresponding to each video. Using multi-view video, different videos are played from each viewpoint.

Description

Glasses-free multi-view stereoscopic display visualization method and device {METHOD AND APPARATUS FOR VISUALIZING GLASS-FREE MULTIVIEW AUTOSTEREOSCOPIC THREE-DIMENSIONAL DISPLAY}

The present disclosure relates to a glasses-free multi-viewpoint stereoscopic display visualization method and device, and more specifically, to generate a random viewpoint image for a random viewpoint based on plenoptic image data for two or more videos to display two or more videos simultaneously. The present invention relates to a method and device for visualizing a glasses-free multi-view stereoscopic display using a glasses-free multi-view stereoscopic display device.

3D content can generally give users a three-dimensional feeling similar to reality, so it is applied and utilized in a variety of fields such as movies, advertisements, and games.

To create 3D content, 3D mesh data is mainly used. When using 3D mesh data, you must scan the subject using a scanning method, create a 3D mesh, and correct the crushed texture. Additionally, in order to realistically express the movement of liquid, a liquid implementation engine or asset must be applied. Additionally, when using 3D mesh data, it is difficult to implement reflective materials such as metal or glass, or irregular deformation of the subject, and it takes a lot of time and money.

The problem to be solved by the present invention is to provide a glasses-free multi-viewpoint that can provide different three-dimensional content depending on the viewing point of view while reducing time and cost by using plenoptic image data acquired through a plenoptic imaging device. The object is to provide a three-dimensional display visualization method and device.

According to one embodiment, a method of visualizing an image through a glasses-free multi-view stereoscopic display device is provided. The glasses-free multi-view stereoscopic display visualization method includes obtaining plenoptic image data for at least one video from a plenoptic imaging device, and generating plenoptic spatial division images for each frame using plenoptic image data for each video. Step, generating a multi-view image corresponding to the number of viewpoints that can be provided by the glasses-free multi-view stereoscopic display device for each frame for each video using a plenoptic spatial division image for each video, and corresponding to each video It includes the step of playing different videos for each viewpoint using a multi-view video for each frame.

According to an embodiment, reflective materials (metal, liquid, glass, etc.), which are difficult to implement in existing mesh-based stereoscopic images, irregular deformation of the subject (cutting of food, etc.), and live-action stereoscopic video are captured using a plenoptic imaging device. After shooting, the plenoptic image data acquired from the plenoptic imaging device is used to generate a viewpoint image for a glasses-free multi-viewpoint stereoscopic display, effectively providing realistic three-dimensional images to users through a glasses-free stereoscopic multiview display. It can provide different stereoscopic images depending on the viewing angle. Through this, it is expected that it can be widely used in the advertising field.

Figure 1 is a diagram showing a glasses-free multi-view stereoscopic display visualization device according to an embodiment.
Figure 2 is a diagram showing an example of a plenoptic imaging device.
FIG. 3 is a diagram showing images observed from different directions for one video played through the glasses-free multi-view stereoscopic display visualization device shown in FIG. 1.
FIG. 4 is a diagram showing images observed from different directions for a plurality of videos played through the glasses-free multi-view stereoscopic display visualization device shown in FIG. 1.
Figure 5 is a diagram showing an actual application example of Figure 4.
FIG. 6 is a diagram showing images allocated to each viewpoint for one frame for the visualization of FIG. 5.
Figure 7 is a flowchart showing a glasses-free multi-view stereoscopic display visualization method according to an embodiment.
Figure 8 is a diagram showing a glasses-free multi-view stereoscopic display visualization device according to another embodiment.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification and claims, when a part is said to “include” a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Now, a glasses-free multi-view stereoscopic display visualization method and device according to an embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram illustrating an autostereoscopic multi-view stereoscopic display visualization device according to an embodiment, and FIG. 2 is a diagram illustrating an example of a plenoptic imaging device.

Referring to FIG. 1 , the glasses-free multi-viewpoint stereoscopic display visualization device includes at least one plenoptic image processing unit 110, a viewpoint image allocation unit 120, a composite rendering unit 130, and a playback unit 140.

The plenoptic image processing unit 110 provides one viewpoint corresponding to the number of viewpoints that can be provided by the glasses-free multi-view stereoscopic display device based on the plenoptic image data for one scene acquired from the plenoptic imaging device 10. Create a multi-view image of a scene. Here, God may mean a video and represents a single piece of independent content.

Specifically, the plenoptic image processing unit 110 includes a plenoptic image acquisition unit 111, a plenoptic image storage unit 112, a spatial division image generation unit 113, a spatial division image storage unit 114, and a viewpoint image generation unit. It may include unit 115.

The plenoptic image acquisition unit 111 acquires plenoptic image data for one video from the plenoptic imaging device 10.

As shown in FIG. 2, the plenoptic imaging device 10 includes a camera array in which a plurality of cameras 1 are arranged.

A plurality of cameras 1 captures a subject 20 and creates a video. The plenoptic image acquisition unit 111 may acquire a plurality of plenoptic image data each captured by a plurality of cameras 1.

Referring again to FIG. 1, plenoptic image data for one video acquired from the plenoptic imaging device 10 is stored in the plenoptic image storage unit 112.

Plenoptic image data can be expressed as intensity with respect to a coordinate system (θ, Φ, λ, t, x, y, z,), where θ and Φ represent the direction of light, and λ is the wavelength (color). , and t represents time. (x, y, z) represents the three-dimensional position of the observer.

Since the plenoptic image data is stored in the plenoptic image storage unit 112 as is the image captured by the plenoptic imaging device 10, the plenoptic image data includes images taken at any three-dimensional position (x, y, z). Video compositing is difficult to perform.

In order to facilitate image synthesis at arbitrary 3D positions (x, y, z), the spatial division image generator 113 generates a plenoptic spatial division image using plenoptic image data. The spatial division image generator 113 divides the plenoptic image data for each camera into objects existing on the screen from a position close to the camera to a position far from the camera by distance, and generates a plenoptic spatial division image for each distance.

Plenoptic spatial division images for each camera distance for one video are stored in the storage unit 114.

The viewpoint image generator 115 generates a multi-viewpoint image corresponding to the number of viewpoints that can be provided by the glasses-free multi-viewpoint stereoscopic display device based on the plenoptic spatial division image for each distance of each camera for one video.

In this way, the plenoptic image processing unit 110 generates a multi-view image for each frame corresponding to the number of viewpoints that the autostereoscopic multi-view stereoscopic display device can provide from one video.

In addition, a multi-viewpoint image for each frame corresponding to the number of viewpoints that can be provided by the autostereoscopic multi-viewpoint stereoscopic display device can be generated from each of the plurality of videos in parallel through the plurality of plenoptic image processing units 110.

In contrast, a single plenoptic image processing unit 110 can generate a multi-view image for each frame corresponding to the number of viewpoints that can be provided by the autostereoscopic multi-view stereoscopic display device from each of a plurality of videos in a sequential manner.

In order to visualize the multi-viewpoint image for each frame corresponding to one video through the unglasses multi-viewpoint stereoscopic display device, the viewpoint image allocation unit 120 assigns the corresponding video to each viewpoint position of the autoglasses multi-viewpoint stereoscopic display device for each frame. Allocate a viewpoint image.

The viewpoint image allocation unit 120 selects each frame of the unglasses multi-viewpoint stereoscopic display device from the multi-viewpoint image for each frame corresponding to each of the plurality of videos, in order to visualize the plurality of videos through the autostereoscopic multi-viewpoint stereoscopic display device. The viewpoint image of any one video can be assigned to the viewpoint position. For example, assume that the field of view (FOV) of a glasses-free multi-view stereoscopic display device is 45 degrees, the number of viewpoints is 45, and 45 multi-view images are generated for each frame corresponding to each of the A and B videos. . The image allocation unit 120 allocates 45 corresponding viewpoint images for each of the 45 viewpoints provided by the glasses-free multi-viewpoint stereoscopic display device for each frame, and each viewpoint position from 1 to 22 for each frame contains the corresponding frame in the A video. The viewpoint images from 1 to 22 are assigned, and the viewpoint images from 23 to 45 of the corresponding frame in the B video are assigned to each viewpoint position from 23 to 45 for each frame.

The composite rendering unit 130 synthesizes the viewpoint images assigned to each viewpoint position of the autostereoscopic multi-view stereoscopic display device for each frame and generates a video corresponding to each viewpoint.

The playback unit 140 plays the video corresponding to each viewpoint. The playback unit 140 includes a glasses-free multi-view stereoscopic display device.

If the image allocation unit 120 allocates the viewpoint image of the video to each viewpoint position of the autostereoscopic multi-view stereoscopic display device for each frame using the multi-viewpoint image for each frame corresponding to one video, the playback unit 140 The video is played at each viewpoint.

Meanwhile, when different viewpoint images of a video are assigned to each viewpoint position of the autostereoscopic multi-view stereoscopic display device for each frame by the video allocation unit 120 using the multi-viewpoint image for each frame corresponding to each of the plurality of videos, the video is played. The video is played from each viewpoint by the unit 140, and at this time, different videos are displayed depending on the viewing angle of the user.

FIG. 3 is a diagram showing images observed from different directions for one video played through the glasses-free multi-view stereoscopic display visualization device shown in FIG. 1.

Referring to FIG. 3, the glasses-free multi-view stereoscopic display visualization device 100 generates a video for each viewpoint using a multi-view image for each frame corresponding to one video, and plays the video for each viewpoint.

Then, the user sees images from different viewpoints in both eyes depending on the viewing direction (a, b, c).

FIG. 4 is a diagram showing images observed from different directions for a plurality of videos played through the glasses-free multi-view stereoscopic display visualization device shown in FIG. 1.

Referring to FIG. 4, the glasses-free multi-view stereoscopic display visualization device 100 plays different videos for each viewpoint using multi-view images for each frame corresponding to each of a plurality of videos.

For example, Assuming that a glasses-free multi-view stereoscopic display device has 45 viewpoints and allocates viewpoint indices from 1 to 45 from left to right, A video is played at viewpoint positions from 1 to 22, and viewpoints from 23 to 45 are played. The B video is played at this location.

Then, the user sees different viewpoint images of video A in both eyes in direction a, and different viewpoint images of video B in both eyes in direction c. Meanwhile, in direction b, the viewpoint image of video A and the viewpoint image of video B may be displayed to both eyes.

Figure 5 is a diagram showing an actual application example of Figure 4.

As shown in FIG. 5, the glasses-free multi-viewpoint stereoscopic display visualization device 100 can provide images from different viewpoints of the video (C, D) according to the viewing angle of the user through the glasses-free multi-viewpoint stereoscopic display device. there is.

FIG. 6 is a diagram showing images allocated to each viewpoint for one frame for the visualization of FIG. 5.

Referring to FIG. 6, the glasses-free multi-viewpoint stereoscopic display visualization device 100 allocates the video from viewpoints 1 to 22 of the corresponding frame in the C video to viewpoint positions 1 to 22 of the autoglasses multi-viewpoint stereoscopic display device for each frame. And, for each frame, the video from viewpoints 23 to 45 of the corresponding frame in the B video is assigned to viewpoint positions 23 to 45, respectively.

In this way, it is possible to provide images from different viewpoints of the video (C, D) depending on the viewing angle of the user, as shown in FIG. 5.

Additionally, if the user moves from viewpoint 1 to 22 of the same video, a three-dimensional effect can be felt from various angles.

Additionally, when shooting with the plenoptic imaging device 10, the sense of space can be maintained as much as possible by creating different videos while maintaining the background. For example, assuming that you want to create multiple videos by filming various udon menu items such as basic udon, tempura udon, curry udon, and kimchi udon in one udon bowl, the background such as the udon bowl and table are fixed and the udon bowl is Only the udon being placed is changed according to the menu and then photographed using the plenoptic photographing device (10). The plurality of videos according to each udon menu filmed in this way are multi-viewpoint images for each frame are generated through the process described in FIG. 1, and different videos for each viewpoint are created using the multi-viewpoint images for each frame corresponding to each of the plurality of videos. It becomes visible.

Figure 7 is a flowchart showing a glasses-free multi-view stereoscopic display visualization method according to an embodiment.

Referring to FIG. 7, the plenoptic image processing unit 110 acquires plenoptic image data for one video from the plenoptic imaging device 10 (S710).

The plenoptic image processing unit 110 generates a plenoptic spatial division image for each frame using plenoptic image data for one video (S720).

The plenoptic image processing unit 110 uses the plenoptic spatial division image for each frame to generate a multi-view image corresponding to the number of viewpoints that can be provided by the glasses-free multi-view stereoscopic display device for each frame (S730).

In this way, a multi-view image is generated for each frame of one video.

If there is another video to be synthesized (S740), the plenoptic image processing unit 110 generates a multi-view image for each frame for the other video in the same manner as described above.

Next, the viewpoint image allocation unit 120 selects the multi-viewpoint image for each frame corresponding to each of the plurality of videos, to visualize the plurality of videos through the autostereoscopic multi-viewpoint stereoscopic display device, frame by frame. A viewpoint image of one video is assigned to each viewpoint position in (S750).

The synthesis rendering unit 130 synthesizes the viewpoint images for each frame assigned to each viewpoint position of the autostereoscopic multi-viewpoint stereoscopic display device and generates a video corresponding to each viewpoint (S760).

The playback unit 140 plays the video corresponding to each viewpoint.

Figure 8 is a diagram showing a glasses-free multi-view stereoscopic display visualization device according to another embodiment.

Referring to FIG. 8 , the glasses-free multi-viewpoint stereoscopic display visualization device 800 may represent a computing device that implements the glasses-free multiview stereoscopic display visualization method described above.

The glasses-free multi-view stereoscopic display visualization device 800 includes at least a processor 810, a memory 820, an input interface device 830, an output interface device 840, a storage device 850, and a network interface device 860. It can contain one. Each component is connected by a common bus 870 and can communicate with each other. Additionally, each component may be connected through an individual interface or individual bus centered on the processor 810, rather than the common bus 870.

The processor 810 may be implemented in various types such as an Application Processor (AP), Central Processing Unit (CPU), Graphic Processing Unit (GPU), etc., and executes instructions stored in the memory 820 or storage device 850. It may be any semiconductor device. The processor 810 may execute a program command stored in at least one of the memory 820 and the storage device 850. This processor 810 has a program memory memory for implementing at least some functions of the plenoptic image processing unit 110, the viewpoint image allocation unit 120, the composite rendering unit 130, and the reproduction unit 140 described in FIG. 1. By storing it in 820, it can be controlled to perform the operations described based on FIGS. 1 to 7.

Memory 820 and storage device 850 may include various types of volatile or non-volatile storage media. For example, the memory 820 may include a read-only memory (ROM) 821 and a random access memory (RAM) 822. The memory 820 or storage device 850 may include a plenoptic image storage unit 112 and a spatial division image storage unit 114. The memory 820 may be located inside or outside the processor 810, and the memory 820 may be connected to the processor 810 through various known means.

The input interface device 830 is configured to provide data to the processor 810.

The output interface device 840 is configured to output data from the processor 810.

The network interface device 860 can transmit or receive signals with other devices through a wired network or wireless network.

At least some of the autostereoscopic multi-view stereoscopic display visualization method according to an embodiment of the present disclosure may be implemented as a program or software running on a computing device, and the program or software may be stored in a computer-readable medium.

In addition, at least some of the autostereoscopic multi-view stereoscopic display visualization method according to an embodiment of the present disclosure may be implemented as hardware that can be electrically connected to a computing device.

Although the embodiments of the present disclosure have been described in detail above, the scope of the rights of the present disclosure is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present disclosure defined in the following claims are also possible. It falls within the scope of rights.

Claims (1)

In a method of visualizing an image through a glasses-free multi-view stereoscopic display device,
Obtaining plenoptic image data for at least one video from a plenoptic imaging device,
Generating a plenoptic spatial segmentation image for each frame using plenoptic image data for each video,
A step of generating a multi-view image corresponding to the number of viewpoints that can be provided by the glasses-free multi-view stereoscopic display device for each frame of each video using a plenoptic spatial division image for each frame, and
A step of playing different videos for each viewpoint using multi-view images for each frame corresponding to each video.
A glasses-free multi-view stereoscopic display visualization method comprising:

Images