KR20210114668A - Apparatus, method and computer program for generating 3d image of target object - Google Patents

Abstract

Provided is a device for generating a stereoscopic image for a target object, which includes: an acquisition unit which acquires a plurality of input images acquired by photographing a target object through a plurality of cameras; a target viewpoint image generation unit which groups the plurality of input images into a plurality of groups, and generates a plurality of target viewpoint images including viewpoint information on the target object based on at least two images included in each group; a synthesized viewpoint image generation unit which generates a synthesized viewpoint image by synthesizing the plurality of target viewpoint images and the background image; and a stereoscopic image generation unit which generates a stereoscopic image based on the synthesized viewpoint image.

Description

Apparatus, method and computer program for generating a stereoscopic image of a target object

The present invention relates to an apparatus, method and computer program for generating a stereoscopic image of a target object.

In general, a plurality of images are input to generate a three-dimensional image using a light field display, a multi-viewpoint display, a super multi-viewpoint display, and the like. The number of input images is determined according to the specification of a display and a stereoscopic image to be generated.

For example, 54,000 (30 (number of views) × 30 (frames) × 60 (seconds)) images must be input to produce a 60-second stereoscopic image using a light field display that displays information from 30 viewpoints. do.

In order to obtain these images through live-action shooting, it is necessary to accurately align and shoot 30 cameras. When the scene to be shot changes, it is necessary to arrange and shoot 30 cameras again, which consumes a lot of time and money. In addition, there is a problem that it is necessary to correct various errors such as an alignment error of the cameras and the brightness, optical axis, and lens distortion between the cameras.

Even when computer graphics technology is used, a significantly larger number of images must be rendered compared to the case of generating a two-dimensional image. That is, the number of images to be rendered and the required time increase exponentially according to the number of viewpoints or the resolution of the image.

For example, assuming that it takes 1 minute to render an 8K image, a 2D image requires 1,800 (30 (frame) × 60 (sec)) minutes, but a 3D image at 30 time points An image requires a time of 54,000 (30 (number of views) × 30 (frames) × 60 (seconds)) minutes. In addition, additional time was required for conversion to a stereoscopic image.

Korea Patent Publication No. 2017-0074164 (published on June 29, 2017)

An object of the present invention is to provide an apparatus, method, and computer program capable of easily generating a stereoscopic image.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention provides an apparatus for generating a stereoscopic image of a target object, and acquires a plurality of input images obtained by photographing the target object through a plurality of cameras a target viewpoint for grouping the plurality of input images into a plurality of groups, and generating a plurality of target viewpoint images including viewpoint information on the target object based on at least two or more input images included in each group and an image generator, a synthesized viewpoint image generator that generates a synthesized viewpoint image by synthesizing the plurality of target viewpoint images and a background image, and a stereoscopic image generator that generates a stereoscopic image based on the synthesized viewpoint image.

In an embodiment, the plurality of cameras may be disposed toward the target object at predetermined intervals on a circle formed around the target object.

In an embodiment, the acquisition unit may further acquire viewpoint information on the target object corresponding to each of the plurality of input images.

In an embodiment, the target viewpoint image generator may remove the background of the plurality of input images by using an alpha channel.

In an embodiment, the stereoscopic image generator may group the synthesized viewpoint images based on viewpoint information on the target object and an arrangement interval of the plurality of cameras.

In an embodiment, the background image may include a plurality of background viewpoint images including background viewpoint information.

In an embodiment, the synthesized viewpoint image generator may generate the synthesized viewpoint image by synthesizing the target viewpoint image and the background viewpoint image based on viewpoint information on the target object and the background viewpoint information.

In an embodiment, the stereoscopic image generator may generate the stereoscopic image by using any one of a light field display, a multi-viewpoint display, and a super multi-viewpoint display.

Another embodiment of the present invention provides a method for generating a stereoscopic image of a target object, comprising: acquiring a plurality of input images obtained by photographing the target object through a plurality of cameras; Grouping into groups, generating a plurality of target viewpoint images including viewpoint information on the target object based on at least two or more input images included in each group, and synthesizing the plurality of target viewpoint images and background images By doing so, the method may include generating a synthesized viewpoint image and generating a stereoscopic image based on the synthesized viewpoint image.

Another embodiment of the present invention provides a computer program stored in a medium including a sequence of instructions for generating a stereoscopic image of a target object. When the computer program is executed by a computing device, the target object is transmitted through a plurality of cameras. to obtain a plurality of input images obtained by photographing the and a sequence of instructions for generating a target viewpoint image of , generating a synthesized viewpoint image by synthesizing the plurality of target viewpoint images and a background image, and generating a stereoscopic image based on the synthesized viewpoint image.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, it is possible to reduce the time and cost required for generating a stereoscopic image.

It can be used in various fields, such as producing advertisement images using stereoscopic images.

1 is a block diagram of an apparatus for generating a stereoscopic image according to an embodiment of the present invention.
2 is a diagram for explaining a method of acquiring an input image according to an embodiment of the present invention.
3 exemplarily shows an input image obtained according to an embodiment of the present invention.
4 is a diagram for explaining a method of generating a target viewpoint image according to an embodiment of the present invention.
5 is a diagram for explaining a method of generating a synthesized viewpoint image according to an embodiment of the present invention.
6 exemplarily shows a stereoscopic image generated according to an embodiment of the present invention.
7 is a flowchart of a method for generating a stereoscopic image according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware. Meanwhile, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, '~' denotes components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

The "network" referred to below means a connection structure capable of exchanging information between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN). , the Internet (WWW: World Wide Web), wired and wireless data networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, etc. are included, but are not limited thereto.

Some of the operations or functions described as being performed by the terminal or device in the present specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for generating a stereoscopic image according to an embodiment of the present invention. Referring to FIG. 1 , the apparatus 100 for generating a stereoscopic image may include an acquisition unit 110 , a target viewpoint image generation unit 120 , a synthesis viewpoint image generation unit 130 , and a stereoscopic image generation unit 140 . .

The stereoscopic image generating apparatus 100 may generate a stereoscopic image of the target object. The stereoscopic image generating apparatus 100 may generate a stereoscopic image of a target object from a plurality of two-dimensional input images. The stereoscopic image generating apparatus 100 may generate a stereoscopic image based on viewpoint information on the target object or the background.

The acquisition unit 110 may acquire a plurality of input images. The plurality of input images may be obtained by photographing a target object through a plurality of cameras, for example. The plurality of input images may be, for example, two-dimensional images. The plurality of input images may be, for example, each of a plurality of cameras simultaneously photographing the target object once.

In an embodiment, the plurality of cameras may be disposed toward the target object at predetermined intervals on a circle formed around the target object.

The acquisition unit 110 may further acquire viewpoint information on the target object corresponding to each of the plurality of input images. Viewpoint information on the target object corresponding to each of the plurality of input images may be generated based on location information of each of the plurality of cameras.

2 is a diagram for explaining a method of acquiring an input image according to an embodiment of the present invention. Referring to FIG. 2 , the plurality of cameras 203 may be disposed toward the target object 201 at predetermined intervals on a circle formed around the target object 201 . For example, 720 cameras may be arranged at intervals of approximately 0.5 degrees on a circle formed around the target object.

According to the present invention, since 720 cameras are arranged at an interval of 0.5 degrees, rotation of the car is expressed through a stereoscopic image to be described later.

3 exemplarily shows an input image obtained according to an embodiment of the present invention. For example, the input image captured by the first camera disposed toward the front of the target object 201 may appear as shown in (a) of FIG. 3 . In this case, numbers may be sequentially assigned to the plurality of cameras 203 in a clockwise direction from the first camera.

For example, an input image photographed through a camera disposed at a distance of 3.5 degrees clockwise from the camera No. 7, that is, the camera No. 1, may appear as shown in (b) of FIG. 3 . For example, an input image captured by camera 20, that is, a camera disposed 10 degrees apart clockwise from camera 1, may appear as shown in FIG. 3(c) .

The acquisition unit 110 may acquire a background image. The background image may include, for example, a plurality of background viewpoint images and background viewpoint information. The background image may be, for example, captured by a plurality of cameras. The background image may be generated, for example, based on a plurality of still images of a moving picture.

The background image may be, for example, a two-dimensional image. The background image may be, for example, an image having a size larger than a target object included in the plurality of input images.

For example, the plurality of background viewpoint images may have respective background viewpoint information, and the background viewpoint information may be generated based on position information and rotation information of a camera that has captured each background viewpoint image.

The target viewpoint image generator 120 may extract a target object from a plurality of input images. The target viewpoint image generator 120 may remove backgrounds of the plurality of input images by using, for example, an alpha channel of computer graphics.

The target viewpoint image generator 120 may group a plurality of input images into a plurality of groups. The target viewpoint image generator 120 may group, for example, eight input images into one group. Each input image may be included in one or more groups in duplicate.

The target viewpoint image generator 120 may generate a plurality of target viewpoint images based on at least two or more input images included in each group. In this case, the target viewpoint image may include viewpoint information on the target object.

4 is a diagram for explaining a method of generating a target viewpoint image according to an embodiment of the present invention. In FIG. 4 , for convenience, viewpoint information (VIEW) of a target object is shown in correspondence with the number of a camera that captured each input image.

Referring to FIG. 4 , the target viewpoint image generator 120 may group images captured by a plurality of adjacent cameras into one group. For example, the target viewpoint image generating unit 120 may group the input images of VIEW 1 ( 411 ) photographed by camera 1 to VIEW 8 ( 413 ) photographed by camera 8 into one group 410 . have. Similarly, the target viewpoint image generating unit 120 groups input images of VIEWs 2 ( 421 ) to 9 ( 423 ) into one group 420 , and sets one input image of VIEWs 711 ( 431 ) to 718 ( 433 ). may be grouped into a group 430 of

In this way, the target viewpoint image generator 120 may group the plurality of input images.

A plurality of groups generated by the target viewpoint image generator 120 may constitute each frame constituting the stereoscopic image.

According to another embodiment, the target viewpoint image generator 120 may group a plurality of input images based on a sense of depth to be expressed in a stereoscopic image to be described later. For example, the target viewpoint image generator 120 may group input images having continuous viewpoint information into one group. As another example, the target viewpoint image generator 120 may group input images having viewpoint information having a predetermined interval into one group.

For example, referring to FIG. 4 , the target viewpoint image generator 120 may group input images of VIEW 1, 3, 5, 7, 9, 11, 13, and 15 into one group. As described above, when the viewpoint information has a predetermined interval, a sense of depth can be better expressed in a stereoscopic image because greater disparity information is included in one group as compared to a case where the viewpoint information is continuous.

According to another embodiment, the target viewpoint image generator 120 may group a plurality of input images based on a sense of rotation of a target object to be expressed in a stereoscopic image to be described later. For example, referring to FIG. 4 , the target viewpoint image generator 120 may group input images of VIEWs 1 to 8 and group input images of VIEWs 3 to 10 . In this case, the sense of rotation in the stereoscopic image may be expressed differently according to viewpoint information on the target object of the input image included in each group.

The synthesis viewpoint image generator 130 may generate a synthesis viewpoint image by synthesizing a plurality of target viewpoint images and a background image.

The synthesized viewpoint image generator 130 may generate a synthesized viewpoint image based on viewpoint information and background viewpoint information on the target object. The background image may include, for example, a plurality of background viewpoint images including background viewpoint information. The synthesized viewpoint image generator 130 may, for example, associate viewpoint information and background viewpoint information on the target object one by one, and synthesize the target viewpoint image and background viewpoint image corresponding to the viewpoint information based thereon.

5 is a diagram for explaining a method of generating a synthesized viewpoint image according to an embodiment of the present invention. In FIG. 5 , for convenience, viewpoint information (VIEW) of a target object is shown in correspondence with the number of a camera that captured each input image.

4 and 5 , the synthesized viewpoint image generator 130 synthesizes the target viewpoint image of the group 410 including the background 1 415 and the input images of VIEW 1 411 to 8 413 . By doing so, the synthesized viewpoint image 510 may be generated. Similarly, the synthesized viewpoint image generating unit 130 synthesizes the target viewpoint image of the group 420 including the background 2 425 and the input images of VIEW 2 421 to 9 423 to generate the synthesized viewpoint image 520 . , and synthesizing the target viewpoint image of the group 430 including the background N 435 and input images of VIEW 711 431 to 718 433 , thereby generating the synthesized viewpoint image 530 . In this way, the synthesized viewpoint image generator 130 may synthesize the target viewpoint image and the background viewpoint image.

The stereoscopic image generator 140 may generate a stereoscopic image based on the synthesized viewpoint image.

In an embodiment, the stereoscopic image generator 140 may generate a stereoscopic image by grouping the synthesized viewpoint images. For example, the stereoscopic image generator 140 may group the synthesized viewpoint images based on viewpoint information on the target object and an arrangement interval of a plurality of cameras.

The stereoscopic image generator 140 may generate a stereoscopic image using any one of a light field display, a multi-viewpoint display, and a super multi-viewpoint display.

6 exemplarily shows a stereoscopic image generated according to an embodiment of the present invention. 6 shows an example of a stereoscopic image generated using an 8K light field display of 36 viewpoints.

The stereoscopic image generator 140 may generate a stereoscopic image as shown in FIG. 6 based on a 6×6 sub-image array (SIA) generated based on the synthesized viewpoint image. The stereoscopic image generator 140 may perform processing such as arrangement adjustment, size adjustment, and position adjustment on the synthesized viewpoint image, and generate a stereoscopic image based thereon.

division conventional way Suggestion method video length 60 seconds 60 seconds number of points 36 point 36 point number of images needed 64,800 (36 (number of views) × 30 (frames) × 60 (seconds)) 780 (720 (viewpoint image)+ 60(background image))

Table 1 shows the number of viewpoint images required to make an image of 60 seconds. In view of this, in the conventional method, 36 viewpoint images are required for each frame. Therefore, to make a 60-second video, 64,800 (36 (number of views) Х 30 (frames) Х 60 (seconds)) images are required.

However, in this method, a 60-second stereoscopic image can be created by processing 780 sheets (720 (viewpoint image) + 60 (background image)). Here, the viewpoint image is generated by shooting each of 720 cameras once.

As described above, when the length of the image to be produced increases, the number of viewpoint images required linearly increases in the conventional method, but in this method, the target viewpoint image is fixed and only the background image needs to be added. You can make a video.

7 is a flowchart of a method for generating a stereoscopic image according to an embodiment of the present invention. The method 700 for generating a stereoscopic image performed by the apparatus 100 for generating a stereoscopic image shown in FIG. 7 includes steps processed in time series by the apparatus for generating a stereoscopic image 100 according to the embodiment shown in FIG. 1 . include Accordingly, even if omitted below, the method for generating a stereoscopic image performed by the stereoscopic image generating apparatus 100 according to the embodiment shown in FIG. 1 is also applied.

In operation S710 , the stereoscopic image generating apparatus 100 may acquire a plurality of input images obtained by photographing a target object through a plurality of cameras.

In operation S720, the apparatus 100 for generating a stereoscopic image may group a plurality of input images into a plurality of groups.

In operation S730, the stereoscopic image generating apparatus 100 may generate a plurality of target viewpoint images including viewpoint information on the target object based on at least two or more input images included in each group.

In operation S740 , the stereoscopic image generating apparatus 100 may generate a synthesized viewpoint image by synthesizing a plurality of target viewpoint images and a background image.

In operation S750, the stereoscopic image generating apparatus 100 may generate a stereoscopic image based on the synthesized viewpoint image.

In the above description, steps S710 to S750 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between the steps may be switched.

The method for generating a stereoscopic image in the stereoscopic image generating apparatus described with reference to FIGS. 1 to 7 may be implemented in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by the computer. have.

Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: stereoscopic image generating device
110: acquisition unit
120: target viewpoint image generation unit
130: synthesized viewpoint image generation unit
140: stereoscopic image generation unit

Claims (17)

An apparatus for generating a stereoscopic image of a target object, the apparatus comprising:
an acquisition unit configured to acquire a plurality of input images obtained by photographing a target object through a plurality of cameras;
a target viewpoint image generator for grouping the plurality of input images into a plurality of groups and generating a plurality of target viewpoint images including viewpoint information on the target object based on at least two or more input images included in each group;
a synthesized viewpoint image generator for generating a synthesized viewpoint image by synthesizing the plurality of target viewpoint images and the background image; and
A stereoscopic image generator for generating a stereoscopic image based on the synthesized viewpoint image
A stereoscopic image generating apparatus comprising a.
The method of claim 1,
wherein the plurality of cameras are disposed toward the target object at predetermined intervals on a circle formed around the target object.
The method of claim 1,
wherein the acquiring unit further acquires viewpoint information on the target object corresponding to each of the plurality of input images.
The method of claim 1,
wherein the target viewpoint image generating unit removes a background of the plurality of input images by using an alpha channel.
The method of claim 1,
and the stereoscopic image generating unit groups the synthesized viewpoint images based on viewpoint information on the target object and an arrangement interval of the plurality of cameras.
The method of claim 1,
The apparatus for generating a stereoscopic image, wherein the background image includes a plurality of background viewpoint images including background viewpoint information.
7. The method of claim 6,
The synthesized viewpoint image generator generates the synthesized viewpoint image by synthesizing the target viewpoint image and the background viewpoint image based on viewpoint information and the background viewpoint information on the target object.
The method of claim 1,
and the stereoscopic image generating unit generates the stereoscopic image by using any one of a light field display, a multi-viewpoint display, and a super multi-viewpoint display.
A method of generating a stereoscopic image of a target object, the method comprising:
acquiring a plurality of input images obtained by photographing a target object through a plurality of cameras;
grouping the plurality of input images into a plurality of groups;
generating a plurality of target viewpoint images including viewpoint information on the target object based on at least two or more input images included in each group;
generating a synthesized viewpoint image by synthesizing the plurality of target viewpoint images and a background image; and
generating a stereoscopic image based on the synthesized viewpoint image
A method of generating a stereoscopic image comprising a.
10. The method of claim 9,
The method of generating a stereoscopic image, wherein the plurality of cameras are disposed toward the target object at predetermined intervals on a circle formed around the target object.
10. The method of claim 9,
The acquiring of the plurality of input images includes acquiring viewpoint information on the target object corresponding to each of the plurality of input images.
10. The method of claim 9,
The generating of the plurality of target viewpoint images includes removing backgrounds of the plurality of input images by using an alpha channel.
10. The method of claim 9,
The grouping may include grouping the synthesized viewpoint image based on viewpoint information on the target object and an arrangement interval of the plurality of cameras.
10. The method of claim 9,
The method for generating a stereoscopic image, wherein the background image includes a plurality of background viewpoint images including background viewpoint information.
15. The method of claim 14,
The generating of the synthesized viewpoint image comprises generating the synthesized viewpoint image by synthesizing the target viewpoint image and the background viewpoint image based on viewpoint information on the target object and the background viewpoint information. How to create a stereoscopic image.
10. The method of claim 9,
The generating of the stereoscopic image may include generating the stereoscopic image using any one of a light field display, a multi-viewpoint display, and a super multi-viewpoint display.
A computer program stored in a medium including a sequence of instructions for generating a stereoscopic image of a target object, the computer program comprising:
When the computer program is executed by a computing device,
Obtaining a plurality of input images obtained by photographing a target object through a plurality of cameras,
grouping the plurality of input images into a plurality of groups,
generating a plurality of target viewpoint images including viewpoint information on the target object based on at least two or more input images included in each group;
A synthesized viewpoint image is generated by synthesizing the plurality of target viewpoint images and background images,
and a sequence of instructions for generating a stereoscopic image based on the synthesized viewpoint image.

Images