KR102404894B1 - Input viewpoint selection system for generating output viewpoints from multi-view image information sources - Google Patents

Abstract

The present invention relates to an input viewpoint selection system for generating an output viewpoint from a multi-viewpoint image information source, comprising: a camera for generating N input images by taking multiple pictures of a subject within a predetermined time range; an image extraction module that interworks with the camera and receives N input images generated through the camera, and extracts K viewpoints from the N images to be used for synthesis; An image synthesizing module for synthesizing the extracted K viewpoints and synthesizing them into N-1 viewpoint values, and an image generator for outputting the synthesized image by changing it into N-1 viewpoint values according to a user's selection. do.

Description

{Input viewpoint selection system for generating output viewpoints from multi-view image information sources}

The present invention relates to an input viewpoint selection system for generating an output viewpoint from a multi-viewpoint image information source. That is, the present invention relates to a method for efficiently selecting a view point to generate a high-quality image, and a system for performing the same.

Until recently, a single fixed-view method, which is a method of viewing a specific camera viewpoint determined by an image producer as it is, has been mainly used for imaging systems, but a multi-view or free viewpoint method that provides a viewpoint desired by a user is being developed.

Here, the multi-view type content means that the user can select and view the desired viewpoint from content that can provide multiple viewpoints by shooting the same object from multiple viewpoints or synthesizing the photographed images. it's a video

On the other hand, the free view method is a method of providing an image of an arbitrary viewpoint desired by the user.

However, in the conventionally used multi-view method and content provided in the free view method, it is impossible to prepare all the viewpoints because there are countless viewpoints desired by the user. You must use a method to create a viewpoint.

On the other hand, the most common method for synthesizing a new viewpoint in 3D is to use color information and depth information defined in the MPEG standard or the like.

Specifically, given the position and internal information of the camera, information of a new viewpoint can be created using the two-dimensional color image information and depth information.

However, due to the difference in the viewpoint of the object that can be seen according to the movement of the viewpoint, a problem occurred that a natural image of the desired viewpoint could not be obtained with only one input image.

In this case, the method of generating output viewpoints using information of all input viewpoints may be the best in terms of image quality, but it is preferable to select only some of the input viewpoints in consideration of the efficiency of the image processing system and network.

Accordingly, there is a need for a method of selecting which viewpoints to effectively synthesize an image, but a method for this has not been developed at present.

At this time, an easily conceivable method is to select input points that are closest to the output time, but in this way, if an input time point that is uniformly close to the distance is selected, an input in a biased direction is selected, that is, an invisible area, that is, , omission of the image is likely to occur, and if the image is output using all input viewpoints as shown in FIG. 2, the image quality may be increased, but the usage of the network used to set the output timing is significantly increased, As shown in FIG. 3 , when an image is output using only a small number of input viewpoints close to the output viewpoint, there may be a problem in that an image having a high quality cannot be obtained.

Accordingly, in a system that generates an image from a multi-view image to a user-defined output viewpoint, when the number of input viewpoints usable for image synthesis is determined, a multi-viewpoint capable of generating an image of (the highest) high quality at the output viewpoint There is an urgent need to develop a technology for an input viewpoint selection system for generating an output viewpoint from an image information source.

An input viewpoint selection system for generating an output viewpoint from a multi-view image information source according to an embodiment of the present invention is to solve the above problems, and an image of a natural arbitrary viewpoint in a situation where a multi-view image taken at a predetermined position is provided. An object of the present invention is to provide an input viewpoint selection system for generating an output viewpoint from a multi-view image information source that can effectively generate a .

In an input viewpoint selection system for generating an output viewpoint from a multi-viewpoint image information source according to an embodiment of the present invention for solving the above problems, a camera for generating N input images by taking multiple pictures of a subject within a predetermined time range; an image extraction module that interworks with the camera and receives N input images generated through the camera, and extracts K viewpoints from the N images to be used for synthesis; It may include an image synthesizing module for synthesizing the extracted K viewpoints and synthesizing them into N-1 viewpoint values, and an image generator for outputting the synthesized image by changing it into N-1 viewpoint values according to a user's selection.

In addition, the camera is provided with a plurality of cameras in all directions of the subject so as to simultaneously photograph the front, rear, and side of the subject, among the N input images generated through the plurality of cameras and transmitted to the image extraction module, the image photographer Only the image of the time set as the set timing of can be transmitted to the image extraction module.

Here, the image extraction module may include an image removal module for removing low-resolution images from the received N images.

In addition, the image synthesizing module derives N-1 viewpoint values from the N synthesized images and stores them in the database of the image synthesizing module, wherein the N-1 viewpoint values are stored in the database of the image synthesizing module. It may be set as a viewpoint having a displacement value closest to the reference value set in , and transmitted to the image generator.

In addition, the image generator may receive a user setting and output the image by varying the angle, position, and height of the synthesized image.

In addition, the image generating unit may output the input image and information about a camera resource used for deriving a viewpoint value and information about a parameter related to resolution together with the image.

In addition, the process of synthesizing K viewpoints by the image synthesis module may be derived through [Equation 1].

[Equation 1]

,

,

는 벡터의 합성값,

는 시점의 벡터값)(here,

is the composite value of the vector,

is the vector value at the time)

In addition, the image synthesis module may adjust the number of K points used for synthesis according to a user's setting value among K viewpoints extracted through the image extraction module.

Here, in the case of using two or more but K or less input points, two input values are synthesized into one input value, and then one input value is added to the synthesized combined value one by one, and finally up to K The input time can be included.

를 도출하고,On the other hand, the process of synthesizing K viewpoints using two or more and K or less input viewpoints is an input set through [Equation 2].

to derive,

[Equation 2]

는, [수학식 3]과 같은 형태로 도출되되,the input set

is derived in the same form as [Equation 3],

[Equation 3]

When the input image and the user's setting value are the same. Alternatively, it is terminated when the summed absolute value is the smallest, and when K < K, the input set h can be updated again through [Equation 2].

The input viewpoint selection system for generating an output viewpoint from a multi-viewpoint image information source according to an embodiment of the present invention may synthesize and provide high-quality arbitrary viewpoint images in a system having limited image synthesis performance.

In addition, when it is possible to selectively transmit an input image according to a viewpoint using a network, it is possible to reduce network usage.

1 is a block diagram showing the overall configuration of an input viewpoint selection system for generating an output viewpoint from a multi-viewpoint image information source according to an embodiment of the present invention.
2 and 3 are exemplary diagrams illustrating problems that may occur when an input viewpoint and a user-defined output viewpoint do not match in the conventional multi-view image content.
4 is an exemplary diagram illustrating an output time synthesized according to an embodiment of the present invention.
5 is an exemplary diagram illustrating a case in which input values are derived as {6, 10}, {6, 11}.
FIG. 6 is an exemplary diagram illustrating the calculated content when the output point position shown in FIG. 5 is (1/8, 1/8) with respect to the time point 6 .
FIG. 7 is an exemplary diagram illustrating the calculated contents when the output point position shown in FIG. 5 is (3/8, 3/8) with respect to the time point 6 .
8 is an exemplary diagram illustrating an algorithm for selecting an input viewpoint set.

Hereinafter, the description of the present invention with reference to the drawings is not limited to specific embodiments, and various modifications may be made and various embodiments may be provided. In addition, it should be understood that the contents described below include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention.

In the following description, terms such as 1st, 2nd, etc. are terms used to describe various components, meanings are not limited thereto, and are used only for the purpose of distinguishing one component from other components.

Like reference numbers used throughout this specification refer to like elements.

As used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In addition, terms such as "comprises", "comprises" or "have" described below are intended to designate the existence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. It should be construed as not precluding the possibility of addition or existence of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, an input viewpoint selection system for generating an output viewpoint from a multi-view image information source according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 8 .

1 is a block diagram showing the overall configuration of an input viewpoint selection system for generating an output viewpoint from a multi-view image information source according to an embodiment of the present invention, and FIGS. 2 and 3 are input viewpoints in conventional multi-view image content It is an exemplary diagram to show a problem that may occur when the output time defined by the user and the user do not match. It is an exemplary diagram to show a case in which the values are derived as {6, 10}, {6, 11}, and FIG. 6 is the output point position shown in FIG. 5 based on the time point 6 (1/8, 1/8) It is an exemplary view for showing the calculated contents when , FIG. 8 is an exemplary diagram illustrating an algorithm for selecting an input viewpoint set.

1 to 8 , an input viewpoint selection system 1 for generating an output viewpoint from a multi-view image information source according to an embodiment of the present invention includes a camera 100, an image extraction module 200, and an image synthesis module. 300 and an image generator 400 may be included.

First, the camera 100 may generate N input images by taking multiple pictures of a subject within a predetermined time range.

In addition, as the camera 100 , a camera 100 capable of recognizing a three-dimensional position of the subject and a distance from the camera 100 to the distance of the subject may be used.

In this case, the distance sensing information on the subject derived through the camera 100 may be used as data for deriving a viewpoint value in the image synthesis module 300 to be described later.

In addition, as the camera 100, a camera for taking a single image or a decoder for taking an image may be used.

In addition, the camera 100 may be provided in plural at all sides of the subject so as to simultaneously photograph the front, rear, and side of the subject.

On the other hand, when the videographer shoots a subject for the purpose of providing VR, the camera 100 is most preferably provided at a position where the subject can be photographed at 360 °, and shows a flat projection image or a fragmentary screen. If you want to shoot content for this purpose, it is most desirable to be provided at a position where the subject can be photographed in a range of 180°.

In addition, the camera 100, from among the N input images generated through the plurality of cameras 100 and transmitted to the image extraction module 200, only the image at the time set as the timing set by the image shooter is the image extraction module ( 200) can be transmitted.

In addition, the input image data transmitted to the image extraction module 200 through the camera 100 may include temporal synchronization information of the captured input image and transmit it together.

At this time, the temporal synchronization information may be included in all captured input images, but this is only a preferred embodiment. It is also possible to provide only temporal synchronization information of the input image extracted through the

Also, the image extraction module 200 may receive an input image photographed by the camera 100 by interworking with the camera 100 .

In addition, the image extraction module 200 may receive the N input images generated through the camera 100 , and then extract K viewpoints from the N images to be used for synthesis.

In this case, the number of input time points extracted as the K number may be used differently depending on the set value of the user using the present invention, and most preferably, it is preferably set to have three or more K values. do.

However, the number of input viewpoints that can be input as three or more is only a minimum number necessary to calculate an output viewpoint, and the number of viewpoints derived through the image extraction module 200 is three or more. In addition, N, which is equal to the total number of input images of the captured input images, may be derived.

In addition, the image extraction module 200 may include an image removal module 210 capable of detecting an image or image having a low resolution among N images received through the camera 100 and removing it. have.

More specifically, the image removal module 210 deletes a low-resolution image among images collected by the input viewpoint selection system 1 for generating an output viewpoint from a multi-viewpoint image information source according to an embodiment of the present invention. , it is possible to secure the storage space in which the low resolution is stored, and to use only the images essential for synthesizing the images, thereby improving the overall speed.

Also, the image synthesis module 300 may synthesize N-1 viewpoint values by synthesizing a plurality of viewpoints extracted by the image extraction module 200 into one viewpoint value.

At this time, the image synthesis module 300 selects an input time of K or less for synthesizing a plurality of viewpoints, but considers the distance and direction of the input viewpoint used for synthesis and the output viewpoint value to obtain a combination with high efficiency. You can choose.

As described above, the viewpoint synthesis method of the input viewpoint selection system 1 for generating an output viewpoint from a multi-view image information source according to an embodiment of the present invention will be described in more detail in addition to the content described above.

First, camera information (a position in three dimensions, a camera characteristic variable) of each input viewpoint is provided as an input value.

Next, N input images are temporally input to the camera, including synchronization information. (Provided through a file or through a network.)

In this case, it is not necessary that N images are provided at every frame time, and only images of K viewpoints to be used for synthesis selected by the user need only be provided.

Next, camera parameter information of the output camera is provided.

Next, the user selects the desired output timing (X, Y, Z position and rotation angle ROLL, PITCH, YAW).

Next, the maximum number of input images K to be used for synthesizing the output images is given.

Next, an input time of K or less for synthesizing the output time is selected, and the most efficient combination is selected in consideration of the distance and direction between the input time and the output time. (At this time, if the input image is a 360-degree image, the output position is considered 3D, and in the case of a flat projection image or a 180-degree image in which content is located in only one direction, a 2-dimensional position is selected.)

라 하고, 출력 시점까지의 변위 벡터를

라 한다면, 다음 수식의 값이 최소가 되는, 즉, 각각의 벡터값의 연산값

의 값이 가장 최소의 값을 입력 시점을 구하여 이에 속한 시점에 대한 값을 입력값으로 사용하고, 이를 수학식으로 나타낼 경우, 하기 개시한 [수학식 1]과 같다.In order to examine the above process in more detail, looking in detail together with the algorithm disclosed in the present invention, first, the output time value is

, and the displacement vector up to the output point is

If , then the value of the following formula is the minimum, that is, the calculated value of each vector value.

When an input time point is obtained with the smallest value of the value of , and a value for a time point belonging to the value is used as an input value, and this is expressed by an equation, it is the same as [Equation 1] described below.

[Equation 1]

,

,

의 값(두개의 변위 벡터

가 합성된 후 도출된 값)은, 입력 시점을 기준으로 주위에 인접한 출력 시점과 합산되는 과정을 반복함으로써, 가장 최신의

값이 도출될 수 있다.On the other hand, two displacement vectors were synthesized and derived

the value of (two displacement vectors

The value derived after ? is synthesized) is the most recent

A value can be derived.

That is, as the distance from the output viewpoint is closer, and the input viewpoints are evenly distributed in all four directions, effective synthesis is possible.

For a more specific example, as shown in FIG. 5 , it is assumed that input viewpoints are given in the form of 16 grids (N = 16), the distance between each grid is 1, and the number of available input viewpoints is 2 In the case (K = 2), the output may be in the form shown in FIG. 5 .

이므로, 벡터 연산에 이용되는 입력 시점 {6, 10}을 선택할 수 있다.6 as an example, when the output time is located at (1/8, 1/8) with respect to the input time 6 at the input time point having a pattern of 16 input points as shown in FIG. 5, the input time point Calculate the value located at the nearest distance from

Therefore, the input time {6, 10} used for vector operation can be selected.

이므로, 벤터 연산에 이용되는 입력 시점 {6, 11}을 선택할 수 있는 것이다.Next, referring to FIG. 7 , when the output time is located at (3/8, 3/8) with respect to the input time 6 , a value located at the closest distance from the input time is calculated, specifically,

Therefore, the input time {6, 11} used for the venture calculation can be selected.

In summary, as shown in FIGS. 6 and 7 , the process of finding the closest vector value with respect to input time 6, synthesizing it, and calculating the synthesized vector value again using nearby input time points By repeating , it is possible to derive a final viewpoint value by synthesizing a viewpoint value calculated by adding the vector of the input viewpoint from the first used input viewpoint with another input viewpoint located nearby.

Accordingly, the input viewpoint selection system 1 for generating an output viewpoint from a multi-viewpoint image information source according to an embodiment of the present invention calculates an output viewpoint of an image using values of all input viewpoints and output viewpoints existing in the image. It has an advantage in that it is possible to obtain an image having the best image quality with the fewest calculation values by using only a synthesis variable located near the input time point instead of the input time point.

In addition, as described above in the above embodiment, when two or more input time points are used, it is possible to finally include up to K input time points while adding the optimal input one by one. It can be represented as shown in Fig.

는, 0 의 값으로 초기화 될 수 있다. (S10)first,

may be initialized to a value of 0. (S10)

가 k-1 개의 개수를 가지는 입력 위치들의 집합일 경우 (S20), 다음 입력 h 의 값은, 하기 개시한 [수학식 2]를 통해 도출될 수 있다. (S30)to the next,

When is a set of input positions having k-1 number (S20), the value of the next input h may be derived through [Equation 2] described below. (S30)

[Equation 2]

는, 하기 개시한 [수학식 3]과 같은 형태로 도출될 수 있다. (S40)On the other hand, the input set derived through this

can be derived in the form of [Equation 3] disclosed below. (S40)

[Equation 3]

Here, when k = k (when the number of input images and the number of input viewpoints of the user is the same) or when the summed absolute value is the smallest, it ends, and when k < k, the input set h is again expressed through [Equation 2] You can repeat the process of updating. (S50)

In addition, N-1 viewpoint values are derived from the synthesized N images, and the derived viewpoint values may be stored in the database DB of the image synthesis module 300 .

Here, the N-1 stored time points may be set as a time point having a displacement value closest to a reference value set in the database DB of the image synthesis module 300 .

Also, a viewpoint having a displacement value closest to the reference value may be transmitted to the image generator 400 to be described later.

In addition, the image generating unit 400 may output the synthesized image to the user by changing it to N-1 viewpoint values according to the user's selection.

In this case, the image generator 400 may cause the user to select a stored viewpoint value, and output images synthesized with each viewpoint according to the user's selection.

Also, the image generating unit 400 may receive user settings and change the angle, position, and height of the synthesized image image and output it to the user.

More specifically, the user may check the image by varying the X, Y, and Z positions of the image output through the image generating unit 400 and the rotation angles ROLL, PITCH, and YAW of the image.

In addition, the image generating unit 400 may output information about a parameter related to a resource and a resolution of the camera 100 used for image synthesis and viewpoint value derivation together with the image.

Accordingly, the input viewpoint selection system 1 for generating an output viewpoint from a multi-viewpoint image information source according to an embodiment of the present invention can provide a high-quality arbitrary viewpoint image synthesis in a system with limited image synthesis performance. , and it is possible to reduce the overall information processing amount of the device constituting the system of the present invention by deriving a viewpoint value based only on information about the condition value required in the process of processing the input image, so that the speed and efficiency are increased. have an effect

The input viewpoint selection system 1 for generating an output viewpoint from a multi-view image information source according to an embodiment of the present invention described above with reference to FIGS. 1 to 8 is not implemented only through an apparatus and/or method, A program for realizing a function corresponding to the configuration of the embodiment of the present invention, the program may be implemented through a recording medium recorded therein, etc., and such implementation can be easily realized by an expert in the technical field to which the present invention belongs from the description of the above-described embodiment. that can be implemented.

In addition, although the embodiment of the present invention has been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also presented. It belongs to the scope of the invention.

1: Input viewpoint selection system for generating output viewpoint from multi-view image information source
100 : camera
200: image extraction module
210: image removal module
300: image compositing module
400: image generating unit
DB: database

Claims (10)

In the input viewpoint selection system for generating an output viewpoint from a multi-view image information source,
a camera for generating N input images by taking multiple shots of a subject within a predetermined time range;
an image extraction module that interworks with the camera and receives N input images generated through the camera, and extracts K viewpoints from the N images to be used for synthesis;
An image synthesizing module that synthesizes the extracted K time points and synthesizes them into N-1 time points, and
and an image generator for outputting the synthesized image by changing it to N-1 viewpoint values according to a user's selection,
The image synthesis module,
Derive N-1 viewpoint values from N synthesized images and store them in the database of the image synthesis module,
The time values stored in the N-1 pieces are,
Input viewpoint selection system for generating an output viewpoint from a multi-view image information source, characterized in that the viewpoint is set as the viewpoint having the closest displacement value to the reference value set in the database of the image synthesis module and transmitted to the image generator
The method of claim 1,
The camera is
It is provided with a plurality of cameras in all directions of the subject so as to simultaneously photograph the front, rear, and side of the subject,
Among the N input images generated through the plurality of cameras and transmitted to the image extraction module, only the image of the time set as the timing set by the image photographer is transmitted to the image extraction module. input point selection system for
3. The method of claim 2,
The image extraction module,
In the received N images,
Input viewpoint selection system for generating an output viewpoint from a multi-view image information source, characterized in that it includes an image removal module for removing a low-resolution image
delete The method of claim 1,
The image generating unit,
Input viewpoint selection system for generating an output viewpoint from a multi-view image information source, characterized in that the angle, position, and height of the synthesized image are varied by receiving user settings and output
The method of claim 1,
The image generating unit,
Input viewpoint selection system for generating an output viewpoint from a multi-viewpoint image information source, characterized in that the information on parameters related to the resolution and the resources of the camera used for deriving the input image and viewpoint value is output together with the image
The method of claim 1,
The process of synthesizing K viewpoints by the image synthesis module is,
Input viewpoint selection system for generating an output viewpoint from a multi-view image information source, characterized in that it is derived through [Equation 1]
[Equation 1]

,

(here,

is the composite value of the vector,

is the vector value at the time)
The method of claim 1,
The image synthesis module,
Input viewpoint selection system for generating an output viewpoint from a multi-viewpoint image information source, characterized in that the number of K viewpoints used for synthesis is adjusted according to a user's setting value among K viewpoints extracted through the image extraction module
9. The method of claim 8,
In the case of using two or more but K or less input times, two input values are synthesized into one input value, and one input value is added to the synthesized composite value one by one, and finally up to K input times Input viewpoint selection system for generating an output viewpoint from a multi-view image information source, characterized in that it includes
9. The method of claim 8,
The process of synthesizing K time points using two or more and K or less input time points is,
Input set through [Equation 2]

to derive,
[Equation 2]

the input set

is derived in the same form as [Equation 3],
[Equation 3]

When the input image and the user's setting value are the same. Alternatively, it is terminated when the summed absolute value is the smallest, and when K < K, the input time selection system for generating an output viewpoint from a multi-view image information source, characterized in that the input set h is updated again through [Equation 2]

Images