WO2019139303A1

WO2019139303A1 - Image synthesizing method and device

Info

Publication number: WO2019139303A1
Application number: PCT/KR2019/000111
Authority: WO
Inventors: 유성열; 김준식; 김규헌; 강전호
Original assignee: 삼성전자 주식회사; 경희대학교 산학협력단
Priority date: 2018-01-12
Filing date: 2019-01-03
Publication date: 2019-07-18
Also published as: KR20190086323A; KR102637913B1

Abstract

An image synthesizing device according to an embodiment of the present disclosure aligns a plurality of frames on the basis of feature points extracted from the plurality of frames respectively corresponding to a plurality of images, confirms an overlapping region comprising the feature points in correspondence between the aligned frames in a region comprising the aligned frames, determines a weight for each pixel on the basis of depth information of the overlapping region, generates a minimum confusion matrix by means of imparting the weight for each pixel to the differential between pixel values of the frames comprised in the overlapping region, generates a boundary by means of the minimum confusion matrix, and synthesizes the aligned frames on the basis of the generated boundary.

Description

Image synthesis method and apparatus

Various embodiments of the present disclosure are directed to a method and apparatus for composing images with different viewpoints into a single image.

Generally, one image acquired through a camera has a limited angle of view. Accordingly, a variety of image synthesis techniques have been studied to generate images having a wider angle of view than a single image, thereby providing a more realistic experience.

As a representative example of the image synthesis techniques, a scale invariant feature transform (SIFT), a speed up robust feature (SURF) and a feature form accelerated segment test (FAST), which extract feature information in an image and synthesize images based on the feature information, And alpha and pyramid blending algorithms to remove the visible boundary after synthesis.

The stitching technique enables the generation of panoramic images without visual errors by sequentially using the algorithms according to the image synthesis technique. The stitching technique has evolved with significant improvements in accuracy. However, the improvement of the accuracy may lead to a decrease in processing speed as the amount of computation for image processing is increased. Such a decrease in the processing speed may limit the use range of the panoramic image generation technique.

According to various embodiments of the present disclosure, it is possible to provide an image synthesis method and apparatus capable of securing versatility by processing speed and accuracy according to image synthesis.

According to various embodiments of the present disclosure, it is possible to provide an image synthesizing method and apparatus for synthesizing images at various different viewpoints to generate a single panoramic image.

According to various embodiments of the present disclosure, it is possible to have a smaller throughput compared to existing algorithms used to generate a single panoramic image, to reduce problems such as image distortion and viewpoint distortion occurring during image synthesis, and to acquire clear panoramic images It is possible to provide a method and an apparatus for synthesizing an image.

According to an embodiment of the present disclosure, a method includes: A method of synthesizing an image, the method comprising: arranging the plurality of frames based on feature points extracted from a plurality of frames corresponding to each of a plurality of images; Determining a weight for each pixel on the basis of the depth information of the overlap region, and calculating a weight for each pixel based on a difference between pixel values of frames included in the overlap region, Generating a minimum error matrix using the minimum error matrix, and composing the aligned frames based on the generated boundary.

According to an embodiment of the present disclosure, there is provided an apparatus comprising: An image synthesizer, comprising: an interface for acquiring a plurality of images; a plurality of frames arranged on the basis of feature points extracted from a plurality of frames corresponding to each of the plurality of images, Determining an overlap area including the minutiae corresponding to each other among the aligned frames, determining a weight for each pixel based on the depth information of the overlap area, and calculating a weight value between the pixel values of the frames included in the overlap area Generating a minimum error matrix by assigning a weight for each pixel to the difference, generating a boundary using the minimum error matrix, and compositing the aligned frames based on the generated boundary.

According to various embodiments of the present disclosure, it is possible to obtain a desired panoramic image by solving the image distortion or the view point distortion generated in the image synthesis with a small throughput.

1 is a block diagram of a video synthesizing apparatus according to an exemplary embodiment.

FIG. 2 (a) is a graph showing an example of a quantization function used for quantizing depth map values according to an embodiment,

FIG. 2 (b) is a graph showing another example of a quantization function used for quantizing the depth map values according to an embodiment,

FIG. 2 (c) is a graph showing another example of the quantization function used for quantizing the depth map values according to the embodiment,

2 (d) is a graph showing a further example of a quantization function used for quantizing depth map values according to an embodiment,

Figure 3 (a) illustrates an overlap region for creating a bottom boundary in accordance with one embodiment,

FIG. 3 (b) illustrates a process of generating a lower boundary according to an exemplary embodiment. FIG.

Figure 4 (a) illustrates an overlap region for creating an upper boundary in accordance with one embodiment,

FIG. 4 (b) illustrates a process of creating a top boundary according to an embodiment,

Figure 5 (a) shows an overlap region for creating a right-hand direction boundary according to one embodiment,

5 (b) illustrates a process of generating a right-direction boundary according to an embodiment,

6 illustrates an overlap region according to one embodiment,

Figure 7 (a) illustrates a first example of a minimum error boundary according to one embodiment,

FIG. 7 (b) is a diagram illustrating a second example of a minimum error boundary according to one embodiment;

7C is a diagram illustrating a third example of a minimum error boundary according to one embodiment,

Figure 7 (d) illustrates a fourth example of a minimum error boundary according to one embodiment,

FIG. 7 (e) illustrates a fifth example of a minimum error boundary according to an embodiment,

FIG. 7 (f) illustrates a sixth example of a minimum error boundary according to an embodiment,

8A is a view illustrating an example of a panoramic image generated using an existing method,

8 (b) is a diagram illustrating an example of a panoramic image generated using a method according to an embodiment,

9 is a flowchart illustrating an image synthesizing method of the image synthesizing apparatus according to an exemplary embodiment;

The operation principle of the present disclosure will be described in detail below with reference to the accompanying drawings. In the following description of the various embodiments of the present disclosure, detailed description of known functions and configurations incorporated herein will be omitted when it may obscure the subject matter of the present disclosure. It is to be understood that the following terms are defined in consideration of the functions of the present disclosure and may vary depending on the user, the intention or custom of the operator, and the like. Therefore, the definition should be based on the overall contents of this disclosure.

The present disclosure describes an image synthesis method and apparatus. Specifically, the present disclosure describes a method and apparatus for generating panoramic images by combining images having different viewpoints into one image. In various embodiments of the present disclosure, a depth map may be used to generate a panoramic image. When the depth map is used as described above, it is possible to reduce an error due to image distortion or the like occurring during image synthesis.

1 is a block diagram illustrating an image synthesizing apparatus according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the image synthesizing apparatus may include an interface unit and at least one processor. The image synthesizing apparatus may include at least one camera. FIG. 1 shows functional blocks configured for image processing in one processor included in the image synthesizing apparatus, for convenience of explanation in accordance with various embodiments of the present disclosure. The functional blocks shown in Fig. 1 may be implemented by a program in one or a plurality of processors. Also, the functional blocks shown in Fig. 1 may be implemented by hardware such as a digital logic circuit.

According to one embodiment, a device (e.g., one processor) for image processing may include a parser 110, a frame extractor 120, and an image processing module 100. The image processing module 100 includes a minimum error matrix generator 130, a minimum error seam generator 140, a minimum error boundary generator 150, and an image synthesizer 160 < / RTI >

The parser 110 may receive metadata related to each of the real images and the real images from one or a plurality of cameras. The real images may include images captured by a plurality of cameras at different points in time or images captured by a camera at a plurality of different points in time. In addition, the real images may include two or more real images for implementing a 3D image.

The parser 110 extracts depth information and position information from each of the input real images, and generates the depth map using the extracted depth information. In this case, the parser 110 provides the inputted real images to the frame extracting unit 120, and provides the generated depth map and the extracted position information to the image processing module 100.

According to one embodiment, the parser 110 parses metadata associated with each of the input real images. The metadata may include various information related to each of the real images, for example, information defined by a virtual world map (VWM) received from a separate system.

The parser 110 may obtain a depth map of each of the real images based on the metadata. The parser 110 may generate a matching position for the real images. The parser 110 may extract a key point and a description from real images, for example. The parser 110 may be used with one or more algorithms to extract feature points and descriptors. The one or more algorithms may include, for example, an algorithm based on FAST (features form accelerated segment test) with an emphasis on speed.

The parser 110 may perform a process of matching minutiae and descriptors extracted from each of the real images, and may determine the positional relationship with the matching minutiae. The parser 110 may detect the position of matching between the real images in consideration of the positional relationship. The parser 110 can identify an area (hereinafter referred to as an overlap area) including corresponding minutiae between the real images based on the detected matched position. The overlap area may be an area including the same subject in multiple frames corresponding to a still image captured at different points in time, for example. For example, the overlap area may be a region having a predetermined area connecting the upper and lower sides of the frame, an area having a predetermined area connecting the left and right sides of the frame, or a predetermined area connecting the left (or right) It may be an area having an area.

The parser 110 may provide the image processing module 100 with the position information on the overlap area and the depth map on the overlap area.

According to the above description, the parser 110 performs an operation for generating a matching position for real images, but the corresponding operation is performed by a separate physical structure such as a matching position generating unit, not the parser 110 .

The frame extracting unit 120 may extract frames from each of the real images provided from the parser 110. [ The frames may be, for example, a unit constituting a still image or a unit constituting an image at a specific time point constituting a moving image. The frame extracting unit 120 may provide the extracted frames to the image processing module 100.

According to one embodiment, the minimum error matrix generator 130, the minimum error boundary generator 140, the minimum error boundary direction selector 150 and the image synthesizer 160 included in the image processing module 100, Each can perform the following operations.

The minimum error matrix generator 130 may be provided with depth maps and position information extracted or generated from each of the frames extracted from each of the real images and the real images. The minimum error matrix generator 130 may generate a minimum error matrix based on the provided frames, the depth map, and the position information. The minimum error matrix may be, for example, a matrix for calculating eigenvalues corresponding to each of the pixels constituting one frame and defining eigenvalues calculated for the pixels constituting the frame.

The minimum error matrix generator 130 may obtain a minimum error matrix, for example, by the following equation (1).

In Equation (1), E (i, j) represents a depth map-based minimum error matrix value at a position (i, j) corresponding to each of the pixels in the overlap region due to the combination of the frame A and the frame B, and O _a (i, j) is a position corresponding to the overlapping area each of the pixels in frame a represents a (the "first pixel location" referred _{to), O B (i, j} ) is the overlapping area of the frame B within the D (i, j) represents a position corresponding to each of the pixels (hereinafter referred to as a second pixel position), and D (i, j) represents a depth map value corresponding to each pixel in the overlap region according to the combination of the frame A and the frame B. And Quant () represents a quantization function.

That is, in Equation (1), a scheme for obtaining an eigenvalue defining each of the pixels included in the overlapping region (overlap region) when combining two frames (frame A and frame B) is proposed have. However, it is a matter of course that the eigenvalues corresponding to each of the pixels included in the overlap region can be implemented by various schemes.

The quantization function in Equation (1) can be used to quantize the depth map values into a plurality of steps (quantization levels). The quantization function may be used such that, for example, the depth map becomes larger as the depth (depth) between the camera and the image is greater. When the above quantization function is used, it is possible to minimize an error that may occur when an object at a long distance is covered by a nearby object when creating a boundary.

When the minimum error matrix generator 130 obtains the minimum error matrix, the minimum error matrix generator 130 outputs the minimum error matrix to the minimum error boundary generator 140. The minimum error matrix may define eigenvalues indicating, for example, the characteristics of each of the pixels included in the overlap region.

The minimum error boundary generator 140 may generate a boundary for each boundary generation direction in the overlap area using the minimum error matrix provided from the minimum error matrix generator 130. [

The boundaries define, for example, connections of pixels that are expected to have a low probability of error occurrence within the overlap region for compositing multiple frames based on a minimum error matrix. This will not only reduce the probability of errors occurring when composing multiple frames, but also reduce the amount of computation required to correct errors during compositing. For example, when two frames are synthesized, the boundaries at which the two frames are synthesized in the pixels with low probability of error are set, thereby reducing the amount of error due to composition and the amount of computation for error correction.

When defining the overlapping area as a rectangle, for example, a direction for generating a boundary in the overlapping area (hereinafter referred to as a " boundary generation direction " Bottom direction, upper direction, and right direction. That is, the starting position of the boundary may be defined by a corner where four pixels (upper edge, lower edge, left edge, right edge) constituting the rectangle defining the overlap region constitute the boundary generating pixel have.

For example, when the starting pixel position of the boundary is the upper edge, the boundary generation direction may be the lower direction, and when the starting pixel position of the boundary is the lower edge, the boundary generation direction may be the upper direction, The boundary generation direction may be the right edge direction.

The following equations (2), (3) and (4) define the determination of the pixels corresponding to the boundaries for each boundary generation direction.

S _ia (i, j) defined as Equation (2) defines a boundary (hereinafter referred to as a lower boundary) generated in the lower direction of the overlap region and S _ib , j) defines a boundary (hereinafter referred to as an upper boundary) generated in the upper direction of the overlap region, and S _j (i, j) defined as in Equation 4 is generated in the right direction of the overlap region (Hereinafter referred to as " right direction boundary "). E (i, j) commonly used in the equations (2), (3) and (4) represents a minimum error matrix. The minimum error matrix E (i, j) may be provided by the minimum error matrix generator 130.

According to an exemplary embodiment, the minimum error boundary generator 140 performs a minimum error search operation to search for positions of pixels having a low probability of generating an error for each boundary generation direction in the overlapping area based on the minimum error matrix. The boundaries can be generated for each boundary generation direction by referring to pixel values (eigenvalues defined for each pixel in the minimum error matrix) around the same row or column as well as the next search row or column in the minimum error search. This makes the boundary generation method more flexible and dramatically increases the range of possible boundaries. In addition, by suggesting three boundaries, starting from the bottom, top, and right edge, we can synthesize the frames considering the boundaries of all directions that can be generated. It considers the synthesis of left and right images in the case of panoramic image, but considering the synthesis of up, down, left, and right images for 360 degree image which is emerging recently, omnidirectional synthesis technique is needed and it is to support this.

A concrete method of generating the upper boundary, the lower boundary, and the right boundary using the equations (2), (3), and (4) 5 (b).

When the upper boundary, the lower edge generation boundary, and the right edge generation boundary are generated in the overlap area, the minimum error boundary generator 140 generates the minimum error boundary And outputs it to the direction selector 150.

Then, the minimum error boundary direction selection unit 150 selects a boundary having the smallest error value among the boundaries generated in each direction of the overlap region. For this, the minimum error boundary direction selection unit 150 may use Equation (5) as follows.

In Equation (5), S (i) represents the generated boundary, N represents the number of pixels for the generated boundary, and SEAMavr represents the error average of the boundary. When SEAMavr is used, an error that can be caused by the generated boundary can be confirmed numerically.

Therefore, the SEAMavr for each direction boundary can be used as a measure for evaluating the boundary. For example, the minimum error boundary direction selection unit 150 may select a direction corresponding to a boundary having the smallest error among the average of the errors with respect to the boundaries of the respective directions. The minimum error boundary direction selection unit 150 outputs information on the boundary of the selected direction (i.e., the boundary having the minimum error) to the image composition unit 160. [

The image synthesizer 160 synthesizes the images in the overlap area based on the information about the boundary input from the minimum error boundary direction selector 150. The image synthesizer 160 synthesizes the real images in the left, right, up, and down directions on the basis of the boundary corresponding to the information, and outputs the panorama image.

1, the minimum error matrix generator 130, the minimum error boundary generator 140, the minimum error boundary direction generator 150, and the image synthesizer 160 are shown as separate components. However, It is to be understood that the components 130, 140, 150, and 160 may be implemented as at least one physical component 100, such as a controller or processor, in accordance with various embodiments.

FIG. 2 (a) is a graph showing an example of a quantization function used for quantizing depth map values according to an embodiment, and FIG. 2 (b) is a graph used to quantize depth map values according to an embodiment A graph showing another example of the quantization function. FIG. 2 (c) is a graph showing another example of a quantization function used to quantize the depth map values according to an embodiment. FIG. 2 (d) is a graph illustrating a quantization function for quantizing depth map values according to an embodiment A graph showing a further example of the quantization function used.

In the graphs shown in FIGS. 2 (a) to 2 (d), the horizontal axis represents depth map values for each pixel expressed by a value between 0 and 255, and the vertical axis represents a weight value value corresponding to each depth map value .

The quantization function may define the relationship between the depth map value and the weight value in various forms as shown in Figs. 2 (a) to 2 (d). As shown in Figs. 2 (a) to 2 (d), the quantization function increases the weight value corresponding to the depth map as the depth increases, that is, as the distance between the camera and the object in the image increases .

In FIGS. 2 (a) to 2 (d), the weight values that can correspond to the depth map values are quantized into 16 levels. However, the weight values are not limited thereto and may be used in various ways. When the above quantization function is used, it is possible to minimize an error that may occur when an object at a long distance is covered by a nearby object when creating a boundary.

Hereinafter, the processes of generating the lower boundary, the upper boundary, and the right boundary will be described with reference to FIGS. 3 (a) through 5 (b).

FIG. 3 (a) is a view illustrating an overlap area for creating a lower boundary according to an embodiment of the present invention, and FIG. 3 (b) is a diagram illustrating a process of creating a lower boundary according to an embodiment.

3 (a), the overlap region may be a rectangular region having 9 x 7 pixels, for example, and the minimum error matrix values (i.e., E (i, j)) may be determined for each pixel position have. As described above with reference to Equation (1), the minimum error matrix values are values to which weights corresponding to the depth map values of the corresponding pixel positions are applied, respectively. Therefore, the smaller the minimum error matrix value, the closer the distance between the camera and the object in the image is.

On the other hand, in Figure 3 the minimum error metric value as shown in (a) are described above as being generated based on equation (1), the lower direction as shown in FIG. 3 (b) threshold (in Equation 2 S _ia ( i, j).

For example, based on Equation (2), E (1, j) values (9, 4, 4, 9, 16, 25, 4, 1) included in the first row of the overlapping region , 2) are generated as S _1a (1, j) values 9, 4, 4, 9, 16, 25, 4, 1, 2 as they are. The values E (2, j) (4, 4, 9, 16, 1, 9, 9, 4, 25) located in the second row of the overlap region are represented by S _2a 13, 20, 10, 13, 10, 5, 26). The starting point for generating the lower boundary values may be a row (e.g., two rows) positioned at the lower end of the update of actual values of the two rows when there are two adjacent rows (e.g., 1 row and 2 rows) . &Lt; / RTI >

The value 4 of E (2, 2) at pixel position (2, 2) is equal to the value of E (1, 2) adjacent to the previous row of pixel position The value of E (2, 2) is added to the smallest value 4 of the values (9, 4, 4) of E (1, 2), E _2a (2, 2).

Further, the value 4 of E (3, 3) at the pixel position 3, 3 corresponds to three E (1, 1), E (1, 2) E (1, 3) values (i.e., the updated value of 8, 13, 20) the summed with a value 8, the value of the E (3, 3) of the value of the added result S _2a (3, 3) of the Lt; / RTI >

When all the updates to E (i, j) are completed in the above manner, the result as shown in FIG. 3 (b) is calculated. The arrows shown in FIG. 3 (b) show the direction in which the values of E (i, j) are added to the corresponding pixels. FIG. 3 (b) shows a case where the direction is the bottom direction of the overlap region.

On the other hand, when S _ia (i, j) as shown in FIG. 3 (b) is determined, a minimum value is detected for each row. The pixels corresponding to the detected minimum value may be connected and determined as a bottom boundary. In the example of FIG. 3 (b), pixels having the minimum values (1, 5, 6, 7, 8, 9, 10) in each of the first to seventh rows may be determined as pixels for the lower boundary.

FIG. 4A is a view showing an overlap area for creating an upper boundary according to an embodiment, and FIG. 4B is a diagram illustrating a process of creating an upper boundary according to an embodiment.

Referring to FIG. 4A, the overlap region may be a rectangular region having 9 x 7 pixels, and minimum error matrix values (i.e., E (i, j)) are determined for each pixel position . As described above with reference to Equation (1), the minimum error matrix values are values to which weights corresponding to the depth map values of the corresponding pixel positions are applied, respectively. Therefore, the smaller the minimum error matrix value, the closer the distance between the camera and the object in the image is.

On the other hand, the 4 minimum error metric value as shown in (a) are described above as being generated based on equation (1), the upper direction as shown in Fig. 4 (b) threshold (in Equation 3 S _ib ( i, j).

(4, 4, 1, 4, 25, 1, 4, 5) included in the last row of the overlap region in FIG. 4 (a) 4) are generated as S _7b (7, j) values (4, 4, 1, 1, 4, 25, 1, 4, 4) as they are. And in the second row from the bottom in the overlap region E (6, j) values (4, 25) S _6b (6, j) values (8, 26, 17, 26, 5, 5, 2 and 5 , 8). The starting point for generating the upper direction boundary values may be a row (e.g., 6 rows) positioned at the upper position where update of substantial values of the two rows is started when two adjacent rows (e.g., row 6, row 7) . &Lt; / RTI >

The value 1 of E (6, 7) at pixel positions 6 and 7 is the sum of the three E (7, 7) adjacent to the next row of pixel positions 6, 7 , E (7, 6), E (7, 7) and E (7, 8) are added to the smallest value 1 among the values 25, _6b (6, 7).

Further, the value 4 of E (5, 4) at the pixel position 5, 4 corresponds to three E (6, 3), E (6, 4) E (6, 5), the values (i.e., the updated value of 17, 26, 5) the summed with a value 5, the value of the E (5, 4) of the said added result value, S _5b (5, 4) of the To " 9 "

When all the updates to E (i, j) as described above are completed, the result as shown in Fig. 4 (b) is calculated. The arrows shown in FIG. 4 (b) show the direction in which the value of E (i, j) is added to the pixel. In FIG. 4 (b), the direction is the upper direction of the overlapping area.

On the other hand, when S _ib (i, j) as shown in FIG. 4 (b) is determined, a minimum value is detected for each row. The pixels corresponding to the detected minimum value may be connected and determined as the upper direction boundary. In the example of FIG. 4 (b), pixels having the minimum values (10, 9, 5, 4, 3, 2, 1) in each of the first to seventh rows may be determined as pixels for the upper direction boundary.

FIG. 5A is a diagram illustrating an overlap region for generating a right-direction boundary according to an embodiment, and FIG. 5B is a diagram illustrating a process of generating a right-direction boundary according to an embodiment.

5 (a), the overlap region may be a rectangular region having 9 x 7 pixels, for example, and the minimum error matrix values (i.e., E (i, j)) may be determined for each pixel position have. As described above with reference to Equation (1), the minimum error matrix values are values to which weights corresponding to the depth map values of the corresponding pixel positions are applied, respectively. Therefore, the smaller the minimum error matrix value, the closer the distance between the camera and the object in the image is.

On the other hand, the 5 minimum error metric value as shown in (a) are described above as being generated based on equation (1), the right end in the same direction as shown in Figure 5 (b) threshold (of equation 4 S _j ( i, j).

For example, based on Equation (4), E (i, 1) values (9, 4, 4, 4, 25, 4, 4) included in the first column of the overlap region in FIG. S ₁ (i, 1) values (9, 4, 4, 4, 25, 4, 4). And in the second column of the overlap region E (i, 2) values (4, 4, 16, 4, 25, 25, 4) S ₂ (i, 2) value (8, 8, 20, 8, 29 , 29, 8). The starting point for generating the right direction boundary values is a row (e.g., two columns) located at the right end where the updating of substantial values of the two columns is started when two adjacent columns (e.g., columns 1 and 2) . &Lt; / RTI >

The value 4 of E (2, 2) at pixel position 2, 2 is equal to the value of E (1, 2) adjacent to the previous column of pixel position 2, 1), E (2, 1 ), E (3, 1) values (9, 4, 4) the summed with a value 4, the value of the E (2, 2) of the S ₂ value of the added result of the (2, 2).

Further, the value 4 of E (3, 3) at the pixel position 3, 3 corresponds to three E (2, 2), E (3, 2), E (4, 2) the values summed and the smallest value of 8 (i.e., the updated value of 8, 20, 8), the E (3, 3) the value of the added result value, S ₂ (3, 3) of the The value is updated to 12.

When all the updates to E (i, j) are completed in the above manner, a result as shown in FIG. 5 (b) is calculated. The arrows shown in FIG. 5 (b) show the direction in which the value of E (i, j) is added to the pixel. In FIG. 5 (b), the direction is the rightward direction of the overlapping area.

On the other hand, when S _j (i, j) as shown in FIG. 5 (b) is determined, a minimum value is detected for each column. The pixels corresponding to the detected minimum value may be connected and determined as a right-direction boundary. In the example of Fig. 5 (b), pixels having the minimum values (4, 8, 9, 10, 14, 15, 9, 13, 17) in columns 1 to 9 can be determined as pixels for the right- .

When the upper boundary, the lower boundary, and the right boundary are determined based on the schemes described with reference to FIGS. 3A to 5B, the probability of occurrence of an error among the determined boundaries based on Equation (5) This lower boundary can be selected. And the real images may be synthesized in the direction corresponding to the selected boundary to be generated as a panoramic image.

6 is a diagram illustrating an overlap region according to one embodiment.

The overlap area 600 may be an area including the same object in the multiple frames 610 and 620 corresponding to the real image shot at different points in time. For example, the overlap area may be a region having a predetermined area connecting the upper and lower sides of the frame, an area having a predetermined area connecting the left and right sides of the frame, or a predetermined area connecting the left (or right) It may be an area having an area.

In FIG. 6, the overlap area 600 has a rectangular shape as an area having a predetermined area connecting left and right sides of the frame. However, the shape of the overlap area 600 is not limited to this, have. On the other hand, when four edges constituting the overlap region 600 are denoted by A, B, C, and D, respectively, minimum error boundaries as shown in FIGS. 7A to 7F can be generated .

FIG. 7A is a diagram illustrating a first example of a minimum error boundary according to an embodiment, FIG. 7B is a diagram illustrating a second example of a minimum error boundary according to an embodiment, and FIG. 7C 7D is a diagram illustrating a fourth example of a minimum error boundary according to an embodiment, and FIG. 7E is a diagram showing a fourth example of a minimum error boundary according to an embodiment FIG. 7 (f) is a diagram illustrating a sixth example of a minimum error boundary according to an exemplary embodiment of the present invention.

A minimum error boundary may be generated when a scheme based on equations (1) to (5) as described above is used, the shape of the minimum error boundary may vary in various forms, as shown in FIGS. 7 (a) .

7 (a) shows the minimum error boundary of the shape extending from the upper end to the right end of the overlap region, Fig. 7 (b) shows the minimum error boundary of the shape extending from the upper end to the lower end of the overlap region, and Fig. 7 ) Represents a minimum error boundary of the form leading from the top of the overlap region to the left end.

7 (d) shows the minimum error boundary of the shape extending from the right end to the bottom end of the overlap region, FIG. 7 (e) shows the minimum error boundary of the shape extending from the left end to the right end of the overlap region, f) represents a minimum error boundary of the shape extending from the left end to the bottom end of the overlap region.

According to the embodiment, since the minimum error boundaries of the overlapping region can be generated in various manners, the corresponding real images are synthesized in the left, right, up, and down directions with respect to the minimum error boundary, Images can be generated.

FIG. 8A is a view showing an example of a panorama image generated using the conventional method, FIG. 8B is an example of a panorama image generated using the method according to the embodiment of the present invention, Fig. When the real images are synthesized based on the minimum error boundaries as described above, errors such as image distortion occurring during image synthesis can be solved.

For example, when an image is synthesized using an existing method in which a depth map value and the like are not considered, there may occur a problem that a partial image of an object is lost as indicated by reference numeral 810 in FIG. 8 (a). However, when the images are synthesized using the above-described method, a panorama image without image loss can be generated as indicated by reference numeral 820 in FIG. 8 (b).

9 is a flowchart illustrating an image synthesizing method of an image synthesizing apparatus according to an embodiment.

Referring to FIG. 9, the image synthesizer receives a depth map corresponding to each of the real images and the real images (operation 910). The depth map of each real image may have the same or different size than the corresponding real image. If the overall system performance is considered, it is more advantageous to use a depth map having the same size as the corresponding real image. This is because, if the real image and the depth map have different sizes, there is a loss of information in the process of matching the sizes, which may affect the performance of the entire system.

The image synthesizer performs an operation to generate a matching position of the real images (operation 920). That is, the image synthesizer performs a process of matching feature points and descriptors extracted from each of the real images, and determines a positional relationship between the matching feature points. In addition, the image synthesizing apparatus generates a position to be matched between the real images in consideration of the positional relationship, and confirms the overlapping region based on the generated position.

The image synthesizer generates a minimum error matrix based on the position information and the depth map of the overlap area (operation 930). As shown in Equation 1, the depth map value for each pixel based on the depth map is changed to a weight value and reflected in the minimum error matrix.

The image synthesizer generates a minimum error boundary using the minimum error matrix (operation 940). For example, the image synthesizer generates a boundary for each direction of the overlapping area based on Equations 2, 3, and 4, and selects a boundary where errors are least likely to occur among the generated boundaries, A minimum error boundary can be generated.

When the minimum error boundary is generated, the image synthesizing apparatus determines whether to generate a composite image as a panorama image (operation 950). If it is determined that the composite image should be generated, the image synthesizer synthesizes the images in the overlap region based on the minimum error boundary to generate the panorama image (operation 960). The operations shown in FIG. 9 may be repeatedly performed for generating the panoramic image.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments, but is capable of various modifications within the scope of the invention. Therefore, the scope of the present invention should not be limited by the illustrated embodiments, but should be determined by the scope of the appended claims and equivalents thereof.

Claims

In the image synthesis method,

Comprising the steps of: arranging the plurality of frames based on feature points extracted from a plurality of frames corresponding to each of a plurality of images;

Identifying an overlap region including feature points corresponding to each other among the aligned frames in an area including the aligned frames;

Determining a weight for each pixel based on the depth information of the overlap region and generating a minimum error matrix by assigning a weight for each pixel to a difference between pixel values of frames included in the overlap region;

Generating a boundary using the minimum error matrix, and compositing the aligned frames based on the generated boundary.
The method according to claim 1,

Wherein the step of aligning the plurality of frames comprises:

Determining a matching position based on the extracted feature points from the plurality of frames, and aligning the plurality of frames based on the determined matching position.
The method according to claim 1,

Wherein the weight of each pixel has a value proportional to a depth map value included in the depth information.
The method according to claim 1,

Generating a boundary using the minimum error matrix,

Generating a update matrix by updating pixel values in units of rows of the minimum error matrix;

Detecting a pixel value having a minimum value for each column of the update matrix and generating the boundary based on pixel values of the detected columns.
5. The method of claim 4,

Updating the pixel values in units of rows of the minimum error matrix,

And adding the pixel value having the smallest value among the adjacent pixel values to the pixel value of each pixel included in the specific row of the minimum error matrix to update the corresponding pixel value,

Wherein the adjacent pixel values are adjacent pixel values included in an upper row or a lower row of the specific row.
The method according to claim 1,

Generating a boundary using the minimum error matrix,

Generating a update matrix by updating pixel values in units of columns of the minimum error matrix;

Detecting a pixel value having a minimum value for each row of the update matrix and generating the boundary based on pixel values of the detected rows.
The method according to claim 6,

Updating the pixel values in units of columns of the minimum error matrix,

And adding the pixel value having the smallest value among the adjacent pixel values to the pixel value of each pixel included in the specific column of the minimum error matrix to update the corresponding pixel value,

Wherein the adjacent pixel values are adjacent pixel values included in a right row or a left row of the specific column.
The method according to claim 1,

Generating a boundary using the minimum error matrix,

Generating a first update matrix by updating pixel values starting from a first row of the minimum error matrix, detecting a pixel value having a minimum value for each column of the first update matrix, Generating a first boundary based on pixel values of each column,

And a second updating matrix is generated by updating pixel values starting from a second row of the minimum error matrix, a pixel value having a minimum value for each column of the second updating matrix is detected, Generating a second boundary based on pixel values of each column,

A pixel update step of generating a third update matrix by updating pixel values starting from a first column of the minimum error matrix, detecting a pixel value having a minimum value for each row of the third update matrix, Generating a third boundary based on pixel values of each column,

Selecting one of the first boundary, the second boundary and the third boundary as a boundary to be used for synthesizing the aligned frames.
9. The method of claim 8,

Selecting one of the first boundary, the second boundary, and the third boundary as a boundary to be used for compositing the aligned frames,

Generating a first value by calculating an average value for pixel values that generate the first boundary, calculating an average value for pixel values that generate the second boundary to generate a second value, Generating a third value by calculating an average value for the pixel values to be generated;

And selecting a boundary corresponding to a smallest value among the first value, the second value and the third value as a boundary to be used for synthesizing the aligned frames.
9. The method of claim 8,

Wherein the step of synthesizing the aligned frames comprises:

And combining the sorted frames in an up, down, left, and right directions based on the selected boundaries.
In the image synthesizer,

An interface unit for acquiring a plurality of images,

A plurality of frames arranged on the basis of feature points extracted from a plurality of frames corresponding to each of the plurality of images, and a plurality of feature points corresponding to the aligned frames, Determining a weight for each pixel on the basis of the depth information of the overlap region, generating a minimum error matrix by assigning a weight for each pixel to a difference between pixel values of frames included in the overlap region, And a processor for generating boundaries using the minimum error matrix and compositing the aligned frames based on the generated boundaries.
An image synthesizer configured to perform the method according to any one of claims 2 to 10.