KR102236222B1 - Method and apparatus of stitching for minimizing parallax using control points in overlapping region - Google Patents

Abstract

Disclosed is an apparatus and method for minimizing parallax stitching using control points of an overlapping area. The parallax minimization stitching method according to an embodiment includes defining a plurality of control points within an overlapping area of a first image and a second image received from a plurality of cameras, and applying a homography to the plurality of control points. And performing geometry correction, defining a plurality of patches based on the plurality of control points, and performing a second order geometry correction by mapping the plurality of patches.

Description

Parallax minimization stitching device and method using control points of overlapping area {METHOD AND APPARATUS OF STITCHING FOR MINIMIZING PARALLAX USING CONTROL POINTS IN OVERLAPPING REGION}

The following embodiments relate to a stitching apparatus and method for minimizing parallax using control points of an overlapping area.

In general, in order to generate a panoramic image, a plurality of cameras must photograph by overlapping a portion of the photographed image. The plurality of cameras extracts feature points from overlapping photographed portions and matches the same feature points to calculate homography information, which is a geometric relationship between adjacent images. Homography information may mean geometric correction information between adjacent images. In planar stitching or cylinder stitching, geometric correction can be performed using homography. However, in 360VR (spherical) stitching, geometry correction can be performed using pitch, yaw, roll, and lens correction information.

Using the calculated homography information, stitching is performed to combine adjacent images. However, in the procedure for calculating homography information, feature point extraction and matching are extracted and matched for the entire image. Therefore, the process of extracting and matching feature points not only takes a lot of time, but also affects the performance of the camera. In addition, image characteristics also affect the stitched image quality. For example, if the number of feature points is extremely insufficient due to image characteristics such as night view, sky, city center, etc., there is a case that the relationship calculation fails due to insufficient amount of basic information to calculate matching and homography information, or the incorrect homography is calculated due to mapping. Can occur. In addition, when a near subject and a distant subject are simultaneously photographed, the stitched image may be distorted due to parallax according to the case where the calculated homography is calculated based on the feature points of the near subject and the case that is calculated based on the feature points of the distant subject. I can.

Parallax may occur when a near object and a far object exist at the same time in the overlapping area, or when physical sensors between a plurality of cameras are far away, and the quality of the image is degraded.

The embodiments may provide a technique for minimizing parallax between a plurality of images in stitching a plurality of images acquired from a plurality of cameras.

In addition, embodiments may provide a technique for performing a first-order geometry correction and a second-order geometry correction on a plurality of control points.

In addition, the embodiments may provide a technique of updating coordinates of a plurality of control points in a look-up table (LUT) based on a first-order geometry correction or a second-order geometry correction.

In addition, embodiments may provide a technique for transforming coordinates corresponding to a panoramic image generated from a plurality of images based on depth information of control points in an overlapping area.

The parallax minimization stitching method according to an embodiment includes the steps of defining a plurality of control points within an overlapping region of a first image and a second image received from a plurality of cameras, and the plurality of control points. Performing a first-order geometric correction by applying homography to the fields, defining a plurality of patches based on the plurality of control points, and mapping the plurality of patches Thereby performing a second order geometry correction.

The performing of the second-order geometry correction may include mapping the plurality of patches using a cost function.

The cost function is a correlation function, a mean squared error (MSE) function, a fast structure similarity (FSSIM) function, and a peak signal-to-noise ratio. (PSNR)) may be one of the functions.

The method may further include converting coordinates of a plurality of control points on which the first-order geometry correction has been performed using depth information.

The converting of the coordinates may include a distance and direction of the matched feature points among a plurality of feature points included in the overlapping area, a distance and direction of a control point of in the overlapping area, and the at least one object ( object) alignment information, FSIM (Feature Similarity Index) of the overlapping region, and HFI_SSIM (High Frequency Information_Structure Similarity Index) of the overlapping region, calculating the depth information have.

The converting of the coordinates includes determining whether the plurality of control points correspond to a short distance or a far distance based on the depth information, lengthening a distance between the control points corresponding to a short distance, and a long distance It may include a step of shortening the distance between the control points corresponding to (shorten).

The method may further include updating coordinates of the plurality of control points in a look-up table (LUT) based on the first geometry correction or the second geometry correction.

The parallax minimization stitching apparatus according to an embodiment includes a receiver for receiving a first image and a second image from a plurality of cameras, and a plurality of control points within an overlapping region of the first image and the second image. points), performing a first-order geometric correction by applying homography to the plurality of control points, defining a plurality of patches based on the plurality of control points, and It includes a controller that performs secondary geometry correction by mapping the patches.

The controller may map the plurality of patches using a cost function.

The cost function is a correlation function, a mean squared error (MSE) function, a fast structure similarity (FSSIM) function, and a peak signal-to-noise ratio. (PSNR)) may be one of the functions.

The controller may convert coordinates of a plurality of control points on which the primary geometry correction has been performed, using depth information.

The controller includes a distance and direction of matched feature points among a plurality of feature points included in the overlapping area, a distance and direction of a control point included in the overlapping area, and alignment of the at least one object. The depth information may be calculated using at least one of (alignment) information, a feature similarity index (FSIM) of the overlapping area, and a high frequency information_structure similarity index (HFI_SSIM) of the overlapping area.

The controller determines whether the plurality of control points correspond to a short distance or a far distance based on the depth information, lengthen the distance between the control points corresponding to the short distance, and determine the distance between the control points corresponding to the far distance. Can be reduced.

The controller may update the coordinates of the plurality of control points in a look-up table (LUT) based on the first geometry correction or the second geometry correction.

1 is a diagram for explaining overall stitching operation of a stitching system according to an exemplary embodiment.
2 is a block diagram of a stitching apparatus for minimizing parallax according to an exemplary embodiment.
3A is a diagram for describing a plurality of control points in a first image.
3B is a diagram for explaining a plurality of control points in a second image. FIG. 4 is a diagram for explaining an operation of a controller calculating depth information.
5 is a diagram for describing an operation of converting coordinates by a controller.
6 shows a plurality of images received by the receiver.
7 shows that parallax occurs in an overlapped area when the controller performs first-order geometry correction on a plurality of images.
8 shows an example of a result of a controller performing a first-order geometry correction and a second-order geometry correction on a plurality of control points within an overlapping area.
9 shows an example of a result of the controller performing a first-order geometry correction on a plurality of images.
10 shows that parallax occurs in an overlapped area when the controller performs first-order geometry correction on a plurality of images.
11 is a diagram for describing an operation of a controller defining a plurality of control points.
12 is a diagram for describing an operation in which the controller performs secondary geometry correction.
13 shows another example of a result of the controller performing a first-order geometry correction and a second-order geometry correction on a plurality of control points within an overlapping area.

Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in the present specification are exemplified only for the purpose of describing embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention They may be implemented in various forms and are not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention can apply various changes and have various forms, the embodiments will be illustrated in the drawings and described in detail in the present specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The terms are only for the purpose of distinguishing one component from other components, for example, without departing from the scope of the rights according to the concept of the present invention, the first component may be referred to as the second component, Similarly, the second component may also be referred to as a first component.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle. Expressions that describe the relationship between components, for example, “between” and “just between” or “directly adjacent to” should be interpreted as well.

The terms used in the present specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that the specified features, numbers, steps, actions, components, parts, or combinations thereof exist, but one or more other features or numbers, It is to be understood that the presence or addition of steps, actions, components, parts, or combinations thereof does not preclude the possibility of preliminary exclusion.

Unless otherwise defined, all terms used herein including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be construed as having a meaning consistent with the meaning of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in the present specification. Does not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. The same reference numerals shown in each drawing indicate the same members.

1 is a diagram for explaining overall stitching operation of a stitching system according to an exemplary embodiment.

Referring to FIG. 1, the stitching system 90 may include a plurality of cameras 111 and 112 and a stitching apparatus 100 for minimizing parallax.

1 illustrates that a plurality of cameras 111 and 112 are implemented outside the parallax minimizing stitching apparatus 100, but are not necessarily limited thereto, and according to an embodiment, the plurality of cameras 111 and 112 minimize parallax. It may be implemented inside the stitching device 100.

That is, the parallax minimization stitching apparatus 100 may perform a stitching (or bonding) operation by acquiring an image captured from an external camera, or a stitching (or bonding) operation by acquiring an image captured from an internal camera. You can do it.

The plurality of cameras 111 and 112 may photograph a plurality of objects 10a, 10b, and 10c, and may generate a plurality of images 51 and 52. The plurality of images 51 and 52 may be one of a static image, a dynamic image, a 2D image, and a 3D image. In order to generate a plurality of images 51 and 52, a plurality of cameras 111 and 112 may be arranged in a radial shape.

In this case, the plurality of cameras 111 and 112 may include a first camera 111 and a second camera 112. The first camera 111 may be a camera in a first position, and the second camera 112 may be a camera in a second position. The position (first position, second position) of the plurality of cameras 111 and 112 is a relative concept according to the camera arrangement. For example, if the first position is left, the second position is right, and if the first position is right, The second position may be on the left.

The first camera 111 and the second camera 112 may be arranged to have the same central point to generate a panoramic image, and have the same angle of view, the same focal length, And may have the same resolution.

In addition, the first camera 111 and the second camera 112 may be arranged to be rotated in a fixed position. That is, the first camera 111 and the second camera 112 may adjust the range of the areas 11, 12, and 13 that are photographed through the rotation operation. For example, by rotating at least one of the first and second cameras 111 and 112, at least one of the size of the overlapping area 12 and the position of the overlapping area 12 may be adjusted. The first camera 111 and the second camera 112 may be rotated manually or automatically in response to a user's input.

The first camera 111 and the second camera 112 may photograph a plurality of objects 10a, 10b, and 10c existing within an effective field of view. In this case, the first camera 111 and the second camera 112 may photograph by overlapping (or overlapping) a predetermined area with respect to the plurality of objects 10a, 10b, and 10c.

For example, the first camera 111 may generate the first image 51 by photographing the objects 10a and 10b included in the field of view of the first camera 111. The second camera 112 may generate the second image 52 by photographing the objects 10b and 10c included in the angle of view of the second camera 112.

At this time, since the first camera 111 and the second camera 112 overlap (or overlap) by a certain angle of view, there may be some objects 10b that are identically photographed in each of the cameras 111 and 112. That is, some of the plurality of objects 10a, 10b, and 10c may be projected (or photographed) equally to the sensors of the first camera 111 and the second camera 112.

The first image 51 includes an object 20a included only in the first image 51 and an overlapping object 20b, and the second image 52 is an object 20c included only in the second image 52 ) And the overlapping object 20c. In addition, the first image 51 includes an area 11 and an overlapping area 12 photographed only in the first image 51, and the second image 52 is an area photographed only in the second image 52 ( 13) and an overlapping area 12.

In addition, the first camera 111 and the second camera 112 may photograph at least one of the plurality of objects 10a, 10b, and 10c with an appropriate overlapping angle of view.

An appropriate overlapping angle of view may mean a reference value of a view angle of a degree capable of calculating the feature points of the object 20b of the overlapping area 12 that the first camera 111 and the second camera 112 commonly photograph. For example, when the first camera 111 and the second camera 112 are set to a field of view less than the appropriate overlapping angle of view, the overlapping area 12 is too small, so that a part of the object 20b is overlapped. It may be located outside of, and the parallax minimizing stitching device 100 may not be able to calculate the feature points of the object 20b.

The feature points may be points necessary for stitching (or bonding) a plurality of adjacent images 51 and 52. Specifically, the feature point may mean a specific point on the same object for confirming the same point of the adjacent images 51 and 52. Accordingly, an appropriate overlapping angle of view may be set on the first camera 111 and the second camera 112 in order to calculate the feature points of the object 20b.

The parallax minimizing stitching device 100 may be a device for performing stitching. The parallax minimizing stitching device 100 may be implemented as an electronic device. The electronic device may be implemented as a personal computer (PC) or a portable device.

Portable devices include a laptop computer, a mobile phone, a smart phone, a tablet PC, a mobile internet device (MID), a personal digital assistant (PDA), an enterprise digital assistant (EDA). , Digital still camera, digital video camera, camcorder, portable multimedia player (PMP), personal navigation device or portable navigation device (PND), television (television), portable game console It may be implemented as a (handheld game console), an e-book, a navigation device, or a smart device. The smart device may be implemented as a smart watch or a smart band. The parallax minimization stitching apparatus 100 may generate a panoramic image based on the plurality of images 51 and 52. The plurality of images 51 and 52 may include a first image 51 and a second image 52.

For example, the parallax minimizing stitching apparatus 100 may create a panoramic image by stitching (or bonding) the first image 51 and the second image 52.

The parallax minimization stitching apparatus 100 defines a plurality of control points within the overlapping area 12 of the first image 51 and the second image 52, and homography on the plurality of control points. ) Can be applied to perform a first-order geometric correction. The primary geometry correction may mean stitching correction. For example, the parallax minimization apparatus 100 may generate a polynomial surface using Bernstein polynomials by using a plurality of control points. The parallax minimization device 100 may generate a Bezier surface as an example of a higher-order surface. The parallax minimization device 100 may change the shape and curvature of the Bezier surface by changing the positions of the plurality of control points.

The parallax minimizing stitching apparatus 100 may define a control point in the overlapped area 12 by a preset division value. For example, the parallax stitching apparatus 100 may define 32 x 18 control points by dividing the overlapping area 12 of 1920 x 1080 pixels based on a preset split value of 60. The parallax minimization stitching device 100 generates an image in real time by associating (or linking) the first image 51 and the second image 52 with Cartesian coordinates of the panoramic image (12K x 2K @ 60 fps). )can do.

In addition, the parallax minimizing stitching apparatus 100 may define a plurality of patches based on a plurality of control points and perform secondary geometry correction by mapping the plurality of patches. The secondary geometry correction may mean a parallax minimization correction.

The parallax minimization stitching apparatus 100 may update coordinates of a plurality of control points in a look-up table (LUT) based on the first geometry correction or the second geometry correction.

That is, the parallax minimization stitching apparatus 100 detects a parallax problem due to different depth information of the object 20b of the first image 51 and the second image 52, and can minimize the parallax. It can be utilized as a measurement tool.

In FIG. 1, only two cameras 111 and 112 are shown for convenience of explanation, but there may be n cameras (n is an integer greater than or equal to 1), and accordingly, there may be n positions. In this case, it may be referred to as an n-th camera at an n-th position according to the relative arrangement (or position) of the n cameras.

That is, regardless of how many cameras exist (eg, one camera or a plurality of cameras), the parallax minimizing stitching apparatus 100 may generate a panoramic image using a plurality of images.

Hereinafter, a stitching operation for minimizing parallax by the parallax minimizing stitching apparatus 100 will be described in detail.

2 is a block diagram of a parallax minimizing stitching apparatus according to an embodiment, FIG. 3A is a diagram for explaining a plurality of control points in a first image, and FIG. 3B is a diagram for explaining a plurality of control points in a second image , FIG. 4 is a diagram for explaining an operation of the controller calculating depth information.

1 to 4, the parallax minimizing stitching apparatus 100 may include a receiver 130, a controller 150, a display 170, and a memory 190.

The receiver 130 may receive a plurality of images 51 and 52 from the first camera 111 and the second camera 112. In this case, the receiver 130 may receive at least one image from the first camera 111 and the second camera 112 through one of a wired communication or a wireless communication network.

For example, wired communication may be a wired LAN, communication using a USB terminal (eg, USB 2.0, USB 3.0), or cable communication, and the wireless communication network is an Internet communication network, an intranet, or Bluetooth. , It may be a local area communication network (LAN), a wireless LAN, or an IEEE 802.11-based Wi-Fi.

The controller 150 may control the overall operation of the stitching apparatus 100 for minimizing parallax. That is, the controller 150 may control the receiver 130, the display 170, and the memory 190.

In addition, the controller 150 may generate a panoramic image by performing a stitching (or bonding) operation for minimizing parallax on the plurality of images 51 and 52. For example, the controller 150 extracts feature points from a plurality of images 51 and 52, and calculates homography of a plurality of images 51 and 52 based on the feature points. , An operation of defining a plurality of control points, an operation of performing a first-order geometric correction on a plurality of control points, an operation of estimating depth information of the overlapping region 12, an operation of estimating the parallax of the overlapping region 12, At least one of an operation of extracting a plurality of patches based on the control points, an operation of performing secondary geometry correction on the plurality of control points, and an operation of stitching the plurality of images 51 and 52 is performed. You can do it.

In this case, the controller 150 may be implemented as a printed circuit board (PCB) such as a motherboard, an integrated circuit (IC), or a system on chip (SoC). For example, the controller 150 may be an application processor.

The controller 150 may set a feature point extraction region on the plurality of images 51 and 52. The feature point extraction area may be an area to which the feature point is to be extracted. For example, the feature point extraction region may be an overlapping region 12 of a plurality of neighboring images 51 and 52, or may be a region set in response to a user's input.

The controller 150 may extract (or define) a feature point from at least one object included in the feature point extraction region. For example, when the feature point extraction region is the overlap region 12 of the first image 51 and the second image 52, the controller 150 selects the feature point from the object 20b included in the overlap region 12. Can be extracted. In this way, when the feature point extraction area is the overlapping area 12 of the first image 51 and the second image 52, the controller 150 extracts the feature points within the limited area, so it uses less resources and performs fast processing. May be possible. The feature point may be a set including at least one point constituting an object.

The controller 150 may generate at least one of homography information and information for a stitching (or bonding) operation through at least one feature point extracted from the first image 51 and the second image 52. In this case, the controller 150 may calculate homography information using a combination line connecting the corresponding feature points in the first image 51 and the second image 52. The homography information may be a relationship between two plane surfaces extracted from the first image 51 and the second image 52, respectively.

In addition, the controller 150 may define a plurality of control points in the overlapped region 12 by a preset division value. The plurality of control points defined by the controller 150 in the first image 51 may be as illustrated in FIG. 3A, and the plurality of control points defined in the second image 52 may be as illustrated in FIG. 3B. The controller 150 may perform geometry correction by moving the positions (coordinates) of the plurality of control points 31 and 33 within the overlapping region 12.

The controller 150 may perform primary geometry correction on a plurality of control points using homography information. The first geometry correction may mean stitching correction. That is, the controller 150 may perform first-order geometry correction by applying homography information to a plurality of control points. As a result of the first geometric correction, the coordinates (positions) of the plurality of control points may be moved (transformed).

The controller 150 may update coordinates of a plurality of control points in a look-up table (LUT). The controller 150 may update coordinates of a plurality of control points in a lookup table (LUT) whenever geometry correction is performed.

The controller 150 may generate the stitched image 40 by performing the first geometry correction. The stitched image 40 may include a plurality of feature points 41, 43, and 45.

The controller 150 may estimate depth information of the stitched image 40 and parallax according to different depth information based on the feature points 41, 43, and 45. The depth information may be relative depth information of the stitched image 40, for example, control points included in the overlapping area 12.

For example, the controller 150 may determine a short distance or a far distance based on the number of feature points 41, 43, and 45 included in the object of the stitched image 40. In the stitched image 40, since the number of feature points increases in the order of the first area 41, the second area 43, and the third area 45, the first area 41 and the second area 43 The depth may be deeper in the order of the third area 45. That is, a distance from the cameras 111 and 112 may be in the order of the first area 41, the second area 43, and the third area 45. The controller 150 may estimate the parallax of the stitched image 40 based on the number of feature points 41, 43, and 45.

The parallax minimum stitching apparatus 100 may also include a depth camera. The controller 150 may acquire precise depth information from a depth image obtained from a depth camera.

The controller 150 may stitch (or bond) the first image 51 and the second image 52 based on the depth information. Stitching (or joining) based on depth information may mean 2D template alignment. For example, the controller 150 may perform stitching (or bonding) based on how far the control point is located from the focal point.

The controller 150 includes homography information, information for stitching (or bonding) operation, the distance and direction of the matched feature points in the overlapping area 12, the distance and direction of the control points in the overlapping area 12, and overlapping. Among the alignment information of at least one object in the area 12, feature similarity index (FSIM) information of the overlapping area 12, and high frequency information_structure similarity index (HFI_SSIM) information of the overlapping area 12 At least one may be used to estimate the depth information and the parallax due to the depth information.

The object alignment information may be information about an arrangement state of the object 20b included in the first image 51 and the object 20b included in the second image 52. The object alignment information may be based on at least one of parallax information, FSIM information, and HFI_SSIM information of the plurality of images 51 and 52. The controller 150 may determine how well at least one object in the overlapping area 12 is matched based on at least one of parallax information, FSIM information, and HFI_SSIM information.

The FSIM information may be an index value indicating feature similarity of the plurality of images 51 and 52, or may be a value used to assess the quality of a panoramic image.

The HFI_SSIM information may be SSIM information based on high frequency information of the plurality of images 51 and 52. HFI_SSIM information and SSIM information indicate the structural similarity of a plurality of images 51 and 52, and evaluate the geometric quality of a panoramic image obtained by stitching (or splicing) a plurality of images 51 and 52. May be the value used.

The controller 150 may acquire HFI_SSIM information from the plurality of images 51 and 52 using the A'Trous filter. The controller 150 may separate low frequency information (LFI) and high frequency information (HFI) of the plurality of images 51 and 52 using the A'Trous filter. The A'Trous filter may be a Starck Murtagh filter.

The matched feature points may be feature points matched in the first image 51 and the second image 52. For example, the controller 150 extracts feature points within the overlapping area 12 from the first image 51 and extracts the feature points within the overlapping area 12 from the second image 52 to extract corresponding feature points. You can match. The controller 150 may estimate the depth information of the plurality of feature points and the parallax due to the depth information by using the correlation between the matched feature points.

The controller 150 may perform a stitching operation by minimizing parallax between the first image 51 and the second image 52 based on the depth information.

In this case, the controller 150 may minimize parallax by converting panorama coordinates corresponding to panorama images generated from the plurality of images 51 and 52 based on depth information. For example, the controller 150 may convert a panoramic image surface of a panoramic image into a nonlinear image surface according to depth information. An operation in which the controller 150 changes panorama coordinates will be described in detail with reference to FIG. 5.

The controller 150 may generate a stitched panoramic image by defining a plurality of patches based on a plurality of control points and performing secondary geometry correction based on the plurality of patches. The secondary geometry correction may mean a parallax minimization correction. The plurality of patches may be polygonal.

The controller 150 may map a plurality of patches using a cost function. The cost function is a correlation function, a mean squared error (MSE) function, a fast structure similarity (FSSIM) function, and a Peak Signal-to-noise ratio ( PSNR)) may be one of the functions.

The display 170 may display the stitched panoramic image. The display 170 may be implemented as a liquid crystal display (LCD). In addition, the display 170 is a touch screen, TFT-LCD (Thin Film Transistor-Liquid Crystal Display), LED (Liquid Emitting Diode) display, OLED (Organic LED) display, AMOLED (Active Matrix OLED) display or flexible (flexible) It can be implemented as a display.

The memory 190 may store at least one of feature points, FSIM information, and HFI_SSIM information of the plurality of images 51 and 52.

The controller 150 may evaluate the geometric quality of the stitched panoramic image by using at least one of feature points, FSIM information, and HFI_SSIM information of the plurality of images 51 and 52 stored in the memory 190. In this case, the controller 150 may transmit only an image in which FSIM information or HFI_SSIM information exceeds a threshold to the display 170.

5 is a diagram for explaining an operation of a controller converting coordinates. Referring to FIG. 5, the controller 150 stitches (or bonds) a plurality of images 51 and 52 based on a plurality of control points. I can. That is, the controller 150 may stitch (or bond) the plurality of images 51 and 52 into one panoramic image based on the calculated homography information.

The controller 150 may transform (or change) corresponding panorama coordinates to generate a panorama image. The area to be subjected to coordinate transformation (or change) may be the overlapping area 12. Instead of transforming (or changing) all the coordinates of the plurality of images 51 and 52, only the coordinates of the overlapped area 12 are transformed (or changed), thereby reducing the amount of data to be processed. That is, the controller 150 may increase the operation processing speed of the stitching apparatus 100 for minimizing total parallax by reducing the amount of processing of.

The controller 150 may transform (or change) coordinates of a plurality of control points corresponding to the depth information in order to minimize parallax between the plurality of images 51 and 52 based on the depth information. For example, if the controller 150 estimates depth information based on the feature point, the controller 150 may transform (or change) the coordinates of the control point corresponding to the feature point.

The controller 150 may determine whether the plurality of control points correspond to a short distance or a far distance based on the depth information. The controller 150 may transform (or change) a distance between control points corresponding to a short distance among a plurality of control points, and may transform (or change) a distance between control points corresponding to a remote distance among a plurality of control points.

An operation of the controller 150 changing (or changing) the distances of the plurality of control points may include an operation of lengthening or shortening the distances between the control points. For example, when the object is determined to be a near object, the controller 150 lengthens the distance between control points corresponding to the near object, and when the object is determined to be a remote object, the control point corresponding to the far object It is possible to shorten the distance between them.

In addition, the controller 150 may increase or decrease the spacing (or distance) between pixels corresponding to the control points. Accordingly, a distance (or distance) between pixels in the overlapping area 12 in which a near object is located may increase, or a gap (or distance) between pixels in the overlapping area 12 in which a distant object is located may be reduced.

In addition, the controller 150 may transform (or change) the panorama coordinates using Bezier Transformation on the plurality of images 51 and 52.

The controller 150 may generate a plurality of control points Px on a Bezier patch 50 to perform Bezier transformation. The overlapping area 12 of the plurality of images 51 and 52 may correspond to the Bezier patch 50. For example, the Bezier patch 50 may include nine control points P0 to P8. The nine control points (P0 to P8) are edge control points (P1, P3, P5, P7), corner control points (P0, P2, P6, P8), and center control points (P4). ) Can be included.

The Bezier patch 50 may include a plurality of subdivisions 53, 55, 57, 59. That is, the Bezier patch 50 may include a plurality of control points P0 to P8, and each of the lower patches 53, 55, 57, and 59 may also include a plurality of control points p0 to p8.

The controller 150 may transform (or change) the panorama coordinates using at least one of the plurality of control points Px and px. For example, when the controller 150 moves the control points Px and px, the panoramic plan view may be converted (or changed).

The controller 150 may transform (or change) the position of at least one of the plurality of control points Px and px based on the depth information. That is, the controller 150 may transform (or change) the positions of the control points Px and px to perform panorama coordinate transformation, and may perform a stitching (or splicing) operation of a plurality of images.

For example, the controller 150 may determine that the control points p0 and p1 of the lower patch 59 are near. Accordingly, the controller 150 may increase the distance (or interval) between the control points p0 and p1. Similarly, the controller 150 may increase the distance (or interval) between the control points p1 and p2.

As a result of the controller 150 performing an operation of transforming (or changing) the panorama coordinates, the plane of the overlapping area 12 may become a nonlinear image surface.

In the above, in the operation of the controller 150 to convert the coordinates, the coordinate conversion operation using Bezier transformation has been described, but the method is not limited thereto, and in addition to Bezier transformation, another method capable of converting coordinates such as Gaussian You can also use them to transform (or change) the panorama coordinates.

6 to 8 show an example of a result of stitching (or bonding) a plurality of images by the parallax minimizing stitching apparatus.

6 shows a plurality of images received by the receiver, FIG. 7 shows that parallax occurs in the overlapping area when the controller performs the first geometrical correction on the plurality of images, and FIG. 8 shows that the controller An example of a result of performing a first-order geometry correction and a second-order geometry correction on the control points of is shown.

6 to 8, the receiver 130 may receive a plurality of images 61, 63, 65, and 67 captured by a plurality of cameras.

The plurality of cameras may photograph objects with an overlapping angle of view, and may generate a plurality of images 61, 63, 65, and 67 having an overlapping area with adjacent images.

If the plurality of images 61, 63, 65, and 67 are simply stitched (or joined), they may be the same as the image 70. Since the image origins of the plurality of cameras cannot be the same, a ghost effect such as the parallax region 73 may occur. Parallax may occur when a distant object and a near object exist simultaneously in the overlapping area.

In order to minimize parallax, the controller 150 may perform a panorama coordinate transformation (or change) by performing a first-order geometry correction and a second-order geometry correction on the overlapped area.

Specifically, the controller 150 defines a plurality of control points within at least one overlapping area of the plurality of images 61, 63, 65, and 67, and applies homography to the plurality of control points, thereby correcting the primary geometry. You can do it. The controller 150 may define a control point based on a preset division value.

The controller 150 may convert coordinates of a plurality of control points on which the first geometric correction has been performed using the depth information. The controller 150 includes homography information, information for stitching (or bonding) operation, distance and direction of matched feature points in the overlapping area, distance and direction of control points in the overlapping area, alignment information of at least one object in the overlapping area, The depth information and the parallax due to the depth information may be estimated using at least one of FSIM information of the overlapping area and HFI_SSIM information of the overlapping area.

In addition, the controller 150 may generate a panoramic image 80 by defining a plurality of patches based on a plurality of control points and performing secondary geometry correction for mapping the plurality of patches. In this case, the controller 150 may map a plurality of patches using a cost function. The cost function can be one of a correlation function, a mean squared error (MSE) function, a fast structure similarity (FSSIM) function, and a maximum signal-to-noise ratio (PSNR) function.

The controller 150 may transmit the panoramic image 80 to the display 170.

9 to 13 show another example of a result of stitching (or bonding) a plurality of images by the parallax minimizing stitching apparatus.

FIG. 9 shows an example of a result of the controller performing the first geometric correction on a plurality of images, and FIG. 10 shows that parallax occurs in the overlapped region when the controller performs the first geometric correction on a plurality of images. , FIG. 11 is a diagram for explaining an operation in which a controller defines a plurality of control points, FIG. 12 is a diagram for explaining an operation in which the controller performs secondary geometry correction, and FIG. 13 is a Another example of the result of performing the first-order geometry correction and the second-order geometry correction on the control points of is shown.

9 to 13, the controller 150 may perform a first-order geometric correction on a plurality of images received by the receiver 130. The controller 150 may stitch (or bond) a plurality of images as shown in FIG. 9. Despite the primary geometry correction of the controller 150, a parallax region 91 may exist in an overlapping region of a plurality of images. The parallax region 91 may be as shown in FIG. 10.

In order to minimize the parallax of the parallax region 91, the controller 150 may define a plurality of control points in an overlapping region of a plurality of images as shown in FIG. 11. The controller 150 may define a control point based on a preset division value.

The controller 150 may convert coordinates of a plurality of control points using depth information. The controller 150 includes homography information, information for stitching (or bonding) operation, distance and direction of matched feature points in the overlapping area, distance and direction of control points in the overlapping area, alignment information of at least one object in the overlapping area, The depth information and the parallax due to the depth information may be estimated using at least one of FSIM information of the overlapping area and HFI_SSIM information of the overlapping area.

The controller 150 may generate a panoramic image by defining a plurality of patches based on a plurality of control points and performing secondary geometry correction for mapping the plurality of patches. In this case, the controller 150 may map a plurality of patches using a cost function. The cost function may be one of a correlation function, a mean squared error (MSE) function, a fast structure similarity (FSSIM) function, and a maximum signal-to-noise ratio (PSNR) function. The result of converting the coordinates of the plurality of control points by the controller 150 using the depth information is as shown in FIG. 12, and thus, as shown in FIG. 13, the parallax may be minimized.

The controller 150 may transmit a panoramic image in which parallax is minimized to the display 170.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments are, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, such as one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (14)

Defining a plurality of control points within an overlapping region of the first image and the second image received from the plurality of cameras;
Performing a first-order geometry correction by applying homography to the plurality of control points;
Converting coordinates of a plurality of control points on which the first-order geometry correction has been performed using depth information;
Defining a plurality of patches based on the plurality of control points; And
Performing secondary geometry correction by mapping the plurality of patches
Including,
Converting the coordinates,
Determining whether the plurality of control points correspond to a short distance or a far distance based on the depth information; And
Adjusting the distance between the plurality of control points based on the determination result
Containing, parallax minimization stitching method.
The method of claim 1,
The step of performing the second-order geometry correction,
Mapping the plurality of patches using a cost function
Parallax minimization stitching method comprising a.
The method of claim 2,
The cost function is a correlation function, a mean squared error (MSE) function, a fast structure similarity (FSSIM) function, and a peak signal-to-noise ratio. (PSNR)) One of the functions, the parallax minimization stitching method.
delete The method of claim 1,
Converting the coordinates,
Distance and direction of matched feature points among a plurality of feature points included in the overlapping area, distance and direction of a control point of A included in the overlapping area, and alignment information of the at least one object , Calculating the depth information using at least one of a feature similarity index (FSIM) of the overlapping region, and a high frequency information_structure similarity index (HFI_SSIM) of the overlapping region
Parallax minimization stitching method comprising a.
The method of claim 1,
The adjusting step,
Lengthening a distance between control points corresponding to a short distance; And
Step of shortening the distance between control points corresponding to a remote distance
Parallax minimization stitching method comprising a.
The method of claim 1,
Updating coordinates of the plurality of control points in a look-up table (LUT) based on the first geometry correction or the second geometry correction.
Parallax minimization stitching method further comprising.
A receiver for receiving a first image and a second image from a plurality of cameras; And
Define a plurality of control points within an overlapping region of the first image and the second image, and perform a first-order geometry correction by applying homography to the plurality of control points. , Using depth information to convert the coordinates of a plurality of control points on which the primary geometric correction has been performed, define a plurality of patches based on the plurality of control points, and the plurality of patches A controller that performs secondary geometry correction by mapping them
Including,
The controller,
A parallax minimization stitching apparatus that determines whether the plurality of control points correspond to a short distance or a far distance based on the depth information, and adjusts a distance between the plurality of control points based on a determination result.
The method of claim 8,
The controller,
Parallax minimization stitching apparatus for mapping the plurality of patches using a cost function.
The method of claim 9,
The cost function is a correlation function, a mean squared error (MSE) function, a fast structure similarity (FSSIM) function, and a peak signal-to-noise ratio. (PSNR)) Stitching device for minimizing parallax, one of the functions.
delete The method of claim 8,
The controller,
Distance and direction of matched feature points among a plurality of feature points included in the overlapping area, distance and direction of control points included in the overlapping area, and alignment information of the at least one object , A parallax minimization stitching apparatus for calculating the depth information using at least one of a feature similarity index (FSIM) of the overlapping region and a high frequency information_structure similarity index (HFI_SSIM) of the overlapping region.
The method of claim 8,
The controller,
To increase the distance between control points corresponding to a short distance (lengthen), reduce the distance between control points corresponding to a distance
Stitching device to minimize parallax.
The method of claim 8,
The controller,
A parallax minimization stitching apparatus for updating coordinates of the plurality of control points in a look-up table (LUT) based on the first geometry correction or the second geometry correction.

Images