KR20210120185A - Feature point matching method and apparatus for 360 degree erp format stitching image generation - Google Patents

Abstract

Provided are a feature point matching method and device for generating a 360-degree omnidirectional ERP format stitching image. The feature point matching method includes the steps of: detecting stitching points between a plurality of input images; applying the predetermined transformation to the detected stitching points; and detecting at least one feature point from the stitching point to which the transformation is applied, and performing feature point matching on the detected feature point. The predetermined transformation may project the stitching point center to an image position of a predetermined projection format.

Description

FEATURE POINT MATCHING METHOD AND APPARATUS FOR 360 DEGREE ERP FORMAT STITCHING IMAGE GENERATION

The present invention relates to a feature point matching method and apparatus for generating a 360-degree omnidirectional ERP format stitching image. Specifically, it relates to a method and apparatus for matching feature points while correcting distortion according to the projection method of the ERP format in the process of stitching a 360-degree omnidirectional image to the ERP format.

The ERP (Equi-Rectangular Projection) format is one of the representative methods for generating a 360-degree omnidirectional stitching image. 1 is a diagram illustrating an ERP format image stitched in a sphere projection method according to an embodiment. The ERP format is an equirectangular projection method that projects a 360-degree image in consideration of a virtual sphere shape, and multiple fisheye lenses or general lenses can be used to view the entire 360-degree space. In particular, if a fisheye lens is used, a space of 180 degrees or more can be captured with one image, and a horizontal and vertical 360-degree stitched image can be generated using at least two images. However, the method using the fisheye lens requires a complex reconstruction algorithm because image distortion occurs.

2 is a diagram illustrating a process of generating a stitched image according to an exemplary embodiment. A general process of generating a stitched image is to first find a part (or overlapping region) connected between images, using an algorithm such as SIFT (Scale-Invariant Feature Transform) and SURF (Speed-Up Robust Features). Feature points such as edges and corners of 201 may be detected. Also, a descriptor may be generated based on the local feature of the feature point, and feature points having similar descriptors may be matched ( 202 ) using a matching technique such as FLANN (Fast Library for Approximate Nearest Neighbors). Also, a homography may be generated using the obtained matching points. Here, the homography may mean a matrix indicating a relationship between images on the same plane. The stitching technique can express images taken by different cameras in one coordinate space using homography. In addition, after determining the connection relationship between the images, all images may be warped and aligned (203) to the reference image plane using the estimated homography. In addition, a stitching image with reduced ghosting or image mismatch can be obtained by naturally joining two images having different textures using the blending technique 204 .

An object of the present invention is to provide a feature point matching method and apparatus for generating a 360-degree omnidirectional ERP format stitching image.

In addition, an object of the present invention is to provide a method and apparatus for matching feature points while correcting distortion according to the projection method of the ERP format in the process of stitching a 360-degree omnidirectional image to the ERP format.

According to the present invention, detecting a stitching point between a plurality of input images; applying a predetermined transform to the detected stitching points; and detecting at least one feature point from the stitching point to which the transformation is applied, and performing feature point matching on the detected feature point, wherein the predetermined transformation includes a predetermined projection of the stitching point center A feature point matching method for generating a 360-degree omnidirectional ERP format stitching image that is projected to the image position of the format may be provided.

The projection format may be characterized as an ERP (Equi-Rectangular Projection) format.

The image location of the predetermined projection format may be a location corresponding to 0 degrees latitude and 0 degrees longitude on a sphere coordinate system.

The performing of the key point matching may include correcting the input image to match the coordinates of the key point.

The method may further include performing blending on the corrected input image.

In addition, according to the present invention, in the feature point matching apparatus for generating a 360-degree omnidirectional ERP format stitching image, the apparatus detects a stitching point between a plurality of input images, and applies a predetermined transformation to the detected stitching point and detecting at least one feature point from the stitching point to which the transformation is applied, and performing feature point matching on the detected feature point, and the predetermined transformation is an image of a predetermined projection format based on the stitching point center A feature point matching device for projecting to a location may be provided.

The projection format may be characterized as an ERP (Equi-Rectangular Projection) format.

The image location of the predetermined projection format may be a location corresponding to 0 degrees latitude and 0 degrees longitude on a sphere coordinate system.

The device may correct the input image to match the coordinates of the feature point.

The device may perform blending on the corrected input image.

According to the present invention, a feature point matching method and apparatus for generating a 360-degree omnidirectional ERP format stitching image may be provided.

In addition, according to the present invention, a method and apparatus for matching feature points while correcting distortion according to the projection method of the ERP format in the process of stitching a 360-degree omnidirectional image to the ERP format may be provided.

Effects obtainable in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which the present disclosure belongs from the description below. will be.

1 is a diagram illustrating an ERP format image stitched in a sphere projection method according to an embodiment.
2 is a diagram illustrating a process of generating a stitched image according to an exemplary embodiment.
3 is a diagram illustrating a process of warping a fisheye lens image to an ERP format image according to a reference of a spherical coordinate system according to an embodiment.
4 is a flowchart of a feature point matching method for generating a 360-degree omnidirectional ERP format stitching image according to an embodiment.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects. Shapes and sizes of elements in the drawings may be exaggerated for clearer description. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] Reference is made to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein with respect to one embodiment may be embodied in other embodiments without departing from the spirit and scope of the invention. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of exemplary embodiments, if properly described, is limited only by the appended claims, along with all scope equivalents to those claimed.

In the present invention, terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component of the present invention is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in between. It should be understood that there may be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

Components shown in the embodiment of the present invention are shown independently to represent different characteristic functions, and it does not mean that each component is composed of separate hardware or a single software component. That is, each component is listed as each component for convenience of description, and at least two components of each component are combined to form one component, or one component can be divided into a plurality of components to perform a function, and each Integrated embodiments and separate embodiments of components are also included in the scope of the present invention without departing from the essence of the present invention.

The terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present invention, terms such as "comprises" or "have" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof. That is, the description of "including" a specific configuration in the present invention does not exclude configurations other than the corresponding configuration, and it means that additional configurations may be included in the practice of the present invention or the scope of the technical spirit of the present invention.

Some components of the present invention are not essential components for performing essential functions in the present invention, but may be optional components for merely improving performance. The present invention can be implemented by including only essential components to implement the essence of the present invention, except for components used for performance improvement, and a structure including only essential components excluding optional components used for performance improvement Also included in the scope of the present invention.

EMBODIMENT OF THE INVENTION Hereinafter, embodiment of this invention is described concretely with reference to drawings. In describing the embodiments of the present specification, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present specification, the detailed description is omitted, and the same reference numerals are used for the same components in the drawings. and repeated descriptions of the same components are omitted.

An overlapping portion may exist in two images captured using a fisheye lens of 180 degrees or more, and the overlapping portion is referred to as a stitching point. In order to combine the images, it is necessary to find a feature point at the stitching point and match the feature point. Distortion problems that occur mainly in the 360-degree omnidirectional image stitching process can be divided into two categories. The first is a ghost phenomenon, which may occur when a plurality of images with large parallaxes are projected onto a single virtual sphere because the camera center points do not coincide. The ghost phenomenon may be solved by obtaining a local homography for each divided area of an object having a large parallax or applying a warping method using depth information. The second is distortion caused by not accurately matching the key points, and may be generated in the process of matching the key points after correcting the image obtained using the fisheye lens and converting it to the ERP image format.

According to the present disclosure, when the camera center point does not match and distortion due to parallax occurring in a near object obtained from a plurality of cameras does not occur, distortion occurring in the process of matching the feature points of the images converted to the RP format A matching point warping method that can reduce .

3 is a diagram illustrating a process of warping a fisheye lens image to an ERP format image according to a reference of a spherical coordinate system according to an embodiment.

In the process of generating a 360-degree omnidirectional stitching image, if a fisheye lens with a wide angle of view is used, after correcting lens distortion, each image is projected based on one virtual spherical coordinate, so that it can be warped in ERP format. In this case, as in the example of FIG. 3 , image distortion may occur from the center of the screen toward the outside (ie, as it moves away from the equator). That is, as the extracted feature points increase as the distance from the central point increases, the search range may be widened, and the probability that the matching point may not be accurately found may increase.

According to the present disclosure, a method of transforming a spherical coordinate system based on each feature point in order to better match the feature point may be provided.

For example, although the distortion is relatively small in the central part of the ERP, a lot of unnatural distortion may occur at the edge of the end of the viewing angle, and the distortion may be more severe in the stitching process. According to the present disclosure, a feature point matching method in consideration of the above phenomenon may be provided.

4 is a flowchart of a feature point matching method for generating a 360-degree omnidirectional ERP format stitching image according to an embodiment.

_{Referring to FIG. 4 , warping transformation T k} such that the center of the 'stitching point' in the k-th overlapping region is projected to a predetermined ERP format image position may be performed on the input image. For example, the predetermined ERP format image location may be a location corresponding to 0 degrees latitude and 0 degrees longitude in spherical coordinates. By using the warping transformation, the stitching point center may be converted to be the ERP center position relatively, and matching may be performed by finding a feature point. In this case, in the process of matching the key points, the input image may be corrected to match the coordinates of the key points. Although the correction method is not specifically described in the present disclosure, when the matching points are very close, a correction method of increasing or decreasing a part of an image in a narrow range may be used.

For example, pre-compensation may store the coordinates p 'of the same feature point aligned after feature point coordinates p _i and matching and calibration of each image transformation i. Further, in order to minimize the loss of information in the original image generated in the conversion coordinates in the original input image to p _i and p 'using the inverse transformation T ^-1 of the _{k ^{T k T k -1 (p i}} ) and T _k ^-1 (p') can be calculated and stored. A by the feature point coordinates and the matching and a correction feature point coordinates of each input image obtained using the process of using a warped transform T _o, it converted to a virtual spherical coordinate system of the set for the stitch image creation T _o (T _k ^{- 1} (p _i )) and T _o (T _k ^-1 (p')) can be found. Warping transformation T _o may mean ERP transformation based on a predetermined spherical coordinate system. The image may be corrected so that the feature points of the ERP-converted images _{fit the corrected feature point T o} (T _k ^-1 (p')). A final stitched image may be generated by performing blending on the corrected image.

Meanwhile, in the example of FIG. 4 , it is assumed that the input image and the camera parameter for which the distortion of the fisheye lens is corrected are obtained without a large error. Using the image in which the distortion of the fisheye lens is corrected and the camera extrinsic and intrinsic parameters, ERP conversion based on the virtual spherical coordinate system and ERP conversion centered on the stitching point may be performed.

According to the present disclosure, it is possible to better match the feature points by warping the stitching points to the central portion of the ERP image having relatively little distortion. In addition, the inverse transformation of the warping transformation is applied to the feature point coordinates and the corrected coordinates in the original input image coordinate system, the coordinates are converted into the coordinates on the ERP format converted coordinate system into the final virtual spherical coordinate system, and the relationship between the feature points is maintained Stitching can be performed while

In the above-described embodiments, the methods are described on the basis of a flowchart as a series of steps or units, but the present invention is not limited to the order of steps, and some steps may occur in a different order or concurrently with other steps as described above. can In addition, those of ordinary skill in the art will recognize that the steps shown in the flowchart are not exclusive, other steps may be included, or that one or more steps in the flowchart may be deleted without affecting the scope of the present invention. You will understand.

The above-described embodiments include examples of various aspects. It is not possible to describe every possible combination for representing the various aspects, but one of ordinary skill in the art will recognize that other combinations are possible. Accordingly, it is intended that the present invention cover all other substitutions, modifications and variations falling within the scope of the following claims.

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM, a DVD, and a magneto-optical medium such as a floppy disk. media), and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

In the above, the present invention has been described with specific matters such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , those of ordinary skill in the art to which the present invention pertains can make various modifications and variations from these descriptions.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and not only the claims described below, but also all modifications equivalently or equivalently to the claims described below belong to the scope of the spirit of the present invention. will do it

Claims (10)

detecting stitching points between a plurality of input images;
applying a predetermined transform to the detected stitching points; and
detecting at least one feature point from the stitching point to which the transformation is applied, and performing feature point matching on the detected feature point;
The predetermined transformation is a feature point matching method for generating a 360-degree omnidirectional ERP format stitching image by projecting the stitching point center to an image position of a predetermined projection format. According to claim 1,
The projection format is a feature point matching method, characterized in that the ERP (Equi-Rectangular Projection) format. 3. The method of claim 2,
The image position of the predetermined projection format is a feature point matching method, characterized in that the position corresponding to 0 degrees latitude and 0 degrees longitude on a sphere coordinate system. According to claim 1,
The step of performing the feature point matching comprises:
and correcting the input image to match the coordinates of the keypoint. 5. The method of claim 4,
and performing blending on the corrected input image. In the feature point matching device for generating a 360-degree omnidirectional ERP format stitching image,
The apparatus detects a stitching point between a plurality of input images, applies a predetermined transform to the detected stitching point, detects at least one feature point at the stitched point to which the transform is applied, and the detection performing feature point matching on the obtained feature points,
wherein the predetermined transformation projects the center of the stitching point to an image position of a predetermined projection format. 7. The method of claim 6,
The projection format is a feature point matching device, characterized in that the ERP (Equi-Rectangular Projection) format. 8. The method of claim 7,
The image position of the predetermined projection format is a feature point matching apparatus, characterized in that the position corresponding to 0 degrees latitude and 0 degrees longitude on a sphere coordinate system. 7. The method of claim 6,
The device is
A feature point matching device that corrects the input image to match the coordinates of the feature point. 10. The method of claim 9,
A feature point matching apparatus for performing blending on the corrected input image.

Images