KR20220136797A - Apparatus and method for image stitching based on priority object - Google Patents

Abstract

The present invention relates to a method for a computing apparatus operated by at least one processor to match images. The method includes the following steps of: detecting feature points from each input image, and matching identical feature points in the input images to extract a common area; detecting objects located in the common area, and calculating an object-based weight matrix proportional to priorities set for the detected objects; calculating boundary line weights based on boundary line position information derived from input images of consecutive previous views, and generating a boundary line generation matrix for the common area based on the boundary line weights and the object-based weight matrix; and matching the input images based on the boundary line generation matrix. Therefore, the present invention is capable of improving the accuracy of image matching.

Description

Priority object-based image matching device and method {APPARATUS AND METHOD FOR IMAGE STITCHING BASED ON PRIORITY OBJECT}

The present invention relates to image registration technology based on deep learning.

Image stitching technology is a technology for synthesizing images taken from multiple angles through multiple cameras into one image, and is mainly used to create panoramic images and 360-degree images.

Panoramic images and 360-degree images generated through image stitching technology have a wider viewing angle than images shot through a conventional camera, which has the advantage of providing a sense of presence to users.

However, in the case of a large parallax between the photographing cameras in the image registration process, parallax distortion such as a phenomenon in which the image junction part spreads, a phenomenon in which a part of an object disappears, or a phenomenon in which objects are overlapped and overlapped may occur. This parallax distortion has a limitation in that it interferes with content immersion by making users feel a sense of heterogeneity.

In order to prevent such parallax distortion, various studies are being conducted. Among them, the seam optimization-based image stitching method (Seam Optimization) sets a weight to the object detection value based on the boundary line for the overlapping area of the input images. It is possible to prevent parallax distortion in objects from occurring.

However, in this boundary line optimization-based image stitching method (Seam Optimization), limitations occur in generating the boundary line depending on the initial position of the boundary line, the performance of the object detector, or the arrangement of objects.

In addition, when synthesizing a video through the image stitching method based on boundary line optimization (Seam Optimization), since each boundary line is generated in every video frame, the position of the boundary line generated in the previous video frame and the boundary line generated in the current video frame is If it is significantly different, a distortion phenomenon in which an object moves rapidly in the synthesized video may occur.

The problem to be solved by the present invention is to detect one or more objects included in input images based on deep learning, generate a boundary line generation matrix based on individual weights according to the priority of the detected object, and based on the boundary line generation matrix to provide an image matching apparatus and method for matching input images.

According to an embodiment of the present invention, there is provided a method of image matching by a computing device operated by at least one processor, wherein each feature point is detected in input images, and a common area is extracted by matching the same feature points in the input images. step, detecting objects located in a common area, calculating an object-based weight matrix proportional to the priority set for the detected objects, based on the location information of the boundary line derived from the input image of consecutive previous views calculating a boundary line weight, generating a boundary line generation matrix in a common area based on boundary line weights and an object-based weight matrix, and matching input images based on the boundary line generation matrix.

According to an embodiment of the present invention, there is provided an image matching apparatus, comprising: a memory including instructions; and a processor executing the instructions to perform image registration between input images, wherein the processor is configured to perform image matching between input images based on feature points detected in the input images. Extracting the common area, calculating the boundary line weight based on the object-based weight matrix proportional to the priority of objects detected in the common area, and the boundary line position information derived from the continuous input image of the previous view, and calculating the boundary line weight and the object The input images are matched through a boundary line generation matrix in the common area generated based on a base weight matrix.

According to the present invention, it is possible to minimize the parallax distortion caused by the parallax between the cameras photographed in the process of matching images captured by multiple cameras.

According to the present invention, since the horizontal weight is set based on the boundary line detected in the previous frame, it is possible to solve the problem of abrupt movement of an object in the synthesized video.

According to the present invention, by using deep learning-based object detection technology during image registration, priority is set according to the importance of the detected object and the boundary line generation matrix is configured to protect important objects from parallax distortion, thereby improving the accuracy of image registration and the user content immersion can be improved.

According to the present invention, it can be used as a universal base technology of image stitching technology using deep learning, and since the deep learning-based object detection technology is used for image registration, a monitoring system and procedure using the existing deep learning-based object detection integration is possible.

Effects obtainable in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which the present disclosure belongs from the description below. will be.

1 is a block diagram showing an image matching apparatus according to an embodiment of the present invention.
2 is a configuration diagram illustrating a priority thing setter according to an embodiment of the present invention.
3 is a flowchart illustrating an image registration method according to an embodiment of the present invention.
4 is an exemplary diagram illustrating a process of extracting a common region from input images according to an embodiment of the present invention.
5 is an exemplary diagram to which a visual perception energy function according to an embodiment of the present invention is applied.
6 is an exemplary diagram illustrating a segment of an object detected according to an embodiment of the present invention.
7 is an exemplary diagram for explaining a priority object-based weight matrix in an embodiment of the present invention.
8 is an exemplary diagram for explaining a configuration for allocating dynamic importance according to an embodiment of the present invention.
9 is an exemplary diagram for explaining a process of applying a horizontal weight according to an embodiment of the present invention.
10 is an exemplary diagram comparing the image registration result according to an embodiment of the present invention and the image registration result of the existing method.
11 is a hardware configuration diagram of an image matching apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. In addition, terms such as “…unit”, “…group”, and “module” described in the specification mean a unit that processes at least one function or operation, which is implemented by hardware or software or a combination of hardware and software. can be

The devices described in the present invention are composed of hardware including at least one processor, a memory device, a communication device, and the like, and a program executed in combination with the hardware is stored in a designated place. The hardware has the configuration and capability to implement the method of the present invention. The program includes instructions for implementing the method of operation of the present invention described with reference to the drawings, and is combined with hardware such as a processor and a memory device to execute the present invention.

As used herein, "transmission or provision" may include not only direct transmission or provision, but also transmission or provision indirectly through another device or using a detour path.

In this specification, expressions described in the singular may be construed as singular or plural unless an explicit expression such as “a” or “single” is used.

1 is a block diagram showing an image matching apparatus according to an embodiment of the present invention.

As shown in FIG. 1 , the image matching apparatus 100 includes a common region extractor 110 , a priority object setter 120 , a boundary line generation matrix generator 130 , and an image matcher 140 .

The common region extractor 110 , the priority object setter 120 , the boundary line generation matrix generator 130 , and the image matcher 140 may be operated by at least one processor. In other words, the common region extractor 110, the priority object setter 120, the boundary line generation matrix generator 130, and the image matcher 140 may be implemented in one computing device or distributed in separate computing devices. can When distributed in a separate computing device, the common area extractor 110 , the priority object setter 120 , the boundary line generation matrix generator 130 , and the image matcher 140 may communicate with each other through a communication interface. . The computing device may be any device capable of executing a software program written to carry out the present invention, and may be, for example, a server, a laptop computer, or the like.

The image matching apparatus 100 may be directly linked with one or more camera modules or may be connected to a database of a camera module to collect or receive captured images. In this case, the captured image represents images including regions overlapping each other.

Hereinafter, captured or inputted images for image matching are named and described as input images.

The common region extractor 110 extracts one or more feature points from the input image, and extracts the common region by matching the extracted feature points between the input images.

Here, the key point corresponds to a robust pixel that is detected as the same point even in rotation, brightness change, and movement transformation of the image, and the common area extractor 110 matches the same point among the respective key points detected in the two input images.

In addition, the common region extractor 110 sets a region capable of including all the matched feature points as an overlap region.

In other words, the common area extractor 110 may set the overlapping area of the maximum size that may include all the matched feature points as the common area.

At this time, the common region extractor 110 calculates a homography for the feature points of the common region, estimates a change relationship between the feature points at the same point, and performs warping to minimize the parallax between input images. can be done

In detail, the common region extractor 110 calculates a warping residual from the feature point matching result between the same points for homography. In addition, the common region extractor 110 may transform the input image based on the distortion residual and the feature point matching result.

For example, assuming that matching is performed between the first input image and the second input image, the slope of the common region set in the second input image may be converted based on the first input image.

The priority object setter 120 detects one or more objects in a common area of input images, and calculates an object-based weight matrix to which weights are applied based on the priority of the detected objects.

The priority object setter 120 inputs the energy value calculated through the visual perception energy function and the common area to the deep learning-based object detection model to obtain the object and background information, and the object detected in the common area and the object The type, location information of the object, and the segment having the same shape as the appearance of the object are acquired.

Mask R-CNN, YOLACT, etc. may be applied to the object detection model used here, and may be implemented with one or more deep learning algorithms.

In addition, the priority object setter 120 sets a priority for each object based on the type and importance of the detected object.

The priority object setter 120 sets the segment of the object itself as an additional weight so that a boundary line is not created in the interior area of the object, and the weight value applied to the interior location of the object is visually recognized to be larger than the weight value of the background. Use the maximum energy value of the energy function.

In addition, the priority object setter 120 generates a priority object based weight matrix (POWM) using the maximum energy value proportional to the priority of the object.

The boundary line generation matrix generator 130 calculates a horizontal weight (PSW, Previous Seam Weight) proportional to the horizontal distance based on the position information of the boundary line derived from the previous viewpoint input image, and combines it with the priority object-based weight matrix to input Creates a Seam Generation Matrix (SGM) of the image.

In addition, the image matcher 140 matches the input images based on the generated boundary line generation matrix.

As described above, the image matching apparatus 100 generates a priority object-based weight matrix proportional to the priority of the detected object using a visual cognitive energy function and a deep learning-based object detection algorithm so that the boundary line of the object object is set inside the object. is not limited to

Also, the image matching apparatus 100 may perform image registration robust to parallax distortion by performing registration between input images using a weighting matrix based on priority objects and a weight based on a boundary position of a previous input image.

Hereinafter, the priority thing setter 120 will be described in detail.

2 is a configuration diagram illustrating a priority thing setter according to an embodiment of the present invention.

As shown in FIG. 2 , the priority object setter 120 includes a boundary line detector 121 , an object detector 122 , an object tracker 123 , and a weight setter 124 , and in addition to the object detection model It may include a learner (not shown) for learning.

The boundary line detector 121 , the object detector 122 , the object tracker 123 , the weight setter 124 , and the learner may be operated by at least one processor. Although the learner is separately implemented for convenience of explanation, it can be implemented so that the learner is included in the priority thing setter 120 .

In addition, as shown in FIG. 2 , the object detection model that has been learned through the learner only needs to be linked with the object detector 122 , so it is not necessarily implemented together.

The boundary line detector 121 acquires object and background information by using the visual perception energy function in the common area extracted by the common area extractor 110 .

Here, the visual perception energy value calculated from the visual perception energy function is defined as the sum of the horizontal (X-axis) pixel change amount and the vertical (Y-axis) pixel change amount with respect to the input image.

Also, the boundary line detector 121 may calculate the maximum energy value of the visual perception energy function. The boundary line detector 121 may calculate a maximum energy value and transmit it to the weight setter 124 in order to set the weight value of the object to be larger than that of the background.

The object detector 122 detects an object in the common area of the input image by using an object detection algorithm that has been trained.

The object detector 122 inputs the visual recognition energy value together with the common area into the object detection algorithm so that the weight is not detected inside the object. Here, the object detection algorithm outputs the type of object detected by the deep learning-based algorithm, location information, and segment of the object shape.

In this case, the object detector 122 may store the obtained object information (type of object, location information, segment, etc.) in a separate database.

In addition, when the importance of each thing is set and stored in the thing importance DB in advance, the thing detector 122 may set the importance value of the corresponding thing in the segment of the thing detected based on the thing importance DB.

The object tracker 123 recognizes a moving object among objects by comparing the location information of the object in the input image of the continuous previous viewpoint with the location information of the same object in the input image of the current viewpoint.

In addition, the object tracker 123 recognizes a moving object in a continuous input image, and sets the weight so that the moving object is assigned a higher weight than the still object.

Accordingly, the object tracker 123 may calculate the dynamic importance by adding the total number of types of objects that can be detected by the object detector 122 based on the importance set for each object. In other words, the number of objects learned by the object detection algorithm is applied as an additional weight value to a moving object.

As such, the object tracker 123 may set the dynamic importance by comparing the positions of the objects from the second frame on the basis of the object information detected in the first frame.

The weight setter 124 determines the priority of objects detected in the common area based on the object information received from the object detector 122, the dynamic importance from the object tracker 123, and the contents of a separate object importance DB, and the set the weight

In other words, the weight setter 124 assigns a high importance value to an object to be preserved from parallax distortion during image matching.

Here, the importance is set according to the type of the object according to one or more setting criteria, such as the purpose of the input images and the speed of the moving object in the input image.

For example, the importance level may be set by criteria such as an object having a relatively high moving speed, an object having a large dissonance effect on parallax distortion, and a traffic-related object. Here, the speed means a relative speed estimated based on location information of an object detected in the image.

In addition, the priority indicates a variable whose integer value is differentially set from 0 according to the priority comparison order of objects detected in the input image.

In other words, the weight setter 124 compares the size of importance set for each object, and sets the priority according to the order in which the importance has a large value.

Also, the weight setter 124 may set the obtained object segment itself as an additional weight. This is to set the weight value applied to the internal position of the object to be always larger than the weight value of the background, and the maximum energy value of the visual perception energy function is collected for each object.

In addition, the weight setter 124 adds the maximum energy value proportional to the priority of the object to the existing visual perception energy function to generate a priority object-based weight matrix.

In order to learn the object detection algorithm, the learner generates learning data by matching an input value with an output value to be obtained. Then, the learning data is input to the object detection algorithm, and the learning is repeatedly performed so that the desired output value is output.

The learner may determine the accuracy of the output result of the object detection algorithm through a separate verification process, and when the accuracy greater than or equal to a threshold is obtained, the learning of the object detection algorithm may be completed.

Next, an image matching method strong against parallax distortion through the image matching apparatus 100 will be described in detail with reference to FIG. 3 .

3 is a flowchart illustrating an image registration method according to an embodiment of the present invention.

The image matching apparatus 100 detects each feature point in the input images and extracts a common area from the input images by matching the same feature point ( S110 ).

The image matching apparatus 100 may detect feature points for one or more input images, respectively, and if the same feature points are matched, a common area may be set to include all the matched feature points.

Next, the image matching apparatus 100 detects objects located in the common area and calculates an object-based weight matrix proportional to the priority set to the detected objects ( S120 ).

The image matching apparatus 100 calculates a visual perception energy function as shown in Equation 1 below to obtain object and background information in the common area to which the warping is applied.

Here, the visual perception energy value E(x,y) is equal to the sum of the pixel change amount on the horizontal axis (X-axis) and the pixel change amount on the vertical axis (Y-axis) with respect to the input image I.

And the image matching apparatus 100 detects the type of object, the position, and the object shape segment from the input image through the deep learning-based object detection algorithm that has been trained.

The image matching apparatus 100 sets respective importance and priority for objects located in a common area among objects detected in the input image. In this case, the image matching apparatus 100 may set based on importance information of objects that are set in advance based on the frequency, importance, and location information of the types detected in the continuous input image. In addition, the image matching apparatus 100 sets priorities in the order of importance.

For example, it is assumed that roadview content is generated by using an object detection algorithm that has been trained for 100 objects such as people, airplanes, trains, traffic lights, signs, birds, cats, and wallets. At this time, the importance of the object is a person (100), an airplane (99), a train (98), ... , a traffic light (89), a sign (88), ... , a bird (24), a cat (23), ... , the wallet (1), etc. can be set.

Such importance may be set by separately input by the administrator or by extracting the types of things highly related to the roadview content through a separate algorithm.

In addition, when a person, a train, a sign, a traffic light, and a cat are detected in the input image through a deep learning-based object detection algorithm in which the priority is set, the priority according to the importance is a person (4), a car (3), a sign (2), a traffic light (1), and a cat (0).

Meanwhile, the image matching apparatus 100 calculates the maximum energy value (E _max ) of the visual perception energy function through Equation (2).

In addition, as shown in Equation 3, the image matching apparatus 100 adds a maximum energy value proportional to the priority of an object to the existing visual perception energy function so that a weight value that can be distinguished between objects can be set. Create a weight matrix (POWM(x,y).

Here, Kp represents a priority value.

Next, the image matching apparatus 100 generates a boundary line generation matrix in the common area based on the boundary line weights set based on the previous input image and the object-based weight matrix ( S130 ).

The image matching apparatus 100 calculates the horizontal weight (PSW(x,y)) proportional to the horizontal distance based on the coordinates of the boundary line generated in the previous view frame by using Equation (4).

In other words, the square of the difference between the horizontal (X-axis) position of the previous boundary line and the horizontal (X-axis) position of the current boundary line may be set as a weight value.

In addition, the image matching apparatus 100 constructs a boundary line matrix S(x,y) by adding POWM and a horizontal weight (PSW) as in Equation 5 to generate a boundary line.

Equation 5 is an equation for generating a boundary line matrix through the addition operation of the POWM calculated for the overlapping region of the input image and the horizontal weight calculated through the boundary line of the previous frame.

And the image matching apparatus 100 generates a boundary line generation matrix (SGM(x,y)) using the following Equation (6).

For example, the boundary line generation matrix SGM is generated by adding the smallest value among the upper three matrix values in the Y-axis direction of the boundary line matrix S.

In other words, the image matching apparatus 100 generates the boundary line generation matrix SGM(x,y) by repeating the process of selecting and adding the lowest values from the top to the bottom.

Next, the image matching apparatus 100 matches the input images based on the boundary line generation matrix ( S140 ).

The image matching apparatus 100 selects the point of the lowest value and repeats the process of selecting the lowest value among the three matrix values located at the top from the point as in Equation 7 to the top, and the boundary line along the minimum path Create Seam(x,y-1).

Equation 7 shows a process of generating a boundary line based on the boundary line generation matrix.

In this way, the image matching apparatus 100 sets different weights according to the type of object by utilizing deep learning-based object detection based on seam optimization-based image stitching, and applies this to the boundary line generation matrix. This can overcome parallax distortion.

4 is an exemplary diagram illustrating a process of extracting a common region from input images according to an embodiment of the present invention.

FIG. 4(a) shows input images in which feature points are matched, and (b) shows a process of extracting a common region based on the input images.

As shown in FIG. 4 , the image matching apparatus 100 extracts feature points from frames #1, #2, and #3 in which the disparity is large, respectively, and then sets a common area through a matching process.

When the same feature points are connected between the #1 frame and the #2 frame, the image matching apparatus 100 may set common areas #1-1 and #2-1 including all the connected feature points. Also, when the same feature points are connected between frame #2 and frame #3, the image matching apparatus 100 may set common areas #2-2 and #3-1 including all the connected feature points.

In addition, the image matching apparatus 100 may transform an image by deriving a transformation relationship between the common regions #1-1 and #2-1.

5 is an exemplary diagram to which a visual perception energy function according to an embodiment of the present invention is applied.

Fig. 5 (a) is an input image, and (b) is an exemplary view showing an input image to which a visual perception energy function is applied.

The method of applying the visual perception energy function is a method of applying an energy value from an image to a boundary line generation matrix and generating a boundary line along a low-frequency region with a small energy value.

When the image matching apparatus 100 applies the visual perception energy function to the input image, the energy value obtained by adding the magnitude of the horizontal change and the magnitude of the vertical change appears as shown in FIG. 5(b), which is similar to the edge of the object in the image .

Referring to (b) of FIG. 5 , if a boundary line is first generated at a red point, the boundary line is generated along the inner region of the object. In this case, since the energy value is defined as the change rate of the pixel, parallax distortion occurs in which the weight is not properly reflected, as in the circled area at the lower right of FIG. 5B , in the object area having a pixel value similar to the background.

6 is an exemplary diagram illustrating a segment of an object detected according to an embodiment of the present invention.

As shown in FIG. 6 , the image matching apparatus 100 may obtain a segmented result of the detected object region by inputting the input images into a learned object detection algorithm (eg, Mask R-CNN). have.

In this case, the image matching apparatus 100 considers only the object corresponding to the common area before performing the warping, and in this case, the value of the visual cognitive energy function may be used as an input value together.

In addition, the image matching apparatus 100 may segment each object and acquire the type of object, location information, and the like.

7 is an exemplary diagram for explaining a priority object-based weight matrix in an embodiment of the present invention.

7 (a) shows an input image, and (b) is an exemplary diagram in which a priority object-based weight matrix is generated with respect to the input image.

As shown in FIG. 7 , the image matching apparatus 100 detects objects by inputting the input image of FIG. 7A to an object detection algorithm that has been trained.

In this case, the image matching apparatus 100 applies the importance and priority assigned to the object as shown in Table 1 below with respect to the detected object.

pole chair train person importance 21 61 98 100 Priority 0 One 2 3

By applying Equation 2 described above, the image matching apparatus 100 calculates an object-based weight matrix and displays it on the input image as shown in FIG. 8(b).

As shown in FIG. 8B , the closer to white, the higher the weight. Therefore, it can be seen that weights are set in the order of person, train, chair, and column.

8 is an exemplary diagram for explaining a configuration for allocating dynamic importance according to an embodiment of the present invention.

As shown in FIG. 8 , the image matching apparatus 100 identifies a moving object based on the importance set for each detected object, and assigns a dynamic importance to the moving object.

8(a) shows object position information in a frame of a previous view input image, and (b) shows object position information in a frame of a current view input image.

The image matching apparatus 100 may check the importance set for each detected object such as a person, an airplane, a train, a motorcycle, a bird, a cat, a traffic sign, or the like. In addition, it is possible to identify a moving object based on a change in object location information for the detected objects.

Accordingly, for a person, an airplane, a train, a car, and a motorcycle identified as moving objects, the dynamic importance is set high based on the number of detected objects.

9 is an exemplary diagram for explaining a process of applying a horizontal weight according to an embodiment of the present invention.

As shown in FIG. 9 , the image matching apparatus 100 derives a horizontal weight matrix using Equation 4 described above with respect to the weights generated based on the boundary line (previous view frame_SGM) generated in the previous view frame. do.

Based on the horizontal distance difference based on the X coordinate of the boundary line in the previous viewpoint frame and the X' coordinate of the boundary line in the current viewpoint frame, (X -X`) ² as a weight for the current viewpoint frame Priority object-based weight matrix (POWM) By adding to , a boundary line generation matrix (SGM) for the current view frame is derived.

Since the boundary line in the current view frame is generated at a position not far from the boundary line of the previous frame, a phenomenon in which an object moves rapidly in the synthesized image can be prevented.

10 is an exemplary diagram comparing the image registration result according to an embodiment of the present invention and the image registration result of the existing method.

10A shows input images photographed at different photographing times and photographing times. The input images have a remarkably small common area compared to the overall image size, and since the column is located in the center of the common area, the boundary line is difficult to generate while bypassing the object, the probability of parallax distortion is high.

Accordingly, (b) of FIG. 10 is an image matched using the existing technology, and (d) is an image matched by the method according to an embodiment of the present invention.

In detail, Fig. 10 (b) is a technique for applying a weight to a pixel value that differs from the average brightness by more than a certain size in the visual perception energy function. Since the pillar is similar to the average brightness of the common area, weight is set for the pillar As this is limited, it can be confirmed that parallax distortion occurs. As such, the method of applying weights based on the existing brightness has a limitation in that a pixel area similar to the average brightness cannot be considered.

On the other hand, referring to (c) of FIG. 10 , it can be seen that parallax distortion does not occur in the column because objects are detected through deep learning and individual weights are set according to the types of objects.

Meanwhile, FIG. 11 is a block diagram illustrating a hardware configuration of an image matching apparatus according to an exemplary embodiment.

Referring to FIG. 11 , the image matching apparatus 100 may be implemented as at least one computing device 200 , and may execute a computer program including instructions described to execute the operation of the present disclosure. .

The hardware of the computing device 200 may include at least one processor 210 , a memory 220 , a storage 230 , and a communication interface 240 , and may be connected through a bus. In addition, hardware such as an input device and an output device may be included. The computing device 200 may be loaded with various software including an operating system capable of driving a computer program.

The processor 210 is a device for controlling the operation of the computing device 200 , and may be various types of processors that process instructions included in a computer program, for example, a central processing unit (CPU), a micro processor (MPU) unit), microcontroller unit (MCU), graphic processing unit (GPU), or the like. Also, the processor 210 may perform an operation on a program for executing the method described above.

The memory 220 stores various data, commands and/or information. The memory 220 may load a corresponding computer program from the storage 230 so that the instructions described to execute the operations of the present disclosure are processed by the processor 210 . The memory 220 may be, for example, read only memory (ROM), random access memory (RAM), or the like. The storage 650 may store various data, computer programs, and the like required to execute the operations of the present disclosure. The storage 230 may non-temporarily store a computer program. The storage 230 may be implemented as a non-volatile memory. The processor 210 is connected to the network through the communication interface 240 .

The computer program includes instructions executed by the processor 210 , and is stored in a non-transitory computer readable storage medium, wherein the instructions are read by the processor 210 . Makes the action of initiation be executed. The computer program may be downloaded over a network or sold in product form. The object detection model may be implemented as a computer program executed by the processor 210 .

As described above, according to the embodiment, parallax distortion caused by parallax between cameras photographed in a process of matching images captured by multiple cameras may be minimized.

In addition, since the weight is set based on the boundary line detected in the previous frame, it is possible to solve the problem of rapidly moving objects in the synthesized video.

In addition, by using deep learning-based object detection technology during image registration, priority is set according to the importance of the detected object and the boundary line generation matrix is configured to protect important objects from parallax distortion, thereby improving the accuracy of image registration and user content immersion. can be improved

It can be used as a universal base technology for image stitching technology using deep learning, and since deep learning-based object detection technology is used for image registration, procedural integration with existing deep learning-based object detection is possible. .

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto. is within the scope of the right.

This study was supported by Korea Electric Power Corporation's 2018 Energy Hub University Cluster Project (Project No.: R18XA02)

Claims (15)

A method of image registration by a computing device operated by at least one processor, the method comprising:
Detecting each feature point in the input images and extracting a common area by matching the same feature points in the input images;
detecting objects located in the common area and calculating an object-based weight matrix proportional to the priority set to the detected objects;
calculating a boundary line weight based on the position information of the boundary line derived from the input image of a continuous previous view, and generating a boundary line generation matrix in the common area based on the boundary line weight and the object-based weight matrix, and
matching the input images based on the boundary line generation matrix;
An image registration method comprising a. In claim 1,
The step of extracting the common region comprises:
An image matching method that converts an input image by estimating a change relationship between feature points at the same point by detecting a point detected as the same point as a feature point even in rotation, brightness change, and movement transformation of an image, and extracting a common area based on the feature point . In claim 1,
Calculating the object-based weight matrix includes:
An image matching method for calculating a visual perception energy value in the common area and detecting objects through an object detection algorithm in which learning of the common area and the calculated visual perception energy value is completed. In claim 3,
Calculating the object-based weight matrix includes:
An image matching method for setting importance based on one or more criteria among a purpose of input images, frequency of appearance in the input image, and speed of a moving object in the input image for each object type obtained by the object detection algorithm. In claim 4,
Calculating the object-based weight matrix includes:
An image matching method for selecting moving objects by comparing the position information of the object in the continuous input image of the previous viewpoint with the position information of the object at the current viewpoint, and setting the dynamic importance based on the importance set for each moving object. In claim 5,
Calculating the object-based weight matrix includes:
Priority is set for each object based on the importance and dynamic importance of the object, the maximum energy value of the object is corrected in proportion to the priority of the object, and then an object-based weight matrix is generated by adding the visual perception energy value Image registration method. In claim 3,
The generating of the boundary line generation matrix comprises:
An image registration method that calculates a boundary line weight proportional to the horizontal distance based on the boundary line coordinates generated from the previous view input image. In claim 1,
The generating of the boundary line generation matrix comprises:
An image matching method for generating a boundary line generation matrix by summing minimum matrix values from one end to the other end based on the object-based weight matrix and boundary line weights. In claim 1,
The step of matching the input images comprises:
An image registration method of generating a boundary line along a smallest value among adjacent pixels at a specific point based on the boundary line generation matrix. An image matching device comprising:
memory containing instructions, and
A processor for performing image registration between input images by executing the instructions,
the processor
Extracting a common area based on the feature points detected in the input images,
Calculating boundary weights based on an object-based weight matrix proportional to the priority of objects detected in the common area and location information of boundary lines derived from an input image of a continuous previous view,
and matching the input images through a boundary line generation matrix in the common area generated based on the boundary line weights and the object-based weight matrix. 11. In claim 10,
the processor
An image matching apparatus for calculating a visual perception energy value in the common area and detecting objects through an object detection algorithm in which learning of the common area and the calculated visual perception energy value is completed. In claim 11,
the processor
For each object type obtained by the object detection algorithm, the importance is set based on one or more criteria among the purpose of the input images, the frequency of appearance in the input image, and the speed of the moving object in the input image,
An image matching apparatus for selecting moving objects by comparing the position information of the object in the continuous input image of the previous viewpoint with the position information of the object at the current viewpoint, and setting the dynamic importance based on the importance set for each moving object . In claim 12,
the processor
Priority is set for each object based on the importance and dynamic importance of the object, the maximum energy value of the object is corrected in proportion to the priority of the object, and then an object-based weight matrix is generated by adding the visual perception energy value , an image matching device. 11. In claim 10,
the processor
An image matching apparatus for calculating the boundary weight in proportion to a horizontal distance based on the coordinates of the boundary line generated from the previous view input image. 11. In claim 10,
the processor
Based on the object-based weight matrix and the boundary line weight, a boundary line generation matrix is generated by summing the minimum matrix values from one end to the other end, and based on the boundary line generation matrix, a boundary line is generated along the smallest value among adjacent pixels at a specific point. , an image matching device.

Images