KR102466998B1 - Method and apparatus for image fusion - Google Patents

Abstract

An image fusion method and apparatus are disclosed. The image fusion method includes: detecting first feature points of an object appearing in the first image frame from a first image frame; transforming the first image frame based on the detected first feature points and predefined reference points; Detecting second feature points of an object appearing in the second image frame from 2 image frames, transforming the second image frame based on the detected second feature points and predefined reference points, and deforming the first image frame and acquiring a combined image by combining the image frame and the deformed second image frame.

Description

Image fusion method and apparatus {METHOD AND APPARATUS FOR IMAGE FUSION}

The description below relates to image processing techniques.

When an image is captured in a low-light environment, it may be difficult to clearly recognize an object in the captured image. As a method for solving this problem, there are a method of taking pictures by increasing the exposure time of a camera and a method of performing image processing such as brightness control on an image taken by a camera. Both methods can make the brightness of the image brighter, but in the former case, motion blur may occur in the image due to the movement of the object, and in the latter case, there is a limit to clarifying the object.

An image fusion method according to an embodiment includes detecting first feature points of an object appearing in a first image frame from a first image frame; transforming the first image frame based on the detected first feature points and predefined reference points; detecting second feature points of an object appearing in the second image frame from a second image frame; transforming the second image frame based on the detected second feature points and the reference points; and obtaining a combined image by combining the transformed first image frame and the transformed second image frame.

The transforming of the first image frame may include transforming the first image frame so that the first feature points come to the positions of the reference points, and the transforming of the second image frame may include transforming the second feature points to the positions of the reference points. The second image frame may be transformed to come into position.

The transforming of the first image frame may include: detecting a first object area in the first image frame based on the detected first feature points; and performing image warping on the first object area based on first feature points included in the detected first object area and the reference points.

The transforming of the second image frame may include: detecting a second object area in the second image frame based on the detected second feature points; and performing image warping on the second object area based on second feature points included in the detected second object area and the reference points.

The acquiring of the combined image may include combining the first object region where the image warping is performed and the second object region where the image warping is performed.

The obtaining of the combined image may include generating the combined image by combining a first pixel value of the transformed first image frame and a second pixel value of the transformed second image frame. have.

An image fusion method according to an exemplary embodiment may include measuring image quality of the first image frame or the transformed first image frame; and determining to detect the first feature points in the first image frame or obtaining the combined image based on the modified first image frame when the measured image quality satisfies a preset condition. A determining step may be further included.

An image fusion method according to an embodiment includes detecting third feature points of an object appearing in a third image frame from a third image frame; transforming the third image frame based on the detected third feature points and the reference points; and combining the obtained combined image and the modified third image frame.

An image fusion device including a processor according to an embodiment, wherein the processor detects first feature points of an object appearing in the first image frame from a first image frame, and determines the detected first feature points and a predefined first feature point. The first image frame is transformed based on reference points, and second feature points of an object appearing in the second image frame are detected from the second image frame, and based on the detected second feature points and the reference points, the second feature points are detected. A combined image may be generated by transforming the second image frame and combining the transformed first image frame and the transformed second image frame.

1 is a diagram for explaining overall operation of an image merging apparatus according to an exemplary embodiment.
2 is a flowchart illustrating an operation of an image fusion method according to an exemplary embodiment.
3 and 4 are flowcharts illustrating examples of an image fusion method according to an exemplary embodiment.
5 is a flowchart illustrating an example in which an image fusion method according to an exemplary embodiment is applied to face authentication.
6 and 7 are diagrams for explaining examples of performing image fusion based on a plurality of image frames according to an exemplary embodiment.
8 is a diagram illustrating a configuration of a video merging device according to an exemplary embodiment.
9 is a diagram illustrating a configuration of a computing device according to an exemplary embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be modified and implemented in various forms. Therefore, the embodiments are not limited to the specific disclosed form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Terms such as first or second may be used to describe various components, but such terms should only be interpreted for the purpose of distinguishing one component from another. In addition, it should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Expressions in the singular should generally be interpreted to mean "one or more" unless otherwise specified. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

In addition, unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 is a diagram for explaining overall operation of an image merging apparatus according to an exemplary embodiment.

Referring to FIG. 1 , an image fusing device 100 is a device that fuses two or more image frames 110 using a processor to generate combined images 122 , 124 , 126 , and 128 . The image fusing apparatus 100, for example, generates a combined image by fusing a current image frame and n (n is a natural number) previous image frames temporally adjacent to the current image frame. For example, assuming that the current image frame is the image frame 116 and the image fusing device 100 generates a combined image based on the three image frames, the image fusing device 100 generates the image frames 112 , 114, 116) are fused to generate a combined image. If the image fusing device 100 generates a combined image based on two image frames, the image fusing device 100 generates the current image frame, the image frame 116, and the previous image frame of the image frame 116. The image frames 114 are fused to create a combined image.

The image frames 110 may include, for example, image frames 112, 114, 116, and 118 sequentially photographed by a camera in time. In this specification, the term 'image frame' may be replaced with 'input video' or 'video'. In the case of an image frame captured in a dark environment with low luminance, it may be difficult to clearly identify an object in the corresponding image frame. In an application area where object identification is important, such as face verification, it is important to clearly identify a face region in a corresponding image frame even in a video frame captured in a dark environment. This is because clear identification of the face area has a great influence on the accuracy of face recognition. As a general solution for overcoming the disadvantages of image frames captured in a low-light environment, there is a method of acquiring image frames by increasing the exposure time of a camera. However, if the exposure time increases, the probability of motion blur appearing in an image frame due to the movement of the camera and/or object increases, which again makes it difficult to identify the object.

In an embodiment, the image merging apparatus 100 may generate a combined image in which brightness is improved and objects are more easily identified by combining image frames captured in a low-light environment. At this time, the image convergence device 100 does not simply combine the image frames, but transforms the image frames based on feature points of each image frame and combines the transformed frames to minimize the occurrence of motion blur. can For example, the image convergence apparatus 100 extracts feature points representing the shape characteristics of an object from image frames, transforms the image frames so that the locations of the extracted feature points come to the location of a reference point, and then combines the transformed image frames. By doing so, a combined image with improved brightness and minimized motion blur can be created. The video convergence device 100 may be used in a user authentication system using biometric information, a financial transaction authentication system, an access control and monitoring system, an object recognition system, and the like.

Hereinafter, the operation of the image merging device 100 will be described in more detail with reference to the drawings.

2 is a flowchart illustrating an operation of an image fusion method according to an exemplary embodiment.

The operations shown in FIG. 2 may be performed sequentially as shown, or some operations may not be performed or the order of operations may be changed without departing from the scope and spirit of the described embodiments. Also, the operations shown in FIG. 2 may be performed in parallel or concurrently. For example, steps 210 and 220 and steps 230 and 240 may be performed in parallel.

In operation 210, the image merging apparatus detects first feature points of an object appearing in the first image frame from the first image frame. Feature points to be extracted may vary according to the type of object. For example, when the object is a face, facial landmarks, such as both end points of the eyes, the center of the eyes, the end points of the nose, and both end points of the mouth, may be extracted from the first image frame. When the object is a thing, a feature point representing a shape feature of the object may be extracted from the first image frame. For example, when the object is a license plate of a vehicle, four corners of the rectangular license plate may be extracted as feature points.

In step 220, the image merging apparatus transforms the first image frame based on the first feature points detected in step 210 and predefined reference points. Here, the positions of the reference points may have previously fixed positions according to the type of object, regardless of positions of the first feature points extracted from the first image frame. For example, when the object is a face, positions of the reference points may be reference positions whose positions are fixed in the face, such as eyes, nose, and mouth.

The image merging apparatus may deform the first image frame so that the first feature points come to the positions of the reference points by using image warping or the like. For example, the image merging apparatus calculates an affine transform matrix indicating a correspondence between the first feature points and the reference points based on the position values of the first feature points and the position values of the reference points, and the calculated affine transform The first image frame may be transformed by applying the matrix to the position value of each pixel of the first image frame. As another example, the process of transforming the first image frame may include detecting a first object region (eg, face region) in the first image frame based on the first feature points, and detecting a first object region included in the detected first object region. A process of performing image warping on the first object area so that the feature points come to the positions of the reference points may be included.

In step 230, the image merging device extracts second feature points (eg, facial landmarks) of objects appearing in the second image frame from the second image frame in the same or similar manner as in step 210. detect

In step 240, the image merging device transforms the second image frame based on the second feature points and reference points in the same or similar manner as in step 220. For example, the image merging apparatus may transform the second image frame so that the second feature points come to the locations of the reference points by using an affine transformation matrix. As another example, the process of transforming the second image frame may include detecting a second object area in the second image frame based on the detected second feature points, and determining that the second feature points included in the detected second object area are reference points. A process of performing image warping on the second object area to come into position may be included. Here, the locations of the reference points are the same as those of the reference points used in step 220 .

In step 250, the image merging apparatus obtains a combined image by combining the transformed first image frame and the transformed second image frame. This process may include, for example, a process of combining a first object region of a first image frame on which image warping is performed and a second object region of a second image frame on which image warping is performed. The image fusing device may generate a combined image by combining a first pixel value of the transformed first image frame and a second pixel value of the transformed second image frame. The combination between the first pixel value and the second pixel value is, for example, the sum, average, weight between the first pixel value and the second pixel value at corresponding positions in the transformed first image frame and the transformed second image frame. It may be a weighted sum or a weighted average, but the scope of the embodiment should not be construed as being limited thereto.

According to another embodiment, image quality of image frames may be considered for selection of image frames to be used for image fusion. The video merging device measures the video quality of the input first video frame or the first video frame transformed in step 220, and when the measured video quality satisfies a preset condition, the first video frame in the first video frame. It may be determined to detect feature points or to obtain a combined image based on the first image frame transformed in step 220 . For example, the image convergence device measures the brightness of the first image frame, and when the measured brightness is smaller than a preset threshold value, determines to detect first feature points in the first image frame or in step 220 It may be determined that a combined image is acquired based on the modified first image frame. When the brightness of the first image frame is greater than or equal to the threshold value, the image merging device may consider that the corresponding first image frame is not captured in a low-light environment and may not use the first image frame to obtain a combined image. Similarly, the image fusion device measures the image quality (eg, brightness, etc.) of the input second image frame or the second image frame transformed in step 240, and the measured image quality satisfies a preset condition. In this case, it may be determined to detect the second feature points in the second image frame or to obtain a combined image based on the transformed second image frame in operation 240 . For example, when the brightness of the second image frame is greater than or equal to a threshold value, the image merging device may not use the corresponding second image frame to obtain a combined image.

In step 260, the image merging device may transform the combined image. Depending on the embodiment, the process of step 260 may be selectively performed. The image merging apparatus may transform the combined image based on a correspondence between reference points and second feature points of a second image frame, which is a recently input image frame, for example. Assuming that the affine transformation matrix is determined based on the locations of the second feature points and the locations of the reference points, and the second image frame is transformed by the corresponding affine transformation matrix in step 240, the image fusion device performs the inverse of the affine transformation matrix. By applying an inverse transform matrix to the combined image, the shape of the object displayed in the combined image may be restored to its original shape. Afterwards, the deformed joint image can be used for the purpose. For example, in face authentication, features for face authentication may be extracted from a deformed combined image.

According to an embodiment, when the third image frame is input to the image merging device, the image merging device converts the third image frame to the third image frame in the same manner as described above with respect to the first image frame and the second image frame. Third feature points of the displayed object may be detected, and the third image frame may be modified based on the detected third feature points and reference points. Assuming that the combined image is generated based on three image frames, the image merging device may acquire a new combined image by combining the combined image obtained in step 250 with the transformed third image frame. have. Combination of the combined image and the modified third image frame in step 250 may be performed by summing, averaging, weighted sum, or weighted average of pixel values at positions corresponding to each other in the combined image and the third image frame.

As described above, the image convergence device transforms the image frames so that the feature points of the image frames come to the fixed positions of the reference points, and then fuses the deformed image frames to provide a combined image in which motion blur and noise are reduced and features of the object are better displayed. can do. When such a combined image is used in an image-based authentication system in which it is important to accurately extract features of an object, recognition accuracy can be improved.

3 is a flowchart illustrating an example of an image fusion method according to an exemplary embodiment.

Referring to FIG. 3 , in step 310, the image convergence device receives an n (n is a natural number) th image frame. In step 320, the video convergence device measures the video quality of the received video frame. For example, the image convergence device may measure the degree of brightness or blur of an image frame. The video quality of a video frame is used to determine whether that video frame needs to be used to create a combined video.

In step 330, the image fusion device determines whether the measured image quality satisfies a preset condition. For example, the video merging device may determine whether brightness of an image frame is less than a threshold value. When the image quality satisfies the preset condition, step 340 is performed. When the image quality does not satisfy the preset condition, step 390 is performed to select the next image frame. Through this, it is possible to prevent image frames photographed in a bright environment from being used in a combined image, and it may be determined whether image frames capable of image fusion are continuously received. This is because image fusion does not need to be performed on image frames that are photographed in a bright environment and have sufficient brightness to define the characteristics of an object. According to an embodiment, when a case in which the video quality does not satisfy a preset condition occurs, the video fusion process so far is terminated, and the next video frame is designated as the first video frame of the video fusion process, so that the video fusion process begins. can be restarted.

In step 340, the image merging apparatus detects the object region and feature points of the target object in the image frame. In step 350, the image merging device generates a deformed image frame by performing image conversion (eg, image warping) on the image frame so that the positions of the feature points detected in the image frame come to the fixed positions of the corresponding reference points. . In an embodiment, the image fusion apparatus extracts a region to be subjected to image transformation from an image frame based on detected feature points, and a transformation matrix for defining a conversion relationship between positions of feature points included in the extracted region and positions of reference points. After determining , a transformed image frame may be generated by applying the determined transformation matrix to an image frame or a corresponding extracted region.

In step 360, the image fusion device obtains a combined image through image fusion. The image merging apparatus may acquire a combined image by image fusion of a previously transformed and stored image frame and a currently transformed image frame. Here, image fusion includes a process of fusing pixel values through a sum, average, weighted sum, or weighted average of pixel values at corresponding positions among transformed image frames.

In operation 370, the image merging apparatus may perform specific image processing on the combined image according to the purpose for which the combined image is used. For example, the image merging apparatus may perform inverse image warping on the combined image in order to restore the shape of the object appearing in the combined image to its original shape. In this case, inverse image warping may be performed so that the feature points of the object appearing in the combined image come to the positions of corresponding feature points in the current image frame.

In step 380, the video merging device determines whether there is a next video frame. If there is a next image frame, in step 390, the image merging device selects the next image frame and performs the above steps 320 to 380 again for the selected next image frame.

4 is a flowchart illustrating another example of an image fusion method according to an exemplary embodiment.

4, steps 410, 420, 430, 440, 450, 460, 470, 480 and 490 Step 310, step 340, step 350, step 320, step 330, step 360, step 370, step 380, step 390 of FIG. Corresponds to, and the description described in relation to FIG. 3 may be applied to the description of each step.

However, the embodiment shown in FIG. 4, unlike the embodiment shown in FIG. 3, measures image quality (440) and determines whether the measured image quality satisfies a preset condition (450). There is a difference in that is performed after step 430 in which image conversion is performed. In addition, in step 420, an object region is detected in the image frame, and in step 430, image transformation such as image warping may be performed so that the positions of feature points come to the positions of corresponding reference points with respect to the detected object region. . Subsequently, in step 440, the image quality (eg, brightness, etc.) of the object region where the image conversion is performed is measured, and in step 450, it is determined whether the image quality satisfies a preset condition. By considering the image quality of the object region, which is the region of interest, image fusion can be applied even to an image frame that is generally bright but the object region may be dark, such as an image frame captured in a backlight environment.

5 is a flowchart illustrating an example in which an image fusion method according to an exemplary embodiment is applied to face authentication.

Face authentication is one of authentication methods for user authentication, and is an authentication method for determining whether a corresponding user is a valid user based on face information of a user who has attempted authentication. The image fusion method described in this specification may be used, for example, to provide an image more clearly representing facial features in an image captured in a low-illuminance environment in a face authentication process. In the case of image-based object recognition technology such as face recognition, how well the features of an object are represented in an image has a great influence on recognition accuracy. As the degree of blur or noise in an image increases, it tends to be difficult to accurately extract features of an object from the image. The image convergence apparatus makes it possible to generate an image with reduced blur and noise while improving the brightness of the image through the feature point-based image convergence process described above.

Referring to FIG. 5 , in step 505, the face authentication device receives an n (n is a natural number) th image frame of a query facial image. The query face image is an image subject to face authentication. In step 510, the face authentication apparatus detects a face area in the image frame and detects facial landmarks in the detected face area. For face region detection, a Haar-based cascade adaboost classifier or a Viola-Jones detector may be used, but the scope of the embodiment is not limited thereto. Depending on the embodiment, an image normalization process such as image cropping and image scaling may be performed on a face region in an image frame.

In step 515, the face authentication device performs image conversion on the face region, and accordingly, the shape of the face region is deformed. For example, the face authentication apparatus may perform image warping on the face region so that locations of landmarks detected in the face region come to positions of reference points in the face region that are predefined.

The face authentication device measures the image quality of the deformed face region in step 520, and determines whether the image quality satisfies a preset condition in step 525. For example, the face authentication device may measure the brightness of the deformed face region and determine whether the measured brightness is smaller than a threshold value.

When the image quality satisfies the preset condition, in step 530, the face authentication device acquires a combined image by combining the face region transformed in the previous image frame and the face region transformed in step 515 through image fusion. can do. This process includes a process of fusing pixel values at positions corresponding to each other in the deformed face regions. Depending on the embodiment, the face authentication apparatus may obtain a combined image by combining face regions transformed not only from two image frames but also from three or more image frames.

In step 535, the face authentication device may perform image processing on the combined image obtained in step 530 to make the combined image more suitable for face authentication. For example, the face authentication device restores the shape of the face to its original shape by performing inverse warping on the combined image, gamma correction to make the facial features appearing in the combined image clearer, Image processing to reduce color variation may be performed.

Afterwards, in step 550, the face authentication device extracts features from the combined image on which the image processing in step 535 is performed. The face authentication device may extract features (or feature vectors) using, for example, a neural network-based feature extractor. When it is determined in step 525 that the image quality of the face region included in the current image frame does not satisfy a preset condition (eg, when the brightness of the detected face region is greater than or equal to a threshold value), the face authentication device performs step ( Without performing steps 530 and 535, features are extracted from the current image frame in step 550. In this case, the image fusion process is initialized, and the image conversion result of the previously stored image frame and the combined image may be removed from a storage such as a memory.

In step 555, the face authentication device determines whether face authentication succeeds or not based on the features extracted in step 550. The face authentication device calculates a matching score based on the similarity between the extracted features and pre-registered features, and if the calculated matching score satisfies a preset condition (eg, if the matching score is greater than a threshold value), face authentication is performed. can be determined to be successful. The matching score may be determined based on a similarity or distance between the feature vector extracted in step 550 and the previously registered feature vector.

In step 560, the face authentication device determines whether face authentication has succeeded. When face authentication is not successful (failed), in step 540, the face authentication device determines whether a next image frame exists. If the next image frame does not exist, the result of face authentication is determined to be authentication failure. If there is a next image frame, in step 545, the face authentication device selects the next image frame and repeats the above process for the next image frame.

Through the above process, even if face authentication is performed based on a face image captured in a low-light environment, it is possible to more accurately perform face authentication even in a low-light environment by performing an image fusion process based on feature points.

6 is a diagram for explaining an example of an image fusion method according to an exemplary embodiment.

Referring to FIG. 6 , as an example of an image fusion method, an example of performing image fusion on a first image frame 612 and a second image frame 614 in which a vehicle license plate is displayed is shown. According to an embodiment, the first image frame 612 and the second image frame 614 may be temporally contiguous image frames or temporally contiguous image frames that are not immediately contiguous. For example, the first image frame 612 and the second image frame 614 are the n−1 (n is a natural number) th image frame and the n th image frame, respectively, or the n−m (m is a natural number greater than or equal to 2) th image frame. and an nth image frame.

When the first image frame 612 is input, the brightness of the first image frame 612 is measured. When it is determined that the first image frame 612 was captured in a dark environment because the measured brightness is smaller than the threshold value, the image fusion device detects feature points 622 in the first image frame 612 . For example, four corners of a license plate may be detected as feature points 622 . An area of the vehicle license plate may be detected in the first image frame based on the detected feature points 622 . Thereafter, the image merging apparatus may generate a modified first image frame 632 by performing image warping so that the positions of the feature points 622 come to the positions of predefined reference points.

The same process as above is performed for the second image frame 614 . When the second image frame 614 is input, the brightness of the second image frame 614 is measured. When the measured brightness is smaller than the threshold value, the image convergence device detects feature points 624 in the second image frame 614 . An area of the vehicle license plate may be detected in the second image frame 614 based on the detected feature points 624 . Thereafter, the image merging apparatus may generate a deformed second image frame 634 by performing image warping so that the positions of the feature points 624 come to the positions of predefined reference points.

Depending on embodiments, measurement of the brightness of an image frame may be performed after image warping is performed. In this case, the image fusion process based on the corresponding image frame may be continuously performed only when the brightness measured for the image frame is smaller than a preset threshold value. If the measured brightness is greater than or equal to the threshold value, the image fusion process may be terminated or initialized.

The image fusing device may generate a combined image 640 by combining the transformed first image frame 632 and the transformed second image frame 634 . For example, the image fusing device may generate a combined image 640 by fusing pixel values of the modified first image frame 632 and pixel values of the modified second image frame 634 . Thereafter, the image convergence device may restore the shape of the license plate shown in the combined image 640 to the original shape shown in the image frame to generate a combined image 650 in which the shape is restored. For example, the image merging apparatus may restore the shape of the license plate by applying an inverse transformation matrix of the transformation matrix used in the above image warping to the combined image 640 .

7 is a diagram for explaining another example of an image fusion method according to an exemplary embodiment.

Referring to FIG. 7 , as another example of the image fusion method, an example of performing image fusion on a first image frame 712 and a second image frame 714 in which a face region appears is illustrated. According to an embodiment, the first image frame 712 and the second image frame 714 may be temporally contiguous image frames or temporally contiguous image frames that are not immediately contiguous. For example, the first image frame 712 and the second image frame 714 are the n−1 (n is a natural number) th image frame and the n th image frame, respectively, or the n−m (m is a natural number greater than or equal to 2) th image frame. and an nth image frame.

When the first image frame 712 is input, the brightness of the first image frame 712 is measured. When the measured brightness is smaller than the threshold value, the image merging device detects facial landmarks 722 as feature points in the first image frame 712 . For example, feature points of major facial parts such as eyebrows, eyes, nose, and mouth may be detected. A face region 732 may be detected in the first image frame 712 based on the detected landmarks 722 . The detected face region 732 may be extracted as a face image 742 corresponding to the face region 732 through image cropping. Thereafter, the image merging device generates a deformed face image 762 by performing image warping on the face image 742 so that the feature points appearing in the face image 742 come to the positions of reference points 750 whose positions are predefined. can For example, image warping is performed so that the position of feature points of the eyebrow, nose, and mouth appearing in the face image 742 come to the position of the reference point 750 corresponding to the eyebrow, nose, and mouth, respectively.

The same process as above is performed for the second image frame 714 . When the second image frame 714 is input, the brightness of the second image frame 714 is measured. When the measured brightness is smaller than the threshold value, the image merging device detects facial landmarks 724 in the second image frame 714 . A face region 734 may be detected in the second image frame 714 based on the detected landmarks 724 . The detected face region 734 may be extracted as a face image 744 corresponding to the face region 734 through image cropping. Thereafter, the image merging device generates a deformed face image 764 by performing image warping on the face image 744 so that the feature points appearing in the face image 744 come to the positions of reference points 750 whose positions are predefined. can

Depending on embodiments, measurement of the brightness of an image frame may be performed after image warping is performed. In this case, the image fusion process based on the corresponding image frame may be continuously performed only when the brightness measured for the image frame is smaller than a preset threshold value. If the measured brightness is equal to or greater than the threshold value, the image fusion process may be terminated or initialized.

The image fusion device may generate a combined image 770 by combining the deformed face image 762 and the deformed face image 764 . For example, the image merging device generates a combined image 770 by summing, averaging, weighted sum, or weighted averaging the pixel values of the deformed face image 762 and the pixel values of the deformed face image 764. can Thereafter, the image merging device may restore the shape of the face shown in the combined image 770 to the original shape displayed in the image frame to generate a combined image 775 in which the shape is restored. The combined image 775 reconstructed in this way may be applied to the second image frame 714 . Through this process, a result image in which brightness is improved in the face area and motion blur and noise are reduced may be obtained. Therefore, when face authentication is performed using the corresponding resultant image, facial features are more accurately extracted, and the accuracy of face authentication can be improved.

In the embodiments of FIGS. 6 and 7 , it has been described that image fusion is performed based on two image frames, but the scope of the embodiment is not limited thereto, and image fusion is performed on three or more image frames. It is also possible.

8 is a diagram illustrating a configuration of a video merging device according to an exemplary embodiment.

Referring to FIG. 8 , the image convergence device 800 receives a series of image frames. The image merging apparatus 800 may detect feature points of an object in image frames and perform image fusion between image frames based on the detected feature points. In this case, the image merging apparatus 800 may determine whether to perform image fusion in consideration of image quality of image frames such as brightness. The image merging device 800 may perform one or more operations described or illustrated in this specification in relation to the image merging method, and may provide a result of face authentication to the user.

The image fusion device 800 may include one or more processors 810 and a memory 820 . Processor 810 may execute instructions for executing one or more operations described with reference to FIGS. 1-4, 6 and 7 . For example, the processor 810 detects first feature points of an object appearing in the first image frame from the first image frame, and transforms the first image frame based on the detected first feature points and predefined reference points. can do. For example, the processor 810 may transform the first image frame by performing image warping so that the first feature points come to the locations of the reference points. As another example, the processor 810 may detect a first object region in the first image frame based on the detected first feature points, and may detect a first object region based on the first feature points and reference points included in the detected first object region. 1 Image warping may be performed on the object area.

Also, the processor 810 may detect second feature points of an object appearing in the second image frame from the second image frame, and transform the second image frame based on the detected second feature points and reference points. For example, the processor 810 may transform the second image frame by performing image warping so that the second feature points come to the locations of the reference points. As another example, the processor 810 may detect a second object area in the second image frame based on the detected second feature points, and may detect a second object area based on the second feature points and reference points included in the detected second object area. Image warping may be performed on the 2-object area.

The processor 810 may generate a combined image by combining the deformed first image frame and the deformed second image frame as described above. The processor 810 combines (eg, sum, average, weighted sum, or weighted average) the pixel values of each pixel of the transformed first image frame with the pixel values of each pixel of the transformed second image frame. You can create video. The processor 810 may transform the combined image based on the correspondence between the reference points and the second feature points of the second image frame. This includes a process of restoring the shape of the object deformed in the image fusion process to its original shape based on the positions of feature points shown in the second image frame.

The memory 820 is connected to the processor 810 and may store instructions executable by the processor 810 and data to be calculated by the processor 810 or data processed by the processor 810 . Memory 820 may include non-transitory computer readable media, such as high-speed random access memory and/or non-volatile computer readable storage media.

9 is a diagram illustrating a configuration of a computing device according to an exemplary embodiment.

Referring to FIG. 9 , a computing device 900 is a device that performs an application domain using the image fusion method described above. In one embodiment, the computing device 900 may correspond to the face authentication device described in FIG. 5 . The computing device 120 may be, for example, an image processing device, a smart phone, a wearable device, a tablet computer, a netbook, a laptop, a desktop, a personal digital assistant (PDA), a set-top box, a home appliance, a biometric door lock, and security device or a vehicle starting device. The computing device 900 may include the functions of the image fusion device 800 of FIG. 8 as they are.

Referring to FIG. 9 , a computing device 900 may include a processor 910, a storage device 920, a camera 930, an input device 940, an output device 950, and a network interface 960. . The processor 910 , the storage device 920 , the camera 930 , the input device 940 , the output device 950 and the network interface 960 may communicate with each other through the communication bus 970 .

Processor 910 executes functions and instructions for execution within computing device 900 . For example, the processor 910 may process instructions stored in the storage device 940 . The processor 910 may perform one or more operations described above through FIGS. 1 to 8 .

The storage device 920 stores information or data required for execution of the processor. The storage device 920 may include a computer readable storage medium or a computer readable storage device. The storage device 920 may store instructions to be executed by the processor 910 and may store related information while software or applications are executed by the computing device 900 .

The camera 930 may capture an image composed of a plurality of image frames. For example, the camera 930 may acquire a query face image composed of a plurality of image frames by capturing a user attempting face authentication.

The input device 940 may receive an input from a user through tactile, video, audio, or touch input. For example, input device 940 may include a keyboard, mouse, touch screen, microphone, or any other device capable of detecting input from a user and passing the detected input to computing device 900 .

The output device 950 may provide an output of the computing device 900 to a user through a visual, auditory, or tactile channel. The output device 950 may include, for example, a display, a touch screen, a speaker, a vibration generating device, or any other device capable of providing an output to a user. The network interface 960 may communicate with an external device through a wired or wireless network.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Claims (20)

In the image fusion method,
detecting first feature points of an object appearing in the first image frame from a first image frame;
transforming the first image frame based on the detected first feature points and predefined reference points;
detecting second feature points of an object appearing in the second image frame from a second image frame;
transforming the second image frame based on the detected second feature points and the reference points; and
Obtaining a combined image by combining the transformed first image frame and the transformed second image frame
including,
The image fusion method,
measuring an image quality of the first image frame or the transformed first image frame; and
When the measured image quality satisfies a preset condition, it is determined to detect the first feature points in the first image frame or to obtain the combined image based on the modified first image frame. step to do
more containing
Video fusion method.
According to claim 1,
Transforming the first image frame,
Transforming the first image frame so that the first feature points come to the positions of the reference points;
Transforming the second image frame,
The image fusion method of transforming the second image frame so that the second feature points come to the locations of the reference points.
In the image fusion method,
detecting first feature points of an object appearing in the first image frame from a first image frame;
transforming the first image frame based on the detected first feature points and predefined reference points;
detecting second feature points of an object appearing in the second image frame from a second image frame;
transforming the second image frame based on the detected second feature points and the reference points; and
Obtaining a combined image by combining the transformed first image frame and the transformed second image frame
including,
Transforming the first image frame,
detecting a first object region in the first image frame based on the detected first feature points; and
Performing image warping on the first object area based on first feature points included in the detected first object area and the reference points;
Transforming the second image frame,
detecting a second object area in the second image frame based on the detected second feature points; and
performing image warping on the second object area based on second feature points included in the detected second object area and the reference points;
Image fusion method comprising a.
According to claim 3,
Acquiring the combined image,
Combining the first object region where the image warping is performed and the second object region where the image warping is performed
Image fusion method comprising a.
According to claim 1,
Acquiring the combined image,
generating the combined image by combining a first pixel value of the transformed first image frame and a second pixel value of the transformed second image frame;
Image fusion method comprising a.
According to claim 5,
The combination between the first pixel value and the second pixel value is
Among the sum, average, weighted sum, or weighted average of first pixel values and second pixel values at corresponding positions in the transformed first image frame and the transformed second image frame Any one, video fusion method.
delete According to claim 1,
The step of measuring the image quality,
Measuring the brightness of the first image frame;
The determining step is
When the brightness of the first image frame is less than a preset threshold, determining to detect the first feature points in the first image frame or acquiring the combined image based on the modified first image frame To determine, the video fusion method.
In the image fusion method,
detecting first feature points of an object appearing in the first image frame from a first image frame;
transforming the first image frame based on the detected first feature points and predefined reference points;
detecting second feature points of an object appearing in the second image frame from a second image frame;
transforming the second image frame based on the detected second feature points and the reference points; and
Obtaining a combined image by combining the transformed first image frame and the transformed second image frame
including,
The image fusion method,
measuring an image quality of the second image frame or the modified second image frame; and
When the measured image quality satisfies a preset condition, it is determined to detect the second feature points in the second image frame or to obtain the combined image based on the modified second image frame. step to do
Image fusion method further comprising a.
According to claim 1,
The image fusion method,
detecting, from a third image frame, third feature points of an object appearing in the third image frame;
transforming the third image frame based on the detected third feature points and the reference points; and
Combining the obtained combined image and the modified third image frame
Image fusion method further comprising a.
According to claim 1,
The step of detecting the first feature points,
Detect facial landmarks in the first image frame;
The step of detecting the second feature points,
The image fusion method of detecting facial landmarks in the second image frame.
According to claim 1,
The image fusion method,
Transforming the combined image based on the correspondence between the second feature points and the reference points
Image fusion method further comprising a.
According to claim 12,
An image fusion method in which a feature for face authentication is extracted from the deformed combined image.
A computer-readable storage medium storing instructions for executing the method of any one of claims 1 to 6 and 8 to 13.
An image fusion device including a processor,
the processor,
Detecting first feature points of an object appearing in the first image frame from a first image frame;
Modifying the first image frame based on the detected first feature points and predefined reference points;
Detecting second feature points of an object appearing in the second image frame from a second image frame;
Modifying the second image frame based on the detected second feature points and the reference points;
generating a combined image by combining the deformed first image frame and the deformed second image frame;
the processor,
The image quality of the first image frame or the modified first image frame is measured, and when the measured image quality satisfies a preset condition, it is determined to detect the first feature points in the first image frame. or determining to obtain the combined image based on the transformed first image frame;
video convergence device.
According to claim 15,
the processor,
generating the transformed first image frame by transforming the first image frame so that the first feature points come to positions of the reference points;
The video merging device of claim 1 , wherein the deformed second image frame is generated by transforming the second image frame so that the second feature points come to positions of the reference points.
An image fusion device including a processor,
the processor,
Detecting first feature points of an object appearing in the first image frame from a first image frame;
Modifying the first image frame based on the detected first feature points and predefined reference points;
Detecting second feature points of an object appearing in the second image frame from a second image frame;
Modifying the second image frame based on the detected second feature points and the reference points;
generating a combined image by combining the deformed first image frame and the deformed second image frame;
the processor,
A first object region is detected in the first image frame based on the detected first feature points, and the first object region is detected based on the first feature points included in the detected first object region and the reference points. Perform image warping on
A second object area is detected in the second image frame based on the detected second feature points, and the second object area is detected based on the second feature points included in the detected second object area and the reference points. An image fusion device that performs image warping on
An image fusion device including a processor,
the processor,
Detecting first feature points of an object appearing in the first image frame from a first image frame;
Modifying the first image frame based on the detected first feature points and predefined reference points;
Detecting second feature points of an object appearing in the second image frame from a second image frame;
Modifying the second image frame based on the detected second feature points and the reference points;
generating a combined image by combining the deformed first image frame and the deformed second image frame;
the processor,
The image merging device of claim 1 , wherein the combined image is generated by combining a first pixel value of the transformed first image frame and a second pixel value of the transformed second image frame.
According to claim 18,
The combination between the first pixel value and the second pixel value is
The image merging device of claim 1 , wherein the deformed first image frame and the deformed second image frame are any one of a sum, an average, a weighted sum, or a weighted average between first pixel values and second pixel values at corresponding positions.
According to claim 15,
the processor,
An image fusion device that transforms the combined image based on the correspondence between the second feature points and the reference points

Images