KR102235127B1 - Adaptive fusion method of multimodal images and apparatus - Google Patents

Abstract

Disclosed are a method for adaptively fusing multi-modal images and a device therefor. The method for adaptively fusing multi-modal images comprises the steps of: acquiring a first image and a second image by individually photographing a target through a plurality of cameras; recognizing an object with respect to the first image and the second image; and generating a fused image by fusing the first image and the second image, by applying different image fusion rules according to the type of the recognized object. In the present invention, multi-modal images can be adaptively fused by applying different fusion rules depending on the type of the object of interest in the image.

Description

An adaptive fusion method of multimodal images and apparatus thereof TECHNICAL FIELD

The present invention relates to a method and apparatus for adaptive fusion of multimodal images.

Image fusion is a process of creating a single image including complementary image information by combining the advantages of different types of image sensors.

The conventional image fusion technology does not use information on the brightness distribution of the types of objects of interest existing in the image, but fused the image depending on the brightness value of the pixel. For this reason, the conventional image fusion technology has a limitation in providing sufficient visual information when used for object recognition in computer vision.

(01) Korean Patent Application Publication No. 10-2009-0103128 (2009.10.01)

The present invention is to provide a method and apparatus for adaptive fusion of multimodal images.

In addition, the present invention is to provide a method and apparatus for adaptive fusion of multi-modal images capable of adaptively fusion of multi-modal images by applying different fusion rules according to the type of an object of interest in an image.

Through this, the present invention is to provide a method and apparatus for adaptive fusion of multimodal images capable of providing sufficient visual information necessary for image processing and analysis when trying to solve a computer vision problem using a fused image. will be.

According to an aspect of the present invention, a method for adaptive fusion of multimodal images is provided.

According to an embodiment of the present invention, the method includes: obtaining a first image and a second image by photographing a target through a plurality of cameras; Recognizing an object for the first image and the second image; And generating a fusion image by fusing the first image and the second image by applying different image fusion rules according to the recognized object type.

The first image is a color image, and the second image is a thermal image.

Before the step of generating the fusion image, the first image and the second image are matched, but after setting any one of the first image and the second image as a reference image, the first image and the second image Another one of the images may be geometrically transformed and matched to match the reference image.

In the step of generating the fusion image, when a plurality of objects of different types are included, a different image fusion rule is applied to each object according to the object type, and the background is an image fusion rule different from the recognized object. The fusion image can be created by applying.

In the generating of the fusion image, different image fusion rules may be applied according to thermal characteristics of the recognized object.

According to another aspect of the present invention, an apparatus for adaptive fusion of multimodal images is provided.

According to an embodiment of the present invention, there is provided an image acquisition unit for obtaining a first image and a second image by photographing a target through a plurality of cameras; An object recognition unit for recognizing an object with respect to the first image and the second image; And an image fusion unit configured to generate a fusion image by fusing the first image and the second image by applying different image fusion rules according to the recognized object type.

The first image and the second image are matched, but after setting any one of the first image and the second image as a reference image, the other one of the first image and the second image is matched with the reference image It may further include an image matching unit for matching by geometric transformation as possible.

When a plurality of objects of different types are included, the image fusion unit applies different image fusion rules to each object according to the object type, and the background is fused by applying an image fusion rule different from the recognized object. You can create an image.

The image fusion unit may apply different image fusion rules according to thermal characteristics of the recognized object.

A plurality of cameras may be equipped with lenses having the same angle of view, and may be installed so that the screens covered when the target is photographed overlap.

By providing a method and apparatus for adaptive fusion of multimodal images according to an embodiment of the present invention, different fusion rules may be applied according to the type of an object of interest in the image to adaptively fusion of multi-modal images.

Through this, the present invention has the advantage of providing sufficient visual information necessary for image processing and analysis when trying to solve a computer vision problem using a fused image.

1 is a flow chart showing a multimodal image fusion method according to an embodiment of the present invention.
2 and 3 are diagrams illustrating images captured by different cameras according to an embodiment of the present invention.
4 and 5 are diagrams showing results of applying the same image fusion rule to a plurality of images.
6 and 7 are diagrams showing results of applying two different fusion rules according to an embodiment of the present invention.
8 is a block diagram showing the configuration of an image fusion device according to an embodiment of the present invention.

Singular expressions used in the present specification include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various elements or various steps described in the specification, and some of the elements or some steps It may not be included, or it should be interpreted that it may further include additional components or steps. In addition, terms such as "... unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flow chart showing a method for fusion of a multimodal image according to an embodiment of the present invention, and FIGS. 2 and 3 are diagrams illustrating images captured by different cameras according to an embodiment of the present invention, and FIG. 4 and 5 are diagrams showing results of applying the same image fusion rule to a plurality of images, and FIGS. 6 and 7 are views showing results of applying two different fusion rules according to an embodiment of the present invention to be.

In step 110, the image fusion device 100 captures a first image and a second image through a plurality of cameras. In this case, it is assumed that the camera photographs the same object.

For convenience, in an embodiment of the present invention, it is assumed that a first image is captured through a general camera and a second image is captured through a thermal imaging camera.

An image captured by a general camera is as illustrated in FIG. 2, and an image captured by a thermal imaging camera is as illustrated in FIG. 3.

As shown in FIGS. 2 and 3, a general RGB camera and a thermal imaging camera may generate a first image and a second image by photographing the same object (subject), respectively.

In an embodiment of the present invention, it is assumed that the RGB camera and the thermal imaging camera are respectively photographing the same object (subject) by mounting a lens having a similar angle of view.

It is assumed that the distance between the two cameras is very small, and the positions of the two cameras are set so that the screens covered by the two cameras almost overlap (overlap).

In operation 115, the image fusion device 100 analyzes the first image and recognizes an object of interest in the first image. That is, the image fusion device 100 may recognize objects of interest in the color image by applying an object recognition algorithm to the color image (first image) captured by the RGB camera. For example, the object recognition algorithm can be applied to all known object recognition algorithms such as SURF and SIFT.

In step 120, the image fusion device 100 matches the object corresponding to the object recognized in the first image with the second image.

For example, when a vehicle is recognized as an object in FIG. 2, the image fusion device 100 matches the recognized vehicle with a second image as well.

In step 120, the image fusion device 100 matches the first image and the second image.

For example, the image fusion device 100 may perform geometric transformation on each of the first image and the second image, and then match the two images (a first image and a second image). At this time, the image fusion device 100 may perform image matching by setting one of the first image and the second image as a reference image and then geometrically converting the other image to match the pixels so as to match the reference image. have.

For example, when the first image is set as the reference image, pixels may be matched to match the first image by geometrically transforming the second image. The first image and the second image captured by different cameras are not completely geometrically matched even though the angles of view are matched as much as possible. Accordingly, in an embodiment of the present invention, a matching process of matching pixels by simple geometric transformation of the other image based on the reference image may be essentially performed.

In step 125, the image fusion device 100 fuses the first image and the second image by applying different fusion rules to objects recognized according to different object types.

For example, as shown in FIGS. 2 and 3, it is assumed that objects recognized in the first image and the second image are a person and a vehicle, respectively. In this case, the image fusion device 100 may apply a first image fusion rule to a human object and a second image fusion rule to a vehicle object. Also, the image fusion device 100 may apply a third image fusion rule to the background.

In this case, the image fusion device 100 may apply different fusion rules in consideration of the thermal characteristics of the recognized object.

For example, FIG. 4 is a diagram showing a result of applying an image fusion rule based on a weighted average in a spatial domain to a first image and a second image. 4 is a result of applying an image fusion rule according to a weighted average in a spatial domain by setting the weight of the first image to 0.7 and applying the weight of 0.3 to the second image.

5 is a result of applying an image fusion rule according to a weighted average in a spatial domain by applying a weight differently from that of FIG. 4. For example, FIG. 5 is a result of applying an image fusion rule according to a weighted average in a spatial domain by applying a weight of 0.3 to a first image and a weight of 0.7 to a second image.

Referring to FIGS. 4 and 5, it can be seen that although detailed visual characteristics of a vehicle are well expressed in FIG. 4, visual information on a person (passenger) is not well expressed. On the other hand, in FIG. 5, the visual information on the pedestrian is relatively well expressed, but it can be seen that the visual information is lost due to too much thermal characteristics of the vehicle.

Therefore, in an embodiment of the present invention, when the thermal characteristic of an object recognized with reference to the second image is relatively high (for example, a car), the weight of the first image is set higher for the vehicle, and the second image is By applying a low weight for the vehicle, visual information about the vehicle can be well expressed. In addition, in the case of passersby (people), since the column characteristics are relatively low compared to cars, in the case of people, the weight of the first image is set low and the weight of the second image is set high so that the visual information of passersby is well expressed can do.

As described above, the image fusion device 100 according to an embodiment of the present invention applies a fusion rule differently from each other when fusion of the first image and the second image to an object recognized in consideration of the thermal characteristics of the object. Can be created.

6 and 7 are results of using two different fusion rules of a spatial domain and a DWT transform domain while applying the same weight. It can be seen that visual information on a vehicle is well expressed in FIG. 6, and visual information on a pedestrian and a background is more faithfully expressed in FIG. 7.

Accordingly, in an embodiment of the present invention, by fusing images by applying different fusion rules according to the recognized object type, there is an advantage in that the images can be fused by preserving visual characteristics suitable for object characteristics.

8 is a block diagram showing the configuration of an image fusion device according to an embodiment of the present invention.

Referring to FIG. 8, an image fusion device 100 according to an embodiment of the present invention includes an image acquisition unit 810, an object recognition unit 815, an image matching unit 820, an image fusion unit 825, and a memory. 830 and a processor 835.

The image acquisition unit 810 acquires images photographed by different types of cameras, respectively.

The object recognition unit 815 recognizes an object by analyzing the acquired image. As already described above, after recognizing an object in any one of a plurality of images, it can be recognized by applying the same to other images.

The image matching unit 820 is a means for matching a plurality of images. The image matching unit 820 may set any one of the plurality of images as a reference image and match the remaining images to match the reference image. That is, the image matching unit 820 may perform simple geometric transformation on the remaining images based on the reference image and may match the remaining images to match the reference image.

The image fusion unit 825 fuses a plurality of images by applying different fusion rules according to the recognized object type.

Since this has already been described above, a redundant description will be omitted.

The memory 830 is a means for storing a command (or program code) necessary to perform the image fusion method according to an embodiment of the present invention.

The processor 835 includes internal components of the image fusion device 100 according to an embodiment of the present invention (for example, an image acquisition unit 810, an object recognition unit 815, an image matching unit 820), It is a means for controlling the image fusion unit 825, the memory 830, and the like.

The apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes a hardware device specially configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

So far, the present invention has been looked at around the embodiments. Those of ordinary skill in the art to which the present invention pertains will be able to understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the above description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

100: video fusion device
810: image acquisition unit
815: object recognition unit
820: image matching unit
825: image fusion unit
830: memory
835: processor

Claims (12)

Obtaining a first image and a second image by photographing an object through a plurality of cameras, respectively-the first image is a color image, and the second image is a thermal image;
Recognizing an object for the first image and the second image; And
Generating a fusion image to preserve visual characteristics suitable for the object type by fusing the first image and the second image by applying different image fusion rules according to the recognized object type,
Before the step of generating the fusion image,
The first image and the second image are matched, but after setting any one of the first image and the second image as a reference image, the other one of the first image and the second image is matched with the reference image Matching by geometric transformation as much as possible,
The step of generating the fusion image,
When a plurality of objects of different types are included, a different image fusion rule is applied to each object according to the object type, and a different image fusion rule different from the recognized plurality of objects is applied as the background to generate a fusion image. Image fusion method, characterized in that to.
delete delete delete The method of claim 1,
The step of generating the fusion image,
An image fusion method comprising generating a fused image by applying different image fusion rules according to the thermal characteristics of the recognized object.
A computer-readable recording medium on which a program code for performing the method according to claim 1 or 5 is recorded.
An image acquisition unit for capturing a target through a plurality of cameras, respectively, to obtain a first image and a second image-the first image is a color image, and the second image is a thermal image;
An object recognition unit for recognizing an object with respect to the first image and the second image; And
An image fusion unit for generating a fusion image by fusing the first image and the second image by applying different image fusion rules according to the recognized object type,
The image matching unit,
The first image and the second image are matched, but after setting any one of the first image and the second image as a reference image, the other one of the first image and the second image is matched with the reference image Matching by geometric transformation as much as possible,
When a plurality of objects of different types are included, a different image fusion rule is applied to each object according to the object type, and a different image fusion rule different from the recognized plurality of objects is applied as the background to generate a fusion image. An image fusion device, characterized in that.
delete delete delete The method of claim 7,
The image fusion unit,
An image fusion device, characterized in that different image fusion rules are applied according to the thermal characteristics of the recognized object.
The method of claim 7,
The plurality of cameras are equipped with lenses having the same angle of view, and are installed so that the screens covered when the target is photographed overlap.

Images