KR102448498B1 - Method and apparatus for removing the noise of IR image - Google Patents

Abstract

According to an embodiment of the present invention, an apparatus and method for removing the noise of an infrared image include: a wavelet transform part for separating a first infrared image including pattern noise into a plurality of frequency components; a noise cancellation neural network for removing pattern noise components from the frequency components through pre-learned artificial intelligence and generating a plurality of final frequency components; and an inverse wavelet transform part for synthesizing the final frequency components and generating a second infrared image from which the pattern noise is removed. The frequency components are composed of a low frequency component, a vertical high frequency component, a horizontal high frequency component, and a bidirectional high frequency component.

Description

Method and apparatus for removing the noise of IR image

The present invention relates to infrared images, and more particularly, to a method and apparatus for removing noise from infrared images in a thermal camera.

All objects radiate heat above absolute 0 degrees (0 K (Kelvin Degree) is -273.15°C), and thermal cameras can create infrared images by visualizing the heat radiated from objects in the infrared region. That is, the thermal camera, unlike a general optical camera that detects light reflected from an object, images the radiant emission of the object, so that the object can be identified even in a limited environment that cannot be seen by the human eye.

For this reason, thermal cameras are being utilized in defense surveillance/perimeter systems and security fields. In particular, the thermal camera is manufactured in the form of an embedded-based edge artificial intelligence (hereinafter referred to as 'AI'), and is mounted on a portable lightweight product (eg, a drone or a vehicle).

However, as the thermal camera becomes smaller, the non-uniformity sensor response, manufacturing process problems, and fixed pattern noise are amplified in bad weather conditions. Such noise not only degrades an infrared image, but also causes degradation of performance when performing a computer vision-based object detection or object tracking method.

As for the noise removal technology developed to prevent such performance degradation, a prior knowledge-based method exists. Here, the prior knowledge-based method includes BM3D (Block-matching and 3D filtering) and a guided filter. BM3D compares the difference between a selected reference block and similar blocks with a threshold. A technology to remove noise A guide filter is a technology to remove noise by comparing the mean and variance of the guide image and the input image.

This noise removal technology has a problem in that the sharpness of the infrared image is blurred by removing all of the object boundary, features, and noise without considering the pattern noise in which a fixed pattern of the infrared image is repeated. For example, as shown in FIG. 1 , the noise removal technique may generate a blurred infrared image 103 by removing not only pattern noise but also sharpness from the infrared image 101 including pattern noise.

Therefore, the need for a method for solving these problems has emerged.

An embodiment of the present invention proposes a method and apparatus for removing only pattern noise while maintaining the boundary and characteristics of an object in an infrared image including pattern noise output from a thermal camera.

In addition, an embodiment of the present invention proposes a method and apparatus for removing only pattern noise while maintaining the boundary and characteristics of an object in an infrared image including pattern noise through pre-learned artificial intelligence.

According to an embodiment of the present invention, an apparatus for removing noise from an infrared image includes: a wavelet transform unit for separating a first infrared image including pattern noise into a plurality of frequency components; a denoising neural network that generates a plurality of final frequency components by removing the pattern noise component from the frequency components through pre-trained artificial intelligence; and an inverse wavelet transform unit generating a second infrared image from which the pattern noise is removed by synthesizing the final frequency components, wherein the frequency components include a low frequency component, a vertical high frequency component, a horizontal high frequency component, and a bidirectional high frequency component. characterized in that

According to an embodiment of the present invention, a method for removing noise from an infrared image includes a process in which a wavelet transform unit separates a first infrared image including pattern noise into a plurality of frequency components, and a noise removal neural network is trained in advance. A process of generating a plurality of final frequency components by removing the pattern noise component from the frequency components through an artificial intelligence, and an inverse wavelet transform unit synthesizes the final frequency components to generate a second infrared image from which the pattern noise is removed. and generating, wherein the frequency components include a low frequency component, a vertical high frequency component, a horizontal high frequency component, and a bidirectional high frequency component.

According to an embodiment of the present invention, only the pattern noise may be removed from the infrared image including the pattern noise output from the thermal camera while maintaining the boundary and characteristics of the object as it is.

And, according to an embodiment of the present invention, only pattern noise can be removed from an infrared image including pattern noise through pre-learned artificial intelligence while maintaining the boundary and characteristics of the object as it is.

In addition, the effects obtainable or predicted by the embodiments of the present invention are to be disclosed directly or implicitly in the detailed description of the embodiments of the present invention. That is, various effects predicted according to an embodiment of the present invention will be disclosed in the detailed description to be described later.

1 is a diagram illustrating an infrared image from which noise is removed according to the related art.
2 is a block diagram of a noise cancellation network according to an embodiment of the present invention.
3 is a diagram illustrating a procedure for removing noise using a noise cancellation network according to an embodiment of the present invention.
4 is a diagram illustrating an infrared image from which noise is removed according to an embodiment of the present invention.
5 is a flowchart for removing noise in a noise cancellation network according to an embodiment of the present invention.
6 is a flowchart for removing noise in a noise canceling neural network according to an embodiment of the present invention.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the embodiments of the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but this may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. . In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

Embodiments of the present invention can apply various transformations and can have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiments, and it should be understood to include all transformations, equivalents and substitutes included in the spirit and scope of the invention. In the description of the embodiments, if it is determined that a detailed description of a related known technology may obscure the subject matter, the detailed description thereof will be omitted.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and are intended to indicate that one or more other It is to be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In an embodiment of the present invention, a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' are integrated into at least one module and implemented with at least one processor (not shown) except for 'modules' or 'units' that need to be implemented with specific hardware. can be

In an embodiment of the present invention, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" with another element interposed therebetween. also includes In addition, when a part "includes" a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

2 is a block diagram of a noise cancellation network according to an embodiment of the present invention.

The denoising network removes only the pattern noise while maintaining the boundary and characteristics of the object in the infrared image including the pattern noise. For example, the pattern noise may indicate noise in which a specific pattern is repeated several times in an infrared image. For example, the specific pattern may have a stripe shape or the like.

Referring to FIG. 2 , the noise canceling network includes a wavelet transform unit 201 , a noise canceling neural network and an inverse wavelet transform unit 211 .

Looking at each component, the wavelet transform unit 201 converts the first infrared image to a plurality of frequency components (eg, a low frequency component, a vertical high frequency component, a horizontal high frequency component, a bidirectional high-frequency components). For example, the first infrared image is an image generated by a thermal camera and may include pattern noise. For example, the wavelet transform unit 201 may divide the first infrared image into a plurality of frequency components using a Haar wavelet method. For example, as shown in FIG. 3 , the wavelet transform unit 201 may divide the first infrared image 301 into four frequency components 303 . For example, the wavelet transform unit 201 may divide the infrared image into a plurality of frequency components using Equation 1 as follows.

은 저주파 성분을 나타내며,

는 수직 고주파 성분을 나타내며,

은 수평 고주파 성분을 나타내고,

는 양방향 고주파 성분을 나타낼 수 있다. 예를 들면,

는 잡음 제거 신경망에 입력되는 적외선 영상의 주파수 성분 집합을 나타낼 수 있다.For example, x may represent an input noise image.

represents the low-frequency component,

represents the vertical high-frequency component,

represents the horizontal high-frequency component,

may represent a bidirectional high-frequency component. For example,

may represent a set of frequency components of an infrared image input to the noise-removing neural network.

That is, the wavelet transform unit 201 may separate the pattern noise component in consideration of the directionality from the infrared image by using Equation (1).

The structure of the denoising neural network uses a U-net-based auto encoder-decoder structure. This structure can concatenate and preserve information lost in the process of converting the size of an infrared image, and is suitable for image segmentation and image restoration fields. For example, as shown in FIG. 3 , the denoising neural network 305 may generate four new frequency components 307 by removing pattern noise from the four frequency components 303 . For example, the denoising neural network 305 may generate new frequency components using Equation (2).

는 잡음 제거 신경망을 나타낼 수 있다.

은 잡음 제거 신경망에서 출력되는 저주파 성분을 나타내며,

는 잡음 제거 신경망에서 출력되는 수직 고주파 성분을 나타내며,

은 잡음 제거 신경망에서 출력되는 수평 고주파 성분을 나타내고,

는 잡음 제거 신경망에서 출력되는 양방향 고주파 성분을 나타낼 수 있다.

는 잡음 제거 신경망에서 출력되는 주파수 성분 집합을 나타낼 수 있다.For example,

may represent a denoising neural network.

represents the low-frequency component output from the noise-removing neural network,

represents the vertical high-frequency component output from the denoising neural network,

represents the horizontal high-frequency component output from the noise-removing neural network,

may represent a bidirectional high-frequency component output from the noise-removing neural network.

may represent a set of frequency components output from the noise-removing neural network.

In more detail, the denoising neural network includes first to fourth convolution units 203 , 205 , 207 , and 209 .

Looking at each component, the first convolution unit 203 receives a plurality of frequency components from the wavelet transform unit 201, and removes pattern noise from the plurality of frequency components using pre-learned artificial intelligence. 1 Generates frequency components. In addition, the first convolution unit 203 reduces the resolution of the generated first frequency components according to a predetermined reduction ratio to generate second frequency components, and converts the generated second frequency components to the second convolution unit 205 . output as

On the other hand, AI is trained using training data to remove pattern noise. For example, the training data may consist of a training pair of ground truths (GT) data and data degraded by pattern noise. For example, GT data may represent an infrared image without pattern noise, and degraded data may represent an infrared image including pattern noise. For example, artificial intelligence can be learned using Equation 3 below.

는 손실 함수를 나타낼 수 있다.

는 패턴 잡음을 포함하는 적외선 영상의 주파수 성분 집합이 잡음 제거 신경망을 통과한 주파수 성분 집합을 나타내고,

는 GT 데이터의 주파수 성분 집합을 나타낼 수 있다. 예를 들면, 주파수 성분 집합은 저주파 성분, 수직 고주파 성분, 수평 고주파 성분 및 양방향 고주파 성분으로 구성될 수 있다.For example,

may represent a loss function.

represents the set of frequency components in which the set of frequency components of the infrared image including pattern noise has passed through the noise-removing neural network,

may represent a set of frequency components of GT data. For example, the frequency component set may include a low frequency component, a vertical high frequency component, a horizontal high frequency component, and a bidirectional high frequency component.

와

사이의 유클리디안(Euclidean) 거리를 줄이는 평균 제곱 오차(Mean Squared Error, 이하 'MSE'라 한다)를 사용하여

가

와 유사해지도록 학습될 수 있다.That is, artificial intelligence

Wow

Using the mean squared error (hereinafter referred to as 'MSE') to reduce the Euclidean distance between

go

can be learned to be similar to

The second convolution unit 205 receives second frequency components from the first convolution unit 203 and removes pattern noise from the second frequency components using pre-learned artificial intelligence to generate third frequency components. create The second convolution unit 205 outputs the generated third frequency components to the third convolution unit 207 .

The third convolution unit 207 receives the third frequency components from the second convolution unit 205 and removes pattern noise from the third frequency components using pre-learned artificial intelligence to generate fourth frequency components. create In addition, the third convolution unit 207 generates fifth frequency components by enlarging the resolution of the generated fourth frequency components according to a predetermined enlargement ratio, and uses the generated fifth frequency components with the fourth convolution unit 209 . output as For example, the preset enlargement ratio may be a reciprocal of the preset reduction ratio. For example, when the preset reduction ratio is 1/2, the preset enlargement ratio may be 2 times.

The fourth convolution unit 209 receives the fifth frequency components from the third convolution unit 207, and removes pattern noise from the fifth frequency components using pre-learned artificial intelligence to remove a plurality of new frequency components. create them The fourth convolution unit 209 outputs new frequency components to the inverse wavelet transform unit 211 .

That is, the denoising neural network performs an operation of down-scaling the resolution of frequency components through the first to fourth convolution units 203 , 205 , 207 , and 209 in order to provide advantages of both local information and global information. The pattern noise can be effectively removed by performing the up-scaling operation only once and performing the noise removal operations using artificial intelligence 4 times.

The inverse wavelet transform unit 211 receives new frequency components from the noise-removing neural network, and synthesizes the new frequency components to generate a second infrared image. For example, the second infrared image may be a clear infrared image from which pattern noise is removed. For example, as shown in FIG. 3 , the inverse wavelet transform unit 211 may generate a second infrared image 309 by synthesizing four new frequency components 307 . For example, the inverse wavelet transform unit 211 may generate the second infrared image using Equation (4).

는 역웨이블릿 변환(IDWT)을 통해 합성된 적외선 영상을 나타낼 수 있다. For example,

may represent an infrared image synthesized through inverse wavelet transform (IDWT).

Through this configuration, according to an embodiment of the present invention, only the pattern noise can be removed while maintaining the boundary and characteristics of the object in the infrared image including the pattern noise output from the thermal camera. And, according to an embodiment of the present invention, only pattern noise can be removed from an infrared image including pattern noise through pre-learned artificial intelligence while maintaining the boundary and characteristics of the object as it is.

4 is a diagram illustrating an infrared image from which noise is removed according to an embodiment of the present invention.

Referring to FIG. 4 , when a portion 403 of the first infrared image 401 including the streaked noise is enlarged, in the enlarged image 409 , cross shapes are present on the roof and streaked noises are present on the outer wall.

On the other hand, when a portion 407 of the second infrared image 405 from which the stripe noise is removed according to the present invention is enlarged, in the enlarged image 411, fourteen shapes exist on the roof, but there are no stripe noises on the outside.

That is, since the noise removal technology of the present invention removes pattern noise from the first infrared image and at the same time maintains detailed information of object, which is an object feature of the first infrared image, as it is, FIG. 4 shows the efficiency of noise removal demonstrate this excellence.

The present invention can provide a high-quality infrared image that can be used in edge artificial intelligence equipment through this noise removal technology. In addition, the present invention can provide a high-quality infrared image without an expensive thermal camera.

5 is a flowchart for removing noise in a noise cancellation network according to an embodiment of the present invention.

Referring to FIG. 5 , in step 501 , the wavelet transform unit 201 of the noise removal network converts the first infrared image to a plurality of frequency components (eg, a low frequency component, a vertical high frequency component, a horizontal high frequency component, and a bidirectional high frequency component). component) is separated. For example, as shown in FIG. 3 , the wavelet transform unit 201 may separate the first infrared image 301 including pattern noise into four frequency components 303 . For example, the wavelet transform unit 201 may divide the first infrared image into a plurality of frequency components by using Equation (1).

In step 503, the denoising neural network of the denoising network generates a plurality of new frequency components by removing the pattern noise component from the plurality of frequency components using the pre-trained artificial intelligence. Hereinafter, a process of generating a plurality of new frequency components will be described in detail with reference to FIG. 6 .

6 is a flowchart for removing noise in a noise canceling neural network according to an embodiment of the present invention.

Referring to FIG. 6 , the first convolution unit 203 of the denoising neural network generates first frequency components by removing a pattern noise component from a plurality of frequency components through artificial intelligence in step 601 .

In step 603, the first convolution unit 203 generates second frequency components by reducing the resolution of the generated first frequency components according to a predetermined reduction ratio.

In step 605, the second convolution unit 205 of the noise-removing neural network generates third frequency components by removing the pattern noise component from the second frequency components through pre-trained artificial intelligence.

In step 607, the third convolution unit 207 of the noise-removing neural network generates fourth frequency components by removing the pattern noise component from the third frequency components through the pre-trained artificial intelligence.

In step 609, the third convolution unit 207 generates fifth frequency components by increasing the resolution of the fourth frequency components according to a predetermined enlargement ratio.

In step 611, the fourth convolution unit 209 of the noise-removing neural network generates new frequency components by removing the pattern noise component from the fifth frequency components through the pre-trained artificial intelligence.

Returning to FIG. 5 , in step 505 , the inverse wavelet transform unit 211 of the noise removal network synthesizes new frequency components to generate a second infrared image from which the pattern noise is removed. For example, as shown in FIG. 3 , the inverse wavelet transform unit 211 may generate a second infrared image 309 by synthesizing four new frequency components 307 . For example, the inverse wavelet transform unit 211 may generate a second infrared image using Equation (4).

Through this process, according to an embodiment of the present invention, only the pattern noise may be removed while maintaining the boundary and characteristics of the object in the infrared image including the pattern noise output from the thermal camera. And, according to an embodiment of the present invention, only pattern noise can be removed from an infrared image including pattern noise through pre-learned artificial intelligence while maintaining the boundary and characteristics of the object as it is.

In the above, preferred embodiments of the present invention have been illustrated and described, but the present invention is not limited to the specific embodiments described above, and it is common in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. Various modifications may be made by those having the knowledge of, of course, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

201: wavelet transform unit
203, 205, 207, 209: first to fourth convolution units
211: inverse wavelet transform unit

Claims (6)

a wavelet transform unit that separates the first infrared image including pattern noise into a plurality of frequency components;
a denoising neural network that generates a plurality of final frequency components by removing a pattern noise component from the frequency components through pre-trained artificial intelligence; and
and an inverse wavelet transform unit for synthesizing the final frequency components to generate a second infrared image from which the pattern noise has been removed,
The frequency components are composed of a low frequency component, a vertical high frequency component, a horizontal high frequency component and a bidirectional high frequency component,
The noise canceling neural network is
A plurality of first frequency components are generated by removing the pattern noise component from the frequency components through the artificial intelligence, and a plurality of second frequency components are generated by reducing the resolution of the first frequency components according to a predetermined reduction ratio. a first convolution unit to generate;
a second convolution unit for generating a plurality of third frequency components by removing the pattern noise component from the second frequency components through the artificial intelligence;
A plurality of fourth frequency components are generated by removing the pattern noise component from the third frequency components through the artificial intelligence, and a plurality of fifth frequency components are generated by expanding the fourth frequency components according to a predetermined magnification ratio. a third convolution unit to generate; and
and a fourth convolution unit generating the final frequency components by removing the pattern noise component from the fifth frequency components through the artificial intelligence.
delete According to claim 1,
The artificial intelligence is pre-learned using the training data composed of the GT (Ground Truths) data and the data degraded by the pattern noise, as a learning pair,
The GT data represents an infrared image without the pattern noise,
The apparatus for removing noise from an infrared image, characterized in that the data degraded by the pattern noise represents an infrared image including the pattern noise.
A process in which the wavelet transform unit separates the first infrared image including the pattern noise into a plurality of frequency components;
A process in which a noise-removing neural network generates a plurality of final frequency components by removing a pattern noise component from the frequency components through pre-trained artificial intelligence, and
and generating, by an inverse wavelet transform unit, a second infrared image from which the pattern noise is removed by synthesizing the final frequency components,
The frequency components are composed of a low frequency component, a vertical high frequency component, a horizontal high frequency component and a bidirectional high frequency component,
The process of generating the final frequency components is,
A process in which a first convolution unit generates a plurality of first frequency components by removing the pattern noise component from the frequency components through the artificial intelligence;
A process of generating, by the first convolution unit, a plurality of second frequency components by reducing the resolution of the first frequency components according to a predetermined reduction ratio;
A process in which a second convolution unit generates a plurality of third frequency components by removing the pattern noise component from the second frequency components through the artificial intelligence;
A process in which a third convolution unit generates a plurality of fourth frequency components by removing the pattern noise component from the third frequency components through the artificial intelligence;
generating, by the third convolution unit, a plurality of fifth frequency components by expanding the fourth frequency components according to a predetermined enlargement ratio; and
and generating, by a fourth convolution unit, the final frequency components by removing the pattern noise component from the fifth frequency components through the artificial intelligence.
delete 5. The method of claim 4,
The artificial intelligence is pre-learned using the training data composed of the GT (Ground Truths) data and the data degraded by the pattern noise, as a learning pair,
The GT data represents an infrared image without the pattern noise,
The method for removing noise from an infrared image, characterized in that the data degraded by the pattern noise represents an infrared image including the pattern noise.

Images