KR102552761B1 - Image processing apparatus and the operating method - Google Patents

Abstract

In the present invention, a lens, an image sensor generating an image corresponding to at least one of visible light and infrared light incident through the lens, and a mixed image generated by the image sensor are machine-learned, and a first image included in the mixed image is performed. The visible ray image and the infrared image are separated from each other, the first visible ray image is applied to a set machine learning model, and the second visible ray image having a second reference illuminance higher than the first reference illuminance and the infrared image are combined to combine the mixed image. An image processing device including a processor for restoring an image is provided.

Description

Image processing apparatus and its operating method {Image processing apparatus and the operating method}

The present invention relates to an image processing device and an operating method thereof, and more particularly, to an image processing device and method of operating the same, which can easily restore color of an image captured in a low-light environment.

In a low-light environment, the brightness of an image taken by a commercial camera is very low, and it may be difficult to check the color and texture of the image.

To solve this problem, in general, images may be obtained and image processing may be performed using the following three methods in a low light environment.

The first method is to additionally use near infrared (NIR) illumination to acquire a low-illuminance image and restore it to a proper-illumination image.

In this case, since the contrast ratio is improved and the strength of the signal is increased, a very clear texture image can be obtained, but there is a disadvantage that the color cannot be confirmed.

That is, since the near-infrared signal is mainly received by a red-type sensor such as red or magenta among image sensors, color noise may occur when a color image is generated.

Therefore, when an image is acquired with near-infrared light, it is common to give up color and convert it into a grayscale image in order to prevent generation of color noise.

The second is a method of restoring an appropriate illumination image through HDR technology after obtaining only a color image at low illumination.

However, in an environment with extremely low illumination, color noise may easily occur because an image of appropriate illumination is not properly restored.

A third method is to acquire a low-illuminance image by additionally using a sensor that receives near-infrared rays in an image sensor and restore an appropriate-illuminance image.

Commercial cameras use RGB Bayer pattern array CFA (color filter array) image sensors. To receive near-infrared signals, RGBN pattern array image sensors are used, or a structure that acquires only near-infrared signals as a separate channel like a multi-spectral camera of cameras can be used.

This has the advantage that color and near-infrared signals can be obtained separately, but has the disadvantage of being traded at a high price.

As described above, when an image is obtained using an image sensor of a commercial camera in a low light environment, the image quality is not good as described in the first and second methods. In order to overcome this, there is a case of using an RGBN array image sensor or a multispectral camera as a third method, but commercialization may be difficult because it is traded at a high price.

Recently, a method for restoring the color of an image obtained from a commercial camera using an image sensor to which CFA such as RGB Bayer pattern is applied in a low light environment and improving image quality is being studied.

An object of the present invention is to provide an image processing device and an operating method thereof that can easily restore the color of an image captured in a low-light environment.

Objects of the present specification are not limited to the above-mentioned purposes, and other objects and advantages of the present specification not mentioned above can be understood by the following description and will be more clearly understood by the examples of the present specification. Further, it will be readily apparent that the objects and advantages of this specification may be realized by means of the instrumentalities and combinations indicated in the claims.

The image processing apparatus according to the present invention includes an image sensor that generates an image corresponding to at least one of visible light and infrared light incident through the lens, and machine learning of a mixed image generated by the image sensor and included in the mixed image. The first visible ray image and the infrared image are separated from each other, and the first visible ray image is applied to a set machine learning model to combine the second visible ray image having a second reference illuminance higher than the first reference illuminance and the infrared image. and a processor for restoring the mixed image.

The processor may include a machine learning unit for machine learning the mixed image and separating the first visible ray image and the infrared image from each other, and applying the first visible ray image to the machine learning model to obtain the second visible ray image. It may include an image generator to generate an image and an image restorer to restore the mixed image by combining the infrared image to remove color noise of the second visible ray image.

The machine learning unit may perform machine learning with DCGAN (Deep Convolutional Generative Adversarial Networks).

The machine learning unit may separate the first visible ray image and the infrared image from each other by iteratively learning the visible ray image and the mixed image captured when the illuminance is higher than the first reference level.

The image generating unit applies the first visible ray image to the machine learning model, performs a convolution operation on the spatial resolution of the first visible ray image with a set reference spatial resolution, and has the reference spatial resolution. The second visible ray image obtained by restoring the color may be generated by performing a convolution operation on a residual signal of the first visible ray image.

The image restoration unit corrects the spatial resolution characteristics by applying the second visible ray image to the FPN model, removes the color noise based on the infrared image, and adds a pre-stored original background image to obtain the mixed image. can be restored

An operating method of an image processing apparatus according to the present invention includes, when a mixed image generated by visible and infrared light is input from an image sensor, a processor separates a first visible ray image and an infrared image included in the mixed image from each other. generating a second visible ray image having higher illuminance than the first visible ray image by applying the first visible ray image to a machine learning model, and mixing the second visible ray image and the infrared image It may include restoring the image.

In the separating from each other, the first visible ray image and the infrared image may be separated from each other by iteratively learning the visible ray image and the mixed image captured when the intensity of illumination is higher than the first reference level.

The step of separating from each other may be machine-learned with DCGAN (Deep Convolutional Generative Adversarial Networks).

The generating of the second visible ray image may include performing a convolution operation on a spatial resolution of the first visible ray image with a set reference spatial resolution, and a residual of the first visible ray image having the reference spatial resolution. The second visible ray image obtained by restoring the color may be generated by performing a convolution operation on the signal.

The restoring of the mixed image may include correcting spatial resolution characteristics by applying the second visible ray image to the FPN model, removing the color noise based on the infrared image, and adding a pre-stored original background image. The mixed image may be restored.

The image processing apparatus and method of operating the same according to the present invention have the advantage of reducing cost by not using an expensive camera such as a multi-spectral camera by applying a commercial camera equipped with a color filter.

In addition, the image processing device and its operating method according to the present invention removes the hot mirror filter from a commercial camera equipped with a color filter in a low-light environment, separates an infrared image from a captured image, and converts an RGB visible ray image to machine-learned brightness. There is an advantage in that an image in which color is restored can be obtained by mixing with an infrared image after improving the contrast ratio.

On the other hand, the effects of the present invention are not limited to the effects mentioned above, and various effects may be included within a range apparent to those skilled in the art from the contents to be described below.

1 is a control block diagram showing a control configuration of an image processing device according to a first embodiment of the present invention.
FIG. 2 is an operation diagram illustrating an operation of the image processing device shown in FIG. 1 .
3 is a structural diagram showing the entire process of the image processing device according to the present invention.
4 and 5 are exemplary diagrams illustrating images restored by the image processing device according to the present invention.
6 is a flowchart illustrating an operating method of an image processing device according to the present invention.
7 is a control block diagram showing a control configuration of an image processing device according to a second embodiment of the present invention.
FIG. 8 is an operation diagram illustrating an operation of the image processing device shown in FIG. 7 .

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term and/or includes a combination of a plurality of related recited items or any one of a plurality of related recited items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Expressions in the singular number include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a control block diagram showing a control configuration of an image processing device according to the present invention, FIG. 2 is an operation diagram showing the operation of the image processing device shown in FIG. 1, and FIG. 3 is an overall process of the image processing device according to the present invention. It is a structural diagram showing

Referring to FIGS. 1 to 3 , the image processing device 100 may include a lens 110 , an image sensor 140 and a processor 150 .

In the embodiment, the image processing device 100 is described as a CCTV capable of capturing a setting space in a low-light environment. Also, although the image processing device 100 describes a low-light environment, it may be used in a high-light environment other than a low-light environment, and is not limited thereto.

The image sensor 140 may be a CCD or CMOS sensor that implements R, G, and B colors.

Here, the image sensor 140 may output an image corresponding to at least one of visible light rays and infrared rays incident from the lens 110 .

The processor 150 may include a machine learning unit 152 , an image generator 154 and an image restoration unit 156 .

The machine learning unit 152 repeatedly learns the visible ray image and the mixed image input from the image sensor 140 and separates the first visible ray image and the infrared image included in the mixed image from each other.

Here, the visible ray image may be an image separated from an infrared image tube, and the mixed image may be an image in which infrared and visible rays are mixed.

That is, the machine learning unit 152 may perform machine learning using DCGAN (Deep Convolutional Generative Adversarial Networks).

DCGAN (Deep Convolutional Generative Adversarial Networks) is unsupervised learning and can be used for specific tasks such as generating images through competition between adversarial neural networks or converting images.

DCGAN can generate data distribution through training data, generate images based on it, and can be classified as generators and discriminators.

The generator may generate an image similar to the original using training data, and the discriminator may learn to determine whether or not the generated image is true by comparing it with the original image.

The machine learning unit 152 may separate the visible ray image and the infrared image from the mixed image through the processor of FIG. 3(a).

That is, FIG. 3(a) shows a deep learning network that separates a visible ray image and an infrared image from a mixed image, and DCGAN can be applied to the visible ray image and the infrared image to separate them.

The image generator 154 may generate the second visible ray image by applying the first visible ray image to a machine learning model.

That is, the image generator 154 applies the first visible ray image to the machine learning model, performs a convolution operation on the spatial resolution of the first visible ray image with a set reference spatial resolution, and performs a convolution operation on the reference spatial resolution. The second visible ray image obtained by restoring the color may be generated by performing a convolution operation on a residual signal of the first visible ray image having spatial resolution.

For the purpose of improving image quality, the image generator 154 may reduce the neural network layer to perform a convolution operation up to 32 × 32 × 256 in order to preserve detailed information such as edges while maintaining the same number of channels to perform a reference spatial resolution. .

After that, the image generator 154 does not operate on all values of the image through the connection of input and output in the residual block for fast learning, but only on the residual signal, thereby speeding up the operation speed and error value of the deep network. It performs a shortcut connection that improves performance by reducing , and adds a 1×1 convolution operation to the residual block to reduce the amount of computation by reducing the dimensionality, thereby speeding up network learning.

The image restoration unit 156 may restore a mixed image by combining infrared images in order to remove color noise of the second visible ray image.

That is, the image restoration unit 156 corrects spatial resolution characteristics by applying the second visible ray image to the set FPN model, removes color noise based on the infrared image, and adds a pre-stored original background image to create a mixed image. can be restored to

The PFN model can be classified as a discriminator of the above-described DCGAN, and can supplement spatial resolution characteristics from low to high levels through a top-down skip connection step.

The image restoration unit 156 represents a deep learning network that reconstructs a mixed image through the processor of FIG. It is possible to achieve color restoration with an image having illuminance).

Finally, the image restoration unit 156 removes color noise by using an infrared image in order to minimize color noise in the second visible ray image, and replaces the background area where no object is located with the previously stored original background image to obtain an image image quality can be improved.

4 and 5 are exemplary diagrams illustrating images restored by the image processing device according to the present invention.

FIG. 4(a) shows a mixed image, FIG. 4(b) shows a first visible ray image, and FIG. 4(c) shows an infrared image.

That is, FIG. 4(a) shows a mixed image including a visible ray image and an infrared image captured by the image sensor 140 in a low light environment, and FIGS. It represents an image obtained by separating a ray image and an infrared image through machine learning.

Fig. 5(a) is a mixed image restored by the conventional method, Fig. 5(b) is a mixed image restored according to the present invention, and Fig. 5(c) shows the original image.

In FIG. 5(a), in the case of the restored mixed image, color information is not properly restored, and the color of the part with low illumination is restored using the difference in illumination. It can be seen that restoration in the environment does not work properly.

6 is a flowchart illustrating an operating method of an image processing device according to the present invention.

Referring to FIG. 6 , the image processing device 100

When the mixed image generated by the visible light and the infrared light is input from the image sensor 140, the processor 150 may separate the first visible ray image and the infrared image included in the mixed image from each other (S110). .

The processor 150 may apply the first visible ray image to a machine learning model to generate a second visible ray image having higher illuminance than the first visible ray image (S120).

Thereafter, the processor 150 may restore a mixed image through the second visible ray image and the infrared image (S130).

That is, the processor 150 may repeatedly learn the visible ray image and the mixed image input from the image sensor 140 to separate the first visible ray image and the infrared image included in the mixed image from each other.

Here, the visible ray image may be an image in which an infrared image is separated, and the mixed image may be an image in which infrared and visible ray are mixed.

The processor 150 may separate the visible ray image and the infrared image from the mixed image through the processor of FIG. 3(a).

That is, FIG. 3(a) shows a deep learning network that separates a visible ray image and an infrared image from a mixed image, and DCGAN can be applied to the visible ray image and the infrared image to separate them.

The processor 150 applies the first visible ray image to the machine learning model, performs a convolution operation on the spatial resolution of the first visible ray image with a set reference spatial resolution, and has the reference spatial resolution The second visible ray image obtained by restoring the color may be generated by performing a convolution operation on a residual signal of the first visible ray image.

For the purpose of improving image quality, the processor 150 may perform a convolution operation only up to 32×32×256 by reducing the number of neural network layers to preserve detailed information such as an edge while maintaining the same number of channels to perform a reference spatial resolution.

After that, the processor 150 does not operate on all values of the image through the connection of input and output in the residual block for fast learning, but only on the residual signal, thereby speeding up the operation speed and reducing the error value of the deep network By performing a shortcut connection that improves performance, and by adding a 1×1 convolution operation to the residual block, it is possible to reduce the amount of computation by reducing the dimensionality, thereby speeding up the network learning speed.

The processor 150 corrects spatial resolution characteristics by applying the second visible ray image to the set FPN model, removes color noise based on the infrared image, and restores the mixed image by adding a pre-stored original background image. there is.

The processor 150 represents a deep learning network that reconstructs a mixed image through the processor of FIG. 3 (b), and converts a second visible ray image to a second reference illuminance higher than the first reference illuminance corresponding to a low-illuminance environment (optimal illuminance). ), color restoration can be achieved.

Finally, the processor 150 removes color noise by using an infrared image in order to minimize color noise in the second visible ray image, and replaces the background area where the object is not located with the original background image stored in the image quality. can improve

7 is a control block diagram showing a control configuration of an image processing device according to a second embodiment of the present invention, and FIG. 8 is an operation diagram showing an operation of the image processing device shown in FIG. 7 .

7 and 8 , the image processing device 200 may include a lens 210, an illuminance sensor 220, a hot mirror filter 230, an image sensor 240, and a processor 250.

In the embodiment, the image processing device 200 will be described as being a CCTV capable of photographing a set space in a low-light environment and a high-light environment.

Here, the lens 210, the image sensor 240, and the processor 250 have the same configuration as the lens 110, the image sensor 140, and the processor 150 shown in FIG. 1, and detailed descriptions are omitted.

The illuminance sensor 220 may measure ambient illuminance of the setting space. Here, the illuminance sensor 220 may be positioned adjacent to the lens 210 and may be implemented as a separate device not included in the image processing device 200, but is not limited thereto.

The hot mirror filter 230 may block infrared rays and pass visible rays.

Here, the hot mirror filter 230 may be moved under the control of the processor 250 .

That is, the hot mirror filter 230 may be positioned between the lens 210 and the image sensor 240 or moved from a position between the lens 210 and the image sensor 240 to another position.

The processor 250 may compare the ambient illuminance input from the illuminance sensor 220 with the first reference illuminance.

When the peripheral illuminance is higher than the first reference illuminance, the processor 250 may maintain the position of the hot mirror filter 230 between the lens 210 and the image sensor 240, as shown in FIG. 8(a).

In addition, when the ambient illumination is lower than the first reference illumination, the processor 250 determines that the environment is low, and the hot mirror filter 230 positioned between the lens 210 and the image sensor 240 as shown in FIG. ) can be moved.

At this time, the movement of the hot mirror filter 230 refers to removing the hot mirror filter 230 between the lens 210 and the image sensor 240 and moving it to another position.

The machine learning unit 252 included in the processor 250 repeatedly learns the visible ray image and the mixed image input from the image sensor 240 to separate the first visible ray image and the infrared image included in the mixed image from each other. can

Here, the visible ray image is an image generated by visible light by blocking infrared rays by the hot mirror filter 230, and the mixed image is an image in which infrared and visible light are mixed by moving the hot mirror filter 230. can

The machine learning unit 252 included in the processor 250 may perform machine learning using DCGAN (Deep Convolutional Generative Adversarial Networks).

The machine learning unit 252 may separate the visible ray image and the infrared image from the mixed image through the processor of FIG. 3(a) described above.

The image generator 254 included in the processor 250 may generate the second visible ray image by applying the first visible ray image to a machine learning model.

The image restoration unit 256 may restore a mixed image by combining infrared images in order to remove color noise of the second visible ray image.

That is, the image restoration unit 256 corrects spatial resolution characteristics by applying the second visible ray image to the set FPN model, removes color noise based on the infrared image, and adds a pre-stored original background image to create a mixed image. can be restored to

The image restoration unit 256 represents a deep learning network that restores a mixed image through the processor of FIG. 3 (b), and can perform color restoration from a second visible ray image to an image having a second reference illuminance (optimal illuminance). there is.

Finally, the image restoration unit 256 removes color noise by using an infrared image in order to minimize color noise in the second visible ray image, and replaces the background area where the object is not located with the previously stored original background image to obtain an image image quality can be improved.

Features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present invention, and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. illustrated in each embodiment can be combined or modified with respect to other embodiments by a person having ordinary knowledge in the field to which the embodiments belong. Therefore, contents related to these combinations and variations should be construed as being included in the scope of the present invention.

In addition, although the above has been described with reference to the embodiments, these are only examples and do not limit the present invention, and those skilled in the art to which the present invention belongs can exemplify the above to the extent that does not deviate from the essential characteristics of the present embodiment. It will be appreciated that various modifications and applications that have not been made are possible. For example, each component specifically shown in the embodiment can be modified and implemented. And differences related to these modifications and applications should be construed as being included in the scope of the present invention as defined in the appended claims.

Claims (11)

lens;
an image sensor generating an image corresponding to at least one of visible light and infrared light incident through the lens;
light sensor;
When the ambient illuminance of the set space measured by the illuminance sensor is higher than the first reference illuminance, it is located between the lens and the image sensor to block infrared rays so that the image sensor generates a visible ray image, and the illuminance sensor a hot mirror filter that is moved to a different location when the measured ambient illumination of the setting space is lower than the first reference illumination, and allows the image sensor to obtain a mixed image of a visible ray image and an infrared image; and
Iteratively learns the visible ray image and the mixed image captured when the illuminance is higher than the first reference level, separates the first visible ray image and the infrared image included in the mixed image from each other, and obtains the first visible ray image. a processor for restoring the mixed image by applying a set machine learning model to combine a second visible ray image having a second reference illuminance higher than the first reference illuminance and the infrared image; including,
image processing device.
According to claim 1,
the processor,
a machine learning unit which performs machine learning on the mixed image and separates the first visible ray image and the infrared image from each other;
an image generating unit generating the second visible ray image by applying the first visible ray image to the machine learning model; and
And an image restoration unit for restoring the mixed image by combining the infrared image to remove color noise of the second visible ray image.
image processing device.
According to claim 2,
The machine learning unit,
Machine learning with DCGAN (Deep Convolutional Generative Adversarial Networks),
image processing device.
delete According to claim 2,
The video generator,
The first visible ray image is applied to the machine learning model, a spatial resolution of the first visible ray image is convoluted with a set reference spatial resolution, and the first visible ray image having the reference spatial resolution Generating the second visible ray image in which the color is restored by performing a convolution operation on the residual signal of the image,
image processing device.
According to claim 2,
The image restoration unit,
Applying the second visible ray image to the FPN model to correct the spatial resolution characteristics, removing the color noise based on the infrared image, and adding a pre-stored original background image to restore the mixed image,
image processing device.
When the peripheral illuminance of the set space measured by the illuminance sensor is higher than the first reference illuminance, the processor places a hot mirror filter between the lens and the image sensor to block infrared rays so that the image sensor generates a visible ray image; When the ambient illuminance of the setting space measured by the illuminance sensor is lower than the first reference illuminance, the hot mirror filter is moved to another position so that the image sensor acquires a mixed image in which visible ray images and infrared images are mixed. image acquisition step;
separating a first visible ray image and an infrared image included in the mixed image by iteratively learning the visible ray image and the mixed image captured when the illuminance is higher than the first reference level;
generating a second visible ray image having higher illuminance than the first visible ray image by applying the first visible ray image to a machine learning model; and
Restoring the mixed image through the second visible ray image and the infrared image,
A method of operating an image processing device.
delete According to claim 7,
The step of separating from each other,
Machine learning with DCGAN (Deep Convolutional Generative Adversarial Networks),
A method of operating an image processing device.
According to claim 7,
Generating the second visible ray image,
A convolution operation is performed on the spatial resolution of the first visible ray image with a set reference spatial resolution, and a convolution operation is performed on a residual signal of the first visible ray image having the reference spatial resolution to restore color. generating the second visible ray image;
A method of operating an image processing device.
According to claim 10,
Restoring the mixed image,
Applying the second visible ray image to the FPN model to correct the spatial resolution characteristics, removing color noise based on the infrared image, and restoring the mixed image by adding a pre-stored original background image,
A method of operating an image processing device.

Images