KR102584573B1 - Integrated image processing apparatus and method for improving image quality using complex sensor information - Google Patents

Abstract

The integrated image processing device of the present invention includes an input unit that receives corresponding detection signals from each of complex sensors including an image sensor, a ToF sensor, and a heat detection sensor; After correcting the image detected through the image sensor based on a preset artificial neural network (ANN) algorithm, the corrected image, 3D distance information output from the ToF sensor, and low-resolution heat output from the heat detection sensor A multi-video signal processing unit that combines image images and outputs them; And learning at least one pre-generated data set for each class to generate reference data and then storing it in advance, and recognizing the image generated by the multiple video signal processor based on the reference data and a preset artificial intelligence algorithm. By including an AI accelerator, image recognition technology based on an artificial neural network (ANN) is applied and processed at the edge, thereby reducing power consumption for communication with the server and processing images quickly, but without human intervention. It has the advantage of being able to process and output images at a level similar to that recognized by the human eye.

Description

Integrated image processing device and method for improving image quality using complex sensor information {INTEGRATED IMAGE PROCESSING APPARATUS AND METHOD FOR IMPROVING IMAGE QUALITY USING COMPLEX SENSOR INFORMATION}

The present invention relates to image processing technology, and more specifically, to an integrated image processing device and method for improving image quality using complex sensor information.

Image processing technology based on complex signals can be used as a core component in various electronic devices/products, including smartphones, and in systems that require multiple sensors, such as future self-driving cars, robots, drones, and HMDs.

As an example of this prior art, Korean Patent Publication No. 10-2018-0128168 discloses an image processing method of an electronic device supporting a dual image sensor system, through a first sensor and a second sensor disposed in the electronic device. detecting data; Comprising: synthesizing an image based on data sensed through the first sensor and data sensed through the second sensor, wherein the first sensor and the second sensor are color sensors, and the second sensor An image processing method for an electronic device is disclosed, wherein the amount of light detected through the first sensor is greater than the amount of light sensed through the first sensor.

According to the patent, by increasing the amount of light acquired through the image sensor compared to the conventional RGB sensor, the SN characteristics of the image are improved in a low-light environment, and by acquiring the image through two color image sensors, the precision of image depth information is improved. There are features that can improve.

However, the above-mentioned conventional technology has a problem in that it is difficult to obtain results similar to those seen with the actual eye due to low precision in image processing such as backlight correction and color temperature correction. Additionally, the above prior art has limitations in processing three-dimensional images by using only a plurality of image sensors.

Therefore, the development of products equipped with various complex sensors such as dual/triple imaging sensors and depth sensors is being actively promoted by leading groups (e.g., Apple's iPhone, etc.).

However, these recent technologies are inefficient in terms of power and area because the processing engines for each sensor are mounted separately and are not functionally integrated. Because of this, recent technologies such as those described above also have the problem of not being able to support real-time services by not being able to process large amounts of data without delay in data access from complex signal sensors.

Korean Patent Publication No. 10-2018-0128168

Therefore, the present invention integrates and processes signals from complex sensors in a single processor to efficiently utilize power and area, thereby improving processing speed, thereby enabling integrated processing of large amounts of data in real time without data access delay. The object is to provide an image processing device and method.

In addition, by using complex sensor information, the present invention can overcome the limitations of systems relying on a single image sensor in various environments such as extremely low light, smoke, fog, or high heat caused by fire, thereby enabling indoor workers such as factories. , We aim to provide an integrated image processing device and method that can be used as an auxiliary means of detection information in various application fields such as firefighters at the scene of a fire or soldiers in operation, thereby expanding the field of use.

In addition, the present invention applies image recognition technology based on an artificial neural network (ANN), implements learned data at the edge using a pre-generated data set for each class, and uses the learned data. By processing the image at the edge, power consumption for communication with the server is reduced and the image is processed quickly, but the integrated image processing that can process and output the image at a level similar to that recognized by the human eye is neglected. and to provide a method thereof.

In order to achieve the above object, the integrated image processing device provided by the present invention includes an input unit that receives corresponding detection signals from each of the complex sensors including an image sensor, a ToF sensor, and a heat detection sensor; After correcting the image detected through the image sensor based on a preset artificial neural network (ANN) algorithm, the corrected image, 3D distance information output from the ToF sensor, and low-resolution heat output from the heat detection sensor A multi-video signal processing unit that combines image images and outputs them; And learning at least one pre-generated data set for each class to generate reference data and then storing it in advance, and recognizing the image generated by the multiple video signal processor based on the reference data and a preset artificial intelligence algorithm. It is characterized by including an AI accelerator.

In addition, in order to achieve the above object, the integrated image processing method provided by the present invention is an integrated image processing device equipped with a complex sensor including an image sensor, a ToF sensor, and a heat sensor, and uses the complex sensor information to image an image. An integrated image processing method for improving image quality, comprising: a learning step of generating and storing reference data for image detection by learning at least one pre-generated dataset for each class; An input step of receiving a corresponding detection signal from each of the complex sensors including the image sensor, the ToF sensor, and the heat detection sensor; A correction step of correcting the image detected through the image sensor based on a preset artificial neural network (ANN) algorithm; A combining step of combining the corrected image, 3D distance information output from the ToF sensor, and a low-resolution thermal image output from the heat detection sensor; And a recognition step of recognizing the image by applying the reference data and the combined image to a preset artificial intelligence algorithm.

The integrated image processing device and method of the present invention as described above efficiently utilize power and area by integrated processing of signals from complex sensors in one processor, thereby improving processing speed, without data access delay. It is effective in processing large amounts of data in real time.

In addition, the integrated image processing device and method of the present invention can overcome the limitations of systems that rely on a single image sensor in various environments such as extremely low light, smoke, fog, or high heat due to fire by using complex sensor information. , This has the effect of expanding the field of use by using it as an auxiliary means of sensing information in various application fields such as indoor workers in factories, firefighters at fire sites, and soldiers on operations.

In addition, the integrated image processing device and method of the present invention apply image recognition technology based on an artificial neural network (ANN), and implement learned data at the edge using a pre-generated data set for each class. By processing the image at the edge using the learned data, power consumption for communication with the server is reduced and the image is processed quickly, but the image is processed at a level similar to that recognized by the human eye. There is an effect that can be printed.

1 is a schematic block diagram of an integrated image processing device according to an embodiment of the present invention.
Figure 2 is a schematic block diagram of a multiple video signal processor according to an embodiment of the present invention.
Figure 3 is a schematic block diagram of a correction unit according to an embodiment of the present invention.
Figure 4 is a schematic processing flowchart of an integrated image processing method according to an embodiment of the present invention.
Figure 5 is a schematic processing flowchart of an image correction step according to an embodiment of the present invention.
Figure 6 is a diagram for explaining the effect of backlight correction according to an embodiment of the present invention compared to the prior art.
FIGS. 7A and 7B are diagrams for explaining the effect of automatic color temperature correction according to an embodiment of the present invention compared to the prior art.
FIG. 8 is a diagram for explaining the effects of noise removal and sharpness correction according to an embodiment of the present invention compared to the prior art.

Below, embodiments of the present invention will be described with reference to the attached drawings, and will be described in detail so that those skilled in the art can easily practice the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Meanwhile, in order to clearly explain the present invention in the drawings, parts unrelated to the description are omitted, and similar parts are given similar reference numerals throughout the specification. In addition, descriptions of parts that can be easily understood by those skilled in the art are omitted even if detailed descriptions are omitted.

Throughout the specification and claims, when it is said that a part includes a certain component, this means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

1 is a schematic block diagram of an integrated image processing device according to an embodiment of the present invention. Referring to Figure 1, the integrated image processing device 100 according to an embodiment of the present invention includes a complex sensor unit 110, an input unit 120, a multiple image signal processor 130, an AI accelerator 140, and reference data. Includes a management DB (150).

The complex sensor unit 110 includes at least one sensor that collects various types of data, and may include a ToF sensor 111, an image sensor 112, and a heat detection sensor 113.

The input unit 120 inputs a corresponding detection signal from each of at least one sensor (e.g., ToF sensor 111, image sensor 112, and heat detection sensor 113) included in the complex sensor unit 110. receive it as At this time, the input unit 120 is composed of one connector, and transmits a signal from each of the at least one sensor (e.g., ToF sensor 111, image sensor 112, and heat sensor 113) to the one Input is received through the connector.

The multiple image signal processing unit 130 combines multiple signals input through the input unit 120 and outputs them as one signal. To this end, the multi-image signal processor 130 first performs a process of correcting the image detected through the image sensor 112 based on a preset artificial neural network (ANN) algorithm, and then sends the corrected image to the ToF sensor. The 3D distance information output from 111 and the low-resolution thermal image output from the heat detection sensor 113 are combined and output.

The AI accelerator 140 recognizes the image generated by the multi-image signal processor 130 based on a preset artificial intelligence algorithm (eg, image recognition algorithm). To this end, the AI accelerator 140 learns at least one pre-generated dataset for each class (e.g., a dataset created by collecting a predetermined number of images for each of smoke, depth, and temperature) to provide reference data. After creation, it is stored in advance, and the image generated by the multi-video signal processor 130 is recognized based on the reference data and a preset artificial intelligence algorithm. That is, the AI accelerator 140 applies the image collected through the complex sensor unit 110 and then corrected through the multi-image signal processor 130 to the artificial intelligence algorithm, and selects one of the pre-learned reference images. It can be recognized as one.

At this time, the reference data may be pre-stored in the AI accelerator 140, or may be implemented as a separate database unit (eg, reference data management DB 150) accessible by the AI accelerator 140.

In addition, the AI accelerator 140 is a processor with a parallel structure, and can receive each signal processing result in parallel and process large amounts of data in real time without data access delay.

Figure 2 is a schematic block diagram of a multiple video signal processor according to an embodiment of the present invention. Referring to Figures 1 and 2, the multiple image signal processing unit 130 includes a correction unit 131 and an image combining unit 132.

The correction unit 131 corrects the image detected through the image sensor 112, and as illustrated in FIG. 3, which will be described later, the resolution correction unit 10, the backlight correction unit 20, and the automatic color temperature correction. It includes a main unit (30), and a noise removal/clearness correction unit (40).

The image combining unit 132 combines the image corrected by the correcting unit 131, the 3D distance information output from the ToF sensor 111, and the low-resolution thermal image output from the heat detection sensor 113. At this time, the method of combining the corrected image with the 3D distance information and the low-resolution thermal image can use various known image combining methods. For example, a method of combining image data or thermal image data from an RGB camera and distance (depth) information, which is point cloud data (PCD) received from a ToF sensor, and a ToF signal and two or more cameras, an RGB camera or a thermal image camera. GC-Net, PSMNet, etc. are learning models consisting of a feature extraction unit that receives left and right disparity images as input and extracts the main features of each disparity image, and a depth learning unit that learns disparity information using the extracted features. The images can be combined by using a method to generate a full depth image.

Figure 3 is a schematic block diagram of a correction unit according to an embodiment of the present invention. Referring to Figure 3, the correction unit 131 includes a resolution correction unit 10, a backlight correction unit 20, and an automatic color temperature correction unit. It includes a correction unit 30 and a noise removal/clearness correction unit 40.

The resolution correction unit 10 converts the low-quality image detected by the image sensor 112 into a high-quality image.

The backlight correction unit 20 receives three or more images with different exposure times from the resolution correction unit 10, and combines the three or more images into one image by applying a preset backlight correction neural network (WDR NN) algorithm. Correct backlight.

The automatic color temperature correction unit 30 automatically corrects the color temperature of the backlight-corrected image in the backlight correction unit 20. To this end, the automatic color temperature correction unit 30 derives a Planck White curve for each object included in the backlight-corrected image, and applies the Planck White curve for each object to a preset color correction neural network (AWB NN). By determining the color temperature for each object, the color temperature for each object is automatically corrected.

The noise removal/sharpness correction unit 40 applies the color temperature corrected image transmitted from the automatic color temperature correction unit 30 to a preset noise removal/sharpness artificial neural network (NR/Sharp NN) to produce the color temperature corrected image. Only noise is removed while maintaining the boundaries included in the image.

FIG. 4 is a schematic processing flowchart of an integrated image processing method according to an embodiment of the present invention, and FIG. 5 is a schematic processing flowchart of an image correction step according to an embodiment of the present invention. With reference to FIGS. 1 to 5 , an integrated image processing method (i.e., an integrated image processing method that improves image quality using complex sensor information) according to an embodiment of the present invention is as follows.

First, in step S110, the AI accelerator 140 learns at least one pre-generated dataset for each class (e.g., a dataset created by collecting a predetermined number of images for each of smoke, depth, and temperature) Generate reference data for image detection and store the results. At this time, the AI accelerator 140 may store the reference data in a separate database (eg, reference data management DB 150) or may store it within the AI accelerator 140 itself.

In step S120, the input unit 120 receives corresponding detection signals from each of the ToF sensor 111, the image sensor 112, and the heat detection sensor 113 included in the complex sensor unit 110.

In step S130, the multiple image signal processor 130 corrects the image detected through the image sensor 112 based on a preset artificial neural network (ANN, Artificial Neural Network) algorithm.

That is, in step S131, the resolution correction unit 10 corrects the resolution of the image. To this end, the resolution correction unit 10 uses the SR (Super Resolution) algorithm developed to improve low-quality images to high-quality images, and by using the SR algorithm based on an artificial neural network, the resolution can be increased more effectively.

Existing traditional super-resolution techniques (SR algorithms) suffer from lowering and deterioration of image quality in detailed parts during the process of restoring high-resolution images, and use polynomial-based interpolation such as bicubic interpolation or linear mapping. , there was a problem in implementing complex and nonlinear low-resolution-high-resolution models. Therefore, in step S131 of the present invention, in order to solve this problem, a multi-layer network is used to precisely analyze the complex non-linear relationship between low-resolution input and high-resolution output to learn convolution filter parameters that can be converted more accurately. , This enabled the creation of optimized high-resolution images.

In step S132, the backlight correction unit 20 corrects the backlight included in the image corrected to high definition in step S131. At this time, the backlight correction unit 20 combines the high-speed shutter in the bright area and the low-speed shutter in the dark area into one, thereby solving the backlight problem and increasing the image recognition rate. To this end, the backlight correction unit 20 uses the WDR (Wide Dynamic Range) algorithm developed to obtain high-quality images even in various lighting environments, and uses the WDR (Wide Dynamic Range) algorithm based on an artificial neural network ( By using WDR NN (WDR Neural Network), backlight can be corrected more effectively.

In particular, in step S132, the backlight correction unit 20 applies the WDR NN algorithm to three or more images with different exposure times among the images corrected to high quality in the resolution correction unit 10 to combine them into one image, The phenomenon of unnatural images appearing during the process of combining multiple images into one has been resolved. At this time, the more images with different exposure times are selected, the more natural images can be created. Therefore, the backlight correction unit 20 can determine the number of images with different exposure times to obtain optimal results by considering both computation time and naturalness.

Meanwhile, compared to cameras, the human eye has the characteristic of being more sensitive to dark areas than to bright areas. This is because the sensor of a camera that receives twice the amount of light linearly recognizes the amount of light as twice the amount of light, but the human eye non-linearly recognizes it as only partially bright. Therefore, when correcting a photo, it is necessary to accurately contrast the bright and dark parts. To this end, in step S132, the backlight correction unit 20 uses a WDR (Wide Dynamic Range) algorithm (WDR NN, WDR Neural Network) based on a backlight artificial neural network. This problem was solved by applying That is, in step S132, the backlight correction unit 20 generates an optimized Contrast Gain Curve based on the images subject to correction and accurately performs Contrast Correction, thereby improving the dark area compared to the existing image.

FIG. 6 is a diagram for explaining the effect of backlight correction of the present invention compared to the prior art. FIG. 6 (a) shows two images with different exposure times selected for backlight correction, and FIG. 6 (a) shows two images with different exposure times selected for backlight correction. b) shows an example of the conventional WDR process (20a), and Figure 6(c) is the result of processing the images illustrated in Figure 6(a) in the manner illustrated in Figure 6(b). Shows an example of an image, and Figure 6(d) shows an example of an artificial neural network (ANN)-based WDR process (20b) according to an embodiment of the present invention, Figure 6(e) shows an example of Figure 6 An example of an image derived as a result of processing the images illustrated in (a) in the manner illustrated in (d) of FIG. 6 is shown.

Referring to (b) of FIG. 6, the conventional WDR process (20a) is a process of collecting the images (i.e., images with different exposure times) illustrated in (a) of FIG. 6 (Long/Short Fusion) ) (21a), a process of combining the images by tone mapping (22a), and a contrast correction process (23a) for the tone mapping result.

Meanwhile, referring to (d) of FIG. 6, the artificial neural network (ANN)-based WDR process (20b) according to an embodiment of the present invention uses the images illustrated in (a) of FIG. 6 (i.e., exposure time A process of collecting different images (Long/Short Fusion) (21b), a process of combining the images by tone mapping (22b), and contrast correction for the tone mapping results ( Contrast Correction (23b) process, applying a backlight correction neural network (WDR NN) (24b) for the image collection process (21b), and generating a correction gain curve (24b) by the backlight correction neural network (WDR NN) (24b) After creating a Contrast Gain Curve (25b), this is reflected in the contrast correction process (23b). This is to determine the ratio of the pixel image in the bright area and the pixel image in the dark area.

For this reason, as illustrated in Figures 6(c) and 6(e), the artificial neural network (ANN)-based WDR progress result (e) according to an embodiment of the present invention is different from the conventional WDR progress result. (c) You can see something more natural.

In step S133, the automatic color temperature correction unit 30 automatically corrects the color temperature for each object included in the backlight-corrected image. That is, in step S133, the automatic color temperature correction unit 30 applies the image output from the backlight correction unit 20 to the artificial neural network to automatically correct the color temperature for each object. To this end, the automatic color temperature correction unit 30 uses the AWB (Auto White Balance) algorithm developed to improve image quality and increase image recognition rate by maintaining the color temperature appropriately, and uses the AWB algorithm based on an artificial neural network. By using , the color temperature can be corrected more effectively.

In particular, in step S133, the automatic color temperature correction unit 30 determines the color temperature by deriving a Planck White curve for each object from the backlight-corrected image, and calculates the Planck White curve for each object using a preset color correction neural network. (AWB NN) is applied to perform a series of processes to correct the color temperature of each object. For example, if the color temperature of any object included in the input image is low, it appears red, so color correction can be performed using a cool blue filter.

FIGS. 7A and 7B are diagrams for explaining the effect of automatic color temperature correction compared to the prior art. FIG. 7A illustrates the conventional AWB process and its results, and FIG. 7B illustrates the conventional AWB process and its results according to an embodiment of the present invention. This illustrates the AWB process and results based on artificial neural networks.

At this time, (a) in Figure 7a is an original photo of a landscape, showing an example of a landscape photo including a sky object (A) and a ground object (B), and (b) in Figure 7a (b) in Figure 7a ( The plank white curve for each object (A, B) for the original photo illustrated in a) is illustrated, and (c) and (d) of Figure 7a are the results of performing the conventional AWB, that is, the results corrected by the conventional AWB. As a result, Figure 7a (c) shows an example in which a blue filter is applied to match the sky color, and Figure 7a (d) shows an example in which a red filter is applied to match the ground color.

Referring to (c) and (d) of Figure 7a, during conventional automatic color temperature correction, when a blue filter is applied to match the color of the sky, the ground is also blue, and when a red filter is applied to match the color of the ground, the sky is also red. You can see that it has less similarity to the original photo.

Meanwhile, (a) in Figure 7b shows an example of a landscape photo including a sky object (A) and a ground object (B), as in (a) in Figure 7a, and (b) in Figure 7b shows ( The process of deriving a Plank White curve for each object (A, B) for the original photo shown in a) and then applying it to AWB (i.e., AWB NN) based on a color correction artificial neural network is illustrated, Figure 7b (c) is the result of performing ANN-based AWB, that is, the result corrected by ANN-based AWB, and shows an example in which different filters are applied according to the color temperature judgment result for each object. That is, (c) in Figure 7b shows an example of color temperature correction by applying different filters to each of the sky object (A) and the ground object (B).

In step S134, the noise removal/sharpness correction unit 40 removes noise included in the automatically color temperature corrected image while maintaining sharpness. That is, in step S134, the noise removal/sharpness correction unit 40 simultaneously identifies a low-light state with a lot of noise in the image and an outdoor state with little noise, and satisfies both low-light and high-light images through different noise removal and sharpness adjustments. to achieve results. To this end, in step S134, the noise removal/sharpness correction unit 40 applies the automatically color temperature corrected image to a noise removal/sharpness artificial neural network (NR/Shart NN), and uses By selecting and applying one of various filters that reduce noise while maintaining the details of the border, noise can be removed while maintaining the clarity of the image.

For example, during the backlight compensation process in step S132, areas where minimum brightness is not provided even in the brightest exposure cut are filled with rough noise values, and noise further increases during the tone mapping process. If you only perform processing to reduce or remove noise, the overall image will be smoother, but the sharpness may be lowered and the outlines may be blurred. Therefore, in step S134, the noise removal/sharpness correction unit 40 corrects the image by applying the NR/Shart NN to remove noise while maintaining sharpness.

FIG. 8 is a diagram for explaining the effects of noise removal and sharpness correction of the present invention compared to the prior art. FIG. 8 (a) shows two images (i.e., a high-frequency image and a low-frequency image) selected for noise removal and sharpness correction. (b) of FIG. 8 shows an example of the conventional noise removal and sharpness correction process 40a, and (c) of FIG. 8 shows the images illustrated in (a) of FIG. 8. An example of an image derived as a result of processing in the manner illustrated in (b) is shown, and (d) in Figure 8 shows the noise removal and sharpness correction process based on an artificial neural network (ANN) according to an embodiment of the present invention ( 40b) shows an example, and FIG. 8(e) shows an example of an image derived as a result of processing the images illustrated in FIG. 8(a) in the manner illustrated in FIG. 8(d).

Referring to (b) of FIG. 8, the conventional noise removal and sharpness correction process 40a is a process of removing noise from the images (i.e., high-frequency images and low-frequency images) illustrated in (a) of FIG. Reduction 41a, sharpness correction 42a, and strength control 43a of the processes 41a and 42a.

Meanwhile, referring to (d) of FIG. 8, the artificial neural network (ANN)-based noise removal and sharpness correction process (40b) according to an embodiment of the present invention is performed on the images illustrated in (a) of FIG. 8 (i.e. , a process of removing noise from high-frequency images and low-frequency images (Noise Reduction) (41b), a sharpness correction process (Sharpness) (42b), and a filter selection process ( Filter Selection) (46a), and may further include Scene Detection and Strength Control processes (43b, 44b, 45b) applying the NR/Shart NN. This is to accurately determine the boundary between noise removal and sharpness correction.

When the image is corrected through a series of processes (S131 to S134) illustrated in FIG. 5, in step S140, the multiple image signal processor 130 processes the corrected image output from the ToF sensor 111. The 3D distance information and the low-resolution thermal image output from the heat detection sensor 113 are combined. At this time, the specific processing process for combining the images can use known technologies.

In step S150, the AI accelerator 140 applies the pre-stored reference data and the image combined in step S140 to a preset artificial intelligence algorithm to recognize the image.

By doing this, the present invention integrates and processes signals from complex sensors in one processor, efficiently utilizing power and area, and thereby improving processing speed, making it possible to process large amounts of data in real time without data access delay. There are features that can be used.

In addition, by using complex sensor information, the present invention can overcome the limitations of systems that rely on a single image sensor, thereby providing sensed information in various application fields such as indoor workers in factories, firefighters at fire sites, and soldiers in operation. It can be used as an auxiliary means, expanding the field of use.

Although the embodiments of the present invention have been described above, the scope of the rights of the present invention is not limited thereto, and the present invention can be easily modified from the embodiments by those skilled in the art in the technical field to which the present invention belongs and is recognized as equivalent. Includes all changes and modifications to the extent permitted.

100: Integrated image processing device 110: Complex sensor unit
111: ToF sensor 112: Image sensor
113: heat detection sensor 120: input unit
130: Multiple video signal processing unit 131: Correction unit
10: Resolution correction unit 20: Backlight correction unit
30: Automatic color temperature correction unit 40: Noise removal/clearness correction unit
132: Image combining unit 140: AI accelerator
150: Standard data management DB

Claims (7)

In an integrated image processing device that improves image quality using complex sensor information,
an input unit that receives corresponding detection signals from each of the complex sensors including an image sensor, a ToF sensor, and a heat detection sensor;
A correction unit is provided to combine a plurality of signals input through the input unit and output the signal as a single signal, and to correct the image detected through the image sensor based on a preset artificial neural network (ANN) algorithm. a multi-image signal processor that combines and outputs a corrected image, 3D distance information output from the ToF sensor, and a low-resolution thermal image output from the heat detection sensor; and
AI that learns at least one pre-generated data set for each class to generate reference data and stores it in advance, and recognizes the image generated by the multi-video signal processor based on the reference data and a preset artificial intelligence algorithm Contains an accelerator,
The correction unit
a resolution correction unit that converts a low-quality image detected through the image sensor into a high-quality image; and
Backlight is corrected by receiving three or more images with different exposure times from the resolution correction unit and combining them into one image, and applying the three or more images to a preset backlight correction neural network (WDR NN) algorithm to create a correction gain curve (Contrast) An integrated image processing device comprising a backlight correction unit that generates a gain curve and then determines the ratio of the pixel image in the bright area to the pixel image in the dark area by reflecting the compensation gain curve in the contrast correction process. The method of claim 1, wherein the multiple video signal processing unit
An integrated image processing device comprising an image combiner that combines the corrected image, 3D distance information output from the ToF sensor, and a low-resolution thermal image output from the heat detection sensor. The method of claim 2, wherein the correction unit
By deriving a Planck White curve for each object included in the backlight-corrected image and applying the Planck White curve for each object to a preset color correction neural network (AWB NN) to determine the color temperature for each object, the object An integrated image processing device further comprising an automatic color temperature correction unit that automatically corrects the star color temperature. The method of claim 3, wherein the correction unit
The color temperature correction image transmitted from the automatic color temperature correction unit is applied to a preset noise removal/sharpness artificial neural network (NR/Sharp NN) to remove noise while maintaining the boundary line included in the color temperature correction image. An integrated image processing device further comprising a noise removal/clearness correction unit. An integrated image processing method in which an integrated image processing device equipped with a complex sensor including an image sensor, a ToF sensor, and a heat sensor improves image quality by using the complex sensor information,
A learning step of generating and storing reference data for image detection by learning at least one pre-generated dataset for each class;
An input step of receiving a corresponding detection signal from each of the complex sensors including the image sensor, the ToF sensor, and the heat detection sensor;
A correction step of correcting the image detected through the image sensor based on a preset artificial neural network (ANN) algorithm;
A combining step of combining the corrected image, the 3D distance information output from the ToF sensor, and the low-resolution thermal image output from the heat detection sensor into one; and
A recognition step of recognizing the image by applying the reference data and the combined image to a preset artificial intelligence algorithm,
The correction step is
A resolution correction step of converting a low-quality image detected through the image sensor into a high-quality image; and
Among the images processed in the resolution correction step, three or more images with different exposure times are received and combined into one image to correct backlight, and the three or more images are applied to a preset backlight correction neural network (WDR NN) algorithm. Integration comprising a backlight correction step of generating a contrast gain curve and then reflecting the contrast gain curve in the contrast correction process to determine the ratio of the pixel image in the bright area and the pixel image in the dark area. Image processing method. The method of claim 5, wherein the correction step is
By deriving a Planck White curve for each object included in the backlight-corrected image and applying the Planck White curve for each object to a preset color correction neural network (AWB NN) to determine the color temperature for each object, the object An integrated image processing method further comprising an automatic color temperature correction step for automatically correcting the star color temperature. The method of claim 6, wherein the correction step is
In the automatic color temperature correction step, the color temperature corrected image is applied to a preset noise removal/sharpness artificial neural network (NR/Sharp NN) to reduce noise while maintaining the boundary line included in the color temperature corrected image. An integrated image processing method further comprising a noise removal/clearness correction step.

Images