KR102625104B1 - Camera Image Compensation Apparatus and Method Thereof - Google Patents

Abstract

An image acquisition unit that acquires a plurality of images including at least one unpolarized visible light image and at least one polarized image of an object, an image recombination unit that recombines at least a portion of the plurality of acquired images, and the recombined image Based on this, a machine-learned boundary value extractor uses an artificial neural network model to extract the boundary value of the target object in the image, and restoration of objects lost or distorted in the unpolarized visible light image based on the output boundary value. An image processing device including an image optimization unit that performs is disclosed.

Description

Image processing apparatus and method for performing camera image compensation {Camera Image Compensation Apparatus and Method Thereof}

The present invention relates to an apparatus and method for improving loss and distortion of camera image information due to reflected light, glare, camera lens contamination, etc.

Cameras are widely used in daily life, and photography takes place in a variety of internal and external environments. Therefore, camera image information may be lost or distorted due to reflected light, glare, contamination of the camera lens (e.g., water droplet contamination), etc.

For example, in the case of in-vehicle camera images, road markings such as lanes may not be recognized as reflected light in the rain, as shown in FIG. 1. Black box and self-driving mobility have high utilization of camera images and are related to safety and/or security, so the importance of camera images is very high. For example, if the camera image of an autonomous vehicle cannot recognize lanes due to reflected light from rain, the autonomous driving function may not be properly performed and there is a possibility of a traffic accident occurring.

In many other fields, loss and distortion of camera image information is a problem in itself and can cause functional deterioration in various applications using cameras.

Accordingly, the need for a method and device to restore the original information by correcting loss and distortion of camera image information due to reflected light, glare, contamination of the camera lens, etc. is increasing.

The purpose of the present invention is to provide an apparatus and method for restoring original information by correcting loss and distortion of camera image information due to reflected light, glare, contamination of the camera lens, etc.

In order to achieve the above-described object, the image processing apparatus of the present invention includes an image acquisition unit that acquires a plurality of images including at least one unpolarized visible light image and at least one polarized image of an object, and the plurality of acquired images. an image recombination unit that recombines at least a portion of and an image optimization unit that restores objects that are lost or distorted in the non-polarized visible light image.

An image processing device according to another aspect of the present invention includes an image acquisition unit that acquires at least one unpolarized visible light image and at least one polarized image of an object, and an image recombination unit that recombines at least some of the acquired plural images. and a boundary value extraction unit that extracts the boundary value of the target object in each image based on the image transmitted from the image recombination unit, and a majority vote method based on the boundary value of the target object in each output image. It includes an image optimization unit that first determines a boundary value and finally determines the boundary value of the target object through secondary fusion of a small number of images that were not considered in the first decision.

An image processing method according to another aspect of the present invention includes an image acquisition step of acquiring a plurality of images including at least one unpolarized visible light image and at least one polarized image of an object, and one of the plurality of acquired images. An image reassembly step of recombining at least a portion of the image, a boundary value extraction step of extracting a boundary value of a target object in the plurality of images, and a search for an object lost or distorted in the unpolarized visible light image based on the output boundary value. It includes an image optimization step for restoration.

According to the present invention, it is possible to recognize necessary information in the image by correcting loss and distortion of camera image information due to reflected light, glare, contamination of the camera lens, etc. and restoring part or all of the lost information.

In particular, in the field of mobility, it is possible to recognize road markings such as lanes by improving the loss and distortion of image information caused by reflected light, glare, etc. that can be encountered in driving situations as shown in Figure 2, and image information while driving. It can increase reliability and contribute to improving driving situation recognition ability.

Figure 1 is an example diagram showing an example in which at least part of camera image information is lost or distorted due to reflected light, glare, contamination of the camera lens, etc.
Figure 2 is an example diagram showing an example of restoring camera image information in which at least part of the image is lost or distorted due to reflected light, glare, contamination of the camera lens, etc. according to the present invention.
Figure 3 is a configuration diagram of an image processing device according to the present invention.
Figure 4 is a conceptual diagram for explaining recombination of camera input images according to the present invention.
Figure 5 is a flowchart of the image processing method according to the present invention.

Hereinafter, the configuration of the present invention will be described in detail with reference to several preferred embodiments. However, this is not intended to limit the present invention to specific embodiments, and the scope of the present invention should be understood to include all changes, modifications, equivalents, or substitutes falling within the technical spirit of the present invention as conceived from the embodiments. do.

Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. The term “and/or” includes any of a plurality of related stated items or a combination of a plurality of related stated items.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. Only when a component is said to be "directly connected" or "directly connected" to another component should it be understood that there are no other components in between.

The terms used in this application are merely used to describe the presented embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of other features, numbers, or steps. , it should be understood that this does not exclude in advance the possibility of the presence or addition of operations, components, parts, or combinations thereof.

All terms used in this specification, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains, and are not idealized or excessively reduced formal terms. It should not be interpreted as meaning, and when the meaning of a term is defined in this specification, the term should be interpreted as defined.

In order to facilitate overall understanding when describing the present invention, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

The image processing device according to the present invention additionally acquires a polarized image in addition to the non-polarized visible light image and restores or augments the lost or distorted original visible light image through image recombination using the polarized image.

For this purpose, as shown in FIG. 3, the image processing apparatus 100 according to the present invention includes an image acquisition unit 110 that acquires multiple images of the object, and an image recombination function that selects and combines at least some of the acquired multiple images. Unit 120, a boundary value extraction unit 130 that is machine-learned to output the boundary value of the target object in the image using an artificial neural network model based on the recombined image, and finally image optimization and decomposition through the boundary value extraction output value. It includes an image optimization unit 140 that restores objects that are lost or distorted in a polarized visible light image.

The image acquisition unit 110 includes a polarized light camera and a non-polarized visible light camera. Other thermal imaging cameras and infrared cameras may also be included.

The cameras that make up the image acquisition unit can be arranged in various ways, such as vertically or horizontally. It is assumed that the placed cameras are fixed, and when the arrangement is changed, the physical calibration information between cameras changes.

Each polarized/unpolarized camera can be one or multiple.

In the following preferred embodiment, the description will be based on a polarization camera that simultaneously extracts polarization angles in four directions: 0 degrees, 45 degrees, 90 degrees, and 135 degrees from one camera. However, it is not limited to this, and of course, a polarization camera that extracts other polarization angles can be used.

Additionally, the polarization camera can be composed of multiple cameras for each spectrum (for example, a 660nm polarization camera, a 470nm polarization camera, etc.).

For all input images, image registration is performed based on the above-described correction information based on one camera, and images for which image registration has been completed based on one camera are defined as input images.

Meanwhile, secondary registration can be further performed through image conversion on the input image. Even if the physical registration of the cameras is perfect, the registration between images may not be accurate due to parameters that occur stochastically due to lens distortion, etc. Taking this into consideration, the degree of matching can be further improved by selectively performing secondary matching.

Looking at a specific example of quadratic registration, first, an affine transformation can be used as a linear transformation (rigid registration). Affine transformations include translation, scaling, rotation, and shearing. In the case of general camera image registration, only rotation and translation are performed in most cases, and results that can be used in most applications are derived.

In addition, in order to expect more detailed (pixel-level) and accurate registration results, non-rigid registration was additionally performed instead of normal camera registration, using b-spline and optical flow. Image registration can be done using (optical flow).

Non-rigid transformation can reflect geometric transformation, but the registration result is greatly affected by the initial state of registration. Although it can achieve good registration for transformation of detailed regions, when applied in cases where broadband registration is not achieved, details Since the registration is limited to an area, a lot of distortion may occur in the registration of the larger frame. So, in the first step, the image to be matched is first aligned in the large frame through linear transformation within the image, and then in the second step, the detailed parts are matched through non-rigid transformation.

The input image may be a snapshot at a certain point in time, but may be acquired to become a time series image over time. In this case, during image recombination, which will be described later, recombination is also performed on the front and rear frames within a predetermined range, which will be explained in detail later.

Meanwhile, an image preprocessor (not shown) for data augmentation of the input image may be further included. In this embodiment, scaling is used as a method of data augmentation. When the image recombination unit 120 and the boundary value extraction unit 130 recombine and learn images, it is desirable to use as many features of the target object as possible. However, the distance between the host vehicle and the target object is long, so the target object is not present in the acquired image. If it is expressed in a small size, the discriminative features of the target object cannot be recognized during image learning, so from an image perspective, the target object's features are less contained compared to the image. In this case, the enlarged image is learned through image enlargement in the image preprocessor so that a greater proportion of the features of the target object in the image are included.

The image recombination unit 120 recombines images acquired through each camera.

In reality, the recombined images are polarized images.

As shown in Figure 4, polarized images and non-polarized images have different combining methods. Figure 4 is a diagram for explaining the image recombination method according to the present invention.

As shown, the unpolarized visible light image has no change in channel configuration from the original image. Constructs a recombinant image with three channels that retains the original R, G, and B channels. Images from non-polarized visible light cameras, infrared cameras, and thermal imaging cameras can be used in their original form without being reconstructed.

In the case of a polarization image, which is an image acquired through a polarization camera, it is recombined into three channels: a polarization image, AoLP (Angle of Linear Polarizatio) image, and DoLP (Degree of Linear Polarizati) image.

At this time, the polarization image, AoLP, and DoLP of the spectrum are combined for each wavelength. For example, a recombinant image with 3 channels is obtained using the polarization image, DoLP, and AoLP of a 660 nm polarization camera, and the same method is obtained for a 470 nm polarization camera.

The type and number of wavelengths of the polarization camera can be configured in various ways, and of course, the optimal configuration can be taken through experiment or simulation for each application (mobility field such as autonomous driving, outdoor indoor/outdoor surveillance, security field, etc.).

Time-series images are recombined taking temporal order into account. However, the frames of the time series video used are limited to those where the movement of the object occurred within a predetermined range. That is, the same object based on the same camera is included in recombination only when it moves less than x pixels based on the center of gravity pixel, and x pixels can be set to, for example, a pixel less than 0.001% of the image resolution. That is, in the case of 2M video, a movement of 10 pixels is considered to have object connectivity, and multiple frames corresponding to it can be used for recombination. In this case, the input image for inference in the boundary value extraction unit, which will be described later, increases, allowing image correction with higher accuracy.

The boundary value extractor 130 uses the recombined image and the original input image to generate information to improve an image that is lost or distorted due to backlight, etc. The image generated by the image recombination unit 120 is received as input, the border of the object included in each of the plurality of images is generated, and the boundary value of the object is extracted. Here, the object may be a dynamic (moving) or static (non-moving) object that can be encountered in a road driving situation, such as a car, pedestrian, traffic light, road fence, or tree in an autonomous vehicle video.

The boundary value extraction unit 130 may be composed of an AI neural network model, and the input image of the artificial neural network is the original recombinant image and unpolarized visible light image constructed through an image recombination process, and may further include original infrared and thermal camera images. there is.

In order to extract boundary values from the recombinant image and the original image, a target object must be selected, and the target object can be selected by the user depending on the purpose. In a preferred embodiment of the present invention, examples can be presented for the purpose of autonomous mobility, and the examples can be selected as vehicles, two-wheeled vehicles, pedestrians, traffic lights, signs, road signs, etc., and segmentation of each object can be added.

Learning is conducted first to extract the boundary value for the selected object, and the learning data is trained with polarized and unpolarized images using the object's boundary value information as the correct answer (GT: ground truth) in the undistorted image. do.

The artificial neural network can use CNN, which is widely used in learning various two-dimensional data such as object classification and object detection in images, but is of course not limited to this.

As mentioned above, the process of preprocessing the image may be preceded for data augmentation.

As another embodiment, the boundary value may be extracted from each final input image using an existing boundary value extraction algorithm without using an AI neural network and transmitted to the image optimization unit 140.

The image optimization unit 140 fuses boundary value information (segmentation) of the target object detected using an artificial neural network in the recombined image through the following process.

1) Primary information fusion: Primary information fusion is performed using majority voting. In other words, multiple results that produce the same result are adopted from a recombinant image of a polarized image in one or more wavelength bands and an unpolarized visible light image. Optionally, images from an infrared camera and a thermal imaging camera can be used together.

Meanwhile, since the main purpose of the present invention is to extract non-visible information missed by visible light cameras, the boundary value obtained through majority voting must be included, and the combination that contains the most unselected polarization information is used. It is desirable to select in secondary information fusion.

2) Secondary information fusion: Selection is made from information not selected in the primary information fusion (information not included in the primary information fusion), but does not include all of the information not included in the primary fusion. Includes some of them. Some of the selection criteria included are combinations that contain the most polarization information, that is, combinations of those with DoLP above a certain level.

Therefore, secondary information fusion is performed including boundary value information of the target object in the range where the DoLP in the image is above a certain threshold. Usually, the DoLP standard threshold for outdoor images is 0.5, and only recombinant images with a DoLP of 0.5 or higher are fused. Of course, other optimal values can be obtained through experiment or simulation and applied as the threshold.

3) The union of the first and second information is defined as the final object boundary value information, the difference between the secondary information fusion result and the first information fusion result is defined as the lost area, and the polarization image corresponding to that area is defined as the missing area. The final image is output by including it in the unpolarized image.

4) When calculating the final image by merging the polarized image and the non-polarized image, the registration information used in the above-mentioned image registration between the two types of images can be used. Alternatively, boundary value information between two types of images can be used.

The corrected image output by the image optimization unit 140 can be delivered to the vehicle's lane keeping system and autonomous driving system to be used for vehicle safety or driving, and can also be delivered to the display unit and provided to the driver.

Hereinafter, an image processing method according to a preferred embodiment of the present invention will be described with reference to FIG. 5.

As shown in FIG. 5, an unpolarized visible light image and a polarized image are acquired from one or more polarized cameras and an unpolarized visible light camera (S110). In addition, thermal and infrared images can also be acquired through thermal and infrared cameras.

Polarization images are acquired through a polarization camera that simultaneously extracts polarization angles in four directions: 0 degrees, 45 degrees, 90 degrees, and 135 degrees, using multiple cameras for each spectrum (e.g., 660 nm polarization camera, 470 nm polarization camera, etc.) Therefore, it is desirable to obtain polarization images for each spectrum.

A plurality of cameras are physically corrected according to physical calibration information between cameras according to installation, and all input images are image registered based on the above-mentioned calibration information based on one camera (S120). Images for which image registration has been completed based on one camera are secured as the final input images for each camera (S130). The input image may be a snapshot at a certain point in time, but may include a plurality of image frames for the same location at a predetermined time to become a time series image according to time.

Once the final input image for each camera is secured, the acquired images are recombined through the image recombination unit 120 (S140).

In the case of a polarization image, which is an image acquired through a polarization camera, it is recombined into three channels: a polarization image, AoLP (Angle of Linear Polarizatio) image, and DoLP (Degree of Linear Polarizati) image.

At this time, the polarization image, AoLP, and DoLP of the spectrum are combined for each wavelength. For example, a recombinant image with 3 channels is obtained using the polarization image, DoLP, and AoLP of a 660 nm polarization camera, and the same method is obtained for a 470 nm polarization camera.

The type and number of wavelengths of the polarization camera can be configured in various ways, and of course, the optimal configuration can be taken through experiment or simulation for each application (mobility field such as autonomous driving, outdoor indoor/outdoor surveillance, security field, etc.).

Time-series images are recombined taking temporal order into account. However, the frames of the time series video used are limited to those where the movement of the object occurred within a predetermined range. That is, the same object based on the same camera is included in recombination only when it moves less than x pixels based on the center of gravity pixel, and x pixels can be set to, for example, a pixel less than 0.001% of the image resolution. That is, in the case of 2M video, a movement of 10 pixels is considered to have object connectivity, and multiple frames corresponding to it can be used for recombination. In this case, the input image for inference in the boundary value extraction unit, which will be described later, increases, allowing image correction with higher accuracy.

Next, the recombined image and the original input image are used to generate information to improve images that are lost or distorted due to backlight, etc. Specifically, the recombined image is received as input, the border of the object included in each of the plurality of images is generated, and the boundary value of the object is extracted (S150).

Here, the object may be a dynamic (moving) or static (non-moving) object that can be encountered in a road driving situation, such as a car, pedestrian, traffic light, road fence, or tree in an autonomous vehicle video.

The image input to the artificial neural network is the original recombinant image and unpolarized visible light image composed through an image recombination process, and may further include original infrared and thermal imaging camera images.

In order to extract the boundary value of an object from the recombinant image and the original image, a target object must be selected, and the target object can be selected by the user depending on the purpose. In a preferred embodiment of the present invention, examples can be presented for the purpose of autonomous mobility, and the examples can be selected as vehicles, two-wheeled vehicles, pedestrians, traffic lights, signs, road signs, etc., and segmentation of each object can be added.

Learning to extract the boundary value for the selected object is preceded, and the learning data is trained with polarized and unpolarized images by using the object's boundary value information as the correct answer (GT: ground truth) in the undistorted image.

The artificial neural network can use CNN, which is widely used in learning various two-dimensional data such as object classification and object detection in images, but is of course not limited to this.

Next, the boundary value information (segmentation) of the target object detected using an artificial neural network in the recombined image is fused and optimized through the following procedure (S160).

1) Primary information fusion: Primary information fusion is performed using majority voting. In other words, multiple results that produce the same result are adopted from a recombinant image of a polarized image in one or more wavelength bands and an unpolarized visible light image. Optionally, images from an infrared camera and a thermal imaging camera can be used together.

Meanwhile, since the main purpose of the present invention is to extract non-visible information missed by visible light cameras, the boundary value obtained through majority voting must be included, and the combination that contains the most unselected polarization information is selected. It is desirable to select from secondary information fusion.

2) Secondary information fusion: Selection is made from information not selected in the primary information fusion (information not included in the primary information fusion), but does not include all of the information not included in the primary fusion. Includes some of them. Some of the selection criteria included are combinations that contain the most polarization information, that is, combinations of those with DoLP above a certain level.

Therefore, secondary information fusion is performed including boundary value information of the target object in the range where the DoLP in the image is above a certain threshold. Usually, the DoLP standard threshold for outdoor images is 0.5, and only recombinant images with a DoLP of 0.5 or higher are fused. Of course, other optimal values can be obtained through experiment or simulation and applied as the threshold.

3) The union of the first and second information is defined as the final object boundary value information, the difference between the secondary information fusion result and the first information fusion result is defined as the lost area, and the polarization image corresponding to that area is defined as the missing area. The final image is output by including it in the unpolarized image.

4) When calculating the final image by merging the polarized image and the non-polarized image, the registration information used in the above-mentioned image registration between the two types of images can be used. Alternatively, boundary value information between two types of images can be used.

The corrected image output through the image optimization step (S160) can be transmitted to the vehicle's lane keeping device and autonomous driving device to be used for vehicle safety or driving, and can also be transmitted to the display unit and provided to the driver.

Although the configuration of the present invention has been described in detail with reference to preferred embodiments of the present invention, this is only an example and various modifications and changes are possible within the scope of the technical idea of the present invention. Therefore, the scope of protection of the present invention should be determined by the description of the patent claims below.

Claims (20)

an image acquisition unit that acquires a plurality of images including at least one unpolarized visible light image and at least one polarized image of the object;
an image reassembly unit that recombines at least some of the acquired plurality of images;
A boundary value extraction unit that is machine-learned to extract the boundary value of the target object in the image using an artificial neural network model based on the recombined image;
The boundary value of the target object is first determined based on the boundary value of the target object in each output image, and the boundary value of the target object is determined through secondary fusion of a small number of images that were not considered in the first decision. Video optimization department makes the final decision
An image processing device including a. The method of claim 1, wherein the image acquisition unit,
An image processing device comprising one or more polarized cameras and a non-polarized visible light camera. The method of claim 2, wherein the polarization camera:
An image processing device that simultaneously extracts polarization angles in four directions: 0 degrees, 45 degrees, 90 degrees, and 135 degrees. The method of claim 2, wherein the polarization camera:
An image processing device consisting of a plurality of cameras for each spectrum. The method of claim 1, wherein the image recombination unit,
An image processing device for recombining one or more of the polarized images acquired by the image acquisition unit into three channels: a polarized image, AoLP, and DoLP. The method of claim 1, wherein the boundary value extractor,
Image processing that receives the unpolarized visible light image and the polarized recombined image transmitted from the image recombination unit as input, generates the border of the object included in each of the unpolarized visible light image and the polarized recombined image, and extracts the boundary value of the object. Device. The method of claim 1, wherein the image optimization unit,
An image processing device that fuses boundary value information in each image detected by the boundary value extraction unit. an image acquisition unit that acquires at least one unpolarized visible light image and at least one polarized image of the object;
an image recombination unit that recombines at least some of the acquired multiple images;
a boundary value extraction unit that extracts a boundary value of a target object in each image based on the image transmitted from the image recombination unit;
The boundary value of the target object is first determined by majority voting based on the boundary value of the target object in each output image, and the target object is determined through secondary fusion of a small number of images that were not considered in the first decision. Image optimization unit that finalizes the boundary value
An image processing device including a. The method of claim 8, wherein the image recombination unit,
An image processing device for extracting DoLP of the polarization image. The method of claim 9, wherein the image optimization unit,
An image processing device that selects an image to be used for secondary fusion based on the DoLP of the polarization image. A plurality of images of an object are acquired using a plurality of cameras, and image acquisition is performed by performing physical correction to correct physical differences between the plurality of cameras and image registration of the plurality of images to obtain a final input image. wealth,
an image reassembly unit that recombines at least a portion of the final input image;
a boundary value extractor that outputs a boundary value of a target object in each image from the final input image;
The boundary value of the target object is first determined based on the boundary value of the target object in each output image, and the boundary value of the target object is determined through secondary fusion of a small number of images that were not considered in the first decision. Image optimization unit that makes the final decision and reflects it on the non-polarized visible light image
An image processing device including a. The method of claim 11, wherein the image acquisition unit,
An image processing device that obtains an input image by performing physical correction to perform image matching on all acquired images based on one camera, based on the physical arrangement information of the plurality of cameras. The method of claim 12, wherein the image acquisition unit,
After the physical correction, the image processing device obtains the final input image by performing two-stage registration between the input images, linear transformation and non-rigid transformation, on the input image. The method of claim 11, wherein the image acquisition unit,
An image processing device that acquires time series images over time. The method of claim 14, wherein the image recombination unit,
An image processing device that performs recombination on images in which the center of gravity pixel for a specific object has moved within a predetermined range with respect to the time series images. According to clause 11,
An image processing device further comprising an image preprocessor that performs scale transformation for data augmentation of the input image data. An image acquisition step of acquiring a plurality of images including at least one unpolarized visible light image and at least one polarized image of the object;
An image reassembly step of recombining at least some of the acquired plurality of images;
A boundary value extraction step of extracting boundary values of target objects in the plurality of images,
The boundary value of the target object is first determined based on the boundary value of the target object in each output image, and the boundary value of the target object is determined through secondary fusion of a small number of images that were not considered in the first decision. An image optimization step of restoring objects lost or distorted in the non-polarized visible light image based on the final and determined boundary value.
An image processing method including. The method of claim 17, wherein the image acquisition step includes:
An image processing method comprising performing physical correction for correcting physical differences between a plurality of cameras with respect to the plurality of images, and performing image registration for the plurality of images. The method of claim 17, wherein the image reassembly step includes:
An image processing method comprising extracting DoLP from the polarization image. The method of claim 17, wherein the image optimization step is:
An image processing method comprising the step of finally determining the boundary value of the target object based on the extracted boundary value and reflecting it in an unpolarized visible light image.