KR20190047243A - Apparatus and method for warning contamination of camera lens - Google Patents

Abstract

The present invention relates to an apparatus and a method for warning contamination of a camera lens which classify a contamination state of a camera lens by merging a space and a frequency domain. The apparatus for warning contamination of a camera lens comprises: an image obtainer for obtaining an image around a vehicle through at least one camera installed in the vehicle; a processor for classifying a contamination state of the camera lens by calculating a multi-domain deep neural network for the image obtained through the image obtainer; and an outputter for outputting a warning according to the contamination state by controlling the processor.

Description

TECHNICAL FIELD [0001] The present invention relates to a camera lens contamination warning device,

The present invention relates to a camera lens contamination warning apparatus and method for detecting contamination of a camera lens through spatial and frequency domain convergence.

2. Description of the Related Art Recently, an image processing system that provides a peripheral image of a vehicle using a camera has been widely applied so that a driver can conveniently check the surrounding situation of the vehicle and easily perform driving and parking. Such an image processing system may cause an error in the image processing if the camera lens is contaminated due to external factors such as weather or dust, so that the reliability of the system may be deteriorated.

Conventionally, in order to detect the contamination of the camera lens, most methods are to use the brightness of the image or to calculate the area of the same area in the moving image, or to use blur phenomenon or edge map of the image. However, this conventional technique is applicable only to a certain kind of pollutants or is limited to be applicable only to images obtained in a specific environment.

[Document 1] KR 1020150076371 A [Document 2] KR 101525516 B1 [Literature 3] US 20150009296 A1 [Document 4] US 9319637 B2 [Literature 5] US 8208043 B2 [Literature 6] US 20150177512 A1

The present invention provides a camera lens contamination warning device and method for classifying the contamination state of a camera lens through spatial and frequency domain convergence.

The present invention also provides a camera lens contamination warning device and method for warning a driver of contamination information.

According to an aspect of the present invention, there is provided an apparatus for alerting a camera lens contamination, comprising: an image acquirer for acquiring an image of a periphery of the vehicle through at least one camera mounted on the vehicle; A processor for performing a multinomial domain neural network operation on the image to classify the contamination state of the camera lens, and an output unit for outputting a warning according to the contamination state under the control of the processor.

Wherein the processor comprises: a pre-processing unit for pre-processing the acquired image; a multidomain-based neural network processing unit for performing the multidomain-based neural network operation on an image preprocessed by the preprocessing unit to calculate a probability value for each contamination state; And a contamination state determiner for determining a contamination state of the camera lens based on the probability value.

The multi-domain neural network processor includes a spatial domain feature extraction unit for extracting spatial domain feature values from the preprocessed image by performing a convolution operation and an integration operation, a DCT (Discrete Cosine Transform) convolution operation, a convolution operation, and an integration And extracting a frequency domain feature value from the preprocessed image, a fusion unit for performing a connection operation to fuse the spatial domain feature value and the frequency domain feature value, And an estimator for performing a Max operation and estimating a probability value for each contamination state based on the characteristic values fused by the fusion unit.

The frequency domain feature extraction unit may set an initial value of a weight of at least one convolution layer to a DCT matrix.

The pollution state determination unit determines a pollution state having the highest probability value among the probability values of the pollution states as the pollution state of the camera lens.

The processor includes a pollution state warning unit for determining whether a classified pollution state is present in a predetermined warning-required pollution state list to determine whether or not a warning is required and controlling the warning output.

And the pollution state warning unit outputs a warning indicating the pollution state if the classified pollution state is present in the preset warning required pollution state list.

Meanwhile, the camera lens contamination warning method according to an embodiment of the present invention includes the steps of acquiring an image through at least one camera mounted on a vehicle, pre-processing an image acquired through the at least one camera, Classifying the contaminated state of the camera lens by performing a multi-domain deep-neural network operation on the image, and outputting a warning according to the classified contamination state.

The pre-processing of the acquired image may include adjusting the size of the acquired image, and normalizing the pixel value of the acquired image.

The classifying the contamination state of the camera lens may include extracting a spatial domain characteristic and a frequency domain characteristic from the preprocessed image, and calculating a probability value for each contamination state based on the fused and merged characteristics.

The step of classifying the contamination state of the camera lens is characterized by extracting the frequency domain characteristic through a DCT convolution operation.

The classifying the contamination state of the camera lens may include classifying the pollution state having the largest probability value among the probability values of the pollution states into a contamination state of the camera lens.

The step of outputting the warning may include outputting a warning indicating the contamination state of the camera lens when the classified pollution state is included in the pre-set alarm-required pollution state list.

The present invention detects contamination of the camera lens through fusion of the spatial domain and the frequency domain, so that contamination can be detected without limitation to certain kinds of pollutants and a specific environment.

In addition, the present invention provides a driver with information on the contamination state by determining the contamination state of the camera lens, which can help the driver to determine whether the camera lens needs to be replaced or cleaned depending on the contamination state, The reliability of the image processing system to be used can be improved.

1 is a block diagram illustrating a camera lens contamination warning device according to an embodiment of the present invention;
2 is a block diagram of the pollution state classifying unit shown in FIG. 1;
3 is a block diagram showing the configuration of the preprocessing unit shown in Fig.
FIG. 4 is a block diagram showing a multi-domain deep-layer neural network processing unit shown in FIG. 2. FIG.
5 is a block diagram showing the configuration of the contamination state warning unit shown in Fig.
FIG. 6 is a diagram for explaining a calculation process for extracting frequency domain characteristics related to the present invention; FIG.
FIG. 7 is an exemplary diagram showing the structure of the frequency domain feature extraction unit shown in FIG. 4; FIG.
FIG. 8 is an exemplary diagram showing the structure of the multi-domain deep-layer neural network processor shown in FIG. 2. FIG.
FIG. 9 is a flowchart illustrating a camera lens contamination warning method according to an embodiment of the present invention. FIG.
10 is a diagram for explaining a multi-domain deep-layer neural network process according to the present invention.
11 is an exemplary view showing an alarm output according to the contamination state of the lens according to the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. It should be noted that, in adding reference numerals to the constituent elements of the drawings, the same constituent elements are denoted by the same reference symbols as possible even if they are shown in different drawings. In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the difference that the embodiments of the present invention are not conclusive.

In describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are intended to distinguish the constituent elements from other constituent elements, and the terms do not limit the nature, order or order of the constituent elements. Also, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

The present invention relates to a system for detecting a contamination state of a camera lens using a multi-domain deep-layer neural network and alerting the driver of the detected contamination state information.

FIG. 1 is a block diagram showing a camera lens contamination warning device according to an embodiment of the present invention. FIG. 2 is a block diagram of a pollution state classifying unit shown in FIG. 1, FIG. 4 is a block diagram showing the multi-domain deep-layer neural network processing unit shown in FIG. 2, FIG. 5 is a block diagram showing the configuration of the pollution status warning unit shown in FIG. 1, FIG. 6 is a diagram for explaining a calculation process for extracting frequency domain characteristics related to the present invention, FIG. 7 is an exemplary view showing a structure of the frequency domain feature extraction unit shown in FIG. 4, FIG. 3 is a diagram illustrating a structure of a multi-domain deep-layer neural network processing unit shown in FIG.

As shown in FIG. 1, the camera lens pollution warning device includes an image acquirer 100, a processor 200, and an output device 300.

The image acquirer 100 acquires images of the surroundings of the vehicle using at least one camera (s) mounted on the vehicle. For example, the image acquirer 100 may acquire external images (front image, rear image, and / or side image) around the vehicle through a front camera, a rear camera, and / or a side camera (side camera). The image acquirer 100 provides the acquired image as input data (input image) of the processor 200.

The camera (not shown) may be a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, a charge priming device (CPD) image sensor, and a charge injection device And may include any one of the image sensors. In addition, the camera (not shown) may include any one of a standard lens, a wide-angle lens, a fish-eye lens, and a telephoto lens.

The processor 200 performs multinomial domain neural network processing on the image acquired through the image acquirer 100 to detect the contamination state of the camera lens and warns according to the detected contamination state. The processor 200 may include a memory (not shown) internally and / or externally. A memory (not shown) may store a program for controlling the operation of the processor 200, setting information, and input / output data of the processor 200.

The processor 200 includes a pollution state classifying unit 210 and a pollution state classifying unit 210. The pollution state classifying unit 210 classifies the pollution state of the camera lens using the multi domain deep- And a warning unit 220.

As shown in FIG. 2, the pollution state classifying unit 210 includes a pre-processing unit 211, a multidomain-based neural network processing unit 212, and a pollution state determining unit 213.

The preprocessing unit 211 preprocesses the acquired image. The pre-processing unit 211 includes an image size adjusting unit 2111 and an image normalizing unit 2112 as shown in FIG.

The image size adjuster 2111 enlarges and / or reduces an image acquired through the image acquirer 100 to use the input image as an input to the multi-domain neural network processor 212. That is, the image size adjustment unit 2111 adjusts the size of the image to use the acquired image as input data of the multidomain-based neural network processing unit 212.

The image normalization unit 2112 normalizes the image scaled by the image size adjustment unit 2111. The image normalization unit 2112 can use any one of known normalization methods.

The image normalization unit 2112 can normalize the image using Equation (1).

Here, x (i, j) is the (i, j) th pixel value of the normalized image and x '(i, j) is the (i, j) th pixel value of the normalized image.

For example, when the image resized by the image size adjusting unit 2111 has pixel values of 0 to 255, the image normalizing unit 2112 normalizes each pixel value of the image to a value between -1 and 1.

The multi-domain neural network processing unit 212 performs a multi-domain neural network operation on an image pre-processed by the preprocessing unit 211. At this time, the final computation result of the multi-domain deep-neural network computation output from the multi-domain neural network processor 212 means a probability value for each pollution state. Here, the pollution state is classified into clean, translucent pollution, and opaque pollution.

Referring to FIG. 4, the multi-domain neural network processing unit 212 includes a spatial domain feature extraction unit 2121, a frequency domain feature extraction unit 2122, a fusion unit 2123, and an estimation unit 2124.

The spatial domain feature extraction unit 2121 performs a convolution operation and a pooling operation to extract spatial domain feature values from the preprocessed image. For example, the spatial domain feature extraction unit 2121 extracts the spatial domain feature extraction unit 2121 from one of the existing deep learning network structures VGG16 (Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." ArXiv preprint arXiv: 1409.1556, 2014 ). ≪ / RTI >

The frequency domain feature extraction unit 2122 is configured in parallel with the spatial domain feature extraction unit 2121. The frequency domain feature extraction unit 2122 extracts frequency domain features from the preprocessed image by performing discrete cosine transform (DCT) convolution, convolution, and integration operations.

DCT convolution in operation for extracting the feature of a frequency domain, the initial value of the weight w _DCT means the convolution operation with which a weight w _DCT learning by setting the DCT matrix (matrix), such as Equation (2) .

이고,here,

ego,

이다.

to be.

x and y are image coordinates on the spatial domain, and u and v are image coordinates on the frequency domain.

The frequency domain feature extraction unit 2122 performs batch normalization (BN) and hyperbolic tangent (tanh) operations after DCT convolution operation as shown in FIG.

Batch normalization is a method for solving the problem that DCT is difficult to learn because the component value of the low frequency band is larger than the component value of the high frequency band. Batch normalization plays a role of normalizing the low frequency band and the high frequency band to have component values of similar size.

The frequency domain feature extraction unit 2122 may be configured by replacing the convolution layer of the existing spatial domain feature extraction unit with a DCT convolution layer. For example, the frequency domain feature extraction unit 2122 can be implemented by replacing the first convolution layer of the VGG 16, which is one of the existing deep learning networks, with the DCT convolution layer as shown in FIG. In other words, in this embodiment, the frequency domain feature extraction unit 2122 is implemented by setting initial values of parameters (weights) of at least one convolution layer of the existing deep learning network to a DCT matrix.

The fusion unit 2123 performs a concatenation operation on the extracted spatial domain characteristic value and the frequency domain characteristic value to fuse. That is, the fusion unit 2123 fuses the extracted spatial domain characteristic and the frequency domain characteristic.

The estimating unit 2124 calculates the probability values of the contaminated states by performing fully connected and softmax operations using the feature values fused in the fusion unit 2123. The estimating unit 2124 estimates the probability of each contamination state based on the fused characteristic values.

The pollution state determiner 213 combines the pollution state list with the results of the multinomial deep-network neural network processor 212 to determine the pollution state. The pollution state determination unit 213 classifies the pollution state into a pollution state having the highest probability value among the pollution state-specific probability values calculated by the estimation unit 2124. [

The pollution state determination unit 213 determines a lens pollution state of the camera that has captured the input image of the processor 200 as the pollution state having the highest probability value among the probability values of the pollution states corresponding to the pollution state list.

The pollution state warning unit 220 includes a pollution warning determination unit 221 and a warning output control unit 222 as shown in FIG.

The pollution warning determining unit 221 determines whether the contamination state of the camera lens classified by the pollution state classifying unit 210 needs to warn the driver of the contamination state information through the warning output. The pollution warning determination unit 221 determines the warning output when the pollution state determined by the pollution state determination unit 213 is included in the alarm-required pollution state list set in advance, If it is not included, it determines the warning power.

The warning output control section 222 controls the pollution warning output in accordance with the determination of the pollution warning determination section 221. [ For example, the warning output control unit 222 controls to output only the time information or the time information and the audible information according to the determination of the contamination warning determination unit 221.

The output device 300 is for outputting information such as time information, auditory information, and / or tactile information, and may include a display, an audio output module, and a haptic module.

The display may be a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, (3D) display, a transparent display, a head-up display (HUD), a touch screen, and a cluster.

The sound output module can output audio data stored in a memory (not shown). The sound output module may include a receiver, a speaker, and / or a buzzer.

The haptic module outputs a signal of a type that the user can perceive by the tactile sense. For example, the haptic module can be implemented as a vibrator to control vibration intensity and pattern.

8, the frequency domain feature extraction unit 2122 is implemented by replacing the weight initial values of at least one convolutional layer of the existing spatial domain feature extraction unit 2121 with a DCT matrix, Domain feature extraction unit 2121 in parallel. Therefore, the camera lens pollution warning apparatus according to the present invention extracts both the spatial domain characteristic and the frequency domain characteristic from the image input from the image acquirer 100, and uses it to classify the pollution state, have.

FIG. 9 is a flowchart illustrating a camera lens contamination warning method according to an embodiment of the present invention. FIG. 10 is a view for explaining a multinomial-depth neural network process according to the present invention, FIG. 2 is a diagram illustrating an example of warning output according to a state; FIG. Here, the camera lens pollution warning device starts with the start of the vehicle and performs lens contamination detection regardless of whether the vehicle is running or not.

The processor 200 of the camera lens pollution warning device acquires the image (s) around the vehicle through the image acquirer 100 (S110). The image acquirer 100 provides an image taken in real time through a camera as input data of the processor 200.

The processor 200 preprocesses the image acquired through the image acquirer 100 (S120). The processor 200 scales and / or scales the acquired image for multi-domain deep-layer neural network processing. In addition, the processor 200 transforms each pixel value of the acquired image into a value between -1 and 1 to normalize the value.

The processor 200 performs multi-domain deep-neural network computation on the preprocessed image to classify the contamination state of the camera lens (S130). The processor 200 extracts spatial domain features and frequency domain features from the preprocessed image and fuses them. 10, the spatial domain feature extraction unit 2121 and the frequency domain feature extraction unit 2122 of the processor 200 extract the spatial domain feature and the frequency feature from the input image, respectively. The fusion unit 2123 of the processor 200 performs a concatenation operation to fuse the extracted spatial domain characteristic and the frequency domain characteristic. The processor 200 calculates a probability value for each contamination state based on the fused characteristic and classifies the contamination state of the camera lens into the contamination state having the largest probability value among the calculated probability values for each pollution state.

The processor 200 determines whether the classified contamination state requires warning (S140). The processor 200 checks whether the classified contamination status is present in the predetermined (predetermined) warning required contamination status list. The processor 200 determines a pollution warning output if the classified pollution condition is present in the pre-set alarm required pollution condition list. On the other hand, the processor 200 determines the pollution warning peak power if the classified pollution state is not present in the preset warning required pollution state list.

The processor 200 outputs a warning indicating that the camera lens is contaminated when a warning about the contamination state is required (S150). At this time, the output device 300 outputs at least one of the time information, the auditory information, and the tactile information according to the control of the processor 200.

On the other hand, if it is determined in S140 that the contamination warning is unnecessary, the processor 200 does not output a warning.

For example, as shown in FIG. 11, the processor 200 does not warn if the result of detecting the contamination state of the camera lens through the image acquired through the image acquirer 100 is normal. Meanwhile, the processor 200 outputs a warning indicating that the lens of the camera that captured the image is contaminated when the contamination by the pollutant having transparency such as the raindrop is detected as a result of analyzing the acquired image. In addition, the processor 200 outputs a warning indicating that the lens of the camera, which has photographed the image, is contaminated when contamination due to opaque contaminants such as mud is detected.

FIG. 11 shows an example in which the same warning is output when the camera lens is contaminated by translucent contaminants or opaque contaminants. However, the present invention is not limited to this.

As described above, the present invention can determine the contamination state of the lens using the image information acquired from the camera for the vehicle, and provide the driver with information on the contamination state.

Further, the present invention recognizes the necessity of replacing and cleaning the camera lens to the driver, thereby improving the reliability of the image processing system using the vehicle camera.

In addition, the present invention can help the driver to decide whether to replace or clean the lens or to determine the urgency of the cleaning according to the contamination state of the camera lens.

In addition, the present invention makes it possible to classify contaminants by using both the frequency property and the spatial property, unlike the existing methods using one characteristic of the contaminant using the multinomial domain neural network through fusion of the spatial domain and the frequency domain .

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention.

Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

100: image acquiring machine
200: processor
210: pollution state classification section
211:
2111:
2112: image normalization unit
212: Multi-domain deep-layer neural network processor
2121: Spatial domain feature extraction unit
2122: Frequency domain feature extraction unit
2123:
2124:
213: contamination state determination unit
220: Pollution status warning section
221: Pollution warning decision unit
222: a warning output control section
300:

Claims (13)

An image acquiring device for acquiring images of the surroundings of the vehicle through at least one camera mounted on the vehicle,
A processor for performing a multinomial domain neural network operation on the image acquired through the image acquirer to classify the contamination state of the camera lens, and
And an output unit for outputting a warning based on the contamination state under the control of the processor.
The method according to claim 1,
The processor includes:
A preprocessor for preprocessing the acquired image,
A multi-domain neural network processor for performing the multi-domain neural network computation on the preprocessed image in the preprocessing unit to calculate a probability value for each pollution state;
And a pollution state determiner for determining a pollution state of the camera lens based on the probability value for each pollution state.
3. The method of claim 2,
Wherein the multi-domain deep-
A spatial domain feature extraction unit for extracting spatial domain feature values from the preprocessed image by performing a convolution operation and an integration operation,
A frequency domain feature extraction unit for extracting a frequency domain feature value from the preprocessed image by performing a discrete cosine transform (DCT) convolution operation, a convolution operation,
A fusion unit for performing a connection operation to fuse the spatial domain feature value and the frequency domain feature value, and
And an estimator for performing a full connection operation and a soft max operation to estimate a probability value for each of the contamination states based on the feature values fused by the fusion unit.
The method of claim 3,
Wherein the frequency domain feature extraction unit comprises:
And sets an initial value of a weight of at least one convolution layer as a DCT matrix.
3. The method of claim 2,
The contamination state determination unit determines,
Wherein the pollution state of the camera lens is determined as a pollution state having a largest probability value among the probability values of the pollution states.
The method according to claim 1,
The processor includes:
And a pollution state warning unit for determining whether the classified pollution state is present in the pre-set alarm-required pollution state list and determining whether or not a warning is required and controlling the warning output.
The method according to claim 6,
The contamination state warning unit,
And if the classified pollution state exists in the pre-set alarm-required pollution state list, outputs a warning indicating the pollution state.
Acquiring an image through at least one camera mounted on a vehicle,
Pre-processing an image acquired through the at least one camera,
Classifying the contaminated state of the camera lens by performing multi-domain deep-neural network computation on the preprocessed image, and
And outputting a warning in accordance with the classified pollution state.
9. The method of claim 8,
Wherein the pre-processing of the acquired image comprises:
Adjusting the size of the acquired image, and normalizing the pixel value of the acquired image.
9. The method of claim 8,
The step of classifying the contamination state of the camera lens comprises:
Extracting a spatial domain characteristic and a frequency domain characteristic from the preprocessed image, and calculating a probability value for each of the pollution states based on fused and fused characteristics.
11. The method of claim 10,
The step of classifying the contamination state of the camera lens comprises:
DCT convolution operation to extract the frequency domain characteristic.
11. The method of claim 10,
The step of classifying the contamination state of the camera lens comprises:
And the pollution state having the largest probability value among the probability values by the pollution state is classified into the pollution state of the camera lens.
9. The method of claim 8,
The step of outputting the warning may include:
And outputting a warning indicating the contamination state of the camera lens when the classified pollution state is included in the preset alarm required pollution state list.

Images