JP5879219B2 - In-vehicle environment recognition system - Google Patents

Description

  The present invention relates to an in-vehicle environment recognition device that recognizes an object based on an image captured by a camera.

  Conventionally, a system that uses a camera dedicated to image recognition has been the mainstream as a vehicle-mounted camera, but in recent years, a system that uses both a camera for presenting video to a user and a camera for image recognition is increasing. Since it is not a dedicated camera for image recognition, a camera installed outside the passenger compartment may be used for image recognition, and it can detect dirt and raindrops and accurately check whether the current lens condition is available for image recognition. The technology to judge is important.

  Patent Document 1 relates to a lane departure warning device that detects a white line on a road using a camera even when there is snow, fog, rain, nighttime light reflection, and the like. It is disclosed that the threshold for partition line recognition is changed as appropriate.

JP 2003-44863 A

  The conventional method improves environmental resistance performance by adjusting recognition parameters according to the appearance of the lane mark on the image when the lane mark state on the road surface is monitored by the in-vehicle camera.

  However, with the conventional method, it was possible to determine that the recognition target object such as a lane mark on the image was not suitable for image recognition. However, whether the road surface is rough or the ambient light source environment It was difficult to determine the main cause whether the lens was dark or the lens was dirty.

  An object of the present invention is to provide an in-vehicle environment recognition device that identifies a factor that makes a recognition object difficult to see and, when the factor can be removed, recognizes the recognition object after removing the factor.

In order to solve the above problems, an in-vehicle environment recognition apparatus according to the present invention includes an imaging unit that acquires an image captured by a camera provided behind the vehicle, a light source environment estimation unit that acquires exposure information from the image, and an image. A feature amount extraction unit that extracts a feature amount from the image, a state determination unit that determines a lens dirt state of the camera based on the relationship between the feature amount and exposure information, and execution of a recognition application based on the lens dirt state and processing the propriety determining, based on the state of the lens contamination, a process of controlling the removal of the lens dirt, possess and the application control unit, a having a feature amount, the image in the background region and the road surface area In this case, at least one feature quantity of a background edge strength feature quantity indicating the magnitude of the brightness difference of the background area and a road edge strength feature quantity showing the magnitude of the brightness difference of the road area is included.

  INDUSTRIAL APPLICABILITY The present invention can provide a vehicle-mounted environment recognition device that identifies a factor that makes a recognition object difficult to see and, when the factor can be removed, recognizes the recognition object after removing the factor.

It is a figure which shows the structure of the vehicle-mounted environment recognition apparatus provided with a lens state determination function. It is a figure which shows the structure of a light source environment estimation part. It is a figure which shows the structure of a feature-value extraction part. It is a figure which shows the structure of a state determination part. Application control unit App execution part It is explanatory drawing of road surface edge strength analysis and background edge strength analysis. It is a figure which shows a brightness | luminance change no area | region. It is a figure which shows the dirt shape analysis method. It is a figure which shows the correlation of background edge intensity | strength and exposure information, and edge intensity fall at the time of cloudiness. It is a figure which shows the time series data of lens cloudiness. It is a figure which shows the flow of the cloudiness determination of a lens state determination part. It is a figure which shows exposure information. It is a figure which shows the flow of the brightness | luminance fixation determination of a lens state determination part. It is a figure which shows the flow of a dirt shape analysis of a lens state determination part.

<Description of FIG. 1>
FIG. 1 is a diagram illustrating a configuration of an in-vehicle environment recognition device 10 having a lens state determination function. The in-vehicle environment recognition apparatus 10 acquires an image captured by an in-vehicle camera provided at the rear of the vehicle by the imaging unit 100. The light source environment estimation unit 200 determines the light source environment around the current camera based on camera exposure information such as shutter speed, aperture, exposure area average brightness, gain, and information on high brightness areas on the image. presume. The feature amount extraction unit 300 extracts an instantaneous value of a feature amount used for determination of the lens state or road surface state of the camera from the image by performing edge strength analysis, luminance analysis, shape analysis, and the like.

  The state determination unit 400 uses the relationship (relationship) between the feature amount extracted by the feature amount extraction unit 300 and the exposure information acquired by the light source environment estimation unit 200 to determine the lens state (lens contamination of the camera). State), background visibility, road surface condition, and the like. From these results, adjustment of parameters for suppressing erroneous detection and non-detection of the application is performed depending on the lens state. If the lens condition further deteriorates, a lens dirt removal process is executed, and if the lens condition further deteriorates, a fail determination is performed, which means that the system recognizes image recognition. If it is determined that the light source environment, the background visibility, and the road surface state are poor, it is understood that the image is temporarily difficult to recognize, and the application control unit 500 performs a response corresponding to the situation. The application execution unit 600 executes an image recognition application based on the adjustment result executed by the application control unit 500.

<Figure 2 Light source environment estimation unit Explanation>
FIG. 2 is a diagram illustrating a configuration of the light source environment estimation unit 200. The light source environment estimation unit 200 estimates information on the light source environment around the vehicle. As the vehicle information, the light source environment estimation unit 200 may manage the information including ON / OFF of the headlight of the host vehicle, the blinker, the brake lamp, and the like.

  First, exposure information of the vehicle's camera is managed by the exposure information unit 210. When the in-vehicle camera automatically controls the shutter speed and gain in accordance with the ambient light source environment, these information itself can be used as one index indicating the ambient brightness. For example, when the surroundings are bright, the camera increases the shutter speed so that it does not fly out. On the other hand, when the surroundings are dark, the shutter speed is slowed down. . Since the exposure time cannot be made longer than the frame rate of the image that can be obtained, adjustment is made so that the image is brightened by gain correction. That is, the darker the shutter speed is slower and the gain for correcting the image brightness tends to be higher. Also, the luminance information itself of the frame used for exposure is also used as useful information for knowing the luminance change around the exposure frame. Using such shutter speed and gain information obtained from the camera, daytime and nighttime determinations, and the brightness of the ambient light source environment are digitized and used for analysis of lens conditions.

  The image high brightness information unit 220 checks whether a high brightness area exists in the image. In particular, the scene used for analysis is, for example, in the case of a western day, when there is a reflection of the road surface on the western side, etc. Judgment with road surface reflection is performed. In the case of night, the image high brightness information unit 220 also grasps the surrounding light sources that flow in the background, such as street lights and store lights. Furthermore, since the headlight information of other vehicles is also a false detection factor for application, lane recognition, etc., in order to grasp the position information of the following vehicle and the vehicle traveling in the adjacent lane for grasping the position of the other vehicle. In addition, an image high brightness area is extracted, and the state determination unit 400 uses the information.

<Figure 3 Feature amount extraction unit Explanation>
FIG. 3 is a diagram illustrating a configuration of the feature amount extraction unit 300. The feature amount extraction unit 300 extracts each feature amount in order to perform road edge strength analysis, background edge strength analysis, luminance analysis, shape analysis, and the like.

  FIG. 7 is an explanatory diagram of edge strength analysis for the road surface edge strength analysis unit 310 and the background edge strength analysis unit 320.

  As shown in FIG. 7, the road surface edge strength analysis unit 310 sets the processing region left RDL and the processing region right RDR at positions assumed to include the white line on the road surface, and histograms the edge strength in the processing region. Thus, a road surface edge strength feature amount indicating the magnitude of the luminance difference of the road surface region is extracted. Thus, the edge strength including the white line on the road surface is analyzed. As shown in the graph of FIG. 7, if a histogram is formed based on the strength of the edge strength, and the average value of the edge strength is taken, the average edge strength in the processing region can be obtained. By dividing the processing area into left and right, it can be used as an index of the visibility of the left and right white lines for each of the left and right white lines to be recognized.

  The processing area position for analyzing the road edge strength is dynamically changed so that the white line and the road surface are included based on the result of the lane recognition. The edge strength may be analyzed. By filtering in the time direction, the visibility of the white line is analyzed as an index of the visibility of the white line in the current camera in a state including lens dirt. However, this visibility does not know whether the white line on the road surface is difficult to see due to rubbing, whether the lens is dirty and difficult to see, or whether the white line is dark and difficult to see. Therefore, the white line visibility is used as one index indicating the visibility of the white line on the current image, and is used as it is for threshold adjustment of the white line recognition application. In addition, by recognizing that the white line is a road such as a broken line or a botsdot, and analyzing its periodicity, the white line strength decreases at intervals where there is no white line. The correction may be performed such that the edge strength at the moment when there is a broken line or a bots dot is used as the visibility of the current white line.

  Next, the background edge strength analysis unit 320 forms a histogram of the edge strength in the background region to extract a background edge strength feature amount indicating the magnitude of the luminance difference in the background region, and this is extracted from the vehicle or pedestrian. This is used as one index for improving visibility. Compared to the high probability that it will always be on the road like a white line, it is unclear where vehicles and pedestrians exist or do not exist. It is difficult to analyze and analyze how much the edge strength of the vehicle is. For this reason, vehicles and pedestrians can be seen easily by using the horizon or the edge of the road edge, which is considered to be always photographed by the in-vehicle camera, and that its own appearance will not change significantly. Use as an index of safety. Here, since the brightness of the edge of the vehicle body greatly changes due to reflection or the like, the edge strength of the vehicle body and the road surface of the own vehicle is significantly different from the visibility of other vehicles and pedestrians and is difficult to use. In addition, although it is almost certain that the white line is a broken line or a solid line, in North America and the like, there are quite a few white lines that are rubbed, so it is difficult to use as a visibility indicator for vehicles and pedestrians. Therefore, we succeeded in using only the distant horizon or road edge as an index of ease of obstacles such as vehicles and pedestrians. Alternatively, the edge of the road edge may be used instead of the horizon. That is, the background edge strength feature amount is obtained from the processing area including the horizon or the road edge in the background area of the image.

  For the daytime, edge strength analysis was performed by dividing the vicinity of the horizon on the screen into three parts: left, right, and center. This is to reduce the influence of the background structure. For example, when there are not many structures in the distance, the edge strength near the horizon in the center of the image is inevitably low, but if you include the left and right areas including the neighboring area, the effect of the presence or absence of the structure Can be reduced. Depending on the application, a region for analyzing edge strength is appropriately selected and used as an edge threshold value at the time of detection.

  In addition, at night, the horizon behind the vehicle is too far away to be seen, and the headlights of the following vehicles may be visible at all times. By using the areas BGL and BGR, stability is ensured. For nighttime, since the visibility of the recognition object varies greatly depending on the lighting environment, an index that evaluates the visibility of the recognition object is an area that includes the road edge close to the processing area of the recognition object. By analyzing the background edge, it is considered easy to analyze the edge strength proportional to the visibility of the edge strength of the recognition target object.

  The luminance analysis unit 350 performs analysis of a region where the luminance change is small in the time direction. When the vehicle-mounted camera is running, the brightness changes naturally because the scenery always flows backward at the vehicle speed except for the vehicle body. Therefore, analyze the possibility of the presence of deposits on the lens, characterized by the fact that it lasts for a certain period of time, which is characterized by the fact that the brightness is always low or high compared to the area where the brightness changes little or the surrounding area of the screen . In FIG. 8, the length of time when the luminance change is small is represented by contour lines. However, when the surrounding environment is dark, the attached matter and the surrounding environment become dark, and there is a possibility that the feature amount of the luminance difference as described above may not be visible. Further, when the vehicle is not moving, it is natural that there is no moving object in the landscape even if the luminance of the entire screen does not change. For this reason, since the determination together with the light source environment and the vehicle behavior information is important for the final determination of the lens state, the state determination unit 400 determines the final lens state together with the light source environment estimation result. In the luminance analysis unit 350, the map information on the screen of the luminance change in the time direction is the final output.

  The shape analysis unit 360 uses edge information on the screen and position information of luminance change. Like the luminance analysis unit 350 above, the fact that the scenery should always change while the vehicle is moving, but the fact that the luminance change in the same direction always exists at the same position on the screen is not a factor in the background. , Judge that there is a high possibility of contamination on the lens. In addition, the shape analysis unit 360 has another type of logic for detecting circular particles such as mud and water droplets adhering to the lens, as shown in FIG. Compare the brightness difference of the points, and if the inner side is brighter, it detects night water drops, mud and day water drops. On the other hand, mud is detected using a characteristic that the inner side is darker on the inside. Similar to the luminance analysis unit 350, the final determination of the lens state using the light source environment and the vehicle behavior information is performed by the state determination unit 400.

<Fig. 4 Explanation of state determination unit>
FIG. 4 is a diagram illustrating a configuration of the state determination unit 400. The state determination unit 400 is a lens state determination unit 410 that determines the dirt state of the lens, a background visibility analysis unit 420 that indicates the visibility of a vehicle or a pedestrian, a sign, etc. reflected in the background, the degree of rubbing of the white line on the road surface, Information on vehicle behavior such as the degree of road surface reflection, wet road surface, road surface condition judgment unit 430 for judging muddy roads and snow conditions without white lines, vehicle behavior of the host vehicle, vehicle speed, rudder angle, yaw rate, etc. ON / OFF information, ON / OFF of the headlight of the host vehicle, and a vehicle information unit 440 that manages vehicle information such as ON / OFF of a winker, a brake lamp, and an engine.

  In the lens state determination unit 410 in the state determination unit 400, the determination of the lens cloudiness state, the lens luminance fixation determination for analyzing that the opaque dirt is attached to the extent that the lens luminance is fixed, and the point of the lens A semi-transparent dirt state is determined such that an object attached in a shape or a specific shape has a transparent rear and the outline of the dirt is raised.

  Determination flows executed by the lens state determination unit 410 are shown in FIGS. FIG. 11-1 shows a flow of determining the lens cloudiness, FIG. 12 shows a flow of determining the luminance fixation, and FIG. 13 shows a flow of analyzing the dirt shape.

  The background visibility analysis unit 420 returns the result of analyzing the edge strength of the background region regardless of the brightness. This is used as an index indicating the visibility when a vehicle or a pedestrian comes at present, regardless of whether the lens is dirty or dark and difficult to see, by analyzing the edge strength.

  Similarly, the road surface state determination unit 430 returns the result of analyzing the edge strength of the road surface region regardless of the brightness. This is an analysis of the edge strength, and whether the lens is dirty, the white line on the road surface is rubbing, or whether it is dark and difficult to see, it is an indicator of the visibility of the feature value of the white line on the current road surface. Show.

  The vehicle information unit 440 handles information such as vehicle speed, engine ON / OFF, headlight ON / OFF blinker, brake lamp lighting, and time measurement by a timer.

<Description of FIG. 11-1>
FIG. 11A describes a method for determining the lens cloudiness state in the lens state determination unit 410.

S1
Let exposure information be input information. The exposure information may be shutter speed, gain, camera high brightness area extraction result information, or exposure frame average brightness value. The timer and vehicle speed from the vehicle information unit 440 are also used. In addition, for the cloudiness determination, an instantaneous edge strength of background edge strength analysis is also received as an input.

S2
Next, the current vehicle speed and exposure information are checked to check whether the vehicle speed or exposure information is appropriate for determining the lens cloudiness state. First, if you analyze the background edge strength when the vehicle is stopped, the vehicle speed check is not preferable because the edge strength of the fixed background will always be observed, and it is unfavorably affected by the background scene too much. The analysis of the cloudiness state will be performed only for SPEEDW_THR or more.

  The exposure information is also analyzed except for scenes where the camera is white or black. As shown in FIG. 11B, the shutter control and the gain are digitized together. While the camera is bright, the shutter speed is fast, and the shutter speed slows as it gets darker. When the shutter speed slows down to a certain value, the gain starts to increase. Using this camera exposure information, it is determined whether the current light source environment around the camera is bright or dark.

  In a dark scene like a dark night, the background is not visible at all. In such a case, whether the lens is clear or cloudy, the background remains dark and cannot be seen, so it is difficult to analyze the lens state itself.

  The same can be said when the screen is too bright and the screen is white. If the entire processing area for analyzing the background edge intensity is whiteout, the lens state cannot be determined even if the background of the area is analyzed. Therefore, it is determined whether the exposure information and the camera high-intensity area are not in the vicinity of the background edge intensity analysis processing area, and whether the information is usable or not. As shown in FIG. 7, the background edge strength analysis processing area is divided into BGL, BGT, and BGR in a design that takes into account the case where the sky is white.

S3
From these determination conditions (1), (2), and (3), when the condition that the background edge strength analysis is considered to be acceptable is selected, NO is selected, and the undetermined timer time of the background edge strength analysis is reset.

S6
If YES, the undetermined timer time for background edge strength analysis is added.

S7
This undetermined timer addition time is determined from the time series data of the background edge intensity analysis to determine whether or not it can be used for analyzing the lens state. To determine whether the data is too old day / night, or to mix with data from several weeks ago, the time-series data is discarded after a certain period of time W, and the lens status is newly diagnosed. By designing so that the current lens state can be accurately reflected.

S8
However, in the case of a lens state in which white turbidity has progressed once, such as when the system gave up, even if the time series data is discarded, the result of the lens white turbid state, which is the lens state that led to the system give up, This time series data reset does not reset. Managed by the application control unit 500 itself.

S4
As shown in FIG. 10A, the ratio of the decrease in edge intensity during cloudiness is obtained from the correlation between the background edge intensity and the exposure information. The edge strength at the time of the clear lens is set to 100, and the rate of decrease is calculated as a numerical value in percentage%. This calculated instantaneous value is accumulated as time series data of lens cloudiness as shown in FIG. Thereby, the lens cloudiness state is determined based on the correlation between the background edge strength feature quantity and the exposure information. A road surface edge strength feature quantity may be used instead of the background edge strength feature quantity.

S5
The lens cloudiness state is updated from the time when data for a certain period of time is collected, using the value filtered in time series as the final lens cloudiness index.

<FIG. 12 Explanation of determination of luminance fixation>
FIG. 12 describes a method for determining whether the lens brightness is fixed in the lens state determination unit 410. The determination of lens luminance fixation is made when there is a region where the background does not move while the host vehicle is traveling, but there is a high possibility that the region is contaminated.

SB1
Let exposure information be input information. The exposure information may be shutter speed, gain, camera high brightness area extraction result information, or exposure frame average brightness value. The timer, vehicle speed, brake lamp, and blinker information from the vehicle information unit 440 are also used.

SB2
Next, the current vehicle speed and exposure information are checked, and it is checked whether the vehicle speed or exposure information is appropriate for determining the lens brightness fixation state. First of all, when checking the vehicle speed, the fixed background is always observed when analyzing the luminance fixation when the vehicle is stopped.If the light source environment does not change, it is determined that the entire screen is fixed luminance and the lens condition is dirty. There is a possibility of judging. For this reason, luminance fixation is analyzed by limiting the vehicle speed to a certain vehicle speed SPEEDB_THR or more.

  The exposure information is also analyzed except for scenes where the camera is white or black. Using the exposure information of the camera, it is determined whether the light source environment around the current camera is bright or dark. In a dark scene like a dark night, the background is not visible at all. In such a case, it is difficult to analyze the lens state because the background does not change darkly regardless of whether the lens is clear or the luminance is fixed.

  The same can be said when the screen is too bright and the screen is white. When a whiteout area continues to exist at a substantially constant position on the screen, for example, when driving with the western sun behind, there is a possibility that a whiteout sky may be erroneously determined to be fixed. is there. Therefore, it is determined whether the exposure information and the camera high-intensity area are not in the processing area, and whether the information is usable or not.

  In the case of nighttime, it is erroneously determined that the brightness of the screen has changed due to the flashing of the vehicle's brake lamp or blinker lamp, etc., and it becomes impossible to determine a dirty screen. Therefore, the determination is made to the YES determination side even when the blinker is lit or when the brake lamp is lit. Further, in order to reduce the light information of other vehicles, the determination may be made as YES and the image not to be analyzed even when there is a light of the following vehicle or an adjacent vehicle.

SB3
From these determination conditions (1), (2), (3), and (4), NO is selected in the condition that the luminance fixation may be analyzed, and the luminance fixation undetermined timer time is reset.

SB6
In the case of YES, the undetermined timer time for luminance fixation analysis is added.

SB7
This undetermined timer addition time is determined from the time series data of luminance fixation analysis to determine whether or not it may be used for analyzing the lens state. Do not confuse the day and night data, or do not confuse it with the data of several weeks ago, discard the time series data after a certain time B, and newly diagnose the lens status By designing, the current lens state can be accurately reflected.

SB8
However, in the case of a lens state in which the fixation of brightness has progressed once, such as when the system gave up, even if the time series data is discarded, the lens brightness fixed state that is the lens state that has resulted in the system give up The result is not reset by this time series data reset. Managed by the application control unit 500 itself.

SB4
A map of a region having no luminance change at the time of fixing the luminance in FIG. 8 is obtained. Here, as shown in FIG. 8, the map is updated by adding small areas with small luminance changes little by little.

SB5
From the point of time when data for a certain period of time is collected, the state of luminance fixation is determined for each recognition processing area of each application.

<Figure 13 Explanation of dirt shape analysis>
FIG. 13 describes a determination method of lens dirt shape analysis in the lens state determination unit 410. The determination of the lens dirt shape analysis is a method of detecting an object in the same edge direction that continues to exist at the same position on the lens, or a circular deposit.

SS1
Let exposure information be input information. The exposure information may be shutter speed, gain, camera high brightness area extraction result information, or exposure frame average brightness value. The timer, vehicle speed, brake lamp, and blinker information from the vehicle information unit 440 are also used.

SS2
Next, the current vehicle speed and exposure information are checked to check whether the vehicle speed or exposure information is suitable for performing lens dirt shape analysis. Since it is assumed that the background moves while the host vehicle is running, the outline of a three-dimensional object or a circular object is extracted from the background when the vehicle is stopped.

  First, when checking the vehicle speed, if the lens dirt shape analysis is performed when the vehicle is stopped, the fixed background will always be observed. There is. For this reason, the lens dirt shape analysis is performed by limiting the vehicle speed to a certain vehicle speed SPEEDS_THR or more.

  The exposure information is also analyzed except for scenes where the camera is white or black. Using the exposure information of the camera, it is determined whether the light source environment around the current camera is bright or dark. In a dark scene like a dark night, the background is not visible at all. In such a case, it is difficult to analyze the lens state because the background is dark and even the outline of the dirt cannot be seen, regardless of whether the lens is clear or dirty.

  The same can be said when the screen is too bright and the screen is white. If the whiteout area continues to exist at a substantially constant position on the screen, the whiteout outline may be erroneously determined as a dirty shape. Therefore, it is determined whether the exposure information and the camera high-intensity area are not in the processing area, and whether the information is usable or not. Further, at night, the brake light of the own vehicle and the blinking of the lamp at the time of the blinker are misjudgment factors, so that a dirty screen cannot be determined appropriately. Therefore, the determination is made to the YES determination side even when the blinker is lit or when the brake lamp is lit. Further, in order to reduce the light information of other vehicles, the determination may be made as YES and the image not to be analyzed even when there is a light of the following vehicle or an adjacent vehicle.

SS3
From these determination conditions (1), (2), (3), and (4), NO is selected and the undetermined timer time of the dirt shape analysis is reset in a condition that the dirt shape analysis is considered to be acceptable.

SS6
In the case of YES, the undetermined timer time of the dirt shape analysis is added.

SS7
The undetermined timer addition time is determined from the time series data of the dirt shape analysis to determine whether or not it can be used to analyze the lens state. Do not confuse the day and night data, or do not confuse it with the data of several weeks ago, discard the time series data after a certain time S, and newly diagnose the lens status By designing, the current lens state can be accurately reflected.

SS8
However, in the case of a lens state in which dirt has progressed, such as once the system has been given up, even if the time series data is discarded, the result of the lens dirt shape analysis that is the lens state in which the system has been given up is This time series data reset does not reset. Managed by the application control unit 500 itself.

SS4
The instantaneous value of the lens dirt shape analysis in FIG. 9 is added to obtain a map of a region that appears to be dirty. Here, as shown in FIG. 9, a map of a region where the circular shape has a luminance change or a dirt shape exists and the image edge direction does not change is updated.

SS5
The state of the dirt shape analysis is determined for each recognition processing area of each application from the time when data for a certain period of time is collected.

<Figure 5 Application Control Unit>
The application control unit 500 performs parameter adjustment and addition of a false detection countermeasure module for maintaining the recognition performance of each application in consideration of a lens dirt state, a decrease in visibility in the surrounding light source environment, and the like. Also, if the lens dirt state is too dirty to implement the image recognition application, the dirt removal hardware control, application give-up and system give-up are executed.

  The lane recognition control unit 510 changes the white line feature amount extraction threshold for lane recognition according to the result of the road surface condition determination unit 430, and performs feature amount extraction with a low threshold when the white line visibility is low. When the visibility is high, the threshold is set high. In addition, when there is a brightness sticking or dirt determined by the dirt shape on the lens, the dirt area is excluded from the processing area for lane recognition to suppress erroneous detection.

  The pedestrian detection control unit 512 adjusts an edge threshold for searching for a pedestrian detection candidate according to the result analyzed by the background visibility analysis unit 420. In the nighttime when pedestrians are difficult to see, the threshold value is reduced, and in the daylight conditions such as noon, the threshold value is raised to suppress false detection. Since the background edge intensity is used, the threshold value increases in the situation where there is a lot of road surface reflection even at night, and the threshold value decreases even in the daytime when the lens is cloudy. This balances false detection and non-detection.

  The entanglement prevention detection control unit 514 adjusts the threshold value of the moving object feature amount to be entangled according to the result analyzed by the background visibility analysis unit 420. At night, when the feature amount of the moving object is difficult to remove, the threshold value is reduced to suppress non-detection, and when it is noon, the threshold value is increased to suppress false detection. Since the background edge intensity is used, the threshold value increases in the situation where there is a lot of road surface reflection even at night, and the threshold value decreases even in the daytime when the lens is cloudy. This balances false detection and non-detection.

  The parking frame detection control unit 516 uses the results of the road surface state determination unit 430, the background visibility analysis unit 420, and the lens state determination unit 410 to adjust the threshold value for extracting the feature amount of the parking frame. Since the road surface condition determination unit 430 can basically analyze the road surface condition on the premise that a white line is captured, whether or not a stable result can be obtained in parking frame recognition in which an application moves at low speeds. It is clear. For this reason, although the result of the road surface condition determination unit 430 is also used to some extent, the threshold value is stably adjusted by using it together with the result of the background visibility analysis unit 420 that is required to be stable to some extent if the vehicle is not stopped. Further, the result of the lens state determination unit 410 is used for correction at the time of threshold adjustment by using the result before the low speed even when the application is not operating at the time of application execution. If it is clear that the lens is cloudy, the threshold of the parking frame feature value is lowered to maintain the detection rate.

  The travel path recognition control unit 518 uses the result of the road surface state determination unit 430 and the result of the lens state determination unit 410 to adjust the threshold used for the travel path feature value extraction. When the road surface pattern is difficult to see, the threshold value is lowered to make it easy to perform matching, etc., so that the travel path can be recognized stably. Conversely, in situations where the road surface is wet and there are many reflections, false recognition is suppressed by raising the threshold value.

  The vehicle detection control unit 520 adjusts the parameter for extracting the feature amount for vehicle detection according to the result analyzed by the background visibility analysis unit 420. At night when it is difficult to see the vehicle, the threshold is lowered, and the threshold is raised when the surroundings are bright, such as when reflected light enters. Even when the surroundings are dark, when the reflected light is inserted, the threshold value is increased to prevent false detection. Since the edge strength of the background is analyzed, the background visibility is determined to be high depending on the reflection state in the background, and as a result, erroneous detection of reflected light can be suppressed. In addition, if there is dirt determined due to the lens brightness fixation or the dirt shape, the dirt area is excluded from the vehicle detection processing area, thereby suppressing erroneous detection.

  The dirt removal hardware control unit 530 determines whether to remove the lens dirt based on the results of the cloudiness, brightness fixation, dirt shape, and the like determined by the lens state determination unit 410. When it is determined that the lens dirt is to be removed, the removal of the lens dirt is performed by controlling a removal member (not shown). Thus, processing for controlling the removal of lens dirt is performed based on the state of lens dirt.

  The application-specific give-up determination unit 540 performs recognition give-up determination for the lane recognition and vehicle detection applications based on the results of the cloudiness, luminance fixation, and dirt shape of the lens state determination unit 410. Since recognition performance differs between lane recognition and vehicle detection, it will not give up at the same time. For example, when dirt is attached only to the upper part of the lens, it is natural that only the vehicle detection is likely to give up. A give-up threshold is determined for each application in accordance with the state of the dirt, and it is determined whether the application is operable. Thus, processing for determining whether or not the recognition application can be executed is performed based on the state of lens contamination.

  In the system give-up determination unit 550, despite the above-described dirt removal performed by the dirt removal hardware control unit 530, dirt is not removed and the lens condition is not improved, and the recognition performance is maintained within an allowable range in each of lane recognition and vehicle detection. If it is determined that it is difficult, the system give-up determination is performed.

<Figure 6 Application Execution Unit>
The application execution unit 600 adjusts each application based on the control content performed by the application control unit 500 and executes the application.

  The lane recognition unit 610 suppresses misrecognition and non-recognition by extracting a white line feature amount based on the threshold adjusted by the lane recognition control unit 510 or adjusting a processing region to recognize a lane.

  The pedestrian detection unit 612 adjusts the feature value extraction threshold for pedestrian candidate detection based on the control content performed by the pedestrian detection control unit 512, and suppresses false detection and non-detection of pedestrian detection. .

  The entanglement prevention detection unit 614 suppresses false detection and non-detection of the entanglement prevention object by adjusting a threshold for optical flow detection based on the control content performed by the entanglement prevention detection control unit 514.

  The parking frame detection unit 616 adjusts a threshold value for extracting the feature amount of the parking frame based on the control content performed by the parking frame detection control unit 516, and suppresses false detection and non-detection of the parking frame detection. .

  The travel path recognition unit 618 adjusts a threshold value for extracting the feature amount of the travel path based on the control content performed by the travel path recognition control unit 518, and suppresses misrecognition and unrecognition of the travel path recognition. .

  The vehicle detection unit 620 suppresses erroneous detection and non-detection by performing vehicle detection based on the threshold value adjusted by the vehicle detection control unit 520.

  As described above, in the present invention, by analyzing a combination of a plurality of feature amounts, a factor that reduces the visibility of the recognition target object is analyzed. By specifying the main factor of the decrease in visibility, it is possible to select a more appropriate coping method.

  In addition, when there is dirt or other deposits on the lens, false detection and non-detection suppression processing is performed with image recognition software processing while the dirt is slight. It is desirable to determine whether to remove dirt or give up the image recognition process as a system. On the other hand, when it is difficult to see the recognition target object due to the influence of the light source environment, if the problem is not easy to see, the detection threshold is reduced to reduce the non-detection. In situations where it is more difficult to see, it is desirable to turn on any illumination that illuminates the recognition object, or to determine that the situation is temporarily difficult to see and to determine a temporary give-up systematically.

  Furthermore, in the case of lane recognition applications, etc., if there is a factor that makes it difficult to see the object to be recognized on the road surface, it is clear that the above-mentioned dirt removal and lighting cannot be used. Therefore, it is desirable to suppress the erroneous detection or to determine that it is desirable for the user that the lane recognition does not operate temporarily, and it is desirable to temporarily give up and stop functioning.

  In this way, by identifying the main factor that makes it difficult to see the recognition object, it is possible to appropriately determine a means for solving this if the main factor can be removed.

10… Automotive environment recognition device
100: Imaging unit
200: Light source environment estimation unit
210 ... Exposure information section
220: Image high brightness information section
300 ... Feature extraction unit
310 ... Road edge strength analysis part
320 ... Background edge strength analysis section
350 ... Brightness analysis unit
360… Shape analysis section
400 ... Status judgment part
410 ... Lens condition judgment part
420 ... Background visibility analysis unit
430 ... Road surface condition judgment unit
440… Vehicle information section
500 ... Application control unit
510 ... Lane recognition controller
512 ... Pedestrian detection controller
514 ... Entrainment prevention detection controller
516 ... Parking frame detection control unit
518 ... Runway recognition control unit
520 ... Vehicle detection control unit
530 ... Dirt removal hardware controller
540 ... Give-up judgment section by application
550 ... System give-up judgment unit
600… Application execution part
610 ... Lane recognition unit
612 ... Pedestrian detection unit
614 ... Entrainment prevention detector
616… Parking frame detector
618 ... Runway recognition unit
620 ... Vehicle detector

Claims (3)

An imaging unit for acquiring an image captured by a camera provided at the rear of the vehicle;
A light source environment estimation unit that obtains exposure information from the image;
A feature amount extraction unit for extracting a feature amount from the image;
A state determination unit that determines a lens dirt state of the camera based on the relationship between the feature amount and the exposure information;
Based on the state of the lens dirt, possess a process of determining whether to execute the recognition application, based on the state of the lens contamination, a process of controlling the removal of the lens dirt, and application control unit having, a ,
When the image is divided into a background region and a road surface region, the feature amount is a background edge strength feature amount indicating the magnitude of the brightness difference in the background region, and a road edge strength feature amount indicating the magnitude of the brightness difference in the road surface region. An on-vehicle environment recognition device characterized by including at least one feature quantity . The in-vehicle environment recognition device according to claim 1 ,
The in-vehicle environment recognition device, wherein the state determination unit determines a lens cloudiness state based on a correlation between the background edge strength feature quantity or the road surface edge strength feature quantity and the exposure information. The in-vehicle environment recognition device according to claim 1 ,
The on-vehicle environment recognition apparatus characterized in that the background edge strength feature amount is obtained from a processing region including a horizon or a road edge as an edge existing on the image.

Images