JP6536946B2 - Imaging device, imaging method, program, and mobile device control system - Google Patents

Description

  The present invention relates to an imaging device, an imaging method, a program, and a mobile device control system.

  Conventionally, in order to detect an object for sensing an object in front of the host vehicle (other vehicle, white line, road surface, person, obstacle, etc.) and an object for sensing such as raindrops attached to the windshield of the host vehicle An image pickup apparatus is known which picks up an image for detecting an attached matter with a single image sensor.

  For example, in Patent Document 1, an imaging frame for acquiring the image for detecting the attached matter every time the imaging frame (frame for sensing) for acquiring the image for sensing (frame for sensing) is taken 30 times consecutively There is disclosed an imaging device which repeats imaging of an attached matter detection frame) once. The imaging apparatus performs imaging while automatically adjusting the exposure amount of the sensing frame by automatic exposure control, while imaging the attached object detection frame with a fixed exposure amount suitable for detecting the attached object.

  In an imaging device that performs imaging of an attached matter detection frame between sensing frames that are continuously imaged, the longer the attached object detection frame time, the longer the interval between the sensing frames before and after the attached matter detection frame. The time interval spreads. The longer the time interval between the sensing frames, the longer the elapsed time from the imaging point of the previous sensing frame to the imaging point of the later sensing frame. Therefore, a change in a situation in front of the host vehicle (a relative position of various objects in front of the host vehicle, etc.) increases between the front and rear sensing frames, causing a problem such as a decrease in object recognition accuracy. Therefore, it is desirable to make the frame time of the deposit detection frame as short as possible.

However, it has been found that the following problem occurs when the frame time of the attached matter detection frame is shortened.
When the sensing frame immediately after the attached matter detection frame is imaged by automatic exposure control, the exposure time of the sensing frame may be set long in a dark environment such as in a tunnel or at night. The exposure time mentioned here means a light reception time from when the light receiving element of the image sensor in the imaging apparatus starts receiving the imaging light to when the light reception is finished. Since the timing at which the light receiving amount data of each light receiving element is output from the image sensor is fixed in advance, when the exposure time is set long, the light reception start timing becomes earlier by that amount.

  Normally, even when the exposure time is set to the upper limit value that can be set by automatic exposure control, the light reception start timing does not overlap with the exposure period of the immediately preceding sensing frame even when the exposure time is set to the upper limit that can be set by automatic exposure control . However, if the frame time of the attached matter detection frame is set to a time shorter than the frame time of the sensing frame in order to shorten the frame time of the attached matter detection frame, the sensing frame immediately after the attached matter detection frame It may happen that the light reception start time at the time when the light reception start time overlaps the exposure period of the deposit detection frame. When such a situation occurs, there may occur a problem that it is not possible to obtain normal received light amount data for at least one of the attached matter detection frame and the sensing frame immediately after that.

  In order to solve the problems described above, the present invention irradiates the light transmitting member with light from a light source to obtain an attached matter detection image for detecting the attached matter attached to the light transmitting member. An object recognition frame for obtaining an object recognition image for receiving light other than the light source and recognizing an object other than the attached object after imaging the attached object detection frame with the image sensor It is an imaging device which repeats a series of imaging operations of imaging a plurality of sheets, and a frame time of the attached matter detection frame is set shorter than a frame time of the object recognition frame, and the attached matter detection frame The exposure time in the frame for object recognition immediately after this is set to an exposure time in which the exposure time of the frame for object recognition does not overlap the exposure time of the frame for detecting an attached matter. The features.

  According to the present invention, an excellent effect is obtained that even when the frame time of the attached matter detection frame is set short, normal received light amount data can be acquired for both the attached matter detection frame and the sensing frame immediately after that. Is played.

FIG. 2 is a schematic view showing a schematic configuration of a mobile device control system in Embodiment 1. It is a schematic diagram which shows schematic structure of the imaging unit in the mobile device control system. It is explanatory drawing which shows schematic structure of the imaging part provided in the same imaging unit. It is explanatory drawing which shows the infrared-light image data which is captured image data for raindrop detection in case the focus of the imaging lens is in the raindrop on the outer wall surface of a windshield. It is explanatory drawing which shows the infrared-light image data which is captured image data for a raindrop detection in, when focusing on infinity. It is an explanatory view showing an example of a filter arranged in an optical filter part corresponding to a raindrop detection image area. It is explanatory drawing which shows the other example of the filter arrange | positioned at the optical filter part corresponding to the image area for raindrop detection. (A) is sectional drawing of the same optical filter, (b) is a front view by the side of the image sensor of the same optical filter. It is explanatory drawing which shows the image example of captured image data. FIG. 2 is an explanatory view showing details of an imaging unit in Embodiment 1. It is a model enlarged view when the optical filter which comprises the same imaging part, and an image sensor are seen from the direction orthogonal to the light transmission direction. It is a figure for demonstrating adhesion thing detection. It is a figure which shows the experimental result of the state to which the raindrop has adhered. It is a figure which shows the experimental result of the state which the raindrop has not adhered. In Embodiment 1, it is an enlarged view which shows the state of the image area | region for raindrops detection in which the amount of raindrops differs. It is the timing chart which showed simply an example of imaging operation in a comparative example. (A) is an explanatory view for explaining a relationship between data reading timing by rolling shutter method and an exposure period in a sensing frame. (B) is an explanatory view for explaining a relationship between data reading timing by rolling shutter method and exposure period in a raindrop detection frame. 7 is a timing chart simply showing an example (configuration example 1) of the imaging operation in the first embodiment. 7 is a timing chart simply showing another example (configuration example 3) of the imaging operation in the first embodiment. It is a timing chart which shows an example of the relation between the line read-out signal in the frame for raindrop detection, and the luminescence timing of a light source. It is explanatory drawing which shows an example of the light emission time of the light source in the exposure period of each line. It is explanatory drawing which shows the other example of the light emission time of the light source in the exposure period of each line. In an experiment example, an example of a picture of a raindrop detection image area containing disturbance light when a light source is ON is shown. In an experiment example, an example of a picture of a raindrop detection image area which does not include disturbance light when the light source is ON is shown. FIG. 24 is an enlarged view of an image portion where light source light is projected in the image example shown in FIG. 23; 5 is a schematic view showing a schematic configuration of an imaging unit in Embodiment 2. FIG. In Embodiment 2, it is an enlarged view which shows the state of the image area | region for raindrops detection in which the amount of raindrops differs.

Embodiment 1
Hereinafter, an embodiment (hereinafter, this embodiment will be referred to as “embodiment 1”) of a mobile device control system to which an imaging device according to the present invention is applied will be described.
The imaging device according to the present invention is not limited to the mobile device control system, and can be applied to, for example, other systems equipped with an object recognition device that performs object recognition based on a captured image.

FIG. 1 is a schematic view showing a schematic configuration of a mobile device control system in the first embodiment.
The mobile device control system is configured to control light distribution of a headlamp, drive control of a wiper, and other in-vehicle devices using a captured image captured by an imaging device mounted on the vehicle 100 such as a mobile vehicle. It controls.

  As shown in FIG. 1, the mobile device control system according to the first embodiment includes an imaging unit 101 having an imaging unit, an exposure control unit 109, an image analysis unit 102, a vehicle travel control unit 108, and a wiper control unit. 106 and a headlamp control unit 103 are mainly provided. Here, the vehicle travel control unit 108, the wiper control unit 106, and the headlamp control unit 103 function as a control unit that controls various in-vehicle devices mounted in the host vehicle 100.

  The imaging unit provided in the mobile device control system according to the first embodiment is provided in the imaging unit 101, and images a region in front of the traveling vehicle 100 in the traveling direction as an imaging region. It is installed near the room mirror of the windshield 105 of 100. The captured image data captured by the imaging unit of the imaging unit 101 is input to the image analysis unit 102. The exposure control unit 109 performs exposure control of the imaging unit of the imaging unit 101.

The image analysis unit 102 analyzes captured image data transmitted from the imaging unit,
Calculates the position, direction, and distance of other vehicles that are ahead of the vehicle 100, detects deposits such as raindrops and deposits on the windshield 105, and detects white lines on the road surface within the imaging area Detect detection objects such as (section lines) or calculate rainfall. In the detection of the other vehicle, the preceding vehicle traveling in the same traveling direction as the own vehicle 100 is detected by identifying the tail lamps of the other vehicle, and the headlamps of the other vehicle are identified in the opposite direction Detect oncoming vehicles.

  The calculation result of the image analysis unit 102 is sent to the headlamp control unit 103. The headlamp control unit 103 generates, for example, a control signal for controlling the headlamp 104 which is an on-vehicle device of the host vehicle 100 from the distance data calculated by the image analysis unit 102. Specifically, for example, it is possible to prevent the driver of another vehicle from being blinded by preventing the strong light of the headlamp of the vehicle 100 from being incident on the eyes of the driver of the preceding vehicle or the oncoming vehicle. The switching of the high beam and the low beam of the headlamp 104 is controlled, and the partial light blocking control of the headlamp 104 is performed so that the driver's view can be secured.

  The calculation result of the image analysis unit 102 is also sent to the wiper control unit 106. The wiper control unit 106 controls the wiper 107, which is an on-vehicle device of the vehicle 100, to remove deposits such as raindrops and deposits attached to the windshield 105 of the vehicle 100. The wiper control unit 106 generates a control signal for controlling the wiper 107 in response to the deposit detection result detected by the image analysis unit 102. When the control signal generated by the wiper control unit 106 is sent to the wiper 107, the wiper 107 is operated to secure the visibility of the driver of the host vehicle 100.

  The calculation result of the image analysis unit 102 is also sent to the vehicle travel control unit 108. Based on the white line detection result detected by the image analysis unit 102, the vehicle travel control unit 108 warns the driver of the own vehicle 100, for example, when the vehicle 100 is out of the lane area divided by the white line. Lane maintenance control such as informing and controlling the steering wheel and the brake of the host vehicle 100 is performed.

FIG. 2 is a schematic view showing a schematic configuration of the imaging unit 101. As shown in FIG.
FIG. 3 is an explanatory view showing a schematic configuration of the imaging unit 200 provided in the imaging unit 101. As shown in FIG.
The imaging unit 101 includes an imaging unit 200, a light source 202, and an imaging case 201 that accommodates these. The imaging unit 101 is installed on the inner wall surface side of the windshield 105 of the vehicle 100. As shown in FIG. 3, the imaging unit 200 includes an imaging lens 204, an optical filter 205, and an image sensor 206. The light source 202 emits light toward the windshield 105, and when the light is reflected by the outer wall surface of the windshield 105, the light is arranged to be incident on the imaging unit 200.

  In the first embodiment, the light source 202 is for detecting a deposit attached to the outer wall surface of the windshield 105 (hereinafter, the case where the deposit is a raindrop will be described as an example). When raindrops 203 adhere to the outer wall surface of the windshield 105, the light emitted from the light source 202 is reflected at the interface between the outer wall surface of the windshield 105 and the outside air, and the reflected light is incident on the imaging unit 200 . From such captured image data of the imaging unit 200, the raindrops 203 attached to the windshield 105 are detected.

  In the first embodiment, the imaging unit 101 covers the imaging unit 200 and the light source 202 together with the windshield 105 with the imaging case 201 as shown in FIG. By covering with the imaging case 201 as described above, even when the inner wall surface of the windshield 105 is cloudy, it is possible to suppress the situation where the windshield 105 covered by the imaging unit 101 becomes cloudy. Therefore, the situation where the image analysis unit 102 misanalyzes due to the fogging of the windshield 105 can be suppressed, and various control operations based on the analysis result of the image analysis unit 102 can be appropriately performed.

  However, when the fogging of the windshield 105 is detected from the captured image data of the imaging unit 200 and, for example, the air conditioning equipment of the own vehicle 100 is controlled, the portion of the windshield 105 facing the imaging unit 200 A passage through which air flows may be formed in part of the imaging case 201 so as to achieve the same situation.

  Here, in the first embodiment, the focal position of the imaging lens 204 is set between infinity or infinity and the windshield 105. Thus, not only when detecting raindrops 203 attached on the windshield 105 but also when detecting a leading vehicle or an oncoming vehicle or detecting a white line, appropriate information is taken from the captured image data of the imaging unit 200. It can be acquired.

  For example, when the raindrops 203 attached on the windshield 105 are detected, the shape of the raindrop image on the captured image data is often circular, so is the raindrop candidate image on the captured image data circular? A shape recognition process is performed to determine whether or not the raindrop candidate image is a raindrop image. When such shape recognition processing is performed, it is between infinity or infinity and the windshield 105 as described above, as the imaging lens 204 is focused on the raindrops 203 on the outer wall surface of the windshield 105. When the in-focus state is achieved, the raindrop shape recognition rate (circular shape) becomes high with some blurring, and the raindrop detection performance is high.

FIG. 4 is an explanatory view showing infrared light image data which is captured image data for raindrop detection when the imaging lens 204 is in focus on the raindrop 203 on the outer wall surface of the windshield 105.
FIG. 5 is an explanatory view showing infrared light image data which is captured image data for detecting raindrops when the image is focused at infinity.
When the imaging lens 204 is in focus on the raindrops 203 on the outer wall surface of the windshield 105, as shown in FIG. 4, even the background image 203a reflected in the raindrops is imaged. Such a background image 203 a causes erroneous detection of the raindrops 203. Further, as shown in FIG. 4, the luminance may increase in a bow shape or the like by only a portion 203b of the raindrop, and the shape of the large luminance portion, that is, the shape of the raindrop image changes depending on the direction of sunlight or the position of a street lamp . In order to correspond to the shape of such a varying raindrop image by shape recognition processing, the processing load is large, and the recognition accuracy is lowered.

  On the other hand, when the camera is in focus at infinity, some defocusing occurs as shown in FIG. Therefore, the reflection of the background image 203a is not reflected on the captured image data, and the erroneous detection of the raindrop 203 is reduced. In addition, the occurrence of slight defocusing reduces the degree to which the shape of the raindrop image changes depending on the direction of sunlight, the position of the streetlight, and the like, and the shape of the raindrop image always has a substantially circular shape. Therefore, the load of the shape recognition processing of the raindrops 203 is small, and the recognition accuracy is also high.

  However, when the camera is in focus at infinity, there may be only one light receiving element for receiving the light of the tail lamp on the image sensor 206 when identifying the tail lamp of the preceding vehicle traveling in the distance. In this case, although the details will be described later, there is a risk that the light of the tail lamp is not received by the red light receiving element that receives the tail lamp color (red), and in that case, the tail lamp can not be recognized and the preceding vehicle can not be detected. In order to avoid such a defect, it is preferable to focus the imaging lens 204 in front of infinity. As a result, the tail lamp of the preceding vehicle traveling far away is defocused, so the number of light receiving elements that receive the light of the tail lamp can be increased, and the recognition accuracy of the tail lamp is improved, and the detection accuracy of the preceding vehicle is improved.

  A light emitting diode (LED), a semiconductor laser (LD), or the like can be used as the light source 202 of the imaging unit 101. Further, as the emission wavelength of the light source 202, for example, visible light or infrared light can be used. However, in order to avoid dazzles the driver or pedestrian of the oncoming vehicle with the light of the light source 202, the wavelength is longer than visible light and the wavelength of the light receiving sensitivity of the image sensor 206 is, for example, 800 nm or more It is preferable to select a wavelength of infrared light region of 1000 nm or less. The light source 202 of Embodiment 1 emits light having a wavelength in the infrared light region.

  Here, when the infrared wavelength light from the light source 202 reflected by the front glass 105 is imaged by the imaging unit 200, the image sensor 206 of the imaging unit 200 may, for example, receive sunlight in addition to the infrared wavelength light from the light source 202. Also, disturbance light with a large amount of light including the infrared wavelength light of Therefore, to distinguish the infrared wavelength light from the light source 202 from the disturbance light of such a large light amount, it is necessary to make the light emission amount of the light source 202 sufficiently larger than the disturbance light, but such a large light emission Using an amount of light source 202 is often difficult.

  Therefore, in the first embodiment, for example, as shown in FIG. 6, a cut filter that cuts light having a wavelength shorter than the light emission wavelength of the light source 202, or a peak of transmittance as shown in FIG. The light from the light source 202 is configured to be received by the image sensor 206 through a band pass filter that substantially matches the emission wavelength of the light source 202. As a result, light other than the emission wavelength of the light source 202 can be removed and received, so that the light quantity from the light source 202 received by the image sensor 206 becomes relatively larger than the disturbance light. As a result, even if it is not the light source 202 with a large light emission amount, it is possible to distinguish the light from the light source 202 from the disturbance.

  However, in the first embodiment, not only the raindrops 203 on the windshield 105 are detected from the captured image data, but also detection of a preceding vehicle or an oncoming vehicle or detection of a white line is performed. Therefore, when the wavelength band other than the infrared wavelength light irradiated by the light source 202 is removed from the entire captured image, the light of the wavelength band necessary for the detection of the preceding vehicle or the oncoming vehicle or the detection of the white line is received by the image sensor 206 It is not possible to interfere with these detections.

  Therefore, in the first embodiment, the image area of the captured image data is a raindrop detection image area for detecting raindrops 203 on the windshield 105, and detection of a preceding vehicle or an oncoming vehicle or detection of a white line. A filter (also referred to as a "raindrop detection filter") that divides the vehicle detection image area and removes wavelength bands other than the infrared wavelength light emitted by the light source 202 only for the portion corresponding to the raindrop detection image area ) Is disposed in the optical filter 205. That is, the raindrop detection image area is an area of captured image data obtained by imaging an area where a raindrop detection filter is present in the imaging area. Further, the vehicle detection image area is an area of captured image data obtained by imaging an area in the imaging area where the raindrop detection filter does not exist.

FIG. 8A is a cross-sectional view of the optical filter 205.
In the optical filter 205 of the first embodiment, as shown in FIG. 8A, a spectral filter layer 224 that transmits infrared light and visible light is formed on the surface on the imaging lens side 205a of the substrate 221. There is. Further, on the surface on the image sensor side 205b of the substrate 221, a polarization filter layer 222, an SOG (Spin On Glass) layer 223, and an infrared light transmission filter 212 which is a raindrop detection filter are formed in order.

  By thus forming the filter layers on both sides of the substrate 221 of the optical filter 205, it is possible to suppress the warping of the optical filter 205. When a multilayer film is formed only on the surface of one side surface of the substrate 221, stress is applied and warpage occurs. However, in the case where multilayer films are formed on both surfaces of the substrate 221 as shown in FIG. 8A, the effect of stress is offset, so that warpage can be suppressed.

  Here, the substrate 221 can be made of a material that is transparent to the visible light range, such as glass, sapphire, or quartz. In the first embodiment, glass, in particular, inexpensive and durable quartz (refractive index 1.46) or tempered glass (refractive index 1.51) is used as the material of the substrate 221.

  The spectral filter layer 224 formed on the surface of the substrate 221 on the side of the optical filter 205a is a filter that transmits both a so-called visible light region of a wavelength range of 400 nm to 670 nm and an infrared light region of a wavelength range of 940 to 970 nm. is there. The visible light region is used to detect vehicle peripheral information, and the infrared light region is used to detect raindrops. In addition, the spectral filter layer 224 does not transmit light in the wavelength range of 700 to 940 nm (a transmittance of 5% or less is desirable). In this case, when this wavelength range is taken in, the image data obtained as a whole becomes red, and it may be difficult to extract a portion showing the red of the tail lamp and the like. Therefore, by forming a filter having a characteristic of cutting infrared light, it is possible to remove light of another color that is a disturbance, for example, it is possible to improve the detection accuracy of the tail lamp.

  The polarization filter layer 222 formed on the surface on the optical filter 205b side is a filter that cuts the S-polarization component and transmits only the P-polarization component. The polarization filter layer 222 can cut disturbance factors and unwanted reflection light (reflection light).

  In the first embodiment, the polarization filter layer 222 is formed of a wire grid polarizer. In the wire grid, conductor wires made of metal such as aluminum are arranged in a lattice at a specific pitch, and the pitch is smaller than that of incident light (for example, wavelength 400 nm to 800 nm of visible light). A relatively small pitch (for example, half or less) reflects most of the light of the electric field vector component oscillating parallel to the conductor wire and the light of the electric field vector component perpendicular to the conductor wire Because it transmits almost, it can be used as a polarizer that produces single polarization.

  In the wire grid polarizer, when the metal wire cross-sectional area is increased, the extinction ratio is increased, and the transmittance is decreased for a metal wire having a predetermined width or more with respect to the period width. In addition, when the cross-sectional shape orthogonal to the longitudinal direction of the metal wire is a tapered shape, the wavelength dispersion of the transmittance and the polarization degree is small in a wide band, and high extinction ratio characteristics are exhibited. In the cross-sectional structure of the wire grid, light is blocked when light in the polarization direction in the groove direction is incident, and light is transmitted when light in the polarization direction in the direction orthogonal to the grooves is incident.

  In Embodiment 1, since a wire grid structure is used as a polarizer of the polarization filter layer 222, the following advantages can be obtained. That is, in the wire grid structure, the sub-wavelength uneven structure of the wire grid may be formed by a well-known semiconductor process, that is, patterning after depositing an aluminum thin film and performing a method such as metal etching. Therefore, since it is possible to adjust the direction of the polarizer by the pixel size (several microns level) of the image pickup device, the transmission polarization axis can be selected in pixel units as in this embodiment. Further, as described above, since the wire grid structure is made of metal such as aluminum, it has excellent heat resistance and is suitable for in-vehicle use.

  In the SOG layer 223 of Embodiment 1, the substrate transparent to light in the use band and the convex portions of the wire grid linearly extended on the substrate are arranged at a pitch smaller than the wavelength of the light in the use band. Between the aluminum convexes, a filling portion filled with an inorganic material having a refractive index lower than or equal to that of the substrate 221 is formed. The filling portion is formed to also cover the convex portion of the wire grid structure.

As a forming material of the SOG layer 223, in order not to degrade the polarization properties of the polarizer of the polarization filter layer 222, it is preferable that the refractive index be a low refractive index material as close as possible to the refractive index of air = 1. For example, a porous ceramic material formed by dispersing fine pores in a ceramic is preferable, and porous silica (SiO ₂ ), porous magnesium fluoride (MgF), porous alumina (Al ₂ O ₃ ), etc. are mentioned. Be Further, the degree of these low refractive indices is determined by the number and size (porosity) of the pores in the ceramic. Among these, when the main component of the substrate 221 is made of quartz crystal or glass of silica, the refractive index is preferably smaller than that of the substrate 221 if porous silica (n = 1.22 to 1.26). .

As a method of forming the SOG layer 223, a method of forming an inorganic coating film (SOG) is used. That is, a solvent in which silanol [Si (OH) ₄ ] is dissolved in alcohol is spin-coated on a substrate, and then the solvent component is volatilized by heat treatment to form a dehydration polymerization reaction of silanol itself.

  By using the SOG layer 223, since the polarization filter layer 222 has a wire grid structure with a sub-wavelength size, its intensity is weaker than that of the infrared light transmission filter 212 formed on the SOG layer 223. In particular, since it is desirable that the optical filter 205 be disposed in close contact with the image sensor 206, there is a possibility that the optical filter 205 and the image pickup device surface of the image sensor 206 contact in its handling. Since the 222 is protected by the SOG layer 223, the optical filter 205 can be mounted without damaging the wire grid structure. The spectral filter layer 224 can also be protected by the SOG layer 223.

  In addition, by providing the SOG layer 223, it is possible to suppress the invading matter to the wire grid portion. The height of the convex portion of the wire grid is generally configured to be half or less of the used wavelength. On the other hand, the spectral filter has a height equal to or several times that of the used wavelength, and as the thickness is increased, the transmittance characteristic at the cutoff wavelength can be made steeper. Further, as the thickness of the SOG layer 223 increases, it becomes more difficult to ensure the flatness of the upper surface thereof, and it is not appropriate to increase the thickness because the uniformity of the filling region is impaired. In the first embodiment, since the infrared light transmission filter 212 is formed after the polarization filter layer 222 is covered with the SOG layer 223, the SOG layer 223 can be stably formed. The infrared light transmission filter 212 formed on the upper surface of the SOG layer 223 can also be formed optimally.

FIG. 8B is a front view of the optical filter 205 on the image sensor side.
FIG. 9 is an explanatory view showing an image example of captured image data.
As shown in FIG. 8B, the optical filter 205 is an area disposed at a position corresponding to the center of the captured image (the area corresponding to the height 2/4 of the imaging area) which is the vehicle detection image area 213. 211, and disposed at a position corresponding to the upper part of the captured image (a part corresponding to 1/4 of the imaging area height) and the lower part of the captured image (a part corresponding to the imaging area 1/4) And the infrared light transmission filter 212 is divided into regions. As the infrared light transmission filter 212, the cut filter shown in FIG. 6 or the band pass filter shown in FIG. 7 is used.

  Headlamps of oncoming vehicles, tail lamps of preceding vehicles, and white lines often exist mainly at the center of the captured image in the vertical direction, and an image of the closest road surface ahead of the vehicle exists at the lower part of the captured image. It is normal that there is an image of the sky ahead of the host vehicle at the top. Therefore, the head lamp of the oncoming vehicle, the tail lamp of the preceding vehicle, and the information necessary for identifying the white line are concentrated at the central portion of the captured image, and the information at the lower and upper portions of the captured image is not very important in the identification. Therefore, when performing detection of an oncoming vehicle, a preceding vehicle, or a white line and detection of raindrops from a single captured image data, as shown in FIG. It is preferable that the area 214 be the remaining center of the captured image be the vehicle detection image area 213, and the infrared light transmission filter 212 be provided correspondingly.

  However, in the first embodiment, the upper part of the captured image is used as the raindrop detection image area 214, and the remaining central part and the lower part of the captured image are used as the vehicle detection image area 213. This is because, as described later, this is suitable for shortening the frame time of the imaging frame for raindrop detection.

  When the imaging direction of the imaging unit 200 is inclined downward, the hood of the host vehicle may intrude into the lower part in the imaging region. In this case, the sunlight reflected by the bonnet of the own vehicle, the tail lamp of the preceding vehicle, and the like become disturbance light, and this is included in the captured image data to cause erroneous identification of the headlamp of the oncoming vehicle, the tail lamp of the preceding vehicle, and the white line. Become. Even in such a case, in the first embodiment, since the cut filter shown in FIG. 6 and the band pass filter shown in FIG. 7 are arranged in the part corresponding to the lower part of the captured image, the sunlight reflected by the bonnet The disturbance light such as the tail lamp of the preceding vehicle is removed. Therefore, the identification accuracy of the headlamp of the oncoming vehicle and the tail lamp of the preceding vehicle and the white line is improved.

  Here, when detecting a leading vehicle, detection of the leading vehicle is performed by identifying a tail lamp on the captured image, but the tail lamp has a smaller amount of light compared to the headlamp of the oncoming vehicle, and disturbance such as a street lamp Since a large amount of light is also present, it is difficult to detect the tail lamp with high accuracy from only simple luminance data. Therefore, it is necessary to use spectral information to identify the tail lamp and to identify the tail lamp based on the amount of received red light. Therefore, in the first embodiment, as described later, a red filter or a cyan filter (a filter that transmits only the wavelength band of the color of the tail lamp), which is matched to the color of the tail lamp, is disposed in the rear stage filter 220 of the optical filter 205, The amount of received red light can be detected.

  However, since each light receiving element constituting the image sensor 206 according to the first embodiment has sensitivity to light in the infrared wavelength band, it is obtained when the light including the infrared wavelength band is received by the image sensor 206. The captured image becomes reddish as a whole. As a result, it may be difficult to identify the red image portion corresponding to the tail lamp. Therefore, in the first embodiment, in the pre-stage filter 210 of the optical filter 205, a portion corresponding to the vehicle detection image area 213 is used as the infrared light cut filter area 211. As a result, since the infrared wavelength band is excluded from the captured image data portion used for identifying the tail lamp, the identification accuracy of the tail lamp is improved.

FIG. 10 is an explanatory view showing the details of the imaging unit 200 in the first embodiment.
The imaging unit 200 mainly includes a sensor substrate 207 including an imaging lens 204, an optical filter 205, and an image sensor 206 having a pixel array in which light receiving elements are two-dimensionally arranged, and an analog output from the sensor substrate 207 A signal processing unit 208 generates and outputs captured image data obtained by converting an electric signal (the amount of light received by each light receiving element on the image sensor 206) into a digital electric signal. Light from an imaging region including a subject (identification target) passes through the imaging lens 204, passes through the optical filter 205, and is converted into an electrical signal according to the light intensity by the image sensor 206. When the electric signal (analog signal) output from the image sensor 206 is input to the signal processing unit 208, the electric signal indicates the brightness (brightness) of each pixel on the image sensor 206 as captured image data. The digital signal is output to the subsequent unit along with the horizontal and vertical synchronization signals of the image.

FIG. 11 is a schematic enlarged view when the optical filter 205 and the image sensor 206 are viewed from the direction orthogonal to the light transmission direction.
The image sensor 206 is a light receiving element using a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like, and uses a photodiode 206A for each pixel. The photodiodes 206A are two-dimensionally arrayed for each pixel, and in order to increase the light collection efficiency of the photodiodes 206A, the microlenses 206B are provided on the incident side of each photodiode 206A. The image sensor 206 is bonded to a PWB (Printed Wiring Board) by a method such as wire bonding to form a sensor substrate 207.

  As an exposure method of the image sensor 206, a global shutter method of receiving light simultaneously (simultaneous exposure) by all the light receiving elements and reading out the signal of each light receiving element, or sequentially receiving light (line exposure) for each predetermined light receiving element line A rolling shutter system that reads out the signal of the element can be adopted. In the first embodiment, an example in which a rolling shutter system is adopted will be described.

  The optical filter 205 and the image sensor 206 may be bonded by a UV adhesive, or UV bonding or thermocompression bonding may be performed in the four side areas outside the effective pixel while the outside of the effective pixel range used for imaging is supported by a spacer. By closely bonding the optical filter 205 and the image sensor 206, the boundary between the raindrop detection image area 214 and the vehicle detection image area 213 becomes clear, and the determination accuracy of the presence or absence of raindrops can be raised.

  Next, the deposit detection for controlling the wiper 107 and the washer according to the first embodiment is possible. Here, the deposits are raindrops, but can also include bird droppings, splashes on the road surface that bounce off from adjacent vehicles, and the like.

FIG. 12 is a diagram for explaining the attached matter detection.
The light source 202 is disposed so that the specularly reflected light on the outer surface of the windshield substantially matches the optical axis of the imaging lens. The rays received by the imaging unit 200 will be described using the marks of rays A to E in the drawing.

  A light ray A is a light ray emitted from the light source 202 and having passed through the windshield, and when no raindrop adheres to the outside as viewed from the imaging unit 200 of the windshield 105, the ray leaks to the outside as it is. In addition, while selecting the light source of the wavelength and light quantity of an eye-safe zone as a light source 202, as shown in FIG. 12, safety is further achieved by outputting light toward upper side.

  The ray B is a ray of the light emitted from the light source 202 which is reflected by the inner wall surface of the windshield 105 when the light ray is incident on the inner wall surface. In general, it is known that the polarization component of light ray B is an S polarization component. Such light is unnecessary light for original raindrop detection and causes erroneous detection. However, in the first embodiment, the S polarization component is cut by the polarization filter layer 222 of the optical filter 205. Therefore, unnecessary light can be removed.

  A ray C is a ray of the light emitted from the light source 202 which is not reflected by the inner wall surface of the windshield 105 but passes through the inside of the windshield 105 and enters the image sensor 206. When raindrops adhere to the outer wall surface of the windshield 105, light incident on the inside of the windshield 105 is multiple-reflected inside the raindrops and transmitted again through the windshield toward the imaging unit 200 side for imaging Arrives at the optical filter 205 of the The component of the light beam C transmitted in the windshield 105 has a P-polarization component more than a S-polarization component, so the light passes through the spectral filter layer 224 and the subsequent polarization filter layer 222 transmits the P-polarization component in a wire Since the groove direction of the grid structure is formed, it also passes through here. At this time, the infrared light transmission filter 212 matched to the wavelength of the light source 202 is formed corresponding to the raindrop detection image area, but the light ray C is transmitted therethrough to reach the image sensor 206.

  The light ray D is not a light source 202 but a light ray that arrives at the imaging unit 200 from the outside of the windshield 105 and reaches the image sensor portion corresponding to the raindrop detection image area. Most of the light ray D is cut by the infrared light transmission filter, so that disturbance light outside the windshield 105 is also cut in the raindrop detection image area.

  The light beam E is a light beam passing through an image area other than the raindrop detection image area, that is, an optical filter portion corresponding to the vehicle detection image area where the infrared light transmission filter 212 is not present. The light beam E transmits only the visible region and infrared light, and reaches only the P-polarization component and reaches the image sensor 206 in a state where unnecessary light is cut, and is detected as a signal for various applications.

  The light source 202 is set to have an angle of incidence on the windshield 105 so that the reflected light on any surface of the raindrop-air interface can be imaged. The layout in which the reflected light from the raindrops is the strongest is, as shown in FIG. 12, when installed at a position substantially opposite to the optical axis of the imaging unit 200 with respect to the normal to the front surface of the windshield 105; It is a case where it arrange | positions so that it may become an optical axis substantially the same as the optical axis of 200, and reflected light is the smallest when the normal line of the windshield 105 and the optical axis of a light source correspond substantially.

  The light source 202 can be arranged to irradiate only the region of the infrared light transmission filter 212. Thereby, noise components from the vehicle detection image area can be avoided. In addition, a plurality of light sources 202 may be provided. In this case, the polarizer pattern of each region of the polarization filter layer 222 is an optical axis of light emitted from the light source 202 having the largest amount of incident light to the polarizer pattern toward the windshield 105, It is set to transmit only the polarization component parallel to the plane formed by the two optical axes with the optical axis of the lens.

  As a light emission method of the light source 202, pulse emission may be performed at a specific timing in addition to continuous emission (also referred to as CW emission). By synchronizing the light emission timing and the imaging timing, the influence of disturbance light can be further reduced. When a plurality of light sources 202 are installed, the plurality of light sources 202 may emit light simultaneously or may emit light sequentially. When light is sequentially emitted, the influence of disturbance light can be further reduced by synchronizing the light emission timing and the imaging timing.

Here, the experimental result which the inventors image | photographed is shown in FIG. 13, FIG.
FIG. 13 is a diagram showing experimental results in a state in which raindrops are attached.
FIG. 14 is a diagram showing experimental results in a state in which no raindrops are attached.
The figure showing this experimental result is an example in which the lower part of the image area is the raindrop detection image area.

FIG. 15 is an enlarged view showing the state of the raindrop detection image area having different amounts of raindrops.
When the standard deviation value of the luminance of the photographed image shown in FIG. 15 is calculated, it can be seen that the values on the left image are 20, 27 and 39, and there is a correlation between the standard deviation value of the luminance and the rainfall. Therefore, in the first embodiment, the image analysis unit 102 calculates a value related to the standard deviation value of luminance from the captured image of the raindrop detection image area, and measures the rainfall based on the standard deviation value. . It should be noted that the variance may be used instead of the standard deviation.

Next, the imaging operation of the imaging unit 200 in the first embodiment will be described.
The imaging unit 200 according to the first embodiment receives an light from the front area of the vehicle 100 to recognize an object (other vehicle, a white line, a road surface, a person, an obstacle, etc.) existing in the front area. The object recognition frame for acquiring the for-use image and the raindrop detection frame for acquiring the raindrop detection image for detecting the raindrop on the windshield by irradiating the light from the light source 202 separately Do. Specifically, a series of imaging operations of imaging one raindrop detection frame each time a plurality of object recognition frames are imaged are repeatedly performed.

FIG. 16 is a timing chart simply showing an example of the imaging operation in the comparative example.
In FIG. 16, the frame synchronization signal, the line readout signal, the frame number, and the exposure period are displayed side by side along the same time axis. For the exposure period, the vertical axis direction corresponds to the position of the light receiving element line. The amount of light received by each light receiving element of the image sensor 206 is read out and output for each light receiving element line according to the line read signal.

FIG. 17A is an explanatory view for explaining the relationship between data reading timing and exposure period according to the rolling shutter method in the sensing frame.
FIG. 17B is an explanatory view for explaining the relationship between data reading timing and exposure period according to the rolling shutter method in the raindrop detection frame.
In the first embodiment (same as in the present comparative example), a rolling shutter method is adopted, and in the sensing frame, as shown in FIG. 17A, the light receiving element lines of the image sensor 206,. The light reception start time according to the set exposure time differs for each light receiving element line based on each timing of the line read signal (horizontal synchronization signal) for each of N + 1, N + 2,. Even in the raindrop detection frame, as shown in FIG. 17B, each timing of the line readout signal (horizontal synchronization signal) of the light receiving element lines..., M, M + 1, M + 2,. As a reference, the light reception start time according to the set exposure time differs for each light receiving element line.

  In the first embodiment (the same as in the comparative example), sensing frames as four object recognition frames for acquiring an image of the vehicle detection image area 213 and an image of the raindrop detection image area 214 are acquired. An imaging operation of sequentially imaging the raindrop detection frame as one attached object detection frame is repeatedly performed.

  The imaging frame of frame number B0 is a sensing frame used for lane maintenance control (LDW: Lane Departure Warning). The imaging frame B0 is an automatic exposure frame in which the exposure time at the time of imaging operation is automatically controlled within a predetermined time range. Therefore, for example, a short exposure time is set in a bright environment such as daytime, and a long exposure time is set in a dark environment such as nighttime.

  The imaging frame of frame number A1 is a sensing frame used for light distribution control (AHB: Auto High Beam) of a headlamp. The imaging frame A1 is a fixed exposure frame in which the exposure time at the time of imaging operation is fixed.

  The imaging frame of frame number B2 is a sensing frame used for collision avoidance control (FCW: Front Collision Warning). The imaging frame B2 is an automatic exposure frame in which the exposure time at the time of imaging operation is automatically controlled within a predetermined time range.

  The imaging frame of the frame number A3 is a sensing frame used for light distribution control of the headlamp as in the case of the imaging frame of the second frame, and is a fixed exposure frame in which the exposure time at the time of imaging operation is fixed.

  The imaging frame of frame number R4 is a raindrop detection frame used for drive control and the like of the wiper. The imaging frame R4 is a fixed exposure frame in which the exposure time at the time of imaging operation is fixed in a very short time. Since one raindrop detection frame for acquiring the image of the raindrop detection image area 214 receives relatively strong light from the light source 202, the signal of the image sensor 206 is prevented from being saturated. In order to reduce the influence of disturbance light other than the illumination light from the light source 202, it is preferable to shorten the exposure time as much as possible. Further, since the light quantity from the light source 202 is substantially constant, the fixed exposure time can be obtained.

  In the first embodiment (the same as in the comparative example), the imaging operation of repeatedly imaging the sensing frame B0 to A3 a predetermined number of times (for example, seven times) and then imaging the raindrop detection frame R4 is repeatedly performed. Do.

  In the first embodiment (similar to the comparative example), as shown in FIG. 16, the frame time Rt of the raindrop detection frame R4 is set shorter than the frame time Ft of the sensing frames B0 to A3. Since the exposure time of the raindrop detection frame R4 is significantly shorter than the exposure time set in the sensing frames B0 to A3 in consideration of receiving strong light from the light source 202 as described above, raindrop detection is performed. The frame time Rt of the frame R4 can be set shorter than the frame time Ft of the sensing frames B0 to A3. Moreover, in the first embodiment, the number of light receiving element lines of the raindrop detection image area imaged by the raindrop detection frame R4 is smaller than that of the vehicle detection image areas imaged by the sensing frames B0 to A3, and Since it is the light receiving element line part (part corresponding to the upper part of the captured image) read out from the sensor first, the frame time Rt of the raindrop detection frame R4 can be set particularly shorter than the frame time Ft of the sensing frames B0 to A3. It is supposed to be.

  However, in the comparative example, the sensing frame B0 immediately after the raindrop detection frame R4 is an automatic exposure frame in which automatic exposure control is performed. Therefore, in a dark environment such as in a tunnel or at night, as shown in FIG. 16, the exposure time of the sensing frame B0 is set long, and the light reception start time of each light receiving element in the sensing frame B0 becomes early. There is. Then, since the frame time Rt of the raindrop detection frame R4 is set short, if the exposure time of the sensing frame B0 immediately after that is set long, as shown in FIG. 16, the raindrop detection frame R4 is A situation may occur where the exposure period EB0 of the sensing frame B0 immediately thereafter overlaps with the exposure period ER4. When such a situation occurs, it is not possible to acquire normal received light amount data for at least one of the raindrop detection frame R4 and the sensing frame B0 immediately after that.

[Configuration Example 1]
FIG. 18 is a timing chart simply showing an example of the imaging operation in the first embodiment.
In the first embodiment, for the sensing frame B0 immediately after the raindrop detection frame R4, the upper limit of the exposure time that can be changed by automatic exposure control is within the exposure period ER4 of the raindrop detection frame R4. The exposure period EB0 'is set equal to or less than the exposure time which does not overlap. Therefore, even in a dark environment such as in a tunnel or at night, the exposure control unit 109 does not set the exposure time of the sensing frame B0 beyond the upper limit. Therefore, as shown in FIG. 18, the exposure period EB0 ′ of the sensing frame B0 does not overlap the exposure period ER4 of the raindrop detection frame R4.

  However, in the first embodiment, since light is emitted from the light source 202 during the exposure period ER4 of the raindrop detection frame R4, the light from the light source 202 does not affect the sensing frame B0 immediately after that. It has a certain margin. Specifically, the end of the exposure period ER4 of the raindrop detection frame R4 (the timing of the line readout signal of the line read last) and the start of the exposure period EB0 of the sensing frame B0 immediately thereafter (the line of the line read last A fixed blank period Gt is provided between the read signal timing and the read signal timing).

  Here, for the sensing frame B0 not positioned immediately after the raindrop detection frame R4, it is not necessary to limit the upper limit of the exposure time that can be changed by automatic exposure control. Therefore, in the first embodiment, under a dark environment, as shown in FIG. 18, the exposure time of the sensing frame B0 not positioned immediately after the raindrop detection frame R4 is set to be long as usual. Therefore, in a dark environment, the exposure time EB0 'of the sensing frame B0 immediately after the raindrop detection frame R4 is different from the exposure time EB0 between the sensing frame B0 which is not immediately after the raindrop detection frame R4. It becomes a thing. If a difference in exposure time occurs between such sensing frames B0, a difference in overall received light amount is generated between the two, which may cause a defect such as a decrease in white line recognition accuracy.

  Therefore, in the first embodiment, when the image analysis unit 102 functions as an image conversion unit and the exposure time EB0 of the sensing frame B0 not immediately after the raindrop detection frame R4 is set to exceed the upper limit value, Based on the exposure time EB0, an image conversion process is performed to convert the captured image captured by the sensing frame B0 immediately after the raindrop detection frame R4 into a captured image corresponding to the exposure time EB0. Specifically, for example, the signal value (light reception amount) output from the image sensor 206 in the sensing frame B0 immediately after the raindrop detection frame R4 is set to the exposure time EB0 'of the sensing frame B0 and the raindrop detection frame It is increased according to the difference from the exposure time EB0 of the sensing frame B0 which is not immediately after R4. As a result, the difference in signal value (the amount of light received) based on the difference in exposure time between the sensing frames B0 is reduced, and a decrease in white line recognition accuracy can be suppressed.

[Configuration Example 2]
Although the first embodiment is an example in which the upper limit value of the exposure time of the automatic exposure control is limited for the sensing frame B0 immediately after the raindrop detection frame R4, the exposure period EB0 of the sensing frame B0 is also limited in the other example. It is possible to prevent the situation in which 'overlaps with the exposure period ER4 of the raindrop detection frame R4.
For example, as for the sensing frame B0 immediately after the raindrop detection frame R4, there is an example in which automatic exposure control is not performed and the exposure time is fixed. The fixed exposure time at this time is set equal to or less than the exposure time in which the exposure period EB0 ′ of the sensing frame B0 does not overlap the exposure period ER4 of the raindrop detection frame R4.

  Also in this example, a difference in exposure time occurs between the sensing frames B0. Therefore, for a captured image captured by the sensing frame B0 set to the fixed exposure time positioned immediately after the raindrop detection frame R4, the automatic exposure controlled sensing frame B0 not positioned immediately after the raindrop detection frame R4 It is preferable to carry out an image conversion process for converting into a captured image corresponding to the exposure time EB0 based on the exposure time EB0.

[Configuration Example 3]
As another example for preventing the situation in which the exposure period EB0 ′ of the sensing frame B0 overlaps the exposure period ER4 of the raindrop detection frame R4, an example shown in FIG. 19 can be given.
In this example, as shown in FIG. 19, the positions of the sensing frame B0 used for lane keeping control and the sensing frame A1 used for light distribution control of a headlamp are interchanged and fixed immediately after the raindrop detection frame R4. The sensing frame A1 for the exposure time is positioned. The fixed exposure time at this time is set equal to or less than the exposure time when the exposure period EA1 of the sensing frame A1 does not overlap the exposure period ER4 of the raindrop detection frame R4.

  In this example, since a difference in exposure time does not occur between the sensing frames A1, there is no need to perform the image conversion process described above.

Next, light emission control of the light source 202 in the first embodiment will be described.
Since the captured image captured in the raindrop detection frame R4 is sufficient for only the raindrop detection image area, the data read out in the raindrop detection frame R4 corresponds to the line corresponding to the raindrop detection image area, that is, the captured image Only the line corresponding to the upper part is sufficient. Therefore, the period during which light is to be emitted by the light source 202 may be only the exposure period of the line corresponding to the raindrop detection image area.

  During a period in which light is emitted from the light source 202, the light also enters the image sensor portion corresponding to the vehicle detection image area, and may become disturbance light of the vehicle detection image area. Therefore, if the light source 202 is controlled to emit light from the light source 202 only during the exposure period of the line corresponding to the raindrop detection image area, even if the frame time of the raindrop detection frame R4 is shortened, the light source 202 Light does not become disturbance light of the sensing frames A3 and B0 located before and after the light.

FIG. 20 is a timing chart showing an example of the relationship between the line read signal and the light emission timing of the light source 202 in the raindrop detection frame R4.
The exposure period of the line portion corresponding to the raindrop detection image area in the raindrop detection frame R4 may be controlled so that the light source 202 always emits light. In this case, data of the light source 202 emitting light can be acquired for all the lines corresponding to the raindrop detection image area. However, in this case, when disturbance light other than the light from the light source 202 is incident on the image sensor 206, it is data including an error of the disturbance light component.

  In the first embodiment, the light source 202 is turned ON and OFF at least once during the exposure period of the line corresponding to the raindrop detection image area in one raindrop detection frame R4. The light emission control of 202 is performed. In the first embodiment, the light source 202 is controlled to emit light so that the light source 202 is turned on / off for each line. For example, the light source 202 may be controlled to emit light so that the light source 202 is turned on / off every two lines. The light emission timing of the light source 202 in the exposure period of each line may be the first timing of the exposure period of each line as shown in FIG. 21 or the end of the exposure period of each line as shown in FIG. It may be time.

Next, experimental examples conducted by the present inventors will be described.
In this experimental example, as in the first embodiment, the light source 202 is controlled to emit light so that the light source 202 is turned ON / OFF for each line, and it is confirmed whether raindrop detection can be appropriately performed.

FIG. 23 shows an image example of a raindrop detection image area including disturbance light when the light source 202 is ON.
FIG. 24 shows an example image of a raindrop detection image area not including disturbance light when the light source 202 is ON.
FIG. 25 is an enlarged view of an image portion where light source light is projected in the image example shown in FIG.
As shown in FIG. 25, it can be seen that high contrast is obtained for each line according to the ON / OFF of the light source 202.

Second Embodiment
Next, another embodiment (hereinafter, this embodiment will be referred to as “embodiment 2”) of a mobile device control system to which the imaging device according to the present invention is applied will be described.
The second embodiment is substantially the same as the first embodiment except that the configuration of the imaging unit 101 is partially different. Therefore, the differences from the first embodiment will be mainly described.

FIG. 26 is a schematic view showing a schematic configuration of the imaging unit 101 in the second embodiment.
In the imaging unit 101 according to the second embodiment, the light emitted from the light source 202 is incident on a prism 230 as a reflective deflection member provided on the inner wall surface of the windshield 105, as shown by the optical path in FIG. Thus, the light is totally reflected on the outer wall surface of the windshield 105 and is incident on the imaging lens 204, and is received by the image sensor 206 on the sensor substrate 207.

  On the other hand, in the place where the raindrops adhere to the outer wall surface of the windshield 105, the light emitted from the light source 202 is transmitted through the outer wall surface of the windshield 105 and is not received by the image sensor 206. In the first embodiment described above, the light emitted from the light source 202 is transmitted through the outer wall surface of the windshield 105, while the light reflected irregularly in the inside of the raindrop is transmitted by the image sensor 206 at the location where the raindrops adhere to the outer wall surface of the windshield 105. Light is received.

  Therefore, in the second embodiment, an image obtained by inverting the image of the raindrop detection image area of the first embodiment described above is obtained. That is, in the first embodiment described above, as shown in FIG. 15, an image with high luminance at the raindrop adhesion site is obtained, while in the second embodiment, the luminance at the raindrop adhesion site is as shown in FIG. Gives a low image. Therefore, in the above-described embodiment, the larger the sum of pixel values is, the larger the amount of rainfall is. However, in the second embodiment, the smaller the sum of pixel values is, the larger the amount of rainfall is.

  As described above, the various processes executed by the mobile device control system according to the first and second embodiments described above can be realized not only by hardware but also by software. In the case of software implementation, a program that causes a computer to function to implement various types of processing is provided by being incorporated in advance in a ROM or the like. In addition, the program is recorded in a computer readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), and the like in an installable or executable file format. It may be configured to be provided. Alternatively, the program may be stored on a computer connected to a network such as the Internet and may be provided by being downloaded via the network. Also, the program may be configured to be provided or distributed via a network such as the Internet.

What has been described above is an example, and the present invention has unique effects in each of the following modes.
(Aspect A)
Raindrop detection for acquiring a deposit detection image for irradiating light from a light source 202 to a light transmitting member such as a windshield 105 and detecting attached matter such as raindrops 203 attached to the light transmitting member After imaging an attached matter detection frame such as the frame R4 by the image sensor 206, light other than the light source is received to recognize an object other than the attached matter, such as a vehicle, a white line, a road surface, a person or an obstacle. An imaging apparatus that repeats a series of imaging operations of imaging a plurality of object recognition frames such as sensing frames B0 to A3 for acquiring an object recognition image for image detection by the image sensor, wherein The frame time Rt of the frame is set shorter than the frame time Ft of the object recognition frame, and the object recognition frame immediately after the attached matter detection frame is set. The exposure time in B0 (A1 in Configuration Example 3) is an exposure time in which the exposure period EB0 '(EA1' in Configuration Example 3) of the object recognition frame does not overlap the exposure period ER4 of the attached object detection frame. It is characterized by being set.
In this embodiment, the frame time Rt of the attached object detection frame R4 is set to be shorter than the frame time Ft of the object recognition frame, but the sensing frame B0 immediately after that in the exposure period ER4 of the attached object detection frame R4. There is no occurrence of overlapping of the exposure periods EB0 'of. If the exposure period EB0 'of the sensing frame B0 immediately thereafter is not overlapped with the exposure period ER4 of the attached object detection frame R4, normal received light amount data of both the raindrop detection frame R4 and the sensing frame B0 immediately thereafter You can get Therefore, even if the frame time Rt of the attached object detection frame R4 is set short, normal received light amount data can be acquired for both the raindrop detection frame R4 and the sensing frame B0 immediately after that.
In addition, as a method of acquiring normal received light amount data for both the attached matter detection frame and the sensing frame immediately after that, the attached matter detection frame and the sensing immediately after that according to the exposure time of the sensing frame. It is conceivable to provide a blanking time with the frame. With this method, it is possible to prevent the light reception start timing of the sensing frame from overlapping the exposure period of the previous attached object detection frame, but for the attached object detection by the amount of blank time provided. The time interval between the sensing frames before and after the frame increases. Therefore, as described above, the situation change in the imaging region becomes large between the front and rear sensing frames, which causes a problem such as a decrease in object recognition accuracy. In addition, with this method, frame timing after providing the blank time is shifted, and for example, in an application that grasps the elapsed time by the number of frames, it becomes impossible to grasp the normal elapsed time. According to this aspect, it is possible to acquire normal received light amount data for both the attached matter detection frame and the sensing frame immediately after that without providing such a blank time.

(Aspect B)
In the mode A, the object recognition frame is an automatic exposure frame B0 or B2 captured in a state where the image sensor is controlled by an automatic exposure control that changes an exposure time, and a fixed image captured by the image sensor at a fixed exposure time The object recognition frame immediately after the attached object detection frame is the fixed exposure frame A1 and includes the object recognition frame during the exposure period of the attached object detection frame. It is characterized in that it is set to a fixed exposure time in which the exposure periods of (1) and (2) do not overlap.
According to this, as described in the configuration example 3, a situation does not occur in which the exposure period EA1 ′ of the sensing frame A1 immediately thereafter overlaps the exposure period ER4 of the attached object detection frame R4. Normal light reception amount data can be acquired for both the raindrop detection frame R4 and the sensing frame A1 immediately after that while keeping the frame time Rt of R4 short.

(Aspect C)
In the mode B, the object recognition image captured in the fixed exposure frame A1 immediately after the attached object detection frame R4 is an object corresponding to the exposure time based on the exposure time when the automatic exposure frame is captured. It is characterized in that it has image conversion means such as an image analysis unit 102 for converting into a recognition image.
According to this, it is possible to convert the object recognition image captured in the fixed exposure frame into an image converted to the exposure time of the automatic exposure frame, so that the processing error caused by the difference in the exposure time between both frames Etc. can be reduced.

(Aspect D)
In the mode A, the object recognition frame includes automatic exposure frames B0 and B2 which are imaged in a state where the image sensor is controlled by automatic exposure control for changing an exposure time, and the attached object detection frame R4 The object recognition frame immediately after is the automatic exposure frame B0, and the upper limit value of the exposure time that can be changed by the automatic exposure control is within the exposure period ER4 of the attached object detection frame R4. It is characterized in that the exposure period EB0 'is set to an exposure time or less not overlapping.
According to this, as described in the configuration example 1, there is no occurrence of overlapping of the exposure period ER0 of the sensing frame B0 immediately after that with the exposure period ER4 of the attached object detection frame R4, so the attached object detection frame Normal light reception amount data can be acquired for both the raindrop detection frame R4 and the sensing frame B0 immediately after that while keeping the frame time Rt of R4 short.

(Aspect E)
In the mode D, when the exposure time EB0 of the automatic exposure frame B0 not immediately after the attached object detection frame is set to exceed the upper limit value, the automatic exposure frame B0 immediately after the attached object detection frame R4 is It is characterized in that it has an image conversion means for converting the picked up object recognition image into an object recognition image corresponding to the exposure time based on the exposure time.
According to this, it is possible to convert the object recognition image captured in the automatic exposure frame B0 immediately after the attached object detection frame R4 into an image converted to the exposure time of the other automatic exposure frame B0. Processing errors and the like resulting from differences in exposure time between the two frames can be reduced.

(Aspect F)
In any one of the above aspects A to E, the attached object detection frame R4 is a part (a line corresponding to the upper part of the image) of the light receiving element group constituting the image sensor, and the object recognition frame The object detection image is acquired using a light receiving element group smaller than the number of light receiving elements used when acquiring an object recognition image.
According to this, it is possible to set the frame time Rt of the attached object detection frame R4 short.

(Aspect G)
In the aspect F, the imaging operation is performed by a rolling shutter system, and the light receiving element group used for imaging the attached object detection image in the attached object detection frame R4 is firstly data from the image sensor Is a light receiving element group to be output.
According to this, it is possible to set the frame time Rt of the attached object detection frame R4 shorter.

(Aspect H)
In any of the above aspects A to G, the imaging operation is performed by a rolling shutter method, and an exposure period and light in which light is emitted from the light source 202 during one attached object detection frame R4 A light source control unit such as an exposure control unit 109 that controls the light source to include an exposure period not irradiated, and having an exposure time in the object recognition frame immediately after the attached object detection frame corresponds to the attached object detection frame The exposure period is set such that the exposure period of the object recognition frame does not overlap the exposure period during which light is emitted from the light source.
According to this, it is possible to eliminate disturbance light using the difference between the light reception amount in the exposure period in which light is irradiated and the light reception amount in the exposure period in which light is not irradiated, and it is possible to improve the detection accuracy It becomes.

(Aspect I)
After imaging the attached matter detection frame for acquiring an attached matter detection image for irradiating the light transmitting member with light from a light source and detecting the attached matter attached to the light transmitting member A series of imaging operations of repeatedly imaging a plurality of object recognition frames for acquiring an object recognition image for receiving an light other than the light source and recognizing an object other than the attached matter; In the imaging method, the frame time of the attached matter detection frame is set to be shorter than the frame time of the object recognition frame, and the exposure time of the object recognition frame immediately after the attached matter detection frame is The exposure time of the object recognition frame is set so as not to overlap the exposure period of the attached matter detection frame.
According to this, even in a dark environment such as in a tunnel or at night, a situation in which the exposure period of the sensing frame immediately after that does not overlap with the exposure period of the attached object detection frame does not occur. Normal light reception amount data can be acquired for both the raindrop detection frame and the sensing frame immediately after that while keeping the frame time of the frame short.

(Aspect J)
After imaging the attached matter detection frame for acquiring an attached matter detection image for irradiating the light transmitting member with light from a light source and detecting the attached matter attached to the light transmitting member A series of imaging operations of repeatedly imaging a plurality of object recognition frames for acquiring an object recognition image for receiving an light other than the light source and recognizing an object other than the attached matter; A program for causing a computer of an imaging device to function, wherein a frame time of the attached matter detection frame is set shorter than a frame time of the object recognition frame, and immediately after the attached matter detection frame. The exposure time in the object recognition frame is set to an exposure time in which the exposure period of the object recognition frame does not overlap the exposure period of the deposit detection frame. Characterized in that it is.
According to this, even in a dark environment such as in a tunnel or at night, a situation in which the exposure period of the sensing frame immediately after that does not overlap with the exposure period of the attached object detection frame does not occur. Normal light reception amount data can be acquired for both the raindrop detection frame and the sensing frame immediately after that while keeping the frame time of the frame short.

(Aspect K)
Based on the attached object detection image and the object recognition image captured by the imaging device, control predetermined devices such as the headlamp 104 mounted on a moving object such as the host vehicle 100, the wiper 107, the warning notification means, and the brake. In a mobile device control system including mobile device control means such as a headlamp control unit 103, a wiper control unit 106, and a vehicle travel control unit 108, the imaging device according to any one of the aspects A to H. An imaging apparatus is used.
According to this, more appropriate device control becomes possible.

100 self-vehicle 101 imaging unit 102 image analysis unit 103 headlamp control unit 104 headlamp 105 windshield 105 wiper control unit 107 wiper 108 vehicle travel control unit 109 exposure control unit 200 imaging unit 201 imaging case 202 light source 203 raindrop 204 imaging lens 205 Optical filter 206 Image sensor 207 Sensor substrate 208 Signal processing unit 213 Vehicle detection image area 214 Raindrop detection image area 230 Prism

JP, 2013-117520, A

Claims (11)

After imaging the attached matter detection frame for acquiring an attached matter detection image for irradiating the light transmitting member with light from a light source and detecting the attached matter attached to the light transmitting member A series of imaging operations of repeatedly imaging a plurality of object recognition frames for acquiring an object recognition image for receiving an light other than the light source and recognizing an object other than the attached matter; An imaging device,
The frame time of the attached matter detection frame is set shorter than the frame time of the object recognition frame,
The exposure time in the object recognition frame immediately after the attached object detection frame is set to an exposure time in which the exposure period of the object recognition frame does not overlap the exposure period of the attached object detection frame. Imaging device. In the imaging device according to claim 1,
The object recognition frame includes an automatic exposure frame for imaging in a state where the image sensor is controlled by automatic exposure control for changing an exposure time, and a fixed exposure frame for imaging with a fixed exposure time by the image sensor,
The object recognition frame immediately after the attached object detection frame is the fixed exposure frame, and is set to a fixed exposure time in which the exposure period of the object recognition frame does not overlap the exposure period of the attached object detection frame. An imaging apparatus characterized in that In the imaging device according to claim 2,
An image for converting an object recognition image acquired in a fixed exposure frame immediately after the attached object detection frame into an object recognition image corresponding to the exposure time based on the exposure time when the automatic exposure frame is captured An image pickup apparatus having a conversion means. In the imaging device according to claim 1,
The object recognition frame includes an automatic exposure frame for imaging in a state where the image sensor is controlled by automatic exposure control for changing an exposure time,
The object recognition frame immediately after the attached object detection frame is the automatic exposure frame, and the upper limit value of the exposure time that can be changed by the automatic exposure control is the object recognition during the exposure period of the attached object detection frame An imaging apparatus characterized by being set to an exposure time equal to or less than an exposure time in which the exposure periods of frames do not overlap. In the imaging device according to claim 4,
When the exposure time of an automatic exposure frame not immediately after the attached object detection frame is set to exceed the upper limit value, the object recognition image captured by the automatic exposure frame immediately after the attached object detection frame is exposed An image pickup apparatus comprising image conversion means for converting into an image for object recognition corresponding to the exposure time based on time. The imaging device according to any one of claims 1 to 5.
The attached object detection frame is a part of the light receiving element group constituting the image sensor, and the number of light receiving element groups smaller than the number of light receiving elements used when acquiring an object recognition image by the object recognition frame An imaging device characterized by acquiring an attached matter detection image using the same. In the imaging device according to claim 6,
The imaging operation is performed by a rolling shutter method,
An image pickup apparatus characterized in that the light receiving element group used for capturing the attached object detection image in the attached object detection frame is a light receiving element group in which data is first output from the image sensor. The imaging device according to any one of claims 1 to 7.
The imaging operation is performed by a rolling shutter method,
A light source control means for controlling the light source such that an exposure period in which light is emitted from the light source and an exposure period in which light is not emitted are included in one deposit detection frame;
The exposure time in the object recognition frame immediately after the attached object detection frame is the exposure time in which the exposure period of the object recognition frame does not overlap the exposure period in which light is emitted from the light source in the attached object detection frame. An imaging device characterized by being set. After imaging the attached matter detection frame for acquiring an attached matter detection image for irradiating the light transmitting member with light from a light source and detecting the attached matter attached to the light transmitting member A series of imaging operations of repeatedly imaging a plurality of object recognition frames for acquiring an object recognition image for receiving an light other than the light source and recognizing an object other than the attached matter; An imaging method,
The frame time of the attached matter detection frame is set shorter than the frame time of the object recognition frame,
The exposure time in the object recognition frame immediately after the attached object detection frame is set to an exposure time in which the exposure period of the object recognition frame does not overlap the exposure period of the attached object detection frame. Imaging method. After imaging the attached matter detection frame for acquiring an attached matter detection image for irradiating the light transmitting member with light from a light source and detecting the attached matter attached to the light transmitting member A series of imaging operations of repeatedly imaging a plurality of object recognition frames for acquiring an object recognition image for receiving an light other than the light source and recognizing an object other than the attached matter; A program for causing a computer of an imaging device to function,
The frame time of the attached matter detection frame is set shorter than the frame time of the object recognition frame,
The exposure time in the object recognition frame immediately after the attached object detection frame is set to an exposure time in which the exposure period of the object recognition frame does not overlap the exposure period of the attached object detection frame. program. In a mobile device control system comprising mobile device control means for controlling a predetermined device mounted on a mobile based on an attached object detection image and an object recognition image captured by an imaging device,
A mobile device control system using the imaging device according to any one of claims 1 to 8 as the imaging device.