JP2018007038A - Electronic device, automobile, imaging device, and image recording device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem in which, it is impossible to set imaging conditions for detecting an object and imaging conditions for generating a display image with a camera mounted on the conventional vehicle.SOLUTION: An electronic device mounted in a car includes: an imaging unit having a first area and a second area in which imaging conditions can be changed and generating a first image signal in the first area and a second image signal in the second area; object detection unit that detects an object from the first image signal; and a display control unit that displays an image on the basis of the second image signal on a display unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an electronic apparatus, an automobile, an imaging apparatus, and an image recording apparatus.

  A camera mounted on a vehicle is known (see Patent Document 1). In a camera mounted on a conventional vehicle, there is a problem that an imaging condition for detecting an object and an imaging condition for generating a display image cannot be set.

JP 2010-79424 A

The electronic device mounted on the vehicle according to the first aspect has a first area and a second area where the imaging condition can be changed, generates a first image signal in the first area, and generates a first image in the second area. An imaging unit that generates two image signals; an object detection unit that detects an object from the first image signal; and a display control unit that displays an image based on the second image signal on a display unit. .
The electronic device mounted on the vehicle according to the second aspect has a first area and a second area where the imaging condition can be changed, generates a first image signal in the first area, and generates a first image in the second area. An imaging unit that generates two image signals; a detection unit that detects whether or not a reflective member is included in the first image signal; and the detection unit that determines whether the second image signal is based on a detection result of the detection unit. And a display unit that displays the image of the part of the reflecting member in a different manner.
The automobile according to the third aspect is equipped with the electronic device.
An automobile according to a fourth aspect includes an imaging unit including an imaging element capable of setting imaging conditions for a plurality of areas, a distance measuring unit that calculates distance information based on an output from a pixel of the imaging element, A setting unit that sets imaging conditions for the plurality of regions based on distance information.
The automobile according to the fifth aspect moves the imaging unit based on an operation unit provided on a movable main body, an imaging unit that performs imaging, a moving speed of the main unit, and an operation amount of the operation unit. And a control unit.
The automobile according to the sixth aspect is based on an image pickup unit including an image pickup device capable of independently setting image pickup conditions of a plurality of areas, a detection unit that detects approach to an intersection, and a detection result by the detection unit. And a setting unit that sets imaging conditions for a plurality of regions of the imaging device.
An image pickup apparatus according to a seventh aspect picks up an image pickup unit including an image pickup element capable of setting image pickup conditions for a plurality of areas, and a portion irradiated with the pattern light of an object irradiated with a predetermined pattern light. And a control unit that controls the imaging condition of the imaging element area to be different from the imaging condition of the imaging element area that captures a portion that is not irradiated with the pattern light.
An electronic device according to an eighth aspect includes an imaging unit including an imaging element capable of setting imaging conditions for a plurality of regions, a transmissive display unit provided in front of a driver of a moving body, and imaging using the imaging unit. And a control unit configured to set imaging conditions for the plurality of regions and to control display of the transmissive display unit based on information about the brightness in front of the moving object obtained from the obtained image.
An image recording apparatus according to a ninth aspect is a second image picked up by a first image pickup unit that picks up an image of the periphery of a moving body and a second image pickup unit provided in a wearing device attached to a passenger of the moving body. An input unit that inputs an image signal, a comparison unit that compares a shooting range of the first image signal captured by the first imaging unit and a shooting range of the second image signal, and a comparison result of the comparison unit A recording unit that records at least one of the first image signal and the second image signal on a recording medium.

It is a schematic block diagram of the driving assistance device of a vehicle. It is a figure explaining the pixel arrangement | sequence and unit area | region of an imaging chip. It is a block diagram which illustrates the composition of a camera. FIG. 4A to FIG. 4C are diagrams illustrating the divided first region and second region. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a figure which shows typically the image on the imaging surface of an imaging chip. FIG. 7A is a diagram schematically showing an image on the imaging surface of the imaging chip, FIG. 7A is a diagram showing a straight traveling time, and FIG. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a flowchart explaining the flow of the control process of the camera which a control part performs. It is a flowchart explaining the detail of an imaging condition setting process. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a figure which shows typically the image on the imaging surface of an imaging chip. It is a figure which illustrates a display image. It is a flowchart explaining the detail of an imaging condition setting process. It is a figure which shows typically the image imaged by an image pick-up element. It is a figure which shows typically the image imaged by an image pick-up element. It is a figure which shows typically the image imaged by an image pick-up element. It is a figure which shows typically the image imaged by an image pick-up element. It is a flowchart explaining the flow of a control process of a camera. It is a flowchart explaining the detail of an imaging condition setting process. It is a flowchart explaining the detail of an imaging condition setting process. 24A and 24B are diagrams illustrating a frame rate according to the image plane movement amount. It is a figure explaining the yaw angle of a camera. It is a flowchart explaining the detail of an imaging condition setting process. FIG. 27A is a diagram for explaining the third embodiment. It is a figure explaining the projection pattern by an auxiliary light source. It is a figure which illustrates the target object irradiated with infrared light. FIG. 28 (a) and FIG. 28 (b) are diagrams for explaining the fourth embodiment. FIG. 29A is a diagram for explaining the fifth embodiment. FIG. 29B is a diagram illustrating the first area and the second area of the imaging surface.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
(First embodiment)
<When to use the camera>
FIG. 1 is a schematic configuration diagram of a driving support device 2 for a vehicle 1 equipped with cameras 3A and 3B according to the first embodiment. In FIG. 1, a driving support device 2 is mounted on a vehicle 1 such as an automobile. The driving support device 2 includes a camera 3A, a camera 3B, a control device 4, a first travel control unit 5, a second travel control unit 6, and the like. In the present embodiment, an example in which an internal combustion engine is used as a drive source will be described, but a motor may be used as a drive source.

  In the present embodiment, the camera 3A and the camera 3B are configured by the same camera. The camera 3A (3B) includes an imaging optical system having a plurality of lenses and a laminated imaging element described later. The camera 3 </ b> A is attached, for example, in front of the ceiling in the passenger compartment, and acquires an image in front of the vehicle 1. The camera 3B is attached, for example, to the rear of the ceiling in the passenger compartment, and acquires an image of the rear of the vehicle 1.

  Each of the cameras 3A and 3B measures the distance to each subject (target object) at a plurality of positions in the shooting screen based on the acquired image (image acquired based on a first area in the image sensor 100 described later) ( Measure distance). The distance measurement is calculated by distance measurement using an image signal from a focus detection pixel provided in the image sensor. Image data and distance measurement data acquired by the camera 3 </ b> A (3 </ b> B) are sent to the control device 4. The camera 3A (3B) may be provided outside the vehicle.

  The control device 4 includes a CPU 4a and a storage unit 4b. Based on various programs stored in the storage unit 4b, the CPU 4a performs various calculations using control parameters stored in the storage unit 4b, detection signals from sensors described later, and the like.

  The first travel control unit 5 performs constant speed travel control and follow-up travel control based on an instruction from the control device 4. The constant speed traveling control is control for causing the vehicle 1 to travel at a constant speed based on a predetermined control program. The follow-up running control is constant with respect to the preceding vehicle when the speed of the preceding vehicle recognized by the control device 4 is equal to or lower than the target speed set in the vehicle 1 during the constant speed running control. It is the control which makes it drive | work in the state holding the distance between two cars.

  The second traveling control unit 6 performs driving support control based on an instruction from the control device 4. Based on a predetermined control program, the driving support control outputs a steering control signal to the steering control device 9 so that the vehicle 1 travels along the road, or avoids the vehicle 1 from colliding with an object. In this control, a brake control signal is output to the brake control device 8.

1 further shows a throttle control device 7, a brake control device 8, a steering control device 9, a steering wheel 10, a turn signal switch 11, a vehicle speed sensor 12, a yaw rate sensor 13, and a display / reproduction device 14. A GPS device 15 and a line-of-sight detection device 16 are shown.
In addition, the communication part 17 is used in the modification 4 mentioned later, and the radar 18 is used in 2nd embodiment. These are not essential components in the first embodiment.

  The throttle control device 7 controls the opening degree of a throttle valve (not shown) according to the depression amount of the accelerator pedal 7a. The throttle control device 7 also controls the opening degree of the throttle valve in accordance with a throttle control signal sent from the first travel control unit 5. The throttle control device 7 further sends a signal indicating the depression amount of the accelerator pedal 7 a to the control device 4.

  The brake control device 8 controls the opening degree of a brake valve (not shown) according to the depression amount of the brake pedal 8a. The brake control device 8 also controls the opening degree of the brake valve in accordance with a brake control signal from the second travel control unit 6. The brake control device 8 further sends a signal indicating the depression amount of the brake pedal 8 a to the control device 4.

  The steering control device 9 controls the steering angle of a steering device (not shown) according to the rotation angle of the steering wheel 10. The steering control device 9 also controls the steering angle of the steering device according to a steering control signal from the second travel control unit 6. The steering control device 9 further sends a signal indicating the rotation angle of the steering wheel 10 to the first traveling control unit 5 and the control device 4, respectively.

  The turn signal switch 11 is a switch for operating a turn signal (winker) device (not shown). The turn signal device is a blinking light emitting device that indicates a change in the course of the vehicle 1. When the turn signal switch 11 is operated by an occupant of the vehicle 1, an operation signal from the turn signal switch 11 is sent to the turn signal device, the second traveling control unit 6, and the control device 4. The vehicle speed sensor 12 detects the vehicle speed V of the vehicle 1 and sends detection signals to the first travel control unit 5, the second travel control unit 6, and the control device 4.

  The yaw rate sensor 13 detects the yaw rate of the vehicle 1 and sends detection signals to the second traveling control unit 6 and the control device 4, respectively. The yaw rate is the rate of change of the rotation angle of the vehicle 1 in the turning direction. The display / reproduction device 14 displays information indicating a control state by the first traveling control unit 5 and the second traveling control unit 6. The display / reproduction device 14 is configured by, for example, a HUD (Head Up Display) that projects information on a windshield. As the display / reproduction device 14, information may be displayed on a room mirror (not shown), or a display unit and a reproduction unit of a navigation device (not shown) may be used.

  The GPS device 15 receives a radio wave from a GPS satellite and calculates a position (latitude, longitude, etc.) of the vehicle 1 by performing a predetermined calculation using information carried on the radio wave. The position information calculated by the GPS device 15 is sent to a navigation device (not shown) and the control device 4.

The line-of-sight detection device 16 is built in, for example, the steering wheel 10. The line-of-sight detection device 16 detects the line-of-sight position of an occupant (for example, a driver) and sends the line-of-sight information indicating the region in which the driver is gazing to the control device 4. For gaze detection, corneal reflection method that detects the direction of gaze by reflecting infrared light on the driver's cornea, limbus tracking method that uses the difference in reflectance to light between the cornea and sclera, and the driver's eyeball There is an image analysis method in which an image is captured by a camera and a line of sight is detected by image processing, and any line of sight detection method may be used. In the case of an autonomous driving vehicle, an occupant may not drive, and an occupant other than the driver may be a target for detecting the line of sight.
Alternatively, the line-of-sight detection device 16 may be incorporated in a spectacle-type wearable device, and the line-of-sight information may be transmitted from the wearable device to the control device 4 by wireless communication.

<Detection of object>
For the detection of the object, an image acquired based on a first area described later is used. The control device 4 performs image processing on the image from the camera 3 </ b> A (3 </ b> B) as follows in order to detect the travel path and the object of the vehicle 1. First, the control device 4 generates a distance image (depth distribution image) based on distance measurement data at a plurality of positions in the shooting screen. The control device 4 performs a well-known grouping process based on the distance image data, and frames (windows) such as three-dimensional road shape data, side wall data, and object data stored in the storage unit 4b in advance. Compared with, white lines (including white line data along the road and white lines crossing the road (stop line: intersection information) data), side walls such as guardrails and curbs that exist along the road, Obstacles are detected by classifying them into other objects such as motorcycles, ordinary vehicles, large vehicles, pedestrians, and utility poles.
In the present description, the white or yellow line drawn on the travel path is called a white line. In addition, a white line including a solid line and a broken line is called.

<Driving support>
The control device 4 recognizes the travel path and the obstacle / obstruction to be obstructed based on the information detected as described above, that is, the white line data, the guardrail side wall data, and the object data, and the recognition result. Then, the second driving control unit 6 performs the driving support control. That is, the vehicle 1 is caused to travel along the road, and the vehicle 1 is prevented from colliding with an object.

<Running control>
For example, the control device 4 estimates the own vehicle traveling path in the following four ways.
(1) Self-vehicle traveling path estimation based on white line White line data on both the left and right sides of the traveling road or one of the left and right sides is obtained from the image acquired by the camera 3A (3B). When the shape of the traveling lane can be estimated, the control device 4 estimates that the own vehicle traveling path is parallel to the white line in consideration of the width of the vehicle 1 and the position of the vehicle 1 in the current lane. .

(2) Self-vehicle travel path estimation based on side data of guardrail, curb, etc. From the images acquired by the camera 3A (3B), the side wall data on either the left or right side of the road or one of the left and right sides is obtained. When the shape of the lane in which the vehicle 1 is traveling can be estimated from the side wall data, the control device 4 considers the width of the vehicle 1 and the position of the vehicle 1 in the current lane, and determines that the own vehicle traveling path is the side wall. Presumed to be parallel.

(3) Self-vehicle traveling path estimation based on the preceding vehicle trajectory The control device 4 estimates the own vehicle traveling path based on the past traveling trajectory of the preceding vehicle stored in the storage unit 4b. The preceding vehicle refers to a vehicle that is closest to the vehicle 1 among objects that travel in the same direction as the vehicle 1.

(4) Self-vehicle traveling path estimation based on traveling state of vehicle 1 The control device 4 estimates the traveling path of the own vehicle based on the driving state of the vehicle 1. For example, the own vehicle traveling path is estimated using the turning curvature based on the detection signal from the yaw rate sensor 13 and the detection signal from the vehicle speed sensor 12. The turning curvature Cua is calculated by Cua = dψ / dt / V. dψ / dt is the yaw rate (speed of change of the rotation angle in the turning direction), and V is the vehicle speed of the vehicle 1.

  The control device 4 estimates the traveling region of the vehicle 1 at the position where the object exists for each object according to a predetermined traveling control program stored in the storage unit 4b based on the own vehicle traveling path. The traveling area and the object position are compared to determine whether or not each object is in the traveling area. The control device 4 further recognizes the preceding vehicle based on the imaging result of the camera 3A (3B). That is, the control device 4 sets the vehicle closest to the vehicle 1 as the preceding vehicle among the objects that exist in the travel region and travel in the forward direction (the same direction as the vehicle 1).

  The control device 4 outputs the inter-vehicle distance information between the preceding vehicle and the vehicle 1 and the vehicle speed information of the preceding vehicle to the first traveling control unit 5 as out-of-vehicle information. Here, the vehicle speed information of the preceding vehicle is based on the vehicle speed V of the vehicle 1 acquired every predetermined time and the image acquired by the camera 3A (3B) every predetermined time in synchronization with the acquisition timing of the vehicle speed V. The distance is calculated based on the change in the distance (inter-vehicle distance) to the preceding vehicle in the measured shooting screen.

  The first travel control unit 5 sends a throttle control signal to the throttle control device 7 so that the vehicle speed V detected by the vehicle speed sensor 12 converges to a predetermined vehicle speed (target speed) set in advance. Thereby, the throttle control device 7 feedback-controls the opening of a throttle valve (not shown), and the vehicle 1 is automatically driven at a constant speed.

  In addition, the first traveling control unit 5 performs the traveling control in the constant speed state when the vehicle speed information of the preceding vehicle input from the control device 4 is equal to or lower than the target speed set in the vehicle 1. Then, a throttle control signal is sent to the throttle control device 7 based on the inter-vehicle distance information input from the control device 4. Specifically, a target value for an appropriate inter-vehicle distance is set based on the inter-vehicle distance from the vehicle 1 to the preceding vehicle, the vehicle speed of the preceding vehicle, and the vehicle speed V of the vehicle 1, and is acquired by the camera 3A (3B). A throttle control signal is sent to the throttle control device 7 so that the inter-vehicle distance measured based on the image converges to the target value of the inter-vehicle distance. As a result, the throttle control device 7 feedback-controls the opening of a throttle valve (not shown), and causes the vehicle 1 to travel following the preceding vehicle.

<Description of Laminated Image Sensor>
The above-described multilayer image sensor 100 provided in the camera 3A (3B) is described in International Publication No. WO13 / 164915 previously filed and published by the applicant of the present application. The imaging element 100 is laminated with a back-illuminated imaging chip that outputs pixel signals corresponding to incident light, a signal processing chip that processes pixel signals, and a memory chip that stores pixel signals. The imaging element 100 is configured to be able to set imaging conditions for each unit area.

  FIG. 2 is a diagram for explaining the pixel array and the unit area 131 of the imaging chip 111. Incident light on the image sensor 100 enters in the positive direction of the Z axis. As shown in the coordinate axes, the right direction on the paper orthogonal to the Z axis is the X axis plus direction, and the upward direction on the paper orthogonal to the Z axis and the X axis is the Y axis plus direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes in FIG.

  In the pixel area of the imaging chip 111, for example, 20 million or more pixels are arranged in a matrix. In the example of FIG. 2, four adjacent pixels of 2 pixels × 2 pixels form one unit region 131. The grid lines in the figure indicate the concept that adjacent pixels are grouped to form a unit region 131. The number of pixels forming the unit region 131 may be any number, for example, 1000 pixels or 1 pixel. Further, the number of pixels may be different between unit areas.

  As shown in the partially enlarged view of the pixel region, the unit region 131 includes a so-called Bayer array composed of four pixels of green pixels Gb and Gr, a blue pixel B, and a red pixel R. The green pixels Gb and Gr are pixels having a green filter as a color filter, and receive light in the green wavelength band of incident light. Similarly, the blue pixel B is a pixel having a blue filter as a color filter and receives light in the blue wavelength band, and the red pixel R is a pixel having a red filter as a color filter and receiving light in the red wavelength band. Receive light.

  In the present embodiment, a plurality of blocks are defined so as to include at least one unit region 131 per block. That is, the minimum unit of one block is one unit area 131. As described above, of the possible values for the number of pixels forming one unit region 131, the smallest number of pixels is one pixel. Therefore, when one block is defined in units of pixels, the minimum number of pixels among the number of pixels that can define one block is one pixel. Each block can control pixels included in each block with different control parameters. In each block, all the unit areas 131 in the block, that is, all the pixels in the block are controlled under the same imaging condition. That is, photoelectric conversion signals having different imaging conditions can be acquired between a pixel group included in a certain block and a pixel group included in another block. Examples of the control parameters include a frame rate, a gain, a thinning rate, the number of addition rows or addition columns to which photoelectric conversion signals are added, a charge accumulation time or accumulation count, a digitization bit number (word length), and the like. The imaging device 100 can freely perform not only thinning in the row direction (X-axis direction of the imaging chip 111) but also thinning in the column direction (Y-axis direction of the imaging chip 111). Furthermore, the control parameter may be a parameter in image processing.

<Explanation of distance measurement>
In the present embodiment, a plurality of light beams passing through different pupil positions of the imaging optical system 31 (FIG. 3) based on distance measurement image signals from focus detection pixels discretely provided on the imaging chip 111. The defocus amount of the imaging optical system 31 is obtained by detecting the shift amounts (phase differences) of a plurality of images due to. Since the defocus amount and the distance to the object correspond one-to-one, the distance from the camera 3A to the object can be obtained.
In the imaging chip 111, normal imaging pixels are provided at pixel positions other than the focus detection pixels. The imaging pixel outputs an image signal for monitoring outside the vehicle.

<Explanation of camera>
FIG. 3 is a block diagram illustrating the configuration of the camera 3A (3B) having the above-described image sensor 100. As shown in FIG. As described above, the configurations of the cameras 3A and 3B are the same. In FIG. 3, the camera 3 </ b> A (3 </ b> B) includes an imaging optical system 31, an imaging unit 32, an image processing unit 33, a work memory 34, a control unit 35, and a recording unit 36. Since the drive part 37 is used in 2nd embodiment, it is not an essential structure in 1st embodiment.

  The imaging optical system 31 guides the light beam from the object scene to the imaging unit 32. The imaging unit 32 includes the imaging device 100 and the driving unit 32a, and photoelectrically converts an image of an object formed on the imaging chip 111 by the imaging optical system 31. The drive unit 32a generates a drive signal necessary for causing the image sensor 100 (image pickup chip 111) to perform independent accumulation control in units of blocks described above. Instructions such as the position and shape of the block, its range, and accumulation time are transmitted from the control unit 35 to the drive unit 32a.

  The image processing unit 33 includes a first image processing unit 30a and a second image processing unit 30b. In the present embodiment, as will be described later, the control unit 35 divides the imaging surface of the imaging chip 111 of the imaging element 100 into a first area and a second area, and imaging conditions that are different in the first area and the second area. To capture the image. In this case, for example, the first image processing unit 30 a executes image processing on the image signal output from the first region of the image sensor 100. The second image processing unit 30b executes image processing on the image signal output from the second area of the image sensor 100.

  FIG. 4A to FIG. 4C are diagrams illustrating the first area and the second area divided by the control unit 35. According to FIG. 4 (a), the first area is constituted by the above-mentioned blocks in odd-numbered columns, and the second area is constituted by the above-mentioned blocks in even-numbered columns. That is, the block in the imaging chip 111 is divided into odd columns and even columns.

  According to FIG. 4 (b), the first region is constituted by the blocks in the odd rows, and the second region is constituted by the blocks in the even rows. That is, the block in the imaging chip 111 is divided into odd rows and even rows.

  According to FIG. 4 (c), the first region is constituted by the above-mentioned block of the odd-numbered row in the odd-numbered column and the above-mentioned block of the even-numbered row in the even-numbered column. In addition, the second region is configured by the above-described block of an odd-numbered row in an even-numbered column and the above-described block of an even-numbered row in an odd-numbered column. That is, in the imaging chip 111, the blocks are divided so as to have a checkered pattern.

  Returning to FIG. 3, the first image processing unit 30 a and the second image processing unit 30 b respectively use the work memory 34 as a work space for the image signal output from the first area and the image signal output from the second area. Perform image processing. For image processing, processing for generating RGB image signals by performing color signal processing (tone correction) on signals obtained by the Bayer array, white balance adjustment, sharpness adjustment, gamma correction, gradation for RGB image signals Includes processing to make adjustments. Also, the image processing includes a process of compressing in a predetermined compression format (JPEG format, MPEG format, etc.) as necessary. The first image processing unit 30a and the second image processing unit 30b further perform color detection of an object included in the image.

  The work memory 34 is a memory that temporarily stores image data before and after image processing. The recording unit 36 records image data and the like on a storage medium constituted by a nonvolatile memory or the like. The control unit 35 includes, for example, a CPU 35a and a storage unit 35b. The CPU 35a controls the overall operation of the camera 3A (3B) in accordance with a control signal from the control device 4. For example, a predetermined exposure calculation is performed based on the image signal imaged by the imaging unit 32, and an instruction is given to the drive unit 32a regarding the accumulation time of the imaging chip 111 necessary for proper exposure. The image data acquired by the camera 3A (3B) and the calculated distance measurement data are sent to the control device 4 (FIG. 1). The storage unit 35b stores a program executed by the CPU 35a and necessary information.

<Block control of image sensor>
The control device 4 causes the image pickup device 100 (image pickup chip 111) of the camera 3A (3B) to perform independent accumulation control in units of the blocks described above. For example, the control device 4 sends an instruction to the control unit 35, divides the image pickup surface of the image pickup chip 111 as described above, sets the first area and the second area, and sets between the first area and the second area. Charge accumulation (imaging) is performed under different imaging conditions. Although the imaging conditions differ between the first area and the second area, the image acquired based on the first area and the image acquired based on the second area each include the same subject (object).

In addition, in the first area (or the second area), the control device 4 sets the fourth area and the third area where the degree of attention is desired to be higher than that of the fourth area, and sets the area between the third area and the fourth area. The charge accumulation (imaging) is performed under different conditions. In this case, the subjects (objects) acquired in the third area and the fourth area are different.
Here, in the third area and the fourth area, in addition to the control parameters described above, the size and position of the third area and the fourth area are also one of the imaging conditions.

<Front camera>
In the present embodiment, the control device 4 sets the first region and the second region as shown in FIG. 4A for the image sensor 100 of the camera 3A that images the front of the vehicle 1, for example. The control device 4 uses the image acquired based on the first region to detect the travel path and the object of the vehicle 1 described above. Further, the control device 4 uses the image acquired based on the second region for display on the display / playback device 14 (FIG. 1).
The control device 4 further sets, as the third region, a region in which the driver is not gazing in the first region or a region where the degree of gazing by the driver is low.

<When turning right>
FIG. 5 is a diagram schematically showing an image of a subject (target object) formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 that is going to turn right at the intersection. Although an inverted inverted image is actually formed, it is shown as an erect image for the sake of clarity. In FIG. 5, an imaging surface (imaging area) 70 of the imaging chip 111 includes an image of an intersection road, an image of an oncoming vehicle 91, and an image of a pedestrian 92.
In the case of division as shown in FIG. 4A, the odd-numbered columns in the image of FIG. 5 correspond to the first region and the even-numbered columns correspond to the second region.

  The control device 4 sets different conditions between the first area and the second area on the imaging surface 70 to perform charge accumulation (imaging). For example, for the unit area 131 corresponding to the first area (FIG. 2), the camera 3A is controlled so as to capture an image with the frame rate set to 120 fps, and the unit area 131 corresponding to the second area is framed. The camera 3A is controlled so as to capture an image with the rate set to 30 fps.

In general, the first region where the frame rate is set high is limited to a shorter charge accumulation time than the second region where the frame rate is set low. For this reason, in a dark environment, the gain is increased to prevent the image from becoming dark. By increasing the gain, the image becomes bright and the object can be detected appropriately. At this time, it is possible to appropriately detect the object even when the noise is increased by increasing the gain. On the other hand, in the second area where the frame rate is set low, the charge accumulation time can be made longer than in the first area, so that an image suitable for display that is bright and has little noise can be obtained without increasing the gain.
In the above description, the frame rate is set to be different between the first region and the second region. However, the gains of the first region and the second region may be made different without changing the frame rate. For example, if the gain of the first region is set higher than the gain of the second region, an object can be appropriately detected in the first region even in dark conditions such as evening or night.

  Generally, a driver of a right turn vehicle turns his / her line of sight to an area 90 including a pedestrian 92 walking on a crosswalk of an oncoming vehicle 91 and a right turn destination road among the objects, and thus the degree of attention to the area 90 increases. Therefore, the control device 4, between the first region on the imaging surface 70, between the left third region 81 not including the region 90 and the right fourth region 82 (shaded) excluding the third region 81. Charge accumulation (imaging) is performed by setting different conditions. For example, the camera 3 </ b> A is controlled to set the third condition for each of the unit areas 131 (FIG. 2) corresponding to the third area 81 of the imaging chip 111 and the fourth area 82. The camera 3 </ b> A is controlled so that the fourth condition is set for each of the unit areas 131 and images are taken.

  A plurality of third regions 81 and a fourth region 82 may be provided, or a plurality of regions having different charge accumulation controls (imaging conditions) may be provided in the third region 81 or the fourth region 82. Furthermore, a rest area where charge accumulation (imaging) is not performed in the third area 81 and the fourth area 82 may be provided.

<When turning left>
FIG. 6 is a diagram schematically showing an image of a subject (object) that is imaged on the imaging chip 111 of the front camera 3A provided in the vehicle 1 about to turn left at the intersection. In FIG. 6, an imaging surface (imaging area) 70 of the imaging chip 111 includes an image of an intersection road, an image of an oncoming vehicle 91, and an image of a pedestrian 92.
In the case of division as shown in FIG. 4 (a), the odd-numbered columns in the image of FIG. 6 correspond to the first region and the even-numbered columns correspond to the second region. The control device 4 controls the camera 3A so as to capture images by setting different frame rates between the first area and the second area, as in the case of a right turn.

  In general, a driver of a left turn vehicle turns his / her line of sight to an area 90 including a pedestrian 92 walking on a pedestrian crossing of a left turn destination road among objects, and thus the degree of attention to the area 90 increases. Therefore, the control device 4, between the first region on the imaging surface 70, between the right third region 81 not including the region 90 and the left fourth region 82 (shaded) excluding the third region 81. Charge accumulation (imaging) is performed by setting different conditions.

<When going straight>
FIG. 7 is a diagram schematically showing an image of a subject (target object) formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 traveling straight on the road. In FIG. 7A, the imaging surface (imaging area) 70 of the imaging chip 111 includes a straight road image, an image of the preceding vehicle 93, and an image of the other vehicle 94 traveling in the adjacent lane.
In the case of division as shown in FIG. 4A, the odd-numbered columns in the image of FIG. 7A correspond to the first region and the even-numbered columns correspond to the second region. The control device 4 controls the camera 3A to set a different frame rate between the first area and the second area and to take an image in the same way as when turning right or turning left.

  In general, a driver of a straight-ahead vehicle focuses his / her line of sight exclusively on the area 90 including the front center, and thus the degree of attention to the area 90 is high. Therefore, the control device 4 includes, between the first regions on the imaging surface 70, the third regions 81 at the left and right ends that do not include the region 90, and the fourth region 82 (shaded) in the center excluding the third region 81. The charge accumulation (imaging) is performed by setting different conditions for.

<At high speed>
FIG. 7B schematically shows an image of a subject (object) formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 having a vehicle speed V higher than that in FIG. 7A. FIG. In FIG. 7B, the imaging surface (imaging area) 70 of the imaging chip 111 includes a straight road image, an image of the preceding vehicle 93, and an image of the other vehicle 95 traveling in the adjacent lane.
In the image of FIG. 7B, odd-numbered columns correspond to the first region, even-numbered columns correspond to the second region, and the control device 4 has different frame rates between the first region and the second region. The point that the camera 3A is controlled so as to capture the image is the same as in the case of the image of FIG.

  In general, the driver moves his / her line of sight farther as the speed of the host vehicle becomes higher, so the range of the region 90 having a high degree of attention is narrower than that at low speed. Therefore, the control device 4 includes, in the first area on the imaging surface 70, a third area 81 that is a peripheral portion of the screen that does not include the area 90 (around the area 90), and a central fourth area that excludes the third area 81. Charge accumulation (imaging) is performed by setting different conditions between the region 82 (shaded).

<When the sun is included in the image>
FIG. 8 is a diagram schematically showing an image formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 traveling toward the morning sun or sunset. In FIG. 8, an image of the sun 96 is included in the imaging surface (imaging area) 70 of the imaging chip 111.
In the image of FIG. 8, the odd number columns correspond to the first region, the even number columns correspond to the second region, and the controller 4 sets different frame rates between the first region and the second region. The point that the camera 3A is controlled so as to capture the image is the same as in the above-described examples.

  In general, when the driver cannot directly view the sun 96, the driver not only deflects his / her line of sight from the sun 96, but also deflects his / her line of sight to the outside of the high brightness region 90 around the sun 96. As a result, the degree of attention to the high luminance region 90 is lowered. Therefore, the control device 4 includes, in the first area on the imaging surface 70, a third area 81 that is the high brightness area 90 (around the sun 96), and a fourth area 82 (shaded) that excludes the third area 81. Charge accumulation (imaging) is performed by setting different conditions during the period.

<When objects that move within the screen are included>
FIG. 9 is a diagram schematically showing an image formed on the imaging chip 111 of the front camera 3A provided in the vehicle 1 approaching the intersection where there is no traffic signal. In FIG. 9, the imaging surface (imaging area) 70 of the imaging chip 111 includes a vehicle 97 that travels straight from the right direction to the left direction of the intersection and an oncoming vehicle 91 that travels straight through the intersection.
In the image of FIG. 9, the odd number columns correspond to the first region, the even number columns correspond to the second region, and the control device 4 sets different frame rates between the first region and the second region. The point that the camera 3A is controlled so as to capture the image is the same as in the above-described examples.

  In general, since the driver moves his / her line of sight in the moving direction of the moving object, the degree of attention to the region 90 including the vehicle 97 traveling leftward is increased. Therefore, the control device 4 includes a third region 81 on the right side (front right) that does not include the region 90 and a fourth region 82 on the left side excluding the third region 81 (shaded) among the first regions on the imaging surface 70. Charge accumulation (imaging) is performed by setting different conditions between and.

<Description of flowchart>
Hereinafter, how to determine the third area 81 and the fourth area 82 will be described with reference to flowcharts (FIGS. 10 and 11). FIG. 10 is a flowchart for explaining the overall flow of the control process of the camera 3 </ b> A executed by the control device 4. A program for executing the processing according to the flowchart of FIG. 10 is stored in the storage unit 4 b of the control device 4. For example, when the power supply is started (system is turned on) from the vehicle 1 or the engine is started, the control device 4 starts a program for performing the processing shown in FIG.

  In step S10 of FIG. 10, the control device 4 determines whether or not the flag a = 0. The flag a is a flag that is set to 1 when the initial setting is completed and set to 0 when the initial setting is not completed. When the flag a = 0, the control device 4 makes a positive determination in step S10 and proceeds to step S20. When the flag a ≠ 0, the control device 4 makes a negative determination and proceeds to step S30.

  In step S20, the control device 4 performs initial setting on the camera 3A, and proceeds to step S30. In the initial setting, a predetermined setting for causing the camera 3A to perform a predetermined operation is performed, and 1 is set in the flag a. Thereby, the camera 3A sets the same imaging condition over the entire imaging surface of the imaging device 100, and starts imaging at a frame rate of 60 frames per second (60 fps), for example.

In step S30, the control device 4 performs an imaging condition setting process and proceeds to step S40. The imaging condition setting process sets, for the imaging device 100 of the camera 3A, a first area composed of odd-numbered blocks and a second area composed of even-numbered blocks, and the first area The third region 81 and the fourth region 82 are set, and the respective imaging conditions are determined for each region. Details of the imaging condition setting process will be described later. In the present embodiment, the first area is set to have a higher frame rate, higher gain, lower thinning rate, and shorter accumulation time than the second area. In the third area 81, the frame rate is increased, the gain is increased, the thinning rate is decreased, and the accumulation time is set shorter than that in the fourth area 82. The camera 3A performs imaging based on this setting and performs the distance measurement (ranging) described above.
Note that it is not necessary to change all of the frame rate, the gain, the thinning rate, the accumulation time, and the like between the first area and the second area, and at least one of them may be changed. Further, it is not necessary to make all the frame rate, gain, thinning rate, accumulation time, and the like different between the third region 81 and the fourth region 82, and at least one of them may be made different.

  In step S40 of FIG. 10, the control device 4 acquires the image data, distance measurement data, and information from each part in the vehicle 1 acquired by the camera 3A after the imaging condition setting process, and proceeds to step S50. In step S50, the control device 4 determines whether or not the setting for displaying information is performed. When the display setting is performed, the control device 4 makes a positive determination in step 50 and proceeds to step S60. When the display setting is not performed, the control device 4 makes a negative determination in step 50 and proceeds to step S70.

  In step S60, the control device 4 sends display information to the display / reproduction device 14 (FIG. 1), and proceeds to step S70. For example, the control device 4 causes the display / reproduction device 14 (FIG. 1) to display an image acquired based on the second region of the image sensor 100 as display information. Further, when the control device 4 detects an object (for example, the oncoming vehicle 91 in FIG. 6) from the image acquired based on the first region, the image (first display) displayed on the display / playback device 14 (FIG. 1). The corresponding object (the oncoming vehicle in the above example) is conspicuous among the images acquired in the two regions. As a conspicuous mode, for example, a frame surrounding the object is displayed, the display brightness of the object is increased to display it brightly, blinking, or colored.

The control device 4 may display the following message on the display / playback device 14 in addition to the image displayed on the display / playback device 14 (FIG. 1) (image acquired in the second region). As an example of the message, for example, when an object (an oncoming vehicle in the above example) is detected from an image acquired based on the first region, the message is “There is an oncoming vehicle ahead”.
Instead of displaying the message on the display / playback device 14, or together with the display of the message, an audio signal for reproducing the message may be transmitted. An audio device of a navigation device (not shown) may be used as the audio reproducing device.

  In step S70, the control device 4 determines whether or not an off operation has been performed. For example, when the control device 4 receives an off signal (for example, a system off signal or an engine off signal) from the vehicle 1, the controller 4 makes an affirmative decision in step S70, performs a predetermined off process, and ends the process of FIG. For example, if the control device 4 does not receive an off signal from the vehicle 1, the control device 4 makes a negative determination in step S70 and returns to step S30. When returning to step S30, the above-described processing is repeated.

<Imaging condition setting process>
Details of the imaging condition setting process (S30) will be described with reference to the flowchart of FIG. In step S31 of FIG. 11, the control device 4 inputs line-of-sight information from the line-of-sight detection device 16 (FIG. 1), and proceeds to step S32A.

  In step S32A, the control device 4 sets a first area composed of odd-numbered blocks and a second area composed of even-numbered blocks on the imaging surface 70 of the imaging chip 111, and sets step S32B. Proceed to In step S32B, the control device 4 sets the area not including the driver's line-of-sight target as the third area 81 in the first area, and sets the area other than the third area 81 as the fourth area 82, and proceeds to step S33. For example, when the driver of the vehicle 1 is gazing exclusively at the front front, the central portion of the first region on the imaging surface 70 is the same as in the case illustrated in FIG. 7A or 7B. The fourth region 82 is used, and the region excluding the central portion is the third region 81.

  In step S <b> 33, the control device 4 inputs line-of-sight information from the line-of-sight detection device 16 (FIG. 1) and determines whether or not there is a line-of-sight movement. For example, when the line-of-sight movement amount is larger than a predetermined value, the control device 4 makes a positive determination in step S33 and proceeds to step S34. If the line-of-sight movement amount is not greater than the predetermined value, the control device 4 makes a negative determination in step S33 and proceeds to step S36.

  In step S <b> 34, the control device 4 determines whether or not the moving direction of the line of sight matches the turning direction of the vehicle 1. The turning direction of the vehicle 1 is detected based on, for example, any one of the operation direction of the turn signal switch 11, the rotation operation direction of the steering wheel 10, and the comparison with the image of the previous frame acquired by the camera 3A. The control device 4 makes a positive determination in step S34 when the moving direction of the line of sight coincides with the turning direction of the vehicle 1, and proceeds to step S35. When the moving direction of the line of sight does not coincide with the turning direction of the vehicle 1, the control device 4 makes a negative determination in step S34 and proceeds to step S39.

  In step S35, as described with reference to FIGS. 5 and 6, the control device 4 in the first region on the imaging surface 70 is in a direction opposite to the direction in which the driver's line of sight is directed when turning right or turning left. The area to be positioned is set as the third area 81, and the area other than the third area 81 (that is, the area in the direction of the line of sight) is set as the fourth area 82, and the process proceeds to step S3E.

  In step S39, which proceeds after making a negative determination in step S34, the control device 4 determines whether or not the movement direction of the line of sight matches the movement direction of the image of the object moving on the imaging surface 70. The moving direction of the image of the object is detected based on, for example, comparison with the image of the previous frame acquired by the camera 3A. When the movement direction of the line of sight matches the movement direction of the image of the object, the control device 4 makes a positive determination in step S39 and proceeds to step S35. When the moving direction of the line of sight does not match the moving direction of the image of the object, the control device 4 makes a negative determination in step S39 and proceeds to step S3A. In step S35 that proceeds with an affirmative determination in step S39, the control device 4 reverses the left direction in which the line of sight follows the vehicle 97 in the first area on the imaging surface 70 as described above with reference to FIG. The region located at is set as the third region 81, and the region other than the third region 81 (that is, the region in the direction of the line of sight) is set as the fourth region 82, and the process proceeds to step S3E.

  In step S <b> 3 </ b> A, the control device 4 determines whether or not a sun image is included in the subject image formed on the imaging surface 70. For example, when there is a region where the value of the image data exceeds a predetermined value, the control device 4 makes an affirmative determination in step S3A and proceeds to step S3B, where there is a region where the value of the image data exceeds the predetermined value. If not, step S3A is negatively determined and the process proceeds to step S3E. When negative determination is made in step S3A, the control device 4 maintains the third area 81 and the fourth area 82 set in step S32B.

  In step S3B, the control device 4 determines whether or not the moved line-of-sight position is away from the position of the image of the sun 96 (FIG. 8). When the line-of-sight position is away from the position corresponding to the sun 96 (the driver is turning the line of sight from the sun), the control device 4 makes a positive determination in step S3B and proceeds to step S3C. The control device 4 makes a negative determination in step S3B when the line-of-sight position is at a position corresponding to the sun 96 (the driver directs the line of sight toward the sun), and proceeds to step S3E. When negative determination is made in step S3B, the control device 4 maintains the third area 81 and the fourth area 82 set in step S32B.

  In step S <b> 3 </ b> C, as described with reference to FIG. 8, the control device 4, among the first regions on the imaging surface 70, the high-intensity region 90 around the image of the sun 96 ( The region near the position of the 96 images) is set as the third region 81, and the region other than the third region 81 is set as the fourth region 82, and the process proceeds to step S3D.

  In step S3D, the control device 4 sends an instruction to the camera 3A, sets the area of the image of the sun 96 to be lower than the gain of the third area 81, and proceeds to step S3E. Specifically, a gain lower than the gain set in the unit region 131 corresponding to the third region 81 is set for the unit region 131 (FIG. 2) corresponding to the image of the sun 96.

  In step S36, which proceeds after making a negative determination in step S33 described above, the control device 4 determines whether or not the vehicle 1 is traveling straight ahead and the driver's line-of-sight position is in the approximate center of the front field. When the vehicle 1 is traveling straight ahead and the driver's line-of-sight position is at a position corresponding to the approximate center of the front field, the control device 4 makes a positive determination in step S36 and proceeds to step S37. If the vehicle 1 is not traveling straight or the driver's line-of-sight position is not at a position substantially corresponding to the center, the control device 4 makes a negative determination in step S36 and proceeds to step S38.

  In step S <b> 37, the control device 4 is located in a direction opposite to the central direction in which the line of sight is directed in the first region on the imaging surface 70, as described with reference to FIGS. 7A and 7B. The peripheral region is set as the third region 81, and the region other than the third region 81 (that is, the region at the approximate center of the screen) is set as the fourth region 82, and the process proceeds to step S3E.

  In step S38, which proceeds after making a negative determination in step S36 described above, the control device 4 determines whether or not the vehicle 1 is decelerating and the driver's line-of-sight position is approximately in the center of the front field. The deceleration of the vehicle 1 is detected based on, for example, any one of the depression signal of the brake pedal 8a, the decrease in the vehicle speed V, and the comparison with the image of the previous frame acquired by the camera 3A. When the vehicle 1 is decelerating and the driver's line-of-sight position is at a position corresponding to the approximate center of the front field, the control device 4 makes a positive determination in step S38 and proceeds to step S3E. When the vehicle 1 is not decelerating or the driver's line-of-sight position is not at a position substantially corresponding to the center, the control device 4 makes a negative determination in step S38 and proceeds to step S3E.

  In step S3E, the control device 4 sends an instruction to the camera 3A, sets the frame rate of the third area 81 higher than the frame rate of the fourth area 82, and proceeds to step S40 in FIG. For example, the frame rate of the third area 81 is 120 frames per second (120 fps), and the frame rate of the fourth area 82 is 30 fps. This is to increase the frequency of acquiring image information about a region where the driver's attention level is low.

<Rear camera>
The control device 4 sets the third region and the fourth region for the camera 3B that captures the rear of the vehicle 1 based on the setting information of the third region 81 and the fourth region 82 for the camera 3A that captures the front of the vehicle 1. Is performed as follows.
Note that the control device 4 also sets the first region and the second region as shown in FIG. 4A even in the camera 3B that captures the rear, and the image of the vehicle 1 described above based on the image acquired based on the first region is set. Used to detect roads and objects. Further, the control device 4 uses the image acquired based on the second region for display on the display / playback device 14 (FIG. 1). The control device 4 further sets a fourth region and a third region in which the degree of attention is to be higher than that in the fourth region in the first region, and accumulates charges (imaging images) under different conditions between the third region and the fourth region. ). For example, the frame rate of the third area of the rear camera 3B is 120 fps, and the frame rate of the fourth area of the rear camera 3B is 30 fps.

<When turning left>
FIG. 12 is a diagram schematically showing an image of a subject (object) that is imaged on the imaging chip 111 of the rear camera 3B provided in the vehicle 1 that is about to turn left at the intersection. As shown in the left side of the vehicle 1 on the left side, the left and right sides are reversed as compared with the case of FIGS. In FIG. 12, the imaging surface (imaging area) 70 of the imaging chip 111 includes an image of the road behind the vehicle 1, an image of the two-wheeled vehicle 98 that follows the vehicle 1, and an image of the vehicle 99 that further follows.

  As described above with reference to FIG. 6, the driver of the left turn vehicle has a high degree of attention to the area 90 on the left front side. Therefore, the control device 4 is arranged on the imaging surface 70 illustrated in FIG. 12 so as to increase the acquisition frequency of the image information of the two-wheeled vehicle 98 that may pass through the left side of the vehicle 1 in the rear where the driver's attention is low. In one region, charge accumulation (imaging) is performed by setting different conditions between the third region 81 on the left rear side and the fourth region 82 (shaded) on the right side excluding the third region 81. According to FIGS. 6 and 12, the front camera 3 </ b> A at the left turn has the third area on the right side as it is forward, and the rear camera 3 </ b> B at the left turn has the third area on the right side as it is rearward.

<When turning right>
In the case of a right turn, the left and right settings may be interchanged with those in a left turn. That is, as described above with reference to FIG. 5, the driver of the right turn vehicle has a high degree of attention to the area 90 on the right front side. Therefore, the control device 4 is configured to increase the frequency of acquiring the image information of the vehicle passing through the right side of the vehicle 1 in the rear where the degree of attention of the driver is low. Charge accumulation (imaging) is performed by setting different conditions between the third region 81 and the left fourth region 82 excluding the third region 81. That is, the camera 3A in front of the right turn has the third area on the left side as viewed forward (FIG. 5), and the camera 3B in the rear of the right turn has the third area on the left side as viewed in the rear.

  The setting of the third area 81 and the fourth area 82 for the rear camera 3B at the time of left turn and right turn is performed before and after the process of step S35 (FIG. 11) for the front camera 3A (may be simultaneous). .

<Deceleration>
FIG. 21 is a diagram schematically illustrating an image of a subject (target object) formed on the imaging chip 111 of the rear camera 3B provided in the vehicle 1 that decelerates on an expressway, for example. In FIG. 21, the imaging surface (imaging area) 70 of the imaging chip 111 includes an image of a road behind the vehicle 1 and an image of the following vehicle 99. In general, a driver of a vehicle who has applied a brake has a high degree of attention to a region 90 (for example, FIG. 7B) including the front center. Therefore, the control device 4 exemplifies the imaging illustrated in FIG. 21 so as to increase the acquisition frequency of the image information of the vehicle 99 that may approach the vehicle 1 when the vehicle 1 decelerates, especially in the rear where the driver's attention is low. Charge accumulation (imaging) by setting different conditions between the third region 81 in the substantially central portion of the first region on the surface 70 and the fourth region 82 (shaded) in the peripheral portion excluding the third region 81. To do.

The setting of the third area 81 and the fourth area 82 for the rear camera 3B at the time of deceleration is performed before or after the process of step S37 for the front camera 3A, which proceeds with an affirmative determination in step S38 (FIG. 11). Yes (or at the same time).
In the above-described embodiment, the sight line of the occupant is detected by the sight line detection device 16, but the frame rate of the entire first region is set high without detecting the sight line of the occupant to detect the object. You may make it perform.

According to the first embodiment described above, the following operational effects can be obtained.
(1) An imaging device that is an example of an electronic device captures a first moving image and a second moving image outside a vehicle under different imaging conditions in a first region on the imaging surface and a second region different from the first region. 3A, a control device 4 that detects an object in the first moving image, and a display / playback device 14 that displays information about the object detected by the control device 4 in a second moving image. Thereby, the imaging condition suitable for the detection of the object can be set for the first region. In addition, the detected object can be visually notified to the occupant.

(2) An imaging device which is an example of an electronic apparatus captures a first moving image and a second moving image outside a vehicle under different imaging conditions in a first region on the imaging surface and a second region different from the first region. 3A that does not include the line-of-sight position of the first area in the first moving image based on the detection result by the line-of-sight detection device 16 and the camera 3A that detects the driver's line-of-sight position. The control device 4 for detecting the object in the above and the display / reproduction device 14 for displaying the information of the object detected by the control device 4 by the second moving image are provided. Thereby, the imaging condition suitable for the detection of the object can be set for the third region 81 having a low degree of attention by the driver. In addition, the detected object can be visually notified to the occupant.

(3) Since the camera 3A makes the frame rate of the first region higher than the frame rate of the second region, the camera 3A acquires image information about the first moving image used for detection of the object as compared with the second moving image used for display. The frequency of doing can be increased.

(4) The imaging device varies imaging conditions of the first moving image in the third area 81 that does not include the line-of-sight position in the first area and the fourth area 82 that includes the line-of-sight position in the first area. The control device 4 is set. Thereby, imaging conditions suitable for the fourth region where the driver's line of sight is directed and the third region where the driver's line of sight is not directed can be set.

(5) The control device 4 sets the frame rate of the third area 81 not including the line-of-sight position in the first area higher than the frame rate of the fourth area 82 including the line-of-sight position in the first area. . Thereby, the frequency which acquires image information about the 3rd field where a driver's line of sight does not turn can be raised compared with the 4th field where a driver's line of sight faces.

  In the above-described embodiment, the camera 3A (3B) is controlled by the control device 4, but a part of the control of the camera 3A (3B) is performed by the control unit 35 of the camera 3A (3B). May be.

(Modification 1)
Based on the image information acquired by the camera 3 </ b> A that captures the front of the vehicle 1, the control device 4 sets the third region 81 and the fourth region 82 with respect to the first region of the camera 3 </ b> B that captures the rear of the vehicle 1. You may make it perform a setting.

  For example, the case where the image of the sun 96 as illustrated in FIG. 8 is detected on the right side of the front image of the vehicle 1 acquired by the camera 3A will be described. In the first modification, the control device 4 gives the gain for the right region (front right) of the first region on the imaging surface 70 to the camera 3A that captures the front of the vehicle 1, and the left side of the first region on the imaging surface 70. Set lower than the gain for the area (front left). On the other hand, for the camera 3 </ b> B that captures the rear side of the vehicle 1, the control device 4 sets the gains for the left and right side regions (rear left and rear right) of the first region on the imaging surface 70 to be equal.

  When the vehicle 1 makes a left turn in the state described above, there is a possibility that an image of the sun 96 appears in the camera 3B that captures the rear of the vehicle 1 after the vehicle 1 makes a left turn. Specifically, an image of the sun 96 appears in the left region (that is, the rear right of the vehicle 1) toward the rear of the vehicle 1 after the left turn. Therefore, the control device 4 sets the gain for the left region (rear right) in the first region on the imaging surface 70 of the camera 3B to be lower than the gain for the right region (rear left) of the first region on the imaging surface 70. Further, the control device 4 is set so that the gains for the left and right side regions (front left and front right) of the first region on the imaging surface 70 are equal for the imaging surface 70 of the camera 3A that images the front of the vehicle 1. To return.

  According to the first modification, by setting the left and right regions in the first region of the camera 3B that captures the rear of the vehicle 1 based on image information acquired by the camera 3A that captures the front of the vehicle 1, For example, it is appropriate to prevent the saturation of the imaging signal, such as predicting in advance (before the left turn in this example) that the image of the sun 96 will appear in the rear camera 3B and setting the gain immediately when turning left. It becomes possible to deal with it.

(Modification 2)
Generally, when the driver's tension increases, the movement range of the driver's line of sight decreases. Therefore, the control device 4 may expand the range of the third region 81 from the normal time in the first region on the imaging surface 70 of the front camera 3A in a scene where the tension of the driver increases. An example of a scene in which the driver's tension increases is traveling at night or traveling during rain. When the headlight is turned on or the wiper is turned on, the control device 4 determines that the driver's tension is increased, and the first region of the front camera 3A is the third region from the normal time. 81 is controlled to expand.

  Alternatively, the driver's tension may be determined based on the driver's body temperature or heart rate. For example, if the seat of the driver's seat is provided with a temperature sensor or a heart rate sensor, and the temperature detection above a predetermined temperature or the heart rate above a predetermined number is detected, the control device 4 determines that the driver's tension has increased. Then, control is performed so that the third area 81 is expanded from the normal time in the first area of the front camera 3A.

  According to the second modification, in the case where the region where the driver's attention level is reduced due to the driver's tension is expanded (in other words, the region where the driver's attention level is high is narrowed), the third region in the first region. 81 can be expanded. If the third area 81 is expanded, when the imaging frequency in the third area 81 is made higher than the imaging frequency in the fourth area 82, an image based on the third area 81 is used to reduce the driver's attention reduction. It can be used to make up.

(Modification 3)
In the above embodiment, the frame rate of the third region 81 of the first region in the camera 3A (camera 3B) is 120 fps, and the frame rate of the fourth region 82 is 30 fps. The example of making it different was explained. Instead, a buffer area may be provided between the third area 81 and the fourth area 82, and the frame rate may be varied in multiple stages. For example, the frame rate is changed in multiple stages by setting the frame rate of the buffer area to an intermediate value (for example, 75 fps) between the frame rate of the third area 81 and the frame rate of the fourth area 82.

  Further, the frame rate in the buffer area may be gradually changed. For example, the frame rate of the buffer region near the boundary with the third region 81 is 90 fps, and the buffer rate of the buffer region near the boundary with the fourth region 82 is 60 fps. By providing such a buffer area, it is possible to avoid a state in which the value of the frame rate changes greatly at the boundary portion.

(Modification 4)
In Modification 4, a communication unit 17 is added in FIG. The communication unit 17 is a wireless communication device that performs narrow area communication called DSRC (Dedicated Short Range Communication), for example. In the modification 4, the communication part 17 acquires the information from the information provision system installed, for example on the road.

  The information providing system includes a camera similar to the camera 3A (3B) mounted on the vehicle 1, and is installed in advance at, for example, an intersection with poor visibility or a corner of a road. When the vehicle 1 enters a point with poor visibility, the information providing system provides a range that cannot be captured from the camera 3A (3B) of the vehicle 1 (for example, a blind spot hidden behind a fence or a building as viewed from the vehicle 1). Shoot at a predetermined frame rate. The information providing system transmits a radio signal carrying the photographed image information to the vehicle 1. The communication unit 17 of the vehicle 1 receives the wireless signal transmitted from the information providing system, and sends the image information carried on the wireless signal to the control device 4.

  For example, when the third area 81 set on the imaging surface 70 of the camera 3A corresponds to the above-described non-photographable range, the control device 4 does not use the information acquired based on the third area of the camera 3A, but the information An object (pedestrian, two-wheeled vehicle, other vehicle, etc.) is detected based on the image transmitted from the providing system.

  According to the modification 4, even when a blind spot is generated in the camera 3A (3B) of the vehicle 1, the detection of the object is performed based on an image acquired by an external camera different from the camera 3A (3B) of the vehicle 1. It becomes possible.

(Modification 5)
FIG. 14 is a diagram schematically illustrating an image of a subject (target object) formed on the imaging chip 111 of the front camera 3 </ b> A provided in the vehicle 1. In FIG. 14, the imaging surface (imaging area) 70 of the imaging chip 111 includes an image of a T-junction positioned in the traveling direction of the vehicle 1. The imaging surface 70 further includes a curve mirror 88A and a curve mirror 88B provided on the T-junction.
In the image of FIG. 14, the odd number columns correspond to the first region, the even number columns correspond to the second region, and the control device 4 sets different frame rates between the first region and the second region. The point of controlling the camera 3A so as to capture an image is the same as in the above-described embodiment.

  In Modification 5, when the control device 4 detects a curve mirror from an image based on the first region acquired by the camera 3A, the control device 4 displays an image of the curve mirror based on the second region acquired by the camera 3A. Display on the playback device 14 (FIG. 1). At this time, the mirror surface portions of the curve mirrors 88A and 88B are greatly enlarged and displayed. When the mirror surface portions of the curve mirrors 88A and 88B are greatly extended, the curvature of the curved image of the mirror surface may be corrected so as to be linear.

  For the enlarged display of the mirror surface portions of the curve mirrors 88A and 88B, a method of reducing the thinning rate of pixel data used for display by so-called electronic zoom processing may be used. By changing the thinning rate low, the resolution can be increased compared to before the change.

However, in the modified example 5, the area where the mirror surfaces of the curved mirrors 88A and 88B are reflected in the second area (referred to as the fifth area) is higher than the area other than the mirror surface (referred to as the sixth area). Set imaging conditions suitable for fine display. For example, the thinning rate of pixels included in the fifth region is set lower than the thinning rate of pixels included in the sixth region, and the frame rate in the fifth region is set higher than the frame rate in the sixth region. The resolution increases as the decimation rate decreases. By increasing at least one of the resolution and the frame rate, imaging conditions suitable for high-definition display are obtained.
Note that it is not necessary to make all the imaging conditions such as the frame rate and the thinning rate different between the fifth area and the sixth area, and at least one of them may be made different.

  FIG. 15 is a diagram illustrating a display screen of the display / playback apparatus 14 (FIG. 1) that displays an image based on the fifth area and the sixth area. In FIG. 15, for example, the curve mirror 88A is greatly enlarged and displayed on the left side of the display screen, and the curve mirror 88B is greatly enlarged and displayed on the right side of the display screen. The shaded area indicates that the image quality is expressed with a coarser image quality than the area corresponding to the mirror surface of the curve mirrors 88A and 88B. The difference in image quality is due to setting different imaging conditions between the fifth area and the sixth area.

  When the control device 4 further detects an object (for example, the two-wheeled vehicle 98 or the vehicle 99 in FIG. 15) from the area where the curved mirrors 88A and 88B are captured in the image acquired based on the fifth area, In the image (FIG. 15) displayed on the playback device 14 (FIG. 1), the corresponding object (two-wheeled vehicle 98, vehicle 99 in the above example) is made conspicuous. As a conspicuous mode, for example, a frame surrounding the object is displayed, the display brightness of the object is increased to display it brightly, blinking, or colored.

The control device 4 may cause the display / reproduction device 14 to display the following message over the image (FIG. 15) displayed on the display / reproduction device 14 (FIG. 1). Examples of the messages are, for example, “a motorcycle is reflected” and “a vehicle is reflected”.
Instead of displaying the message on the display / playback device 14, or together with the display of the message, an audio signal for reproducing the message may be transmitted. An audio device of a navigation device (not shown) may be used as the audio reproducing device.

<Description of flowchart>
Hereinafter, a method for determining the third region 81 and the fourth region 82 in Modification 5 will be described with reference to a flowchart (FIG. 16). FIG. 16 is a flowchart illustrating the details of the imaging condition setting process (S30B). The control device 4 executes an imaging condition setting process (S30B) subsequent to the imaging condition setting (S30) in FIG. 11 or instead of the imaging condition setting (S30) in FIG.

In step S301 in FIG. 16, the control device 4 sets a first area configured by odd-numbered blocks and a second area configured by even-numbered blocks on the imaging surface 70 of the imaging chip 111. Then, the process proceeds to step S302.
Note that step S301 is omitted when the imaging condition setting of FIG. 16 is executed following the imaging condition setting (S30) of FIG.

  In step S <b> 302, the control device 4 determines whether a curve mirror is detected from an image based on the first region of the image sensor 100. The detection of the curve mirror is performed by using a template matching method or by determining that the mirror surface of the curve mirror is located at a predetermined range from the road surface. When the control device 4 detects the curve mirror from the image based on the first region, the control device 4 makes a positive determination in step S302 and proceeds to step S303. If the control device 4 does not detect the curve mirror from the image based on the first region, the control device 4 makes a negative determination in step S302 and proceeds to step S306.

  In step S303, the control device 4 sets the area corresponding to the mirror surface of the curved mirror in the second area as the fifth area, and sets the area other than the fifth area as the sixth area, and proceeds to step S304.

  In step S304, the control device 4 sends an instruction to the camera 3A, sets the frame rate of the fifth area higher than the frame rate of the sixth area, and proceeds to step S305. For example, the frame rate of the fifth area is 120 frames per second (120 fps), and the frame rate of the sixth area is 30 fps. This is to increase the frequency of acquiring image information reflected on the mirror surface of the curved mirror that is difficult for the driver to understand.

  In step S305, the control device 4 sends an instruction to the camera 3A to set the pixel thinning rate included in the fifth region to be lower than the pixel thinning rate included in the frame rate of the sixth region. Proceed to step S40. This is because the image information reflected on the mirror surface of the curved mirror, which is difficult for the driver to understand, is acquired with high definition.

  In step S306, which proceeds after making a negative determination in step S302, the control device 4 sends an instruction to the camera 3A to set uniform imaging conditions for the entire second region, and proceeds to step S40 in FIG. For example, the frame rate is set to 30 fps, and a normal thinning rate is set.

  In the fifth modification, the control device 4 performs the following step 60B instead of step S60 in FIG. That is, in step S60B, the control device 4 sends display information to the display / playback device 14 (FIG. 1). For example, the control device 4 displays an image acquired based on the fifth region and the sixth region of the image sensor 100 on the display / reproduction device 14 (FIG. 1) as display information. Specifically, as illustrated in FIG. 15, the curve mirrors 88 </ b> A and 88 </ b> B among the images acquired in the fifth region are greatly enlarged and displayed on the display screen of the display / playback device 14 (FIG. 1).

The control device 4 further displays and reproduces when an object (for example, the two-wheeled vehicle 98 and the vehicle 99 in FIG. 15) is detected from the region where the curve mirrors 88A and 88B are captured in the image acquired based on the fifth region. 14 (FIG. 1), the corresponding objects (the two-wheeled vehicle 98 and the vehicle 99 in the above example) are conspicuous.
Since the image reflected on the mirror mirror is a mirror image, the control device 4 may display an image obtained by inverting the left and right of the image displayed on the display / playback device 14 (FIG. 1).

  According to the modification 5, the frequency which acquires the image information reflected on the mirror surface of the curve mirrors 88A and 88B that is difficult for the driver to understand can be increased. Further, it is possible to acquire the image information reflected on the mirror surfaces of the curved mirrors 88A and 88B, which is difficult for the driver to understand, with high definition. The acquired image information reflected on the mirror surfaces of the curve mirrors 88A and 88B can be displayed on the display / reproduction device 14 (FIG. 1) in an easily understandable manner.

(Modification 6)
In the above description, as a distance measurement performed by the cameras 3A and 3B, a method of calculating by distance measurement using an image signal from a focus detection pixel provided in the image sensor 100 is used. You may use the method of measuring distance using the image of 1 sheet. Further, a method of measuring distance using a millimeter wave radar separately from the cameras 3A and 3B may be used.

(Second embodiment)
In the second embodiment, a radar 18 is added in the driving support device 2 of FIG. The radar 18 is configured by a millimeter wave radar device, for example. The radar 18 detects an object or an obstacle in each area in association with a plurality of areas (called areas) on the shooting screen of the camera 3A. As a detection method for each area, a method of scanning the detection target area while changing the direction of the radar antenna (not shown) with an actuator, or a detection target area while switching multiple reception antennas corresponding to each area with a switch etc. However, any method may be used.

The communication unit 17 of the second embodiment performs wireless communication (optical beacon, radio wave) with an external device such as an information providing system installed on a road or a VICS (registered trademark: Vehicle Information and Communication System), for example. (Including beacon and visible light communication). Any communication method may be used.
In the second embodiment, the camera 3B that captures the rear is not an essential configuration.

<Explanation of camera>
In the second embodiment, a drive unit 37 is added in the block diagram of FIG. The drive unit 37 is configured by a motor, for example. The drive unit 37 changes the horizontal direction (yaw angle) of the camera 3 </ b> A in accordance with an instruction from the control unit 35. The orientation of the camera 3A is adjusted so that the object far from the front of the vehicle 1 is captured at the approximate center of the screen, that is, the vanishing point is aligned with the approximate center of the screen. The control unit 35 controls the yaw angle based on this direction.

<Block control of image sensor>
The control device 4 causes the imaging device 100 (imaging chip 111) of the camera 3A to perform independent accumulation control for each unit region 131 described above. Specifically, the control device 4 sets a non-priority area, a priority area, and a top-priority area on the imaging surface of the imaging chip 111, and performs charge accumulation (imaging) under different conditions between them. Here, in addition to the control parameters at the time of imaging described above, the size and position of the non-priority area, priority area, top priority area, whether or not to perform distance measurement, and the yaw angle of the imaging unit 32 are also one of the imaging conditions. It is.

FIG. 17 is a diagram schematically illustrating an image of a subject (target object) formed on the image sensor 100 of the camera 3 </ b> A that images the front of the vehicle 1. Although an inverted inverted image is actually formed, it is shown as an erect image for the sake of clarity.
Note that in some drawings referred to in the description of the second embodiment, there are reference numerals common to those used in the above-described drawings, but the description of this embodiment is applied.

In FIG. 17, the imaging surface of the imaging chip 111, the non-priority area 81, the priority area 82, and the highest priority area 83 in the imaging chip 111 are illustrated. The non-priority area 81, the priority area 82, and the highest priority area 83 perform charge accumulation (imaging) under different conditions.
Note that a pause region where charge accumulation (imaging) is not performed in the imaging chip 111 may be provided.

  The control device 4 sets the non-priority area 81, the priority area 82, and the highest priority area 83 based on the above-described depth distribution, the classification result of the object and the obstacle, and the like. For example, the non-priority area 81 is set so as to reduce the degree of attention with respect to a distant object, and the area including the preceding vehicle 70 is set as the highest priority area 83 so that the degree of attention with respect to the preceding vehicle 70 is increased. The priority area 82 is set so that the area other than the non-priority area 81 and the highest priority area 83 has a medium degree of attention.

  FIG. 18 is a diagram schematically illustrating an image of a subject (target object) when the entire imaging surface of the image sensor 100 is set as the priority area 82. In FIG. 18, the control device 4 sets the entire imaging surface of the imaging chip 111 as the priority area 82 (all area common control). Based on the distance image (depth distribution image) generated based on distance measurement data at a plurality of positions in the shooting screen, the control device 4 corresponds to a distance from the vehicle 1 to the subject equal to or greater than a predetermined threshold (for example, 300 m). Extract the area to be used.

  FIG. 19 is a diagram illustrating a case where a part of the imaging surface is changed to the non-priority area 81 from the state of FIG. The control device 4 changes the extraction area (that is, the distance from the vehicle 1 to the subject corresponds to 300 m or more) to the non-priority area 81. The control device 4 sets the above-described control parameters different between the non-priority area 81 and the priority area 82, and sets the non-priority area 81 not to read the distance measurement image signal. When the image signal for distance measurement is not read, power consumption is reduced.

For example, the control device 4 sets the frame rate lower for the unit area 131 (FIG. 2) of the image sensor 100 included in the non-priority area 81 than in the priority area 82. Based on the idea of lowering the degree of attention in the distance.
Further, the control device 4 may set the image acquired in the non-priority area 81 so as not to perform image processing such as tracking processing by the image processing unit 33.

FIG. 20 is a diagram illustrating a case where a part of the imaging surface is changed to the highest priority area 83 from the state of FIG. The control device 4 changes the entire imaging surface from the priority area 82 to the highest priority area 83 when the vehicle 1 travels through the intersection (approaching the intersection, entering the intersection, and exiting the intersection). Based on the idea of raising the degree of attention when driving at intersections. However, the area including the pedestrian crossing 92 that passes through the intersection when turning left (or when turning right) is left as the priority area 82. The reason for not including the pedestrian crossing 92 in the highest priority area 83 is as follows. That is, in general, since the occupant driving the vehicle 1 pays attention to the pedestrian crossing 92 that passes therethrough, it is based on the idea that there is little need to increase the degree of attention to the pedestrian crossing 92 by the camera 3A. In the present embodiment, the attention level by the camera 3 </ b> A is increased by setting, as the highest priority area 83, a region where the attention level by the occupant driving is low. This increases the degree of attention to other vehicles such as the vehicle 91 that makes a right turn from the oncoming lane in FIG.
Note that the control device 4 may change a region where the distance from the vehicle 1 to the subject is equal to or more than a predetermined threshold (for example, 300 m) to the non-priority region 81 based on the distance image (depth distribution image). .

  For example, the control device 4 sets the frame rate higher for the unit region 131 (FIG. 2) of the image sensor 100 included in the highest priority region 83 than in the priority region 82. When the non-priority area 81 is set, the frame rate is set lower for the unit area 131 (FIG. 2) of the image sensor 100 included in the non-priority area 81 than in the priority area 82. .

  Note that the control device 4 uses the sight line information sent from the sight line detection device 16 as the priority area 82 in addition to the area including the pedestrian crossing 92 recognized based on the image acquired by the camera 3A. The area may be the priority area 82.

  The depth distribution image used when the control device 4 sets the non-priority area 81, the priority area 82, and the highest priority area 83 is a first depth distribution image that is generated based on distance measurement data based on an image signal for distance measurement. Alternatively, a second depth distribution image generated based on detection information for each area by the radar 18 may be used. In the present embodiment, the first depth distribution image is used when the image movement (referred to as the image plane movement amount) on the imaging surface of the image sensor 100 is equal to or less than a predetermined value, and the image plane movement amount exceeds the predetermined value. A second depth distribution image is used. Details of the image plane movement amount will be described later.

<Description of flowchart>
Hereinafter, a flow of control processing of the camera 3A executed by the control device 4 will be described with reference to flowcharts (FIGS. 21, 22, 23, and 26). The program executed by the control device 4 is stored in the storage unit 4b of the control device 4. For example, when the power supply is started from the vehicle 1 or the engine is started, the control device 4 starts a program for performing the processing shown in FIG.

  In step S10 of FIG. 21, the control device 4 determines whether or not the flag a = 0. The flag a is a flag that is set to 1 when the initial setting is completed and set to 0 when the initial setting is not completed. When the flag a = 0, the control device 4 makes a positive determination in step S10 and proceeds to step S20. When the flag a ≠ 0, the control device 4 makes a negative determination and proceeds to step S30.

  In step S20, the control device 4 performs initial setting on the camera 3A, and proceeds to step S30. The initial setting is a predetermined setting for causing the camera 3A to perform a predetermined operation. In this example, the camera 3A sets the same imaging condition over the entire imaging surface of the imaging device 100, and starts imaging at a frame rate of 60 frames per second (60 fps), for example.

In step S30, the control device 4 performs an imaging condition setting process and proceeds to step S40. In the imaging condition setting process, the non-priority area 81 and the highest priority area 83 are set in addition to the priority area 82, and the imaging conditions are set for the unit area 131 corresponding to each area. Details of the imaging condition setting process will be described later.
In addition, when setting each imaging condition for the non-priority area 81, the priority area 82, and the highest priority area 83, it is not necessary to make all of the frame rate, gain, thinning rate, accumulation time, and the like different between the areas. , You can just make at least one different.

  In step S40, the control device 4 sends an instruction to the camera 3A to drive the unit areas 131 of the image sensor 100 corresponding to the non-priority area 81, the priority area 82, and the highest priority area 83 under predetermined conditions, respectively. Get acquisition and ranging calculations. Thereby, distance measurement information based on the image data and the image signal for distance measurement is acquired. Further, the control device 4 causes the image processing unit 33 or the control unit 35 of the camera 3A to perform predetermined image processing and image analysis.

  In step S50, the control device 4 determines whether or not the setting for displaying information is performed. When the display setting is performed, the control device 4 makes a positive determination in step S50 and proceeds to step S60. When the display setting is not performed, the control device 4 makes a negative determination in step S50 and proceeds to step S70.

In step S60, the control device 4 sends display information to the display / reproduction device 14 (FIG. 1), and proceeds to step S70. The display information is information according to the state of the vehicle 1 analyzed in the imaging condition setting process (S30). For example, “the oncoming vehicle is waiting to turn right”, “emergency stop”, “turn right” The message “turn left” is displayed on the display / playback device 14 or a navigation device (not shown).
Instead of sending display information, or together with sending display information, an audio signal for reproducing the message may be sent to an audio playback device (not shown). In this case as well, an audio playback unit of a navigation device (not shown) may be used as the audio playback device.

  In step S70, the control device 4 determines whether or not an off operation has been performed. For example, when the control device 4 receives an off signal (for example, an engine off signal) from the vehicle 1, the control device 4 makes a positive determination in step S <b> 70, performs a predetermined off process, and ends the process of FIG. 21. For example, if the control device 4 does not receive an off signal from the vehicle 1, the control device 4 makes a negative determination in step S70 and returns to step S30. When returning to step S30, the above-described processing is repeated.

<Imaging condition setting process>
Details of the imaging condition setting process (S30) will be described with reference to the flowchart of FIG. In the present embodiment, the moving amount of the image plane of the object shown on the image sensor 100, the depth distribution, the driving state of the vehicle 1 (changes in the vehicle speed V, the operating state of the steering wheel, etc.), the driving environment (the road condition, the driving state) The imaging conditions are set for the unit areas 131 of the imaging device 100 corresponding to the non-priority area 81, the priority area 82, and the highest priority area 83, respectively.

In step S31 of FIG. 22, the control device 4 acquires distance information based on the image signal acquired in the state where the same imaging condition is set over the entire imaging surface of the imaging element 100, and sets 0 to the flag b. Proceed to step S32. The distance information is the first depth distribution image. The flag b is a flag that sets 1 when a predetermined value (for example, 1.0) is substituted for a correction value α, which will be described later, and sets 0 when a value 0 is substituted for the correction value α. Here, flag b = 0 is set as an initial condition.
Note that the second depth distribution image based on detection information for each area by the radar 18 may be used as the distance information.

  In step S32, the control device 4 determines whether or not the distance d from the vehicle 1 to the object is longer than a predetermined distance D (for example, 300 m). For example, the control device 4 divides the shooting screen into a plurality of areas (for example, vertical 8 × horizontal 8 = 64 areas) based on the distance information, and determines whether d> D is satisfied for each area. To do. When there are a plurality of objects in the area, the distance to the closest subject is set as the distance d in the area.

  If there is even one area where d> D is established, the control device 4 makes a positive determination in step S32 and proceeds to step S33. If there is no area where d> D is established, the control device 4 makes a negative determination in step S32 and proceeds to step S34. When a negative determination is made in step S32, the setting of the priority area 82 is maintained for the entire imaging surface of the imaging element 100 (common control for all areas).

  In step S <b> 33, the control device 4 performs area-specific control for setting the priority area 82 and the non-priority area 81 on the imaging surface of the image sensor 100. The control device 4 sets the area where d> D is established as the non-priority area 81, and maintains the setting of the priority area 82 for areas where d> D is not established, and proceeds to step S34.

  In step S34, the control device 4 determines whether or not the vehicle 1 is outside the intersection area. The control device 4 refers to position information from the GPS device 15, map information from a navigation device (not shown), and external information such as white lines, pedestrian crossings, and traffic lights recognized from an image acquired by the camera 3A. If 1 is determined to be outside the intersection, an affirmative determination is made in step S34 and the process proceeds to step S35 in FIG. If the control device 4 determines that the vehicle 1 is within an intersection, the control device 4 makes a negative determination in step S34 and proceeds to step S301 in FIG.

FIG. 23 is a flowchart for explaining processing when the vehicle 1 travels outside an intersection. In step S35 of FIG. 23, the control device 4 calculates the image plane movement amount Δdiff. The image plane movement amount Δdiff is the image movement amount on the imaging surface of the image sensor 100 as described above. For example, the control device 4 calculates the image plane movement amount Δdiff using the vehicle speed V, the rotation angle θ of the steering wheel, and the predetermined coefficient μ as in the following equation (1), and the process proceeds to step S36.
Δdiff = (μ × V) × θ (1)

  In step S36, the control device 4 determines whether or not the turn signal device is off. For example, if the control device 4 determines that the turn signal device is off based on the operation signal of the turn signal switch 11 (FIG. 1), the control device 4 makes a positive determination in step S36 and proceeds to step S37. When determining that the turn signal device is on, the control device 4 makes a negative determination in step S36 and proceeds to step S38.

When proceeding to step S37, it is assumed that the vehicle 1 travels along a road outside the intersection. In step S <b> 37, the control device 4 sets the imaging condition of the imaging device 100 as follows. That is, the frame rate is set for each of the non-priority area 81, the priority area 82, and the highest priority area 83 with reference to FIG. 24A based on the image plane movement amount Δdiff calculated in step S35.
In the case where the three areas of the non-priority area 81, the priority area 82, and the highest priority area 83 are not set, such as when the entire area of the imaging surface is set as the priority area 82, the control device 4 performs FIG. Settings are made for the corresponding area in (A).

  According to FIG. 24A, four frame rates can be set so that different degrees of attention can be set for the object depending on the difference in the absolute value of the image plane movement amount Δdiff. Further, the four frame rates can be set for the non-priority area 81, the priority area 82, and the highest priority area 83, respectively. In general, the faster the vehicle speed V of the vehicle 1 and the smaller the radius of curvature that the vehicle 1 travels (the larger the rotation angle θ of the steering wheel), the faster the image flow (optical flow) that appears on the image sensor 100. By setting the frame rate according to the magnitude of the absolute value of the image plane movement amount Δdiff, it is possible to set the frame rate suitable for the change in the image captured on the image sensor 100.

  In addition, the frame rate can be set individually for the non-priority area 81, the priority area 82, and the highest priority area 83, so that each of the non-priority area 81, the priority area 82, and the highest priority area 83 is noticed. The frame rate can be set appropriately.

The movable range of the yaw angle Y in FIG. 24A is a movable range when the drive unit 37 described above changes the yaw angle Y of the camera 3A in accordance with an instruction from the control unit 35. In general, when the vehicle 1 turns to the left, the flow of an image (optical flow) reflected on the image sensor 100 increases from the left to the right of the screen. On the other hand, when the vehicle 1 turns to the right, the flow of the image shown on the image sensor 100 increases from the right to the left of the screen. For this reason, the control device 4 performs yaw angle control that moves the shooting direction of the camera 3A according to the magnitude and direction of the rotation angle θ of the steering wheel so that an image on the upstream side of the optical flow is quickly captured on the image sensor 100. Do. If the rotation angle θ of the steering wheel and the predetermined coefficient λ are used, the yaw angle Y is expressed by the following equation (2).
Y = λ × θ (2)
However, since the excessive movement of the yaw angle Y is not appropriate, the movable range of the yaw angle Y is limited by determining the movable range of the yaw angle Y in the present embodiment.

  According to FIG. 24A, the yaw angle movable range is 4 so that the limit value of the movable range of the yaw angle Y increases stepwise as the absolute value of the image plane movement amount Δdiff calculated in step S35 increases. Determined in stages.

  Since the flag b = 0 described above in step S37, the correction value α = 0 in FIG. That is, correction for narrowing the movable range of the yaw angle Y is not performed.

  FIG. 25 is a schematic diagram illustrating the yaw angle 30 of the camera 3A. Generally, when the turning speed of the vehicle 1 is high, the image plane movement amount Δdiff (absolute value) increases, so that the imaging unit yaw angle 30 (absolute value) is also greatly changed. FIG. 25 illustrates a case where the vehicle 1 turns to the left, and the image plane movement amount Δdiff is positive in the right direction. The control device 4 controls the yaw angle 30 so that the camera 3A is directed in the turning direction of the vehicle 1 (left in this example).

  Furthermore, as illustrated in FIG. 24A, the control device 4 obtains distance information using the radar 18 when the absolute value of the image plane movement amount Δdiff is a predetermined value (for example, 20) or more. That is, the second depth distribution image based on the detection information for each area by the radar 18 is used for the determination process in step S32. As described above, when the absolute value of the image plane movement amount Δdiff is large, the distance information is obtained by using the radar 18 having better response.

  As described above, when the imaging condition for setting the imaging condition of the imaging device 100 is changed, the control device 4 ends the process of FIG. 23 and proceeds to step S40 of FIG.

When a negative determination is made in step S36 described above and the process proceeds to step S38, it is assumed that the vehicle 1 changes the course by operating a turn signal device in an alley outside the intersection, for example. In step S38, the control device 4 sets the imaging condition of the imaging device 100 as follows. That is, the frame rate is set for each of the non-priority area 81, the priority area 82, and the highest priority area 83 with reference to FIG. 24B based on the image plane movement amount Δdiff calculated in step S35.
In the case where the three areas of the non-priority area 81, the priority area 82, and the highest priority area 83 are not set, such as when the entire area of the imaging surface is set as the priority area 82, the control device 4 performs FIG. Settings are made for the corresponding area in (B).

  According to FIG. 24B, attention is paid to the priority area 82 and the highest priority area 83 by increasing the frame rate setting for the priority area 82 and the highest priority area 83, for example, compared to the case of FIG. The degree is configured to be higher than in the case of FIG.

  The movable range of the yaw angle Y in FIG. 24B is a movable range when the drive unit 37 described above changes the yaw angle Y of the camera 3A in accordance with an instruction from the control unit 35. According to FIG. 24B, the limit value of the movable range of the yaw angle Y is set in FIG. 24B so that the image on the upstream side of the optical flow appears on the image sensor 100 faster than in the case of FIG. It is configured to be larger than in the case of A).

  Since the flag b = 0 described above in step S38, the correction value α = 0 in FIG. That is, correction for narrowing the movable range of the yaw angle Y is not performed.

  FIG. 26 is a flowchart for explaining processing when the vehicle 1 travels in an intersection. In step S301 in FIG. 26, the control device 4 calculates the image plane movement amount Δdiff in the same manner as in step S35, and proceeds to step S302.

  In step S302, the control device 4 acquires traffic information transmitted from an information providing system (not shown) installed on the road via the communication unit 18, and proceeds to step S303. Thereby, the vehicle 1 acquires information on the presence or absence of a vehicle on the opposite lane at the intersection, information on the right or left turn of the vehicle, information on a signal displayed on the traffic signal at the intersection, and the like.

  In step S303, the control device 4 refers to position information from the GPS device 15, map information from a navigation device (not shown), and external information such as white lines, pedestrian crossings, and traffic lights recognized from the images acquired by the camera 3A. Then, it is determined whether the vehicle 1 is in a straight lane at the intersection, a left turn lane, or a right turn lane, and the process proceeds to steps S304, S307, or S313, respectively.

<For straight lane>
In step S304, the control device 4 determines whether or not an oncoming right turn vehicle exists at the intersection. The control device 4 determines the presence or absence of a right turn vehicle in the opposite lane based on the traffic information acquired in step S302, and also determines the presence or absence of a right turn vehicle in the opposite lane based on the image acquired by the camera 3A. If the control device 4 determines that there is a right turn vehicle in the oncoming lane in at least one of the above determinations, it makes a positive determination in step S304 and proceeds to step S305. If the controller 4 does not determine that there is a right turn vehicle in the oncoming lane, the controller 4 makes a negative determination in step S304 and proceeds to step S306.

  In step S305, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, the entire imaging surface is set as the highest priority area 83 (all area common control), and based on the image plane movement amount Δdiff calculated in step S301, the highest priority area 83 is referred to with reference to FIG. Set the frame rate for.

  The movable range of the yaw angle Y and the change of the yaw angle Y are the same as in step S38. Since the flag b = 0 in step S305, correction for narrowing the movable range of the yaw angle Y is not performed, as in step S38.

  Further, as illustrated in FIG. 24B, the control device 4 detects the radar 18 when the absolute value of the image plane movement amount Δdiff is equal to or larger than a determination threshold (for example, 10) that is smaller than that in FIG. Get distance information using. That is, switching is performed so as to obtain distance information using the radar 18 when the image plane movement amount Δdiff is smaller than that in FIG.

  As described above, when the imaging condition for setting the imaging condition of the imaging device 100 is changed, the control device 4 ends the process of FIG. 26 and proceeds to step S40 of FIG.

  In step S306, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, the entire imaging surface is set as the highest priority area 83 (all area common control), and based on the image plane movement amount Δdiff calculated in step S301, with reference to FIG. Set the frame rate for.

The movable range of the yaw angle Y and the change of the yaw angle Y are the same as in step S37. Since the flag b = 0 in step S306, the correction for narrowing the movable range of the yaw angle Y is not performed as in the case of step S37. Further, as illustrated in FIG. 24A, the control device 4 detects the radar 18 when the absolute value of the image plane movement amount Δdiff is greater than or equal to a determination threshold (for example, 20) that is larger than that in FIG. Get distance information using. That is, switching is performed so as to obtain distance information using the radar 18 when the image plane movement amount Δdiff is larger than that in FIG.
As described above, when the imaging condition for setting the imaging condition of the imaging device 100 is changed, the control device 4 ends the process of FIG. 26 and proceeds to step S40 of FIG.

<For left turn lanes>
In step S307, the control device 4 determines whether or not an oncoming right turn vehicle exists at the intersection. The control device 4 determines the presence or absence of a right turn vehicle in the opposite lane based on the traffic information acquired in step S302, and also determines the presence or absence of a right turn vehicle in the opposite lane based on the image acquired by the camera 3A. If the control device 4 determines that there is a right turn vehicle on the oncoming lane in at least one of the above determinations, the control device 4 makes a positive determination in step S307 and proceeds to step S308. If the controller 4 does not determine that there is a right turn vehicle in the oncoming lane, the controller 4 makes a negative determination in step S307 and proceeds to step S309.

  In step S308, the control device 4 performs yaw angle correction setting. The control device 4 sets 1 to the flag b and proceeds to step S309. Thereby, the process of correcting the value of the yaw angle Y with the correction value α is effective. When the movable range of the yaw angle Y is increased as the absolute value of the image plane movement amount Δdiff is increased, the direction of the camera 3A is directed in the turning direction of the vehicle 1, while the other vehicles in the opposite lane are captured by the camera 3A. There is also a high risk of disengagement. Therefore, the control device 4 provides the correction value α so as to narrow the movable range of the yaw angle Y when there is another vehicle in the oncoming lane, compared with the case where there is no other vehicle. Correct the movable range. This is because the camera 3A reliably acquires the information of the vehicle in the opposite lane while turning the direction of the camera 3A in the turning direction of the vehicle 1.

  In step S309, the control device 4 sets the imaging conditions of the imaging device 100 as follows. That is, the entire imaging surface is set as the highest priority area 83 (all area common control), and based on the image plane movement amount Δdiff calculated in step S301, the highest priority area 83 is referred to with reference to FIG. Set the frame rate for.

  Since the flag b = 1, the control device 4 corrects and narrows the limit value of the movable range of the yaw angle with the correction value α as described above.

  In step S310, the control device 4 determines whether or not the pedestrian crossing pedestrian signal to be passed in the case of a left turn is a crossing permission signal (for example, a green light). The control device 4 determines the presence / absence of a crossing permission signal based on the traffic information acquired in step S302, and also determines the presence / absence of a green signal based on the image acquired by the camera 3A. If the control device 4 determines that it is a crossing permission signal in at least one of the above determinations, it makes a positive determination in step S310 and proceeds to step S311. If the control device 4 does not determine that the signal is a crossing permission signal, the control device 4 makes a negative determination in step S311, ends the processing of FIG. 26, and proceeds to step S40 of FIG.

In step S <b> 311, the control device 4 performs area-specific control for setting the priority area 82 and the highest priority area 83 on the imaging surface of the image sensor 100 based on the line-of-sight information sent from the line-of-sight detection device 16. The control device 4 sets the area corresponding to the sight line of the occupant as the priority area 82, maintains the highest priority area 83 for the area not corresponding to the sight line of the occupant, and proceeds to step S312.
As described above, the control device 4 may set the area including the pedestrian crossing 92 recognized based on the image acquired by the camera 3A as the priority area 82 (FIG. 20).

  In step S312, the control device 4 sets the imaging condition of the imaging device 100 as follows. That is, based on the image plane movement amount Δdiff calculated in step S301, the frame rates for the priority area 82 and the highest priority area 83 are set with reference to FIG.

  Since the flag b = 1, the control device 4 corrects and narrows the limit value of the movable range of the yaw angle Y with the correction value α as described above. The control device 4 ends the process of FIG. 26 and proceeds to step S40 of FIG.

<Right turn lane>
In step S313, the control device 4 determines whether there is an oncoming straight vehicle at the intersection. The control device 4 determines whether or not there is a vehicle traveling straight in the opposite lane based on the traffic information acquired in step S302, and also determines whether or not there is a vehicle traveling straight in the opposite lane based on the image acquired by the camera 3A. If the control device 4 determines that there is a straight vehicle in the oncoming lane in at least one of the above determinations, it makes a positive determination in step S313 and proceeds to step S314. If the controller 4 does not determine that there is a straight vehicle on the oncoming lane, the controller 4 makes a negative determination in step S313 and proceeds to step S315.

  In step S314, the control device 4 performs yaw angle correction setting. The control device 4 sets 1 to the flag b and proceeds to step S315. Thereby, the process of correcting the value of the yaw angle Y with the correction value α is effective. This is because the camera 3A reliably acquires the information of the vehicle in the opposite lane while turning the direction of the camera 3A in the turning direction of the vehicle 1.

  In step S315, the control device 4 sets the imaging condition of the imaging device 100 as follows. That is, the entire imaging surface is set as the highest priority area 83 (all area common control), and based on the image plane movement amount Δdiff calculated in step S301, the highest priority area 83 is referred to with reference to FIG. Set the frame rate for.

  Since the flag b = 1, the control device 4 corrects and narrows the limit value of the movable range of the yaw angle with the correction value α as described above.

  In step S316, the control device 4 determines whether or not the pedestrian crossing pedestrian signal that will be passed in the case of a right turn is a crossing permission signal (for example, a green light). The control device 4 determines the presence / absence of a crossing permission signal based on the traffic information acquired in step S302, and also determines the presence / absence of a green signal based on the image acquired by the camera 3A. If the control device 4 determines that it is a crossing permission signal in at least one of the above determinations, it makes a positive determination in step S316 and proceeds to step S317. If the control device 4 does not determine that the signal is a crossing permission signal, the control device 4 makes a negative determination in step S316, ends the processing in FIG. 26, and proceeds to step S40 in FIG.

In step S <b> 317, the control device 4 performs area-specific control for setting the priority area 82 and the highest priority area 83 on the imaging surface of the image sensor 100 based on the line-of-sight information sent from the line-of-sight detection device 16. The control device 4 sets the area corresponding to the sight line of the occupant as the priority area 82, maintains the highest priority area 83 for the area not corresponding to the sight line of the occupant, and proceeds to step S318.
Note that, as described above, the control device 4 may set the area including the pedestrian crossing recognized based on the image acquired by the camera 3 </ b> A as the priority area 82.

  In step S318, the control device 4 sets the imaging condition of the imaging device 100 as follows. That is, based on the image plane movement amount Δdiff calculated in step S301, the frame rates for the priority area 82 and the highest priority area 83 are set with reference to FIG.

  Since the flag b = 1, the control device 4 corrects and narrows the limit value of the movable range of the yaw angle Y with the correction value α as described above. The control device 4 ends the process of FIG. 26 and proceeds to step S40 of FIG.

According to the second embodiment described above, the following operational effects can be obtained.
(1) The vehicle 1 has a first depth distribution image based on the imaging unit 32 including the imaging element 100 that can independently set the imaging conditions of the plurality of unit regions 131 and the output from the pixels of the imaging element 100. And a control device 4 that sets the imaging conditions of the plurality of unit regions 131 based on the first depth distribution image. Thereby, for example, an area where the distance from the vehicle 1 to the subject is equal to or greater than a predetermined threshold (for example, 300 m) is set as the non-priority area 81, and the other is set as the priority area 82. Conditions can be set.

(2) The movable vehicle 1 includes the steering wheel 10, and the control device 4 includes a plurality of unit regions 131 based on the vehicle speed V of the vehicle 1, the rotation angle θ of the steering wheel, and the first depth distribution image. Set the imaging conditions. Accordingly, it is possible to set appropriate imaging conditions for the imaging unit 32 based on the image plane movement amount Δdiff and the first depth distribution image.

(3) The vehicle 1 moves the imaging unit 32 based on the steering wheel 10 provided in the movable vehicle 1, the imaging unit 32 that performs imaging, the vehicle speed V of the vehicle 1, and the operation amount of the steering wheel 10. And a control device 4 to be operated. Thereby, the yaw angle control of the imaging unit 32 can be appropriately performed so that the upstream image of the image flow (optical flow) captured on the image sensor 100 is captured quickly.

(4) Since the imaging unit 32 includes the imaging device 100 that can independently set the imaging conditions of the plurality of unit regions 131, different imaging conditions can be set for each unit region 131.

(5) Since the control device 4 moves the imaging unit 32 according to the operation of the steering wheel 10, the yaw angle control of the imaging unit 32 can be performed in real time with respect to the rotation operation of the steering wheel 10.

(6) Since the control device 4 determines the movement amount of the imaging unit 32 based on the vehicle speed V and the rotation angle θ of the steering wheel 10, the yaw angle control of the imaging unit 32 can be appropriately performed.

(7) The vehicle 1 includes an imaging unit 32 including the imaging element 100 that can independently set imaging conditions of the plurality of unit regions 131, a control device 4 that detects approach to an intersection, and detection by the control device 4. Based on the result, the control device 4 that sets the imaging conditions of the plurality of unit regions 131 of the imaging device 100 is provided. Thereby, different imaging conditions can be set when the vehicle 1 approaches the intersection and when the vehicle 1 is away from the intersection. Furthermore, it is possible to set appropriate imaging conditions for the vehicle 1 traveling on the left turn lane, the straight lane, and the right turn lane at the intersection.

(8) Since the radar 18 that detects an object around the vehicle 1 is provided, a second depth image can be obtained separately from the first depth distribution image calculated by the control unit 35.

(9) The vehicle 1 includes a line-of-sight detection device 16 that detects the line of sight of an occupant. Accordingly, for example, the imaging condition of the imaging unit 32 can be set in consideration of the sight line of the occupant, such as increasing the frame rate for an area that the occupant is not careful about.

(10) The control device 4 of the vehicle 1 sets the highest priority area 83 for the area other than the attention area 82 including the pedestrian crossing 92 that passes when the vehicle 1 turns left (or right) at the intersection, and sets the frame rate. The image is picked up with a high or low thinning rate. As a result, an area where the degree of attention by the driving occupant is low is increased as the highest priority area 83 by the imaging unit 32. Specifically, as shown in FIG. 20, another vehicle such as a vehicle 91 turning right from the oncoming lane at the time of a left turn is monitored by the imaging unit 32.

(11) The control device 4 of the vehicle 1 sets the imaging condition of the imaging unit 32 based on the image plane movement amount Δdiff indicating the magnitude of the flow of the image captured on the imaging device 100 of the imaging unit 32. Thereby, for example, the imaging condition for the imaging unit 32 can be appropriately set such that the frame rate is changed depending on the magnitude of the image plane movement amount Δdiff.

(12) Since the control device 4 calculates the image plane movement amount Δdiff based on the product of the vehicle speed V and the rotation angle θ of the steering wheel, it can appropriately calculate the size of the flow of the image shown on the image sensor 100.

(13) The control device 4 controls the yaw angle Y of the imaging unit 32 based on the direction and magnitude of the image plane movement amount Δdiff. As a result, the yaw angle control of the imaging unit 32 can be appropriately performed so that the image on the upstream side of the optical flow is quickly captured on the imaging surface of the imaging unit 32.

(14) The control device 4 changes the frame rate of the imaging unit 3 based on the magnitude of the image plane movement amount Δdiff. Thereby, the imaging conditions for the imaging unit 32 can be appropriately set such that the frame rate is increased as the image plane movement amount Δdiff is larger.

(15) The control device 4 corrects the moving range of the yaw angle of the imaging unit 32 to be narrow due to the presence of a right turn vehicle in the opposite lane or a straight vehicle in the opposite lane. Thereby, the information of the vehicle in the opposite lane can be reliably acquired by the imaging unit 32 while the direction of the imaging unit 32 is directed in the turning direction of the vehicle 1.
In the above-described embodiment, the camera 3A is controlled by the control device 4, but a part of the control of the camera 3A may be performed by the control unit 35 of the camera 3A.

(Modification 1 of the second embodiment)
In the above-described embodiment, the example in which the frame rate is different between the priority area 82 and the highest priority area 83 has been described. Instead, a buffer area may be provided between the priority area 82 and the highest priority area 83, and the frame rate may be varied in multiple stages. For example, the frame rate is changed in multiple stages by setting the frame rate of the buffer area to an intermediate value between the frame rate of the priority area 82 and the frame rate of the highest priority area 83. By providing such a buffer area, it is possible to avoid a state in which the value of the frame rate changes greatly at the boundary portion.

(Modification 2 of the second embodiment)
In the above description, as a distance measurement performed by the camera 3A, a method of calculating by distance measurement using an image signal from a focus detection pixel provided in the image sensor 100 is used. You may use the method of measuring distance using an image.

(Third embodiment)
The third embodiment is an example in which distance measurement is performed using two images by a stereo camera. In particular, a case where the auxiliary light source 22 is used will be described. FIG. 27A is a diagram for explaining the third embodiment. In FIG. 27 (a), the auxiliary light source 22 is provided at the front portion of the vehicle 1, and irradiates the front of the vehicle 1 traveling parallel to the road surface at a predetermined ground height (for example, 50 cm). The auxiliary light source 22 emits infrared light 22B having a predetermined projection pattern within the field angle of view taken by the stereo camera 3S. The vehicle 1 is the same as the vehicle illustrated in FIG. 1, and a stereo camera 3S is provided instead of the camera 3A.

  FIG. 27B is a diagram for explaining a projection pattern by the auxiliary light source 22. In FIG. 27 (b), the infrared light 22 </ b> B is irradiated on the low-contrast wall 40 provided in front of the vehicle 1. The projection pattern is bilaterally symmetric with respect to the central portion of the wall 40, and the pattern stripes are configured to be uneven. In the example of FIG. 27 (b), the interval between the stripes is gradually increased from the central portion of the wall 40 toward the left and right ends.

  For example, when an infrared LED is used as the auxiliary light source 22, the pattern of the light emitting part of the LED chip can be configured in a striped pattern, and the pattern can be projected. Further, the infrared light 22B from the auxiliary light source 22 may be projected in a stripe pattern by using a lens and a prism.

On the other hand, the left and right cameras constituting the stereo camera 3S are equipped with image sensors similar to the image sensor 100 described above. In each imaging device, among RGB pixels having a predetermined array such as a Bayer array, for example, IR pixels in which a filter that receives infrared light is disposed at the position of the G pixel are disposed at a predetermined ratio. Thereby, the stereo camera 3S can acquire two left and right infrared images based on the image signal from the IR pixel. The filter disposed in the IR pixel has a characteristic of transmitting the wavelength of the infrared light 22B emitted from the auxiliary light source 22 and blocking other wavelength regions.
The position where the IR pixel is arranged is not limited to the position of the G pixel in the Bayer array, but may be the position of the R pixel or the position of the B pixel.

  The control device 4 performs distance measurement using a stereo image obtained by the infrared light 22B acquired by the stereo camera 3S. FIG. 27 (c) is a diagram illustrating an object (two persons) irradiated with infrared light 22B. When the outside of the vehicle 1 has a predetermined brightness, the control device 4 drives all the RGB pixels of the imaging elements of the left and right cameras to acquire an RGB stereo image at a predetermined frame rate. The control device 4 detects a front object on which the vehicle 1 travels based on the RGB stereo image.

  The control device 4 drives an IR pixel corresponding to the detected object that is irradiated with the infrared light 22B and does not correspond to the object irradiated with the infrared light 22B. Do not drive. The control device 4 performs distance measurement based on the RGB stereo image, and when the distance measurement based on the RGB stereo image is difficult, performs the distance measurement to the target object based on the stereo image by the infrared light 22B.

  When the outside of the vehicle 1 is less than the predetermined brightness, the control device 4 drives all the IR pixels of the imaging elements of the left and right cameras, and acquires a stereo image by the infrared light 22B at a predetermined frame rate. . The control device 4 detects an object in front of which the vehicle 1 travels based on the stereo image of the first frame by the infrared light 22B. Then, the control device 4 drives the detected object and the IR pixel corresponding to the predetermined range including the object for the subsequent frames and does not drive the other IR pixels. The control device 4 performs distance measurement to the object based on the stereo image by the infrared light 22B.

  The control device 4 further increases the frame rate of the corresponding IR pixel as the distance to the detected object is shorter, and decreases the frame rate of the corresponding IR pixel as the distance to the detected object is longer. As described above, since the IR pixels are selectively driven and the frame rates of the IR pixels are varied according to the distance to the object, all of the IR pixels of the imaging elements of the left and right cameras are increased. The power consumption can be reduced as compared with the case of driving at the frame rate.

The light emission timing of the auxiliary light source 22 that emits the infrared light 22B may be always emitted, or may be intermittent light emission that emits light at the timing of driving the IR pixels of the imaging elements of the left and right cameras.
Alternatively, the infrared light 22B may be emitted when a predetermined condition is determined based on an RGB stereo image. For example, when there is a wall in front of the vehicle 1 (T-junction, dead end, etc.), when the object (person) is wearing plain clothes, when the object is a car, the object has a regular pattern. When it is held (such as a shutter), infrared light 22B is emitted from the auxiliary light source 22.

According to the third embodiment described above, the following operational effects can be obtained. That is, the imaging apparatus includes a stereo camera 3S including an imaging element that can independently set imaging conditions for a plurality of areas, and infrared light 22B of an object irradiated with infrared light 22B having a predetermined pattern. And a control device 4 that controls the imaging condition of the region of the imaging element that captures the irradiated portion and the imaging condition of the region of the imaging element that captures the portion not irradiated with the infrared light 22B to be different. Thereby, based on the stereo image by the stereo camera 3S, distance measurement can be appropriately performed even at night or in backlight. Further, since the intervals between the stripes of the projection pattern by the infrared light 22B are made non-uniform, it is easy to detect the parallax even when projected onto the plane in front of it.
Moreover, it is effective not only at night but also in the case of fog.

(Fourth embodiment)
FIG. 28 (a) and FIG. 28 (b) are diagrams for explaining the fourth embodiment. In FIG. 28A, the vehicle 1 is the same as the vehicle illustrated in FIG. 1, except that a transmissive liquid crystal filter 21 is laminated on the windshield glass 20 (excluding the region through which the light flux to the camera 3 passes). Yes. In the fourth embodiment, information that is difficult for the driver to visually recognize (necessary light) is acquired by the camera 3A.

  In FIG. 28 (b), a head-up display as a display / playback device 14 is provided in the dashboard. The head-up display 14 alerts the driver 30. Further, the transmissive liquid crystal filter 21 (FIG. 28A) suppresses light that is dazzling for the driver 30 (unnecessary light).

  The transmissive liquid crystal filter 21 is a filter capable of controlling the visible light transmittance for each region. In the present embodiment, when the control device 4 detects the presence of an object (sun 35 such as morning sun or sunset) having a predetermined brightness or higher, among the liquid crystal elements constituting the transmissive liquid crystal filter 21 The transmittance of the liquid crystal element of the transmissive liquid crystal filter 21 corresponding to the sun 35 and the surrounding area in the scenery observed by the driver 30 is lowered. Thereby, the excessive unnecessary light with respect to the driver | operator 30 eyes is suppressed.

  In addition to the sun 35, the reflected light reflected by the building and the headlight of the oncoming vehicle during night driving may be targeted for suppression as excessive unnecessary light. Here, it is assumed that the relationship between the position of the eyes of the driver 30 based on the physique and the sitting position of the driver 30 and the scenery that the driver 30 observes through the transmissive liquid crystal filter 21 is stored in advance.

  The head-up display 14 is a display device that displays information displayed on a liquid crystal panel or the like on the windshield glass 20 as a virtual image. The driver 30 observes information by the head-up display 14 projected so as to be superimposed on the scenery outside the vehicle 1. In the present embodiment, when the control device 4 detects the presence of an object (a pedestrian, a vehicle (including a car, a two-wheeled vehicle), etc.), a frame including the object in the scenery observed by the driver 30 ( P2 may be displayed by the head-up display 14.

In addition, even when the presence of an object (a road sign, a signal, a pedestrian crossing, a gutter, etc.) is detected by the control device 4, a frame (which may be bordered) P1 including the object in the scenery observed by the driver 30 May be displayed by the head-up display 14.
The relationship between the position of the eyes of the driver 30 based on the physique of the driver 30 and the sitting position, the scenery observed by the driver 30 through the windshield glass 20, and the projection position by the head-up display 14 is stored in advance. It is assumed that

(Division)
The control device 4 performs classification based on the image acquired by the camera 3A, thereby performing control of the transmittance with respect to the transmissive liquid crystal filter 21, display control with respect to the head-up display 14, and control with respect to the gain with respect to the camera 3A.

(Excessive light area)
The control device 4 detects an object having a predetermined brightness or more as follows. For example, the position information and time information of the vehicle 1 calculated by the GPS device 15, the direction information of the vehicle 1 detected by the navigation device 18, the position information of the sun 35 prepared in advance (observation point and altitude (elevation angle) and Based on the information indicating the azimuth) and the angle of view information of the camera 3A, an area (referred to as an excessive light area) corresponding to unnecessary excessive light from the sun 35 or the like in the image acquired by the camera 3A is detected. As described above, the control device 4 reduces the transmittance of the liquid crystal element corresponding to the excess light region in the transmissive liquid crystal filter 21. Further, the control device 4 sets the gain corresponding to the excessive light region on the imaging surface to be low for the imaging device 100 of the camera 3A.

(Insufficient light area)
When the object of the image acquired by the camera 3A is darker than the predetermined brightness, the control device 4 determines that the light from the object is insufficient light. The control device 4 sets a high gain in an area corresponding to insufficient light (referred to as an insufficient light area) in the image sensor 100 of the camera 3A. As a result, the object of the image acquired by the camera 3A becomes brighter and it becomes easier to recognize the object (pedestrian, vehicle (including cars, two-wheeled vehicles), signs, etc.).
In addition to partially increasing the gain of the image sensor 100, the frame rate may be partially increased. If it is determined that there is no object, the gain and the frame rate may be returned to the values before the change.

(Required light area)
The control device 4 determines that the light from the object is necessary light when an object (a road sign, a signal, a pedestrian crossing, a gutter, etc.) is included in the image acquired by the camera 3A. The control device 4 sets the gain of a region (referred to as a necessary light region) corresponding to the necessary light in the imaging device 100 of the camera 3A to be partially high. Thereby, the object of the image acquired by the camera 3A becomes brighter, and information on the object (road sign, signal, pedestrian crossing, side gutter, etc.) becomes easy to understand.
Note that when the image acquired by the camera 3A is brighter than a predetermined brightness, the control device 4 may return the gain and the frame rate to values before the change.

Further, as described above, the control device 4 causes the head-up display 14 to display frames (which may be bordered) P1 and P2 including the respective objects with respect to the insufficient light region and the necessary light region. Thereby, it is possible to inform the driver 30 of the presence of the object or to call attention.
In addition to the display of the frame and border, or in addition to the display of the frame and border, a previously stored mark (for example, the same mark as a road sign, a mark that looks like a person, a mark that looks like a vehicle, etc.) It may be displayed.

  The control device 4 may perform pixel addition for the same area instead of increasing the gain of the insufficient light area or the necessary light area on the imaging surface of the image sensor 100 or increasing the gain. Pixel addition means that a plurality of pixel signals are added and handled as one pixel signal. For example, the control device 4 pixel-adds images of a plurality of frames (for example, four images) with successive acquisition timings to obtain one image. In pixel addition, corresponding pixel signals between a plurality of frames are added together.

  After detecting the object by temporarily increasing the gain of the insufficient light area and the necessary light area as described above or performing pixel addition between a plurality of frames as described above, the control device 4 moves the object. The gain may be lowered to the extent that tracking can be performed, or the number of frames for pixel addition may be reduced.

  According to the fourth embodiment described above, the following operational effects can be obtained. That is, the electronic apparatus includes an imaging unit 32 including an imaging element 100 that can independently set imaging conditions for a plurality of areas, a head-up display 14 provided on the front surface of the driver 30 of the vehicle 1, and an imaging unit. And a control device 4 that performs setting of imaging conditions for a plurality of regions and control of display on the head-up display 14 based on information about the brightness in front of the vehicle 1 obtained from the image captured at 32. For example, by setting the gain of the region corresponding to the insufficient light in the image sensor 100 to be high, the object of the image acquired by the camera 3A becomes bright, and the object (pedestrian, vehicle (including a car and a motorcycle), It becomes easier to recognize signs, etc.). In addition, the driver 30 can be notified of the presence of the object and can be alerted.

(Fifth embodiment)
In the fifth embodiment, the wearable camera 19 worn by the driver 30 and the camera 3A mounted on the vehicle 1 cooperate. FIG. 29A is a diagram for explaining the fifth embodiment. In FIG. 29A, the camera 3A has a function as a drive recorder. For example, when power supply from the vehicle 1 is started (system on) or the engine is started, imaging starts at a predetermined frame rate. To do. The wearable camera 19 starts imaging at a predetermined frame rate from when the wearable camera 19 is attached to the driver 30, for example. The wearable camera 19 sequentially transmits captured images to the control device 4 via a built-in wireless communication unit. The control device 4 can record images acquired by the camera 3 </ b> A and the wearable camera 19 in the storage unit 4 b of the control device 4.

The camera 3A divides the imaging surface of the imaging chip 111 of the imaging device 100 into a first area (normal imaging area) and a second area (special imaging area), and different imaging is performed between the first area and the second area. Imaging can be performed by setting conditions. According to FIG. 29 (b), the first area is constituted by the blocks in the odd numbered columns, and the second area is constituted by the blocks in the even numbered columns. That is, the block in the imaging chip 111 is divided into odd columns and even columns.
The division mode of the first area and the second area is not limited to the example of FIG. 29 (b), but may be divided into odd and even lines as shown in FIG. 4 (b). It can also be divided into a checkered pattern.

  For example, when the vehicle travels at night, the control device 4 performs normal imaging in the first region of the camera 3A, and performs imaging by setting the gain in the second region of the camera 3A higher than the gain of the first region. Make it.

  For example, in the backlit state during daytime running, the control device 4 causes normal imaging in the first region of the camera 3A and sets the gain in the second region of the camera 3A to be lower than the gain of the first region. To take an image. In addition to setting the gain low, the accumulation time may be set short.

  The control device 4 does not drive the pixels in the second region in the camera 3A when the vehicle does not correspond to a night driving or a daylight backlight state. The control device 4 performs only normal imaging using the first region in the camera 3A.

  The wearable camera 19 is equipped with a line-of-sight detection unit and can detect the line of sight of the driver 30. The line-of-sight information detected by the line-of-sight detection unit is transmitted from the wearable camera 19 to the control device 4 by wireless communication together with the captured image via the wireless communication unit.

  Based on the line of sight of the driver 30, the control device 4 determines whether or not the imaging range by the wearable camera 19 is included in the imaging range by the camera 3A. And the control apparatus 4 records only the image acquired by the camera 3A in the memory | storage part 4b of the control apparatus 4, when the imaging range by the wearable camera 19 is contained in the imaging range by the camera 3A.

  In addition, when a part of the imaging range by the wearable camera 19 is included in the imaging range by the camera 3A, the control device 4 controls both the image acquired by the camera 3A and the image acquired by the wearable camera 19. Is stored in the storage unit 4b.

  Furthermore, when the imaging range of the wearable camera 19 is not included in the imaging range of the camera 3A, the control device 4 displays both the image acquired by the camera 3A and the image acquired by the wearable camera 19 of the control device 4. Record in the storage unit 4b.

According to the fifth embodiment described above, the following operational effects can be obtained.
(1) The image recording device includes a camera 3A that captures the surroundings of the vehicle 1, a control device 4 that inputs a second image captured by the wearable camera 19 attached to the driver 30 of the vehicle 1, and a camera 3A. The control device 4 that compares the imaging range of the captured first image and the imaging range of the second image, and at least one of the first image and the second image is stored in the storage unit 4b based on the comparison result of the control device 4. And a control device 4 for recording. For example, when at least a part of the imaging range by the wearable camera 19 is not included in the imaging range by the camera 3A, both the first image acquired by the camera 3A and the second image acquired by the wearable camera 19 are displayed. By recording in the storage unit 4b of the control device 4, information that cannot be obtained only by the camera 3A can be left.
On the other hand, when the entire imaging range of the wearable camera 19 is included in the imaging range of the camera 3A, only the first image is recorded in the storage unit 4b of the control device 4, thereby reducing the usage amount of the storage unit 4b. It can be kept low.

(2) In addition, when imaging is performed in the second area in the camera 3A, an image obtained by normal imaging in the first area and an image obtained by imaging under different conditions in the second area Since both are recorded, it is possible to leave information that is difficult to understand in normal imaging. Furthermore, when imaging is not performed in the second area of the camera 3A, only the image obtained by normal imaging in the first area is recorded, so that the usage amount of the storage unit 4b can be reduced.

(Modification)
In each embodiment described above, an image acquired by the camera 3A mounted on the vehicle 1 may be shared with other vehicles other than the vehicle 1 or shared with a traffic control center. For example, an image acquired by the camera 3A is transmitted to another vehicle or a traffic control center by a wireless communication device. Transmission from the vehicle 1 to another vehicle may be performed by inter-vehicle communication, or may be transmitted from the vehicle 1 via a traffic control center.

  When an image is transmitted from the vehicle 1 to the traffic control center, information generated based on the image received by the traffic control center from the vehicle 1 may be distributed to other vehicles. For example, when an obstacle is reflected in the image received from the vehicle 1, the control device of the traffic control center analyzes the image and displays a message “There is a fallen object near the prefectural road XX line △△ 4 chome.” Generate and distribute to other vehicles.

  When the image including the obstacle on the road described above is acquired by the camera 3A of the vehicle 1, the control device 4 sets a region of interest in the imaging chip 111 for the obstacle, and other regions (forward attention regions). (For example, increase the resolution, frame rate, etc.). Thereby, detailed information of the obstacle in the attention area can be obtained. In addition, by capturing only the region of interest (that is, a part of the image) with high definition, the image data to be acquired is compared with the case of capturing all the regions (entire image) on the imaging chip 111 with high definition. The amount can be kept small.

  By reducing the amount of data, it is possible to reduce the amount of communication when an image is transmitted from the vehicle 1 to another vehicle or a traffic control center. Further, when an image acquired by the camera 3A is stored as a Drave recorder, the capacity of the recording medium to be recorded can be reduced by reducing the data amount.

  In the case where the amount of communication when the image is transmitted from the vehicle 1 to another vehicle or the traffic control center is reduced, the entire area of the imaging chip 111 is imaged with high definition and only the area for the obstacle is cut out. Alternatively, a part of the image may be transmitted from the vehicle 1 to another vehicle or a traffic control center.

  Examples of sharing an image acquired by the camera 3A mounted on the vehicle 1 with other vehicles other than the vehicle 1 and the traffic control center include the presence or absence of an accident, the occurrence of a disaster, in addition to the presence or absence of the obstacle. It can also be applied to the presence / absence of construction, the presence / absence of construction, the presence / absence of traffic jams, the presence / absence of inspections, the presence / absence of chain attachment restrictions, etc.

  DESCRIPTION OF SYMBOLS 1 ... Vehicle, 2 ... Driving assistance apparatus, 3A, 3B ... Camera, 4 ... Control apparatus, 4a, 35a ... CPU, 4b, 35b ... Memory | storage part, 10 ... Steering wheel, 12 ... Vehicle speed sensor, 14 ... Display and reproduction | regeneration apparatus , 16 ... gaze detection device, 17 ... communication unit, 18 ... radar, 31 ... imaging optical system, 32 ... imaging unit, 35 ... control unit, 70 ... imaging surface, 81 ... first region, non-priority region, 82 ... first 2 areas, priority areas, 83 ... top priority area, 100 ... imaging element, 111 ... imaging chip

Claims (19)

In electronic devices mounted on cars,
An imaging unit that has a first area and a second area that can change imaging conditions, generates a first image signal in the first area, and generates a second image signal in the second area;
An object detection unit for detecting an object from the first image signal;
An electronic apparatus comprising: a display control unit that displays an image based on the second image signal on a display unit. The electronic device according to claim 1,
The display control unit is an electronic device that displays information on the object detected by the object detection unit on the display unit. The electronic device according to claim 1 or 2,
The imaging unit has a frame rate of the first region higher than a frame rate of the second region;
Electronics. The electronic device according to any one of claims 1 to 3,
In the imaging unit, the gain of the first region is higher than the gain of the second region.
Electronics. The electronic device according to any one of claims 1 to 4,
A line-of-sight detection unit for detecting the line-of-sight position of the vehicle occupant;
The target object detection unit is an electronic device that detects the target object in a region that does not include the line-of-sight position in the first region in the first image signal based on a detection result by the line-of-sight detection unit. The electronic device according to claim 5,
A setting unit that sets the imaging condition of the first image signal differently in an area that does not include the line-of-sight position in the first area and an area that includes the line-of-sight position in the first area;
Electronic equipment comprising. The electronic device according to claim 6,
The setting unit sets a frame rate of an area not including the line-of-sight position in the first area higher than a frame rate of an area including the line-of-sight position in the first area;
Electronics. In electronic devices mounted on cars,
An imaging unit that has a first area and a second area that can change imaging conditions, generates a first image signal in the first area, and generates a second image signal in the second area;
A detection unit for detecting whether or not a reflective member is included in the first image signal;
Based on the detection result by the detection unit, a display unit that displays an image of the part of the reflection member in the second image signal in a different manner;
With electronic equipment. The electronic device according to claim 8,
When the detection unit detects that the reflection member is included, the display unit displays a resolution of an area in which the reflection member is included in the second image signal. Displaying the image of the part of the reflecting member at a higher resolution than the case where the inclusion is not detected,
Electronics. The electronic device according to any one of claims 1 to 9,
The imaging unit images a common object in the first image signal and the second image signal.
Electronics.   The motor vehicle carrying the electronic device as described in any one of Claims 1-10. An imaging unit including an imaging element capable of setting imaging conditions of a plurality of regions;
A distance measuring unit that calculates distance information based on an output from a pixel of the image sensor;
A setting unit for setting imaging conditions of the plurality of regions based on the distance information;
Automobile equipped with. The automobile according to claim 12,
It has an operation part provided on the movable main body,
The setting unit sets imaging conditions for the plurality of regions based on a moving speed of the main body, an operation amount of the operation unit, and the distance information;
Automobile. An operation unit provided on the movable main body;
An imaging unit for imaging;
A control unit that moves the imaging unit based on a moving speed of the main body and an operation amount of the operation unit;
A car equipped with. 15. The automobile according to claim 14,
The control unit moves the imaging unit according to an operation of the operation unit.
Automobile. An imaging unit including an imaging element capable of setting imaging conditions of a plurality of regions;
A detection unit for detecting approach to the intersection;
A setting unit that sets imaging conditions of a plurality of regions of the imaging device based on a detection result by the detection unit;
A car equipped with. An imaging unit including an imaging element capable of setting imaging conditions of a plurality of regions;
Imaging conditions of the area of the imaging element that images the portion irradiated with the pattern light of the object irradiated with the predetermined pattern light, and imaging of the area of the imaging element that images the part not irradiated with the pattern light A control unit that controls the conditions to be different from each other;
An imaging apparatus comprising: An imaging unit including an imaging element capable of setting imaging conditions of a plurality of regions;
A transmissive display unit provided in front of the driver of the moving body;
A control unit that performs setting of imaging conditions of the plurality of regions and control of display of the transmissive display unit, based on information on the brightness of the front of the moving object obtained from the image captured by the imaging unit;
With electronic equipment. A first imaging unit that images the periphery of the moving body;
An input unit for inputting a second image signal captured by a second imaging unit provided in a mounting device mounted on an occupant of the mobile body;
A comparison unit that compares the imaging range of the first image signal captured by the first imaging unit with the imaging range of the second image signal;
A recording unit that records at least one of the first image signal and the second image signal on a recording medium according to a comparison result of the comparison unit;
An image recording apparatus comprising:

Images