JP2020188476A - Imaging apparatus - Google Patents

Abstract

To properly set imaging conditions of an imaging apparatus.SOLUTION: The imaging apparatus includes an imaging unit mounted on a vehicle and imaging the exterior of the vehicle, and an imaging control unit for controlling imaging conditions of the imaging unit based on the position of a steering wheel of the vehicle.SELECTED DRAWING: Figure 9

Description

The present invention relates to an imaging device.

The driving environment of the vehicle is detected based on the image acquired by the camera mounted on the vehicle, and based on the detected driving environment data, automatic driving control such as following the preceding vehicle, warning, braking, steering support, etc. A technique for providing driving support has been developed (see Patent Document 1).
In the prior art, a solid-state image sensor such as a CCD is used in an in-vehicle camera. In-vehicle cameras that continue to acquire images of roads and the like play an important role in automatic driving control and driving support, but there are not many proposals for cameras that are premised on being installed in vehicles, and it cannot be said that the usability of the cameras is sufficient. It was.

JP-A-2010-79424

The image pickup device according to the present invention is an image pickup device mounted on a vehicle, and includes an image pickup unit, an input unit for inputting information from the outside, and an image pickup control unit for controlling the image pickup unit. The unit changes the imaging conditions of the imaging unit based on the information input to the input unit.

It is a schematic block diagram of the driving support device of a vehicle. It is a block diagram which illustrates the structure of the control device. It is sectional drawing of the laminated type image sensor. It is a figure explaining the pixel arrangement and the unit area of an image pickup chip. It is a figure explaining the circuit of a unit area. It is a block diagram which shows the functional structure of an image sensor. It is a figure which illustrates the position of the focus detection pixel on the image pickup surface of an image sensor. It is an enlarged view of the area including a part of a focus detection pixel line. It is a block diagram which illustrates the structure of the camera which has an image sensor. It is a figure which illustrates the image pickup surface of the image pickup chip, the image pickup area and attention area, and the rest area. It is a flowchart explaining the flow of the control process of a camera executed by a control unit. It is a flowchart explaining the detail of the initial setting process. It is a figure which illustrates the table of the initial setting value. It is a figure which illustrates the image pickup surface of the image pickup chip, the image pickup area and attention area, and the rest area. It is a figure which illustrates the image pickup surface of the image pickup chip, the image pickup area and attention area, and the rest area. It is a figure which illustrates the image pickup surface of the image pickup chip, the image pickup area and attention area, and the rest area. It is a flowchart explaining the detail of the running assist setting process. It is a figure explaining the flag Em. It is a figure explaining the distance Z. FIG. 20A is a diagram illustrating a change in the position and size of a region of interest when turning right at an intersection on a general road. FIG. 20B is a diagram illustrating the movement of the position and the change in size of the region of interest when changing lanes while accelerating on the highway. It is a figure explaining the turn signal direction and resizing of a region of interest. It is a flowchart explaining the process at the time of operation of the turn signal switch by the modification 1.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
<Camera usage scene>
FIG. 1 is a schematic configuration diagram of a driving support device 2 of a vehicle 1 equipped with a camera 3 according to an embodiment of the present invention. In FIG. 1, a driving support device 2 is mounted on a vehicle 1 such as an automobile. The driving support device 2 includes a camera 3, a control device 4, a first traveling control unit 5, a second traveling control unit 6, and the like. In this description, an example in which an internal combustion engine is used as a drive source will be described, but a motor may be used as a drive source.

The camera 3 includes an image pickup optical system having a plurality of lenses and an image pickup element (in this embodiment, a stacked image pickup element (see FIG. 3)), and is attached to the front of the ceiling in the vehicle interior, for example. The camera 3 is directed to the front of the vehicle 1, and its mounting height (distance from the ground to the camera 3) is adjusted to, for example, 1.4 (m). The camera 3 acquires an image of the traveling direction of the vehicle 1, and measures the distance (distance measurement) to each subject (object) at a plurality of positions on the shooting screen based on the acquired image. The distance measurement is calculated by a distance measurement calculation using an image signal from a focus detection pixel provided in the stacked image sensor. The focus detection pixel and distance measurement will be described later. The image data and the distance measurement data acquired by the camera 3 are transmitted to the control device 4. The cameras 3 may be provided outside the vehicle, the cameras 3 inside and outside the vehicle may be linked, and the number of cameras 3 may be appropriately set. As an example, the camera 3 outside the vehicle may be used for detecting the white line, which will be described later, and the cameras 3 inside and outside the vehicle may be used for recognizing an object or an obstacle.

The control device 4 includes a CPU 4a and a storage unit 4b as illustrated in FIG. Based on various programs stored in the storage unit 4b, the CPU 4a performs various calculations using control parameters stored in the storage unit 4b and detection signals from each sensor described later.

The first travel control unit 5 performs constant speed travel control and follow-up travel control based on an instruction from the control device 4. The constant speed running control is a control for running the vehicle 1 at a constant speed based on a predetermined control program. The follow-up travel control is constant with respect to the preceding vehicle when the speed of the preceding vehicle recognized by the control device 4 is equal to or less than the target speed set in the vehicle 1 when the constant speed traveling control is performed. It is a control to drive while maintaining the inter-vehicle distance.

The second travel control unit 6 performs driving support control based on an instruction from the control device 4. Based on a predetermined control program, the driving support control outputs a steering control signal to the steering control device 9 so that the vehicle 1 travels along the road, and prevents the vehicle 1 from colliding with an object. It is a control that outputs a brake control signal to the brake control device 8.

FIG. 1 further shows a throttle control device 7, a brake control device 8, a steering control device 9, a steering wheel 10, a turn signal switch 11, a vehicle speed sensor 12, a yaw rate sensor 13, and a display device 14. The GPS device 15, the shift lever position detection device 16, and the microphone 17 are shown in the figure.

The throttle control device 7 controls the opening degree of a throttle valve (not shown) according to the amount of depression of the accelerator pedal 7a. The throttle control device 7 also controls the opening degree of the throttle valve according to the throttle control signal transmitted from the first travel control unit 5. The throttle control device 7 further sends a signal indicating the amount of depression of the accelerator pedal 7a to the control device 4.

The brake control device 8 controls the opening degree of a brake valve (not shown) according to the amount of depression of the brake pedal 8a. The brake control device 8 also controls the opening degree of the brake valve in response to the brake control signal from the second travel control unit 6. The brake control device 8 further sends a signal indicating the amount of depression of the brake pedal 8a to the control device 4.

The steering control device 9 controls the steering angle of a steering device (not shown) according to the rotation angle of the steering wheel 10. The steering control device 9 also controls the steering angle of the steering device in response to the steering control signal from the second travel control unit 6. The steering control device 9 further sends a signal indicating the rotation angle of the steering wheel 10 to the first travel control unit 5 and the control device 4, respectively.

The turn signal switch 11 is a switch for operating a turn signal (winker) device (not shown). The turn signal device is a blinking light emitting device that indicates a change of course of the vehicle 1. When the turn signal switch 11 is operated by the occupant of the vehicle 1, the operation signal from the turn signal switch 11 is sent to the turn signal device, the second travel control unit 6, and the control device 4, respectively. The vehicle speed sensor 12 detects the vehicle speed V of the vehicle 1 and sends a detection signal to the first travel control unit 5, the second travel control unit 6, and the control device 4, respectively.

The yaw rate sensor 13 detects the yaw rate of the vehicle 1 and sends the detection signal to the second travel control unit 6 and the control device 4, respectively. The yaw rate is the rate of change of the rotation angle of the vehicle 1 in the turning direction. The display device 14 displays information indicating a control state by the first travel control unit 5 and the second travel control unit 6. The display device 14 is composed of, for example, a HUD (Head Up Display) that projects information on a windshield. As the display device 14, a display unit of a navigation device (not shown) may be used.

The GPS device 15 receives radio waves from GPS satellites and calculates the position (latitude, longitude, etc.) of the vehicle 1 by performing a predetermined calculation using the information carried on the radio waves. The position information calculated by the GPS device 15 is transmitted to a navigation device or a control device 4 (not shown). The shift lever position detecting device 16 detects the position of a shift lever (for example, parking (P), reverse (R), drive (D), etc.) not shown, which is operated by the occupant of the vehicle 1. The position information of the shift lever detected by the shift lever position detection device 16 is transmitted to the control device 4.

The microphone 17 is composed of, for example, a front microphone, a right side microphone, and a left side microphone. The front microphone has a directivity that exclusively collects the sound in front of the vehicle 1. The right-side microphone has a directivity that exclusively collects the sound on the right side of the vehicle 1. The left-side microphone has a directivity that exclusively collects the sound on the left side of the vehicle 1. Each sound information (front, right side, left side) collected by the microphone 17 is transmitted to the control device 4, respectively.

<Detection of object>
The control device 4 performs image processing on the image from the camera 3 as follows in order to detect the traveling path and the object of the vehicle 1. First, the control device 4 generates a distance image (depth distribution image) based on distance measurement data at a plurality of positions in the shooting screen. The control device 4 performs a well-known grouping process based on the distance image data, and frames (windows) such as three-dimensional road shape data, side wall data, and object data stored in the storage unit 4b in advance. White line data (including white line data along the road and white line (stop line: intersection information) data crossing the road), side wall data such as guard rails and edge stones existing along the road are extracted and targeted. Objects / obstacles are classified and extracted into other objects such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, and electric poles.
In this description, the white or yellow line drawn on the road is called a white line. In addition, it is called a white line including a solid line and a broken line.

<Driving support>
The control device 4 recognizes the traveling path and the object / obstacle that becomes an obstacle based on each of the information extracted as described above, that is, the white line data, the guard rail side wall data, and the object data, and the recognition result. The second travel control unit 6 is made to perform the above-mentioned driving support control based on the above. That is, the vehicle 1 is driven along the road to prevent the vehicle 1 from colliding with the object.

<Driving control>
The control device 4 estimates the vehicle traveling path in the following four ways, for example.
(1) Vehicle travel path estimation based on white line White line data on both the left and right sides of the driving path or either the left or right side is obtained from the image acquired by the camera 3, and the vehicle 1 travels from these white line data. When the shape of the lane in which the vehicle is located can be estimated, the control device 4 estimates that the vehicle traveling path is parallel to the white line in consideration of the width of the vehicle 1 and the position of the vehicle 1 in the current lane.

(2) Vehicle travel path estimation based on side wall data of guardrails, curbs, etc. From the images acquired by the camera 3, side wall data on both the left and right sides of the lane or on either the left or right side is obtained. When the shape of the lane in which the vehicle 1 is traveling can be estimated, the control device 4 considers the width of the vehicle 1 and the position of the vehicle 1 in the current lane, and the vehicle traveling path is parallel to the side wall. Estimate.

(3) Own vehicle traveling path estimation based on the preceding vehicle trajectory The control device 4 estimates the own vehicle traveling path based on the past traveling trajectory of the preceding vehicle stored in the storage unit 4b. The preceding vehicle refers to the vehicle closest to the vehicle 1 among the objects traveling in the same direction as the vehicle 1.

(4) Own vehicle travel path estimation based on the travel locus of the vehicle 1 The control device 4 estimates the own vehicle travel path based on the driving state of the vehicle 1. For example, the vehicle traveling path is estimated using the turning curvature based on the detection signal by the yaw rate sensor 13 and the detection signal by the vehicle speed sensor 12. The turning curvature Cua is calculated by Cua = dψ / dt / V. dψ / dt is the yaw rate (speed of change of the rotation angle in the turning direction), and V is the vehicle speed of the vehicle 1.

The control device 4 estimates the traveling area of the vehicle 1 at the position where the object exists for each object according to a predetermined traveling control program stored in the storage unit 4b based on the own vehicle traveling path. The traveling area and the position of the object are compared to determine whether or not each object is within the traveling area. The control device 4 further recognizes the preceding vehicle based on the image pickup result of the camera 3. That is, the control device 4 sets the vehicle closest to the vehicle 1 as the preceding vehicle among the objects existing in the traveling region and traveling in the forward direction (the same direction as the vehicle 1).

The control device 4 outputs the inter-vehicle distance information between the preceding vehicle and the vehicle 1 and the vehicle speed information of the preceding vehicle to the first traveling control unit 5 as outside-vehicle information. Here, the vehicle speed information of the preceding vehicle is distance-measured based on the vehicle speed V of the vehicle 1 acquired at predetermined time intervals and the image acquired by the camera 3 at the predetermined time intervals in synchronization with the acquisition timing of the vehicle speed V. Calculated based on the change in the distance to the preceding vehicle (inter-vehicle distance) on the shooting screen.

The first travel control unit 5 sends a throttle control signal to the throttle control device 7 so that the vehicle speed V detected by the vehicle speed sensor 12 converges to a preset predetermined vehicle speed (target speed). As a result, the throttle control device 7 feedback-controls the opening degree of the throttle valve (not shown) to automatically drive the vehicle 1 at a constant speed.

Further, when the vehicle speed information of the preceding vehicle input from the control device 4 when the first travel control unit 5 is performing the travel control in the constant speed state is equal to or less than the target speed set in the vehicle 1. , The throttle control signal is transmitted to the throttle control device 7 based on the inter-vehicle distance information input from the control device 4. Specifically, an appropriate target value of the inter-vehicle distance is set based on the inter-vehicle distance from the vehicle 1 to the preceding vehicle, the vehicle speed of the preceding vehicle, and the vehicle speed V of the vehicle 1, and the image acquired by the camera 3 is used. A throttle control signal is transmitted to the throttle control device 7 so that the inter-vehicle distance measured based on the distance converges to the target value of the inter-vehicle distance. As a result, the throttle control device 7 feedback-controls the opening degree of the throttle valve (not shown) to cause the vehicle 1 to follow the preceding vehicle.

<Explanation of stacked image sensor>
The stacked image sensor 100 provided in the camera 3 described above will be described. The laminated image sensor 100 is described in Japanese Patent Application No. 2012-139026 previously filed by the applicant of the present application. FIG. 3 is a cross-sectional view of the stacked image sensor 100. The image sensor 100 includes a back-illuminated image pickup chip 113 that outputs a pixel signal corresponding to incident light, a signal processing chip 111 that processes the pixel signal, and a memory chip 112 that stores the pixel signal. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are laminated and electrically connected to each other by a conductive bump 109 such as Cu.

As shown in the figure, the incident light is mainly incident in the Z-axis plus direction indicated by the white arrow. In the present embodiment, in the image pickup chip 113, the surface on the side where the incident light is incident is referred to as a back surface (imaging surface). Further, as shown in the coordinate axes, the left direction of the paper surface orthogonal to the Z axis is defined as the X-axis plus direction, and the Z-axis and the front direction of the paper surface orthogonal to the X-axis are defined as the Y-axis plus direction. In some subsequent figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

An example of the image pickup chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back surface side of the wiring layer 108. The PD layer 106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally and accumulating charges according to incident light, and a transistor 105 provided corresponding to the PD 104.

A color filter 102 is provided on the incident side of the incident light in the PD layer 106 via the passivation film 103. The color filter 102 has a plurality of types that transmit different wavelength regions from each other, and has a specific arrangement corresponding to each of the PD 104. The arrangement of the color filters 102 will be described later. A set of a color filter 102, a PD 104, and a transistor 105 forms one pixel.

A microlens 101 is provided on the incident side of the incident light in the color filter 102 corresponding to each pixel. The microlens 101 collects incident light toward the corresponding PD 104.

The wiring layer 108 has a wiring 107 that transmits a pixel signal from the PD layer 106 to the signal processing chip 111. The wiring 107 may have multiple layers, and may be provided with passive elements and active elements.

A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with the plurality of bumps 109 provided on the facing surfaces of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are aligned by pressurization or the like. The bumps 109 are joined together and electrically connected.

Similarly, a plurality of bumps 109 are arranged on the surfaces of the signal processing chip 111 and the memory chip 112 facing each other. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressurized, so that the aligned bumps 109 are joined to each other and electrically connected.

The bonding between the bumps 109 is not limited to Cu bump bonding by solid phase diffusion, and micro bump bonding by solder melting may be adopted. Further, for example, about one bump 109 may be provided for one block described later. Therefore, the size of the bump 109 may be larger than the pitch of the PD 104. Further, in the peripheral region other than the pixel region in which the pixels are arranged, a bump larger than the bump 109 corresponding to the pixel region may be provided together.

The signal processing chip 111 has TSVs (Through Silicon Vias) 110 that connect circuits provided on the front and back surfaces to each other. The TSV 110 is preferably provided in the peripheral region. Further, the TSV 110 may also be provided in the peripheral area of the image pickup chip 113 and the memory chip 112.

FIG. 4 is a diagram illustrating a pixel arrangement of the image pickup chip 113 and a unit area 131. In particular, the state in which the imaging chip 113 is observed from the back surface (imaging surface) side is shown. For example, 20 million or more pixels are arranged in a matrix in the pixel area. In the example of FIG. 4, 16 pixels of adjacent 4 pixels × 4 pixels form one unit region 131. The grid lines in the figure show the concept that adjacent pixels are grouped to form a unit region 131. The number of pixels forming the unit region 131 is not limited to this, and may be about 1000, for example, 32 pixels × 64 pixels, more or less.

As shown in the partially enlarged view of the pixel region, the unit region 131 of FIG. 4 includes four so-called Bayer arrays including four pixels of green pixels Gb, Gr, blue pixels B, and red pixels R in the vertical and horizontal directions. The green pixels Gb and Gr are pixels having a green filter as the color filter 102, and receive light in the green wavelength band among the incident light. Similarly, the blue pixel B is a pixel having a blue filter as a color filter 102 and receives light in the blue wavelength band, and the red pixel R is a pixel having a red filter as the color filter 102 and having a red wavelength band. Receives light.

In the present embodiment, a plurality of blocks are defined so that each block includes at least one unit area 131, and each block can control the pixels included in each block with different control parameters. That is, it is possible to acquire an imaging signal having different imaging conditions between the pixel group included in a certain block and the pixel group included in another block. Examples of control parameters are frame rate, gain, thinning rate, number of rows or columns to be added to add pixel signals, charge accumulation time or number of times, digitization bits (word length), and the like. The image sensor 100 can freely thin out not only the row direction (the X-axis direction of the image pickup chip 113) but also the column direction (the Y-axis direction of the image pickup chip 113). Further, the control parameter may be a parameter in image processing after acquiring an image signal from a pixel.

FIG. 5 is a diagram illustrating a circuit in the unit region 131. In the example of FIG. 5, one unit region 131 is formed by 9 pixels of 3 adjacent pixels × 3 pixels. As described above, the number of pixels included in the unit region 131 is not limited to this, and may be less or more. The two-dimensional positions of the unit region 131 are indicated by reference numerals A to I.

The pixel reset transistor included in the unit region 131 is configured to be individually on / off for each pixel. In FIG. 5, a reset wiring 300 for turning on / off the reset transistor of the pixel A is provided, and a reset wiring 310 for turning on / off the reset transistor of the pixel B is provided separately from the reset wiring 300. Similarly, the reset wiring 320 for turning on / off the reset transistor of the pixel C is provided separately from the reset wirings 300 and 310. Dedicated reset wiring for turning on / off each reset transistor is also provided for the other pixels D to I.

The transfer transistors of the pixels included in the unit area 131 are also configured to be individually on / off for each pixel. In FIG. 5, a transfer wiring 302 for turning on / off the transfer transistor of pixel A, a transfer wiring 312 for turning on / off the transfer transistor of pixel B, and a transfer wiring 322 for turning on / off the transfer transistor of pixel C are separately provided. Dedicated transfer wiring for turning on / off each transfer transistor is also provided from the other pixel D to the pixel I.

Further, the pixel selection transistor included in the unit region 131 is also configured to be individually on / off for each pixel. In FIG. 5, the selection wiring 306 for turning on / off the selection transistor of pixel A, the selection wiring 316 for turning on / off the selection transistor of pixel B, and the selection wiring 326 for turning on / off the selection transistor of pixel C are separately provided. Dedicated selection wiring for turning on / off each selection transistor is also provided for the other pixels D to I.

The power supply wiring 304 is commonly connected by pixels A to pixels A included in the unit area 131. Similarly, the output wiring 308 is commonly connected by pixels A to pixels A included in the unit area 131. Further, the power supply wiring 304 is commonly connected between the plurality of unit areas, but the output wiring 308 is individually provided for each unit area 131. The load current source 309 supplies current to the output wiring 308. The load current source 309 may be provided on the imaging chip 113 side or the signal processing chip 111 side.

By turning on / off the reset transistor and the transfer transistor of the unit region 131 individually, the charge including the charge accumulation start time, the accumulation end time, and the transfer timing is independently from the pixel A included in the unit region 131 with respect to the pixel I. Accumulation can be controlled. Further, by turning on and off the selection transistors of the unit region 131 individually, the pixel signal of the pixel I can be output from each pixel A via the common output wiring 308.

Here, a so-called rolling shutter method is known in which charge accumulation is controlled in a regular order with respect to rows and columns for pixels A to I included in the unit region 131. When a pixel is selected for each row by the rolling shutter method and then a column is specified, pixel signals are output in the order of "ABCDEFGHI" in the example of FIG.

By configuring the circuit with the unit region 131 as a reference in this way, the charge accumulation time can be controlled for each unit region 131. In other words, pixel signals with different frame rates can be output between the unit regions 131. Further, in the imaging chip 113, the unit region 131 included in the other area is rested while the unit region 131 included in a part of the area is charged (imaging), so that the unit region 131 included in the other area is rested in a predetermined area of the imaging chip 113. Only the image can be taken and the pixel signal can be output. Further, it is also possible to switch the area for performing charge storage (imaging) between frames (the target area for storage control) to sequentially perform imaging in different areas of the imaging chip 113 and output a pixel signal.

FIG. 6 is a block diagram showing a functional configuration of the image sensor 100 corresponding to the circuit illustrated in FIG. The analog multiplexer 411 sequentially selects nine PD 104s forming the unit area 131, and outputs each pixel signal to the output wiring 308 provided corresponding to the unit area 131. The multiplexer 411 is formed on the imaging chip 113 together with the PD 104.

The pixel signal output via the multiplexer 411 is CDS and A / by a signal processing circuit 412 formed on the signal processing chip 111 that performs correlated double sampling (CDS) analog / digital (A / D) conversion. D conversion is performed. The A / D converted pixel signal is passed to the demultiplexer 413 and stored in the pixel memory 414 corresponding to each pixel. The demultiplexer 413 and the pixel memory 414 are formed on the memory chip 112.

The arithmetic circuit 415 processes the pixel signal stored in the pixel memory 414 and hands it over to the image processing unit in the subsequent stage. The arithmetic circuit 415 may be provided on the signal processing chip 111 or the memory chip 112. In addition, although FIG. 6 shows the connection for one unit area 131, in reality, these exist for each unit area 131 and operate in parallel. However, the arithmetic circuit 415 does not have to exist for each unit area 131. For example, even if one arithmetic circuit 415 processes sequentially while referring to the value of the pixel memory 414 corresponding to each unit area 131 in order. Good.

As described above, the output wiring 308 is provided corresponding to each of the unit regions 131. Since the image sensor 100 has an image pickup chip 113, a signal processing chip 111, and a memory chip 112 stacked on top of each other, by using an electrical connection between the chips using bumps 109 for these output wirings 308, each chip is placed in the plane direction. Wiring can be routed without making it large.

<Explanation of distance measurement>
FIG. 7 is a diagram illustrating the positions of the focus detection pixels on the image pickup surface of the image pickup device 100. In the present embodiment, the focus detection pixels are discretely provided side by side along the X-axis direction (horizontal direction) of the image pickup chip 113. In the example of FIG. 7, 15 focus detection pixel lines 60 are provided at predetermined intervals. The focus detection pixels constituting the focus detection pixel line 60 output an image signal for distance measurement. In the imaging chip 113, ordinary imaging pixels are provided at pixel positions other than the focus detection pixel line 60. The imaging pixel outputs an image signal for monitoring outside the vehicle.

FIG. 8 is an enlarged view of a region including a part of one of the focus detection pixel lines 60. In FIG. 8, a red pixel R, a green pixel G (Gb, Gr), and a blue pixel B, a focus detection pixel S1, and a focus detection pixel S2 are exemplified. The red pixel R, the green pixel G (Gb, Gr), and the blue pixel B are arranged according to the above-mentioned Bayer arrangement rules.

The square-shaped regions exemplified for the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B indicate the light receiving region of the imaging pixel. Each imaging pixel receives a light flux passing through the exit pupil of the imaging optical system 31 (FIG. 9). That is, the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B each have a square mask opening, and the light passing through these mask openings reaches the light receiving portion of the imaging pixel. ..

The shape of the light receiving region (mask opening) of the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B is not limited to a quadrangle, and may be, for example, a circle.

The semicircular region exemplified for the focus detection pixel S1 and the focus detection pixel S2 indicates a light receiving region of the focus detection pixel. That is, the focus detection pixel S1 has a semicircular mask opening on the left side of the pixel position in FIG. 8, and the light passing through the mask opening reaches the light receiving portion of the focus detection pixel S1. On the other hand, the focus detection pixel S2 has a semicircular mask opening on the right side of the pixel position in FIG. 8, and the light passing through the mask opening reaches the light receiving portion of the focus detection pixel S2. As described above, the focus detection pixel S1 and the focus detection pixel S2 receive a pair of light fluxes passing through different regions of the exit pupil of the imaging optical system 31 (FIG. 9), respectively.

The position of the focus detection pixel line on the image pickup chip 113 is not limited to the position illustrated in FIG. 7. Further, the number of focus detection pixel lines is not limited to the example of FIG. 7. Further, the shape of the mask opening in the focus detection pixel S1 and the focus detection pixel S2 is not limited to a semicircle, and for example, a square light receiving region (mask opening) in the imaging pixel R, the imaging pixel G, and the imaging pixel B. The part) may be divided into a rectangular shape in the horizontal direction.

Further, the focus detection pixel line in the image pickup chip 113 may be provided by arranging the focus detection pixels along the Y-axis direction (vertical direction) of the image pickup chip 113. An image pickup device in which an image pickup pixel and a focus detection pixel are arranged in a two-dimensional manner as shown in FIG. 8 is known, and detailed illustration and description of these pixels will be omitted.
In the example of FIG. 8, a configuration in which the focus detection pixels S1 and S2 receive one of a pair of light fluxes for focus detection, that is, a so-called 1PD structure, has been described. Instead, for example, as disclosed in Japanese Patent Application Laid-Open No. 2007-282107, a so-called 2PD structure may be adopted, in which the focus detection pixels receive both of a pair of light fluxes for focus detection. By adopting the 2PD structure in this way, it is possible to read the image data from the focus detection pixels as well, and the focus detection pixels do not become defective pixels.

In the present embodiment, a pair of images with a pair of luminous flux passing through different regions of the imaging optical system 31 (FIG. 9) based on the image signals for distance measurement output from the focus detection pixel S1 and the focus detection pixel S2. The focus adjustment state (defocus amount) of the imaging optical system 31 is calculated by detecting the image shift amount (phase difference) of.

In general, the pair of images approach each other in a so-called front pin state in which the imaging optical system 31 forms a sharp image of an object (for example, a preceding vehicle) in front of the planned focal plane, and conversely, the objects are behind the planned focal plane. In the so-called rear focus state that connects the sharp images of, they move away from each other. The pair of images relatively match the in-focus state that forms a sharp image of the object on the planned focal plane. Therefore, the relative positional deviation amount of the pair of images corresponds to the distance (depth information) to the object.

Since the defocus amount calculation based on the phase difference is known in the field of cameras, detailed description thereof will be omitted. Here, since the defocus amount and the distance to the object have a one-to-one correspondence, the distance from the camera 3 to each object can be obtained by obtaining the defocus amount for each object. That is, the distance to the object can be measured (distance measurement) at a plurality of positions on the shooting screen. The relationship between the amount of defocus and the distance to the object is prepared in advance as a mathematical formula or a look-up table and stored in the non-volatile memory 35b (FIG. 9).

<Camera description>
FIG. 9 is a block diagram illustrating the configuration of the camera 3 having the above-mentioned image sensor 100. In FIG. 9, the camera 3 has an image pickup optical system 31, an image pickup unit 32, an image processing unit 33, a work memory 34, a control unit 35, and a recording unit 36.

The imaging optical system 31 guides the light flux from the field of view to the imaging unit 32. The image pickup unit 32 includes the image pickup element 100 and the drive unit 32a, and photoelectrically converts an image of an object imaged on the image pickup chip 113 by the image pickup optical system 31. The drive unit 32a generates a drive signal necessary for causing the image sensor 100 (imaging chip 113) to perform independent storage control in block units as described above. Instructions such as the position and shape of the block, its range, and the accumulation time are transmitted from the control unit 35 to the drive unit 32a.

The image processing unit 33 cooperates with the work memory 34 to perform image processing on the image data captured by the image capturing unit 32. The image processing unit 33 also performs color detection of an object included in the image, in addition to image processing such as contour enhancement processing and gamma correction.

The work memory 34 temporarily stores image data and the like before and after image processing. The recording unit 36 records image data or the like on a storage medium composed of a non-volatile memory or the like. The control unit 35 is composed of, for example, a CPU, and controls the entire operation of the camera 3 in response to a control signal from the control device 4. For example, a predetermined exposure calculation is performed based on the image signal captured by the image pickup unit 32, and the accumulation time of the image pickup chip 113 required for proper exposure is instructed to the drive unit 32a.

The control unit 35 includes a distance measuring calculation unit 35a and a non-volatile memory 35b. As described above, the distance measurement calculation unit 35a measures the distance to the object (distance measurement) at a plurality of positions on the shooting screen. The image data acquired by the camera 3 and the distance measurement data calculated by the camera 3 are sent to the control device 4 (FIG. 1). The non-volatile memory 35b stores a program executed by the control unit 35a and information necessary for distance measurement.

<Block control of image sensor>
The control device 4 causes the image sensor 100 (imaging chip 113) of the camera 3 to perform independent accumulation control in block units as described above. Therefore, the following signals are input to the control device 4 from each part of the vehicle 1 (FIG. 2).
(1) Depressed amount of the accelerator pedal 7a A signal indicating the depressed amount of the accelerator pedal 7a is input from the throttle control device 7 to the control device 4.
(2) Depressed amount of the brake pedal 8a A signal indicating the depressed amount of the brake pedal 8a is input from the brake control device 8 to the control device 4.

(3) Rotation angle of the steering wheel 10 A signal indicating the rotation angle of the steering wheel 10 is input from the steering control device 9 to the control device 4. The ratio of the rotation angle of the steering wheel 10 to the steering angle of the steering device depends on the gear ratio of the steering.
(4) Vehicle speed V of vehicle 1
The detection signal from the vehicle speed sensor 12 is input to the control device 4.

(5) Operation signal of the turn signal switch 11 The operation signal of the turn signal switch 11 is input to the control device 4.
(6) Shift lever operation position A signal indicating the shift lever operation position detected by the shift lever position detection device 16 is input to the control device 4.

(7) Position information of vehicle 1 The position information determined by the GPS device 15 is input from the GPS device 15 to the control device 4.
(8) Sound information around the vehicle 1 Sound information collected by the microphone 17 from the front, the right side, and the left side of the vehicle 1 is input to the control device 4, respectively.

FIG. 10 shows an imaging surface of the imaging chip 113, a region (imaging region 81 and attention region 82) for accumulating charges (imaging) on the imaging chip 113, and charge accumulation (imaging) in the row and column directions. It is a figure which illustrates the region which does not exist (pause region 83). The region of interest 82 is a region in which charge accumulation (imaging) is performed under conditions different from those of the imaging region 81. The size and position of the imaging region 81 and the region of interest 82 on the imaging chip 113 are also one of the imaging conditions.

The control device 4 controls the unit region 131 included in the imaging region 81 so as to set the first condition for imaging, and also sets the second condition for the unit region 131 included in the region of interest 82. Control to set conditions and image. Further, the control device 4 pauses the unit region 131 included in the pause region 83 so as not to take an image. A plurality of areas of interest 82 may be provided, or the imaging conditions may be different between the plurality of areas of interest. Moreover, it is not necessary to provide the rest area 83.

<Explanation of flowchart>
Hereinafter, how to determine the imaging region 81 and the region of interest 82 will be described with reference to the flowcharts (FIGS. 11, 12, and 17). FIG. 11 is a flowchart illustrating a flow of control processing of the camera 3 executed by the control device 4. The program for executing the process according to the flowchart of FIG. 11 is stored in the storage unit 4b of the control device 4. When the power supply is started from the vehicle 1 or the engine is started, for example, the control device 4 starts a program for performing the process according to FIG.

In step S10 of FIG. 11, the control device 4 determines whether or not the flag a = 0. The flag a is a flag in which 1 is set when the initial setting is completed and 0 is set when the initial setting is not completed. When the flag a = 0, the control device 4 determines affirmatively in step S10 and proceeds to step S20, and when the flag a ≠ 0, determines negatively in step S10 and proceeds to step S30.

In step S20, the control device 4 performs the initial setting process and proceeds to step S30. The details of the initial setting process will be described later. In step S30, the control device 4 performs the traveling assist setting process and proceeds to step S40. In the traveling assist setting process, the image pickup region 81 and the attention region 82 are determined for the image pickup element 100. The details of the driving assist setting process will be described later.

In step S40, the control device 4 sends an instruction to the camera 3 to drive the image pickup region 81 and the attention region 82 of the image pickup device 100 under predetermined conditions to acquire an image. In the present embodiment, for example, when the vehicle speed V increases from 0, the frame rate of the region of interest 82 is increased, the gain is increased, the thinning rate is decreased, and the accumulation time is shortened as compared with the imaging region 81. Set. As a result, the image is taken by the camera 3, and the distance is measured (distance measurement) at a plurality of positions on the shooting screen as described above.
It is not necessary to make all of the frame rate, gain, thinning rate, accumulation time, etc. different between the imaging region 81 and the region of interest 82, and at least one may be different. The control device 4 may be set so that the region of interest 82 is not thinned out.

In step S45, the control device 4 acquires image data and ranging data from the camera 3 and proceeds to step S50. In step S50, the control device 4 determines whether or not the setting for displaying information is made. When the display setting is made, the control device 4 positively determines step 50 and proceeds to step S60. If the display setting has not been made, the control device 4 negatively determines step 50 and proceeds to step S70.

In step S60, the control device 4 sends display information to the display device 14 (FIG. 1) and proceeds to step S70. The display information is information according to the state of the vehicle 1 determined in the driving assist setting process (S30), for example, "stopping", "emergency stop", "turn right", "turn left". The message "Masu" is displayed on the display device 14.
Instead of transmitting the display information, or together with the transmission of the display information, an audio signal for reproducing the above message may be transmitted to an audio reproduction device (not shown). In this case as well, the audio device of the navigation device (not shown) may be used as the audio reproduction device (not shown).

In step S70, the control device 4 determines whether or not the operation has been turned off. When the control device 4 receives, for example, an off signal (for example, an engine off signal) from the vehicle 1, the control device 4 positively determines step S70, performs a predetermined off process, and ends the process according to FIG. When the control device 4 does not receive the off signal from the vehicle 1, for example, the control device 4 negatively determines step S70 and proceeds to step S80. In step S80, the control device 4 waits for a predetermined time (for example, 0.1 second) and returns to step S30. When returning to step S30, the above-mentioned process is repeated.

<Initial setting process>
FIG. 12 is a flowchart illustrating the details of the step S20 initial setting process of the flowchart of FIG. In step S21 of FIG. 12, the control device 4 inputs the position information of the vehicle 1 from the GPS device 15 (FIG. 1) and proceeds to step S22. In step S22, the control device 4 sets a flag indicating whether the lane in which the vehicle 1 travels is the left or right of the road, that is, the left or right lane, based on the latitude and longitude included in the position information. Specifically, the country name in which the vehicle 1 is used is determined based on the latitude and longitude. Then, referring to a database (not shown), a flag indicating whether the road in the country is left-handed or right-handed is set. A database showing the relationship between the country name and the left and right of the lane is stored in advance in the storage unit 4b.

In step S23, the control device 4 sets a flag indicating the mounting position (right or left) of the steering wheel (steering wheel 10) in the vehicle 1 and proceeds to step S24. Information indicating whether the steering wheel is right or left is stored in advance in the storage unit 4b as specification information of the vehicle 1. In step S24, the control device 4 determines the initial setting value based on the table illustrated in FIG. 13 and proceeds to step S25. The order of step S21 and step S23 may be changed.

According to FIG. 13, there are four types of "1" to "4" depending on the combination of the mounting position (right or left) of the steering wheel 10 on the vehicle 1 and the position of the lane (right or left) on the road. Initial setting values are prepared. When driving on the left side with the right steering wheel, the initial setting value is "4".

In step S25, the control device 4 sets the initial position of the region of interest 82. The initial position of the region of interest 82 is a position corresponding to the initial setting value. Specifically, when the initial setting value is "1", the initial position of the attention area 82 is (Xq1, Yq), and when the initial setting value is "2", the initial position of the attention area 82 is (Xq2, Yq). When the initial setting value is "3", the initial position of the attention region 82 is (Xq3, Yq), and when the initial setting value is "4", the initial position of the attention region 82 is (Xq4, Yq).

In this description, in the coordinate system representing the imaging region 81, the position of the region of interest 82 is represented by the coordinates (Xq, Yq) at the center of the region of interest 82. FIG. 10 exemplifies the attention area 82 when the initial setting value is “4”, and since the right steering wheel is for left-hand traffic, the attention area 82 is on the driver's seat side (to the right) in the left-hand traffic zone. The initial position (Xq4, Yq) is set so as to set.

FIG. 14 exemplifies the attention area 82 when the initial setting value is “1”, and since the left steering wheel is for right-hand traffic, the attention area 82 is on the driver's seat side (to the left) in the right-hand traffic zone. The initial position (Xq1, Yq) is set so as to set.

FIG. 15 exemplifies the attention area 82 when the initial setting value is “3”, and since the left steering wheel is used for left-hand traffic, the attention area 82 is on the driver's seat side (to the left) in the left-hand traffic zone. The initial position (Xq3, Yq) is set so as to set.

FIG. 16 exemplifies the attention area 82 when the initial setting value is “2”, and since the right steering wheel is for right-hand traffic, the attention area 82 is on the driver's seat side (to the right) in the right-hand traffic zone. The initial position (Xq2, Yq) is set so as to set.

In step S26 of FIG. 12, the control device 4 sets the initial size of the region of interest 82. In the present embodiment, the initial size (Px (X-axis direction) × Py (Y-axis direction)) of the region of interest 82 is determined based on the size (dimensions) of the object (for example, the preceding vehicle). When the image acquired by the camera 3 includes the preceding vehicle, the control device 4 obtains the image height of the preceding vehicle captured by the imaging chip 113, the focal length of the known imaging optical system 31, and the distance measurement. The size (dimensions) of the preceding vehicle is estimated based on the distance L from the vehicle 1 to the preceding vehicle. Then, the number of pixels constituting the image obtained on the imaging chip 113 when the preceding vehicle having the estimated size (for example, width 3 (m) × height 1.4 (m)) is imaged from 1 (m) behind. (Px (X-axis direction) x Py (Y-axis direction)) is the initial size.

Px and Py are calculated by the following equations (1) and (2).
Px = ox × L ... (1)
Py = oy × L… (2)
However, ox is the number of pixels in the X-axis direction constituting the image of the preceding vehicle imaged by the image pickup chip 113 at a distance of L (m). oy is the number of pixels in the Y-axis direction constituting the image of the preceding vehicle imaged by the image pickup chip 113 at a distance of L (m). L is the inter-vehicle distance from the vehicle 1 to the preceding vehicle. In the coordinate system representing the imaging region 81, the Yq representing the initial position is the height center of the image obtained on the imaging chip 113 when the preceding vehicle 1 (m) away is imaged (preceding in this example). It corresponds to a part with a height of 0.7 (m) of a car).

In step S27, the control device 4 sends display information to the display device 14 (FIG. 1), sets the flag a to 1, and ends the process according to FIG. The display information is information indicating that the initial setting process has been completed. For example, the display device 14 displays the message "Initial setting is completed".

<Running assist setting process>
FIG. 17 is a flowchart illustrating the details of the traveling assist setting process. In step S310 of FIG. 17, when the position information of the shift lever input from the shift lever position detection device 16 (FIG. 1) is “P” (parking), the control device 4 positively determines step S310 and steps. Proceed to S320. If the position information of the shift lever input from the shift lever position detection device 16 (FIG. 1) is not “P”, the control device 4 negatively determines step S310 and proceeds to step S420. The determination in step S310 may be applied even when the shift lever is "N" (neutral).

In step S320, the control device 4 inputs the vehicle speed V from the vehicle speed sensor 12 and proceeds to step S330. The control device 4 changes the frame rate of the region of interest 82 according to, for example, the vehicle speed V. As described above, when the frame rate of the attention region 82 is set higher than the frame rate of the imaging region 81, the control device 4 sets the frame rate of the attention region 82 higher as the vehicle speed V increases, and the vehicle speed V The frame rate of the region of interest 82 is set lower as the number decreases. In this case, the control device 4 may apply control so that the frame rate of the imaging region 81 other than the region of interest 82 is also proportional to the vehicle speed V. In step S330, the control device 4 inputs the amount of depression of the brake pedal 8a from the brake control device 8 (FIG. 1) and proceeds to step S340.

In step S340, the control device 4 determines whether or not the flag Em = 0 based on the vehicle speed V and the depression amount (depression angle) of the brake pedal 8a. The flag Em is a flag set as illustrated in FIG. 18 based on the vehicle speed V and the amount of change in the amount of depression (depression angle) of the brake pedal 8a. In the present embodiment, the case where Em = 1 is determined to be an emergency brake (sudden brake), and the case where Em = 0 is determined to be a normal brake. When Em = 0, the control device 4 positively determines step S340 and proceeds to step S350. When Em = 1, the control device 4 negatively determines step S340 and proceeds to step S430.
Instead of the amount of change in the amount of depression (depression angle) of the brake pedal 8a, Em = 1 may be determined based on the amount of change in the opening degree of the brake valve (not shown). Further, Em = 1 may be determined based on the amount of change in the vehicle speed V, or Em = 1 may be determined based on the amount of change in the reduction ratio of the transmission (not shown).

In step S350, the control device 4 inputs the amount of depression of the accelerator pedal 7a from the throttle control device 7 (FIG. 1) and proceeds to step S360. In step S360, the control device 4 inputs the rotation angle θ of the steering wheel 10 from the steering control device 9 and proceeds to step S370. In step S370, the control device 4 determines whether or not the steering operation has been performed. When the rotation angle θ is larger than the predetermined value, the control device 4 positively determines step S370 and proceeds to step S380, and when the rotation angle θ is equal to or less than the predetermined value, the control device 4 negatively determines step S370 and proceeds to step S440.

In step S380, the control device 4 calculates the movement amount Xdist of the region of interest 82 in the X-axis direction by the following equation (3) based on the rotation angle θ of the steering wheel 10 and the vehicle speed V.
Xdist = θ × (V × 0.2)… (3)
According to the above equation (3), the larger the steering angle of the steering device (that is, the rotation angle θ of the steering wheel 10) and the larger the vehicle speed V, the larger the movement amount Xdist.

In step S390, the control device 4 calculates the position (X coordinate) of the attention region 82 during traveling by the following equation (4) based on the initial position (XqN, Yq) of the attention region 82 set in the initial setting process. To do.
Xq = XqN + Xdist… (4)
However, N is any value of the initial setting values 1 to 4 determined in the initial setting process. Xdist is the amount of movement of the region of interest 82 in the X-axis direction calculated in step S380, and corresponds to the number of pixels in the X-axis direction. By the process of step S390, the position of the region of interest 82 changes according to the steering operation. The position of the region of interest 82 also changes depending on the magnitude of the vehicle speed V.

In step S400, the control device 4 calculates the position (Y coordinate) of the attention region 82 during traveling by the following equation (5) based on the initial position (XqN, Yq) of the attention region 82 set in the initial setting process. To do.
Yq = Yq + P (Z)… (5)
However, P (Z) is the amount of movement of the region of interest 82 in the Y-axis direction, and is the number of pixels in the Y-axis direction corresponding to the depth Z (m). For example, it represents how many pixels a road image having a depth of 20 (m) corresponds to in the Y-axis direction. The relationship P (Z) between the depth Z and the number of pixels is stored in advance in the storage unit 4b (FIG. 2).

Generally, when the traveling direction is imaged on a flat straight road, the number of pixels in the Y-axis direction corresponding to the image of the road on the image pickup chip 113 increases as the depth Z (m) from the vehicle 1 becomes deeper. Therefore, the value of Yq corresponding to the height center of the image when the preceding vehicle 1 (m) away is imaged is increased as the notable preceding vehicle becomes farther (that is, the depth Z becomes deeper).

The control device 4 determines the depth Z of the preceding vehicle to be noted by the following equation (6).
Z = Za + Zb ... (6)
However, Za is the braking distance (m) on a dry road, and Zb is the braking distance (m) on a wet road surface. Za and Zb are based on the values illustrated in FIG. In the present embodiment, the position of the attention region 82 is determined so that the preceding vehicle having a depth Z in front of the vehicle 1 (that is, a position Z (m) away from the vehicle 1) is included in the attention region 82. This is based on the idea of focusing on the distance farther than the distance required to stop when the emergency brake is applied. The value of the depth Z (Za + Zb) corresponding to the vehicle speed V is stored in advance in the storage unit 4b (FIG. 2). According to the process of step S400, the position of the region of interest 82 changes according to the change in the vehicle speed V.
With respect to the attention region 82 whose position has been changed in this way, at least one such as a frame rate, a gain, a thinning rate, and an accumulation time is made different between the imaging region 81 and the attention region 82.

In step S410, the control device 4 sets the size (X_wid, Y_wid) of the attention region 82 during traveling based on the initial size (Px × Py) of the attention region 82 set in the initial setting process by the following equation (7) and It is calculated according to (8), and the process according to FIG. 17 is completed.
X_wid = Px / Z ... (7)
Y_wid = Py / Z ... (8)
However, Px is the number of pixels in the X-axis direction set in step S26, and Py is the number of pixels in the Y-axis direction set in step S26. According to the above equations (7) and (8), the size (X_wid, Y_wid) of the attention area 82 during traveling is such that the farther the preceding vehicle to be noted is (the depth Z becomes deeper), the initial the attention area 82 It is smaller than the size (Px × Py). According to the process of step S410, the size of the region of interest 82 changes according to the change in the vehicle speed V. With respect to the attention region 82 whose size has been changed in this way, at least one such as a frame rate, a gain, a thinning rate, and an accumulation time is made different between the imaging region 81 and the attention region 82.

In step S420 in which the above-mentioned step S310 is negatively determined and the process proceeds, the control device 4 performs the stopped setting process and ends the process according to FIG. The setting process during stop determines the position of the attention region 82 so that, for example, the preceding vehicle 1 (m) away is included in the attention region 82. Further, the size of the attention region 82 in the X-axis direction is maximized so that the object at a position close to the side portion of the vehicle 1 is included in the attention region 82 as much as possible.

In step S430 in which the above-mentioned step S340 is negatively determined and the process proceeds, the control device 4 performs the setting process at the time of sudden braking determination and ends the process according to FIG. In the setting process at the time of sudden braking determination, for example, thinning out in the region of interest 82 is stopped, the frame rate is increased to the maximum, the accumulation time is shortened, and the gain is set high. The control device 4 may also increase the frame rate of the imaging region 81 other than the region of interest 82. Further, the control device 4 records the image acquired by the camera 3 in the recording unit 36 for a predetermined time (for example, 5 seconds to 15 seconds) after the negative determination in step S340. Give instructions.

The control device 4 after the sudden stop further moves the attention area 82 to the initial position of the attention area 82 set in the initial setting process (step S25 in FIG. 12), and sets the attention area 82 in the initial setting process (step S26 in FIG. 12). The size of the attention area 82 is changed to the initial size (Px × Py) of the attention area 82. As a result, the position and size of the attention area 82 changed by the vehicle speed V during traveling returns to the position and size suitable when the vehicle is stopped.

In step S440 in which the above-mentioned step S370 is negatively determined and the process proceeds, the control device 4 sets the position (X coordinate) of the region of interest 82 during traveling so as not to move. That is, when the rotation angle θ of the steering wheel 10 is equal to or less than a predetermined value, θ ← 0 and the value of Xdist is also 0. That is, when the operating angle of the steering wheel 10 is less than a predetermined value, the position (X coordinate) of the region of interest 82 is maintained. Therefore, it is useful for reducing the processing load at the time of a minute operation other than the turning operation.

FIG. 20A is a diagram illustrating the movement of the position of the region of interest 82 and the change in the size of the region of interest 82 when turning right at an intersection of a general road. According to the above driving assist setting process, when the vehicle 1 is waiting for a right turn behind the preceding vehicle, the position of the attention area 82A is in the initial position, and the size of the attention area 82A is the initial size (Px × Py). Is almost the same as. When the driver starts the steering operation to the right while the vehicle 1 is moving forward, the position of the attention area 82B moves diagonally upward to the right. Since the vehicle speed V is low, the size of the region of interest 82B is also substantially the same as the initial size (Px × Py).

FIG. 20B is a diagram illustrating the movement of the position of the attention region 82 and the change in the size of the attention region 82 when changing lanes while accelerating to the right overtaking lane on the highway. According to the travel assist setting process, when the vehicle 1 travels at high speed, the position of the attention region 82A is above the initial position, and the size of the attention region 82A is smaller than the initial size (Px × Py). When the driver steers to the right while the vehicle 1 is accelerating, the position of the region of interest 82B moves diagonally upward to the right. Since the vehicle speed V is high, the size of the attention area 82B is further reduced. Note that FIG. 20 is an example of the case of left-hand traffic, and can be appropriately used for a left turn of right-hand traffic and a lane change of right-hand traffic. Further, the line-of-sight detection device (for example, the steering wheel is provided with the line-of-sight detection device) (not shown) detects the driver's line of sight, and the area that the driver is not gazing at or the area that becomes a blind spot is set as the area of interest 82. You may try to do it. The line-of-sight detection includes a corneal reflex method in which infrared rays are reflected by the driver's cornea to detect the user's line-of-sight direction, a limbus tracking method that uses the difference in reflectance for light between the cornea and the sclera, and an eyeball. There is an image analysis method in which an image is captured by a camera and the line of sight is detected by image processing, and any line of sight detection method may be used.

According to the above-described embodiment, the following effects can be obtained.
(1) The vehicle 1 has a control device 4 that recognizes at least one of the specifications of the mounted vehicle 1 and the operation of the operation unit of the vehicle 1, at least the area of interest 82, and the imaging area 81. Since the image pickup unit 32 that captures the outside of the image and the control device 4 that sets the image pickup condition of the attention region 82 and the image pickup condition of the image pickup region 81 differently based on the recognition result by the control device 4 are provided. The imaging conditions of the camera 3 can be set appropriately.

(2) Since the control device 4 recognizes the mounting position (right or left) of the steering wheel (steering wheel 10) in the vehicle 1, it is possible to appropriately set the imaging conditions of the camera 3 according to the riding position of the driver. it can.

(3) Since the setting unit sets the frame rate of the attention region 82 and the frame rate of the imaging region 81 differently according to the position of the steering wheel 10, for example, the attention region 82 on the driver's seat side (right). The imaging conditions of the camera 3 can be appropriately set, such as increasing the frame rate.

(4) A control device 4 for detecting information on the vehicle speed V of the vehicle 1 is provided, and the control device 4 has an imaging condition of the region of interest 82 and an imaging condition of the imaging region 81 according to the detection result of the information on the vehicle speed V. Is set differently, so that the imaging conditions of the camera 3 can be appropriately set according to the vehicle speed V.

(5) The control device 4 changes at least one imaging condition of the imaging condition of the region of interest 82 and the imaging condition of the imaging region 81 when the information regarding the vehicle speed V increases and decreases, so that, for example, The faster the vehicle speed V, the higher the frame rate, and the imaging conditions of the camera 3 can be set appropriately.

(6) When the rotation angle θ of the steering wheel (steering wheel 10) exceeds a predetermined value, the control device 4 has at least one of the frame rate of imaging in the region of interest 82 and the frame rate of imaging in the imaging region 81. Since the frame rate is changed high, the imaging conditions of the camera 3 can be changed in the case of the turning operation.

(7) Since the control device 4 for transmitting display information to the display device 14 of the vehicle 1 based on the image pickup result by the image pickup unit 32 is provided, necessary information can be provided to the occupants of the vehicle 1.

(8) When the rotation angle θ of the steering wheel (steering wheel 10) does not reach a predetermined value, the control device 4 receives at least one frame of the imaging frame rate of the attention region 82 and the imaging frame rate of the imaging region 81. Since the rate setting is maintained, it is possible to avoid changing the imaging conditions during a minute operation other than the turning operation. As a result, for example, it is possible to prevent the frame rate of the region of interest 82 from being changed more finely than necessary, which helps reduce the processing load.

(9) The control device 4 sets the imaging conditions of the region of interest 82 and the imaging conditions of the imaging region 81 to be different from each other in terms of imaging frame rate, gain, thinning, pixel signal addition, accumulation, and bit length. , The size of the imaging region, and at least one of the positions of the imaging region are included, so that the imaging conditions of the camera 3 can be set appropriately.

(10) The control device 4 changes at least one center position between the center position of the attention region 82 and the center position of the imaging region 81 based on the detection result of the information regarding the vehicle speed V, so that the vehicle speed V changes. As a result, the imaging conditions of the camera 3 can be appropriately set, such as changing the position of the region of interest 82.

(11) Since the control device 4 changes at least one size of the area of interest 82 and the size of the imaging area 81 based on the detection result of the information regarding the vehicle speed V, attention is paid as the vehicle speed V changes. The imaging conditions of the camera 3 can be appropriately set, such as changing the size of the area 82.

(12) Since the control device 4 sets the imaging region 81 surrounding the region of interest 82, the imaging conditions of the camera 3 can be appropriately set.

(13) A steering wheel (steering wheel 10) is provided as an operation unit of the vehicle 1, and the control device 4 has at least one of the center position of the attention region 82 and the center position of the imaging region 81 based on the operation of the steering wheel. Since the two center positions are changed, the imaging conditions of the camera 3 can be appropriately set, such as changing the position of the region of interest 82 according to the change in the course of the vehicle 1. In the above-described embodiment, the camera 3 is controlled by the control of the control device 4, but a part of the control of the camera 3 may be performed by the control unit 35 of the camera 3.

The following modifications are also within the scope of the present invention, and one or more of the modifications can be combined with the above-described embodiment.
(Modification example 1)
In the traveling assist setting process, the control device 4 may be configured to change the position of the attention region 82 and the size of the attention region 82 according to the operation signal from the turn signal switch 11. As illustrated in FIG. 21, the control device 4 changes the size of the region of interest 82 or pays attention based on the initial setting value determined in step S24 and the direction of the turn signal by operating the turn signal switch 11. The imaging conditions of the area 82 are set.

For example, to explain with reference to FIG. 10, in the case of right-hand steering and left-hand traffic and the initial setting value is "4", the control device 4 includes the left end of the road in the area of interest 82 if the turn signal is in the left direction. Such control is performed. Specifically, the region of interest 82 in FIG. 10 is expanded to the left. The area of interest 82 is expanded to the left to prevent a entanglement accident when turning left. On the contrary, the control device 4 controls so that the oncoming lane is included in the region of interest 82 if the turn signal is in the right direction. Specifically, the region of interest 82 in FIG. 10 is expanded to the right.

Explaining with reference to FIG. 14, in the case of left steering wheel and right side traffic and the initial setting value is "1", the control device 4 includes the oncoming lane in the attention area 82 if the turn signal is in the left direction. Take control. Specifically, the region of interest 82 in FIG. 14 is expanded to the left. On the contrary, the control device 4 controls so that the right end of the road is included in the region of interest 82 if the turn signal is in the right direction. Specifically, the region of interest 82 in FIG. 14 is expanded to the right. The reason for expanding to the right is to prevent a entanglement accident when turning right.

Explaining with reference to FIG. 16, in the case of right-hand steering and right-hand traffic and the initial setting value is "2", the control device 4 includes the oncoming lane in the area of interest 82 if the turn signal is in the left direction. Take control. Specifically, the region of interest 82 in FIG. 16 is greatly expanded to the left. On the contrary, the control device 4 controls so that the right end of the road is included in the region of interest 82 if the turn signal is in the right direction. Specifically, the region of interest 82 in FIG. 16 is slightly expanded to the right. The reason for expanding to the right is to prevent a entanglement accident when turning right.

Explaining with reference to FIG. 15, in the case of left steering wheel and left traffic and the initial setting value is "3", the control device 4 includes the left end of the road in the area of interest 82 if the turn signal is in the left direction. Take control. Specifically, the region of interest 82 in FIG. 15 is slightly expanded to the left. The reason for expanding to the left is to prevent a entanglement accident when turning left. On the contrary, the control device 4 controls so that the oncoming lane is included in the region of interest 82 if the turn signal is in the right direction. Specifically, the region of interest 82 in FIG. 15 is greatly expanded to the right.

FIG. 22 is a flowchart illustrating processing at the time of operation of the turn signal switch 11 according to the first modification. When an operation signal is input from the turn signal switch 11 during the travel assist setting process, the control device 4 activates the process according to FIG. 22 as a subroutine. In step S510 of FIG. 22, the control device 4 determines whether or not the turn signal direction is leftward. The control device 4 positively determines step S510 and proceeds to step S520 when the turn signal direction is leftward, and negatively determines step S510 when the turn signal direction is rightward and proceeds to step S530.

In step S520, the control device 4 determines whether or not the position of the lane is on the left. The control device 4 determines affirmatively in step S520 and proceeds to step S550 in the case of left-hand traffic, and negatively determines step S520 in the case of right-hand traffic and proceeds to step S540.

In step S540, the control device 4 controls the imaging unit 32 so as to include the oncoming lane in the region of interest 82, and ends the process according to FIG. In step S550, the control device 4 controls the imaging unit 32 so as to include the left end of the road in the region of interest 82, and ends the process according to FIG.

In step S530, the control device 4 determines whether or not the position of the lane is on the left. The control device 4 positively determines step S530 and proceeds to step S560 in the case of left-hand traffic, and negatively determines step S530 and proceeds to step S570 in the case of right-hand traffic.

In step S560, the control device 4 controls the imaging unit 32 so as to include the oncoming lane in the region of interest 82, and ends the process according to FIG. In step S570, the control device 4 controls the imaging unit 32 so as to include the right end of the road in the region of interest 82, and ends the process according to FIG.

When the turn signal switch 11 is turned off after executing the process shown in FIG. 22, the control device 4 cancels the size change of the region of interest 82 shown in FIG. 22. Further, when the turn signal switch 11 is turned on, even if the vehicle speed V is 0, the frame rate of the attention region 82 is increased, the gain is increased, the thinning rate is decreased, and the accumulation time is increased as compared with the imaging region 81. May be set short. However, when at least one of the frame rate, the gain, the thinning rate, the accumulation time, and the like is made different between the imaging area 81 and the area of interest 82, only the different imaging conditions are changed.

According to the modification 1 described above, the imaging condition of the region of interest 82 and the imaging condition of the imaging region 81 are set differently according to the operation of the turn signal switch 11, so that, for example, when turning left or right at an intersection. In the above, the oncoming lane is included in the attention area 82 so that the oncoming vehicle can be reliably detected, or the road end is included in the attention area 82 so as to prevent a entanglement accident, and the attention area 82 is appropriately set. be able to. Further, the imaging conditions can be appropriately set in the imaging region 81 and the region of interest 82, such as setting the frame rate of the region of interest 82 higher than that of the imaging region 81.

(Modification 2)
In the traveling assist setting process, the control device 4 determines the position of the attention region 82 and the attention region 82 according to the change in the distance between the object such as a two-wheeled vehicle, an ordinary vehicle, a large vehicle, and a pedestrian and the vehicle 1. It may be configured to be resized.

In the second modification, for example, when the vehicle 1 approaches the preceding vehicle and the distance L (inter-vehicle distance) to the preceding vehicle becomes short, the control device 4 determines the position of the attention region 82 so that the preceding vehicle is included in the attention region 82. To do. Here, the change in the inter-vehicle distance L from the vehicle 1 to the preceding vehicle is photographed by the distance measuring calculation unit 35a of the camera 3 based on the images acquired by the camera 3 at predetermined time intervals in synchronization with the acquisition timing of the vehicle speed V. It is obtained by sequentially measuring the distance L (inter-vehicle distance) to the preceding vehicle on the screen.

In the above equation (5), the control device 4 uses the inter-vehicle distance L instead of the depth Z to calculate the position (Y coordinate) of the region of interest 82 during traveling. As a result, when the preceding vehicle is imaged on a flat straight road, the value of Yq indicating the position of the region of interest 82 in the Y-axis direction on the imaging chip 113 increases as the inter-vehicle distance L increases, and the inter-vehicle distance L increases. It decreases as it gets shorter.

Further, in the control device 4, when the inter-vehicle distance L changes and becomes shorter, the preceding vehicle is largely captured by the camera 3, so the size of the attention region 82 is set large. On the contrary, the control device 4 sets the size of the attention region 82 to be small because the preceding vehicle appears small on the camera 3 when the inter-vehicle distance L changes and becomes long. The control device 4 calculates the size of the region of interest 82 during traveling by substituting the inter-vehicle distance L instead of the depth Z in the above equations (7) and (8).

According to the process according to FIG. 17 described above, the size of the region of interest 82 is set small when the vehicle speed V becomes high. However, in the modification 2, when the vehicle-to-vehicle distance L to the preceding vehicle is short even if the vehicle speed V is high, Since the size of the attention region 82 is set large, the preceding vehicle can be appropriately included in the attention region 82. Therefore, as compared with the case where the size of the region of interest 82 is continuously set small, it becomes easier to detect a change in the traveling state of the preceding vehicle based on the image acquired by the camera 3.
Regarding the attention region 82 whose size and position have been changed in this way, at least one such as a frame rate, a gain, a thinning rate, and an accumulation time may be different between the imaging region 81 and the attention region 82.

(Modification 3)
When the control device 4 detects an object around the vehicle 1 such as a two-wheeled vehicle, an ordinary vehicle, a large vehicle, or a pedestrian, in addition to the existing attention area 82, the control device 4 sets the attention area 82 including the object. It may be newly set. In the third modification, when the detected object moves, the control device 4 newly sets the area of interest 82 including the object. For example, when the distance between the detected object and the vehicle 1 approaches within a predetermined distance, the control device 4 newly sets a region of interest 82 including the object. Then, when the distance between the detected object and the vehicle 1 is more than a predetermined distance, the control device 4 cancels the setting of the attention area 82 including the object.
Regarding the attention region 82 set in this way, at least one such as a frame rate, a gain, a thinning rate, and an accumulation time may be different between the imaging region 81 and the attention region 82.

According to the third modification, the control device 4 for detecting a moving object around the vehicle 1 based on the information from the camera 3 is provided, and the control device 4 takes an image of the region of interest 82 based on the detection result of the moving object. Since at least one imaging condition of the condition and the imaging condition of the imaging region 81 is changed, the imaging condition of the camera 3 can be appropriately set according to the presence or absence of a moving object.

Further, when it is detected that the distance between the vehicle 1 and the moving object is approaching within a predetermined distance based on the information from the camera 3, the control device 4 determines the frame rate of the imaging of the region of interest 82 and the imaging region 81. Since at least one frame rate with the frame rate of the imaging of the above is changed to a high value, it becomes easy to detect a change in the moving state of the moving object based on the image acquired by the camera 3.

(Modification example 4)
The area of interest 82 may be newly set based on the color of the image acquired by the camera 3. The control device 4 sets a region of the image including the red object as the region of interest 82. By adding a region including a red object to the region of interest 82, for example, a red light, a railroad crossing alarm, a red light of an emergency vehicle, and the like can be included in the region of interest 82.
Regarding the attention region 82 set in this way, at least one such as a frame rate, a gain, a thinning rate, and an accumulation time may be different between the imaging region 81 and the attention region 82.

(Modification 5)
The area of interest 82 may be newly set based on the sound information collected by the microphone 17 of the vehicle 1. For example, when the level of sound information on the right side of the vehicle 1 is input in excess of a predetermined value, the control device 4 expands the attention area 82 to the right side of the imaging area 81, or newly expands the attention area 82 to the right side of the imaging area 81. The area of interest 82 is set. The area of interest 82 is provided on the right side in order to collect information on the outside of the vehicle 1 on the right side.

Further, for example, when the level of sound information on the left side of the vehicle 1 is input in excess of a predetermined value, the control device 4 expands the attention area 82 to the left side of the imaging area 81, or expands the attention area 82 to the left side of the imaging area 81. The area of interest 82 is newly set. The area of interest 82 is provided on the left side in order to collect information on the left side of the vehicle 1.
Regarding the attention region 82 set in this way, at least one such as a frame rate, a gain, a thinning rate, and an accumulation time may be different between the imaging region 81 and the attention region 82.

(Modification 6)
In the above embodiment, the case of setting the attention region 82 including the preceding vehicle has been described, but the attention region 82 including the oncoming vehicle of the vehicle 1 may be set. In the sixth modification, the control device 4 recognizes the vehicle closest to the vehicle 1 as an oncoming vehicle from among the objects existing in the above-mentioned traveling region and traveling in the opposite direction (opposing the vehicle 1).

The control device 4 sets the region corresponding to the position of the oncoming vehicle as the region of interest 82 in the image acquired by the camera 3. The control device 4 sets the area of interest 82 including the license plate of the oncoming vehicle and the face of the driver driving in the driver's seat of the oncoming vehicle.

By setting the area including the license plate of the oncoming vehicle and the face of the driver of the oncoming vehicle as the attention area 82, the oncoming vehicle approaching the vehicle 1 can be appropriately included in the attention area 82.
Regarding the attention region 82 set in this way, at least one such as a frame rate, a gain, a thinning rate, and an accumulation time may be different between the imaging region 81 and the attention region 82.

(Modification 7)
In the above description, the example in which the imaging region 81 includes the region of interest 82 (surrounding the region of interest 82) has been described, but the imaging region 81 and the region of interest 82 may be set side by side. By moving the boundary line between the imaging region 81 and the region of interest 82 to the left or right, the size and position of the imaging region 81 and the region of interest 82 can be changed. Regarding the attention region 82 set in this way, at least one such as a frame rate, a gain, a thinning rate, and an accumulation time may be different between the imaging region 81 and the attention region 82.

In the above description, as the distance measurement performed by the camera 3, a method of calculating by a distance measurement calculation using an image signal from a focus detection pixel provided in the image sensor 100 is used, but two images are measured by a stereo camera. A method of measuring the distance using an image may be used. Further, a method of measuring the distance using a millimeter wave radar separately from the camera 3 may be used.

Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of the present invention are also included within the scope of the present invention.

1 ... Vehicle 2 ... Driving support device 3 ... Camera 4 ... Control device 4b ... Storage unit 5 ... First travel control unit 6 ... Second travel control unit 7 ... Throttle control device 7a ... Accelerator pedal 8 ... Brake control device 8a ... Brake pedal 9 ... Steering control device 10 ... Steering wheel 11 ... Turn signal switch 12 ... Vehicle speed sensor 14 ... Display device 15 ... GPS device 16 ... Shift lever position detection device 17 ... Mike 31 ... Imaging optical system 32 ... Imaging unit 32a ... Drive unit 33 ... Image processing unit 34 ... Work memory 35 ... Control unit 35a ... Distance measurement calculation unit 36 ... Recording unit 60 ... Focus detection pixel line 81 ... Imaging area 82, 82A, 82B ... Attention area 83 ... Pause area 100 ... Imaging Element 113 ... Imaging chip

Claims (6)

An image pickup device installed in a car
Imaging unit and
Input section where information is input from the outside and
An imaging control unit that controls the imaging unit is provided.
The imaging control unit is an imaging device that changes the imaging conditions of the imaging unit based on the information input to the input unit. In the imaging device according to claim 1,
Information related to the distance to surrounding objects is input to the input unit, and
The imaging control unit is an imaging device that changes the imaging conditions of the imaging unit based on the information input to the input unit. In the imaging device according to claim 2,
Information related to the distance to the vehicle traveling around the vehicle is input to the input unit.
The imaging control unit is an imaging device that changes the imaging conditions of the imaging unit based on the information input to the input unit. In the imaging device according to claim 2,
Information related to the distance to pedestrians around the vehicle is input to the input unit.
The imaging control unit is an imaging device that changes the imaging conditions of the imaging unit based on the information input to the input unit. In the imaging device according to any one of claims 1 to 4.
The imaging control unit is an imaging device that changes the imaging speed of the imaging unit based on the information input to the input unit. In the imaging device according to any one of claims 1 to 4.
The image pickup control unit is an image pickup device that changes the thinning rate of pixels of the image pickup unit based on the information input to the input unit.

Images