JP7380666B2 - Vehicles and driving support devices - Google Patents

Description

The present invention relates to a vehicle and a driving support device .

The driving environment of the vehicle is detected based on images captured by a camera installed in the vehicle, and based on the detected driving environment data, automatic driving control such as following the vehicle in front, warning, braking, steering support, etc. Technology for providing driving support has been developed (see Patent Document 1).

Japanese Patent Application Publication No. 2010-79424

However, in the future, it is expected that there will be a mix of cars with automatic driving control and cars that are driven manually by drivers, but there have not been many proposals regarding this point.

According to a first aspect of the present invention, the vehicle is a vehicle equipped with a driving support device, and the driving support device includes an imaging unit that images the outside of the vehicle, and an object existing outside the vehicle. a distance measurement unit that measures the distance to the vehicle; a positioning unit that measures the current position of the vehicle; and a positioning unit that provides driving support for the vehicle based on information from the imaging unit, the distance measurement unit, and the positioning unit. It has a control unit that generates data, and a reception unit that receives information transmitted from another vehicle imaged by the imaging unit, and the control unit is configured to transmit data to other vehicles based on information from the reception unit. It is determined whether or not the vehicle is running automatically, and the frame rate and thinning rate, which are the imaging conditions of the imaging unit, are changed according to the determined result.
According to a second aspect of the present invention, the vehicle is a vehicle equipped with a driving support device, and the driving support device includes an imaging unit that images the outside of the vehicle, and an object existing outside the vehicle. a distance measurement unit that measures the distance to the vehicle; a positioning unit that measures the current position of the vehicle; and a positioning unit that provides driving support for the vehicle based on information from the imaging unit, the distance measurement unit, and the positioning unit. a control unit that generates data, and the control unit determines whether or not the other vehicle is a vehicle that is running in automatic operation based on the information from the imaging unit. Depending on the result , the frame rate and thinning rate, which are the imaging conditions of the imaging unit, are changed.
According to a third aspect of the present invention, the driving support device includes: an input section into which information obtained by capturing an image of the outside of a vehicle is input; a distance measuring section that measures a distance to an object existing outside the vehicle; a positioning unit that measures the current position of the vehicle; a control unit that generates data for providing driving support for the vehicle based on information from the input unit, the ranging unit, and the positioning unit; and an imaging unit a receiving unit that receives information transmitted from another vehicle imaged by the other vehicle, and the control unit is configured to control the vehicle in which the other vehicle is running automatically based on the information from the receiving unit. The frame rate and thinning rate, which are the imaging conditions of the imaging unit, are changed according to the determined result.
According to the fourth aspect of the present invention, the driving support device includes: an input section into which information obtained by capturing an image of the outside of the vehicle is input; a distance measuring section that measures the distance to an object existing outside the vehicle; a positioning unit that measures the current position of the vehicle; and a control unit that generates data for providing driving support for the vehicle based on information from the input unit, the ranging unit, and the positioning unit. , the control unit determines whether the other vehicle is a vehicle running in automatic operation based on information from the imaging unit, and sets imaging conditions for the imaging unit according to the determined result. Change frame rate and thinning rate .

FIG. 1 is a block diagram showing the configuration of an imaging system according to an embodiment. FIG. 2 is a diagram illustrating the arrangement of traffic lights at an intersection. FIG. 2 is a diagram illustrating an example of a traffic light for an automobile. FIG. 2 is a cross-sectional view of a stacked image sensor. FIG. 3 is a diagram illustrating a pixel array and a unit area of an imaging chip. FIG. 3 is a diagram illustrating a circuit of a unit area. FIG. 2 is a block diagram showing the functional configuration of an image sensor. FIG. 3 is a diagram illustrating the position of focus detection pixels on an imaging plane. FIG. 3 is an enlarged view of a region including part of a focus detection pixel line. It is a flow chart explaining control by a control part of a car. FIG. 11(a) is a diagram explaining the positional relationship of cars, FIG. 11(b) is a diagram schematically showing the subject image of the front camera, and FIG. 11(c) is a diagram schematically showing the subject image of the rear camera. FIG. It is a flow chart explaining control by a control part of a traffic light. FIG. 2 is a diagram illustrating control of an imaging unit of a traffic light for an automobile. FIG. 2 is a diagram illustrating control of an imaging unit of a traffic light for an automobile. It is a figure explaining the control of the imaging part of the traffic light for pedestrians. FIG. 16(a) is a diagram illustrating an imaging scene by the imaging unit of a pedestrian traffic light, and FIG. 16(b) is a diagram illustrating setting of imaging conditions. It is a figure which illustrates the imaging scene by the imaging part of the traffic light for cars.

FIG. 1 is a block diagram illustrating the configuration of an imaging system 1 including an imaging device according to an embodiment. The imaging system 1 uses a car 10, another car 20, a traffic signal generation device 30, and a traffic light 40.
Note that in place of the traffic light 40 or in combination with the traffic light 40, an information providing system installed on the road or VICS (registered trademark: Vehicle Information and Communication System) may be used.

(Car 10)
The automobile 10 includes a vehicle operation section 11, a GPS device 12, a navigation system 13, an optical system 14, a photoelectric conversion section 15, a communication section 16, a storage section 17, a sensor 18, and a control section. It is equipped with 19. Note that the automobile 10 has a basic configuration as an automobile, although the explanation will be omitted.

The vehicle operation unit 11 includes various operation members related to the operation of the vehicle, such as a steering wheel, a turn signal switch, a shift lever, an accelerator, a brake, and a switch for switching between an automatic driving mode and a manual driving mode.

The GPS device 12 calculates the position (longitude, latitude, etc.) of the vehicle 10 based on signals obtained by receiving radio waves from GPS satellites. The position information calculated by the GPS device 12 is output to the navigation system 13 and the control unit 19.

The navigation system 13 detects the current location of the vehicle 10 from a GPS device 12 or the like, obtains map data corresponding to the current location from a storage medium or network, displays it on the LCD monitor, and displays the driving route to the input destination. This is a system that guides you through the process. This navigation system 13 includes an operation section that accepts operations from the user, the aforementioned liquid crystal monitor, a speaker that provides voice guidance, a reading section that reads map data, and the like.

The optical system 14 includes a plurality of lenses, and forms a subject image on the photoelectric conversion unit 15. When the optical system 14 is directed toward the front of the vehicle 10, the photoelectric conversion unit 15 acquires an image in the traveling direction of the vehicle 10. When the optical system 14 is directed toward the rear of the vehicle 10, the photoelectric conversion unit 15 acquires an image in the direction opposite to the traveling direction of the vehicle 10. The optical system 14 has an angle of view corresponding to a plurality of driving lanes (two lanes, three lanes, etc.).
Note that a stereo camera may be constructed by providing a plurality of optical systems 14.

The photoelectric conversion unit 15 is configured by stacking an imaging chip that outputs a pixel signal in response to light incident from the optical system 14, a signal processing chip that processes the pixel signal, and a memory chip that stores the pixel signal. An image sensor 100 is provided. As will be described in detail later, in the image sensor 100, imaging conditions (including cases in which no imaging is performed) can be set individually for each pixel or for each unit area made up of a plurality of pixels (for example, 16 pixels x 16 pixels).

In this embodiment, the optical system 14 and the photoelectric conversion section 15 constitute the imaging section 5 of the camera, and the imaging section 5 of the camera is configured to take images of objects (moving bodies, obstacles, etc.) around the car 10, and to take images of objects (moving bodies, obstacles, etc.) on the road. (including lines of other colors). The automobile 10 is equipped with a front imaging section 5 that obtains an image of the front of the automobile 10, and a rear imaging section 5 that obtains an image of the rear of the automobile 10.
In this description, a line drawn on the driving path, such as a white line, is referred to as a white line. In addition, both solid lines and broken lines are referred to as white lines.
Although not shown, a radar may be provided, and surrounding objects may be detected using the radar and the imaging section 5 (optical system 14, photoelectric conversion section 15).

The communication unit 16 performs wireless communication (including optical beacons, radio beacons, and visible light communication) with external devices such as other cars 20 and traffic lights 40. Any communication method may be used.

The storage unit 17 is configured with a nonvolatile semiconductor memory such as a flash memory, and stores various programs and control parameters for driving the automobile 10 (including automatic driving).

The sensor 18 includes various sensors such as one or more vehicle speed sensors and yaw rate sensors. The vehicle speed sensor detects the vehicle speed V of the automobile 10 and sends a detection signal to the control unit 19 and the like. The yaw rate sensor detects the yaw rate of the vehicle 10 and sends a detection signal to the control unit 19 and the like. The yaw rate is the rate at which the rotation angle of a vehicle or the like changes in the turning direction.

The control unit 19 controls the entire automobile 10, and includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. In this embodiment, the control unit 19 sets and controls imaging conditions for each unit area of the image sensor 100 of the photoelectric conversion unit 15. Further, when the automatic driving mode is set by the vehicle operation unit 11, the control unit 19 detects white lines on the road using the imaging unit 5 (optical system 14, photoelectric conversion unit 15), and uses the imaging unit 5 to detect white lines on the road. The vehicle 10 detects moving objects, obstacles, etc. around the vehicle 10, and performs automatic driving to the destination input to the navigation system 13 in cooperation with the navigation system 13.

In this embodiment, the automatic driving mode refers to operating the steering wheel, accelerator, brake, turn signal switch, shift lever, etc., all automatically under the control of the control unit 19. Furthermore, manual driving refers to the driver operating the steering wheel, accelerator, brakes, turn signal switch, shift lever, etc., and the transmission may be an automatic transmission or a manual transmission. Furthermore, in the automatic driving mode, in addition to fully automatic driving in which all driving is controlled by the control unit 19, even when the user is operating the vehicle operation unit 11, the control unit 19 controls the imaging unit 5, the GPS 12, and the communication unit 16. It also includes semi-automatic driving in which the vehicle 10 is stopped or decelerated based on the output of the sensor 18 or the like to avoid a collision or the like. Thereby, the user can enjoy driving the automobile 10 while ensuring safety. Semi-automatic driving also includes a case where the control unit 19 controls some operations of the steering wheel, accelerator, brake, turn signal switch, shift lever, etc. in place of the driver.

(20 other cars)
The other vehicle 20 includes a communication section 21, a vehicle operation section 22, a storage section 23, an imaging section 24, a control section 25, and the like, and the functions of each section are similar to those of the automobile 10. Although other vehicles 20 are also omitted in FIG. 1, they have the basic structure of an automobile. However, there may be other vehicles 20 that are not equipped with the communication section 21. Further, the other cars 20 include a mixture of automatically driven cars and manually driven cars. Among these, vehicles equipped with at least an automatic driving mode can communicate with each other through the communication unit 21, and can send and receive information regarding whether they are driving automatically or manually, and image data acquired by the imaging unit 24. It is configured.

(Traffic signal generation device 30)
The traffic signal generation device 30 is a device that controls the signal lights displayed on the display section 42 of the traffic light 40, and includes a signal information generation section 31, a storage section 32, a communication section 33, and a control section 34. There is. The traffic signal generation device 30 may be installed at each of a plurality of traffic lights 40 provided at an intersection or the like, but a single traffic signal generation device 30 may control the plurality of traffic lights 40.

The signal information generation unit 31 generates traffic signals based on the types and installation positions of a plurality of traffic lights 40 installed at intersections, etc., and instructions from a traffic control center (not shown) regarding traffic.

The storage unit 32 is configured by, for example, a nonvolatile semiconductor memory such as a flash memory, and stores various programs for the traffic signal generation device 30, control parameters, and the like.

The communication unit 33 transmits the traffic signal generated by the signal information generation unit 31 to each of one or more traffic lights 40 by wire or wirelessly. Furthermore, the communication unit 33 transmits and receives information to and from the traffic control center.

The control unit 34 controls the entire traffic signal generation device 30, and includes a CPU, RAM, ROM, and the like. Further, the control unit 34 can perform an analysis of traffic conditions based on traffic volume and the like, and can control the signal information generation unit 31 based on the analysis.

(Traffic light 40)
The traffic light 40 includes a communication section 41, a display section 42, an optical system 43, a photoelectric conversion section 44, a storage section 45, and a control section 46. Although only one traffic light 40 is shown in FIG. 1, a plurality of traffic lights 40 are normally provided. For example, in the case of an intersection, as illustrated in FIG. 2, there are four traffic lights 40a for cars and eight traffic lights 40b for pedestrians. The traffic lights 40 each receive a traffic signal from the traffic signal generation device 30 according to the installed position, and turn on or blink the indicator light of the display unit 42.

The communication unit 41 receives the traffic signal generated by the signal information generation unit 31 by wire or wirelessly. Further, the communication unit 41 transmits and receives various information such as vehicle driving information and traffic information to and from the automobile 10, other cars 20, and other traffic lights 40.

The display section 42 has a signal light, and displays the signal light according to the traffic signal received by the communication section 41. Specifically, the signal lights are turned on, flashed, or turned off in order to permit or restrict the movement of vehicles traveling on the road and pedestrians crossing the road, such as proceeding or stopping. Note that the display unit 42 may turn on not only red lights, yellow lights, and blue lights, but also arrow lights indicating whether it is possible to go straight, turn left, or turn right at an intersection.

In this embodiment, the optical system 43 and the photoelectric conversion section 44 constitute an imaging section 50 of the camera. The optical system 43 includes a plurality of lenses, and forms a subject image on the photoelectric conversion unit 44 . When the imaging unit 50 is provided in a traffic light 40a for an automobile, the optical system 43 is mainly used to obtain images of the vehicle. When the imaging unit 50 is provided in the traffic light 40b for pedestrians, the optical system 43 is mainly used to obtain images of pedestrians (including bicycles). The imaging unit 50 is provided near the display unit 42 (signal light).

The photoelectric conversion unit 44 is configured by stacking an imaging chip that outputs a pixel signal in response to light incident from the optical system 43, a signal processing chip that processes the pixel signal, and a memory chip that stores the pixel signal. An image sensor 100 is provided. It has the same configuration as the photoelectric conversion unit 15 in the vehicle 10 described above. The imaging device 100 can set imaging conditions according to traffic signals for each pixel or for each unit area made up of a plurality of pixels (for example, 16 pixels x 16 pixels).

FIG. 3 is a diagram illustrating a traffic light 40a for automobiles placed at an intersection. In FIG. 3, an imaging section 50 is installed near the bottom of a display section 42a of a traffic light 40a. The imaging unit 50 has, for example, a wide-angle lens as the optical system 43, and has an angle of view that includes four lanes on both sides of a plurality of travel lanes (two lanes on each side, etc.) at the intersection illustrated in FIG.

The storage unit 45 is configured by, for example, a nonvolatile semiconductor memory such as a flash memory, and stores image data etc. acquired by the photoelectric conversion unit 44.

The control unit 46 controls the entire traffic light 40, and includes a CPU, RAM, ROM, and the like. In the present embodiment, the control unit 46 controls the display of the signal light on the display unit 42 in accordance with traffic signals, and also controls imaging using the image sensor 100 of the photoelectric conversion unit 44.

Note that if the imaging range by the imaging unit 50 of the automobile traffic light 40a includes the imaging range by the imaging unit 50 of the pedestrian traffic light 40b, the imaging unit 50 (optical system 43 and photoelectric conversion section 44) may be omitted.

<Description of stacked image sensor>
The stacked image sensor 100 provided in the imaging section of the above-described automobile 10, other vehicle 20, and traffic light 40 will be described. Note that this stacked image sensor 100 is described in WO 13/164915, which was previously filed and published by the applicant of the present application. FIG. 4 is a cross-sectional view of the stacked image sensor 100. The image sensor 100 includes a back-illuminated image sensor chip 113 that outputs a pixel signal corresponding to incident light, a signal processing chip 111 that processes the pixel signal, and a memory chip 112 that stores the pixel signal. These imaging chip 113, signal processing chip 111, and memory chip 112 are stacked and electrically connected to each other by conductive bumps 109 made of Cu or the like.

Note that, as shown in the figure, the incident light mainly enters in the positive Z-axis direction indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light enters is referred to as the back surface (imaging surface). Further, as shown in the coordinate axes, the left direction of the paper plane perpendicular to the Z axis is the X-axis plus direction, and the front direction of the paper plane orthogonal to the Z axis and the X axis is the Y-axis plus direction. In the following several figures, the coordinate axes are displayed based on the coordinate axes in FIG. 4 so that the orientation of each figure can be understood.

An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back side of the wiring layer 108. The PD layer 106 is two-dimensionally arranged and includes a plurality of PDs (photodiodes) 104 that accumulate charges according to incident light, and transistors 105 provided corresponding to the PDs 104.

A color filter 102 is provided on the incident light side of the PD layer 106 with a passivation film 103 interposed therebetween. The color filter 102 has a plurality of types that transmit different wavelength regions, and has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A set of color filter 102, PD 104, and transistor 105 forms one pixel.

On the incident light side of the color filter 102, a microlens 101 is provided corresponding to each pixel. The microlens 101 condenses incident light toward the corresponding PD 104.

The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multilayered and may include passive elements and active elements.

A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with the plurality of bumps 109 provided on opposing surfaces of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are aligned by applying pressure or the like. The bumps 109 are joined together and electrically connected.

Similarly, a plurality of bumps 109 are arranged on mutually opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressurized or the like, so that the aligned bumps 109 are joined and electrically connected.

Note that the bonding between the bumps 109 is not limited to Cu bump bonding by solid phase diffusion, but may also employ microbump bonding by solder melting. Further, for example, about one bump 109 may be provided for one block, which will be described later. Therefore, the size of the bumps 109 may be larger than the pitch of the PDs 104. Further, a bump larger than the bump 109 corresponding to the pixel region may also be provided in a peripheral region other than the pixel region where pixels are arranged.

The signal processing chip 111 has TSVs (through-silicon vias) 110 that connect circuits provided on the front and back surfaces to each other. Preferably, the TSV 110 is provided in the peripheral area. Further, the TSV 110 may also be provided in the peripheral area of the imaging chip 113 and the memory chip 112.

FIG. 5 is a diagram illustrating the pixel array and unit area 131 of the imaging chip 113. In particular, the state in which the imaging chip 113 is observed from the back surface (imaging surface) side is shown. For example, more than 20 million pixels are arranged in a matrix in the pixel area. In the example of FIG. 5, 16 adjacent pixels (4 pixels×4 pixels) form one unit area 131. The grid lines in the figure represent the concept that adjacent pixels are grouped to form a unit area 131. The number of pixels forming the unit area 131 is not limited to this, and may be about 1000, for example, 32 pixels x 64 pixels, or may be larger or smaller.

As shown in the partial enlarged view of the pixel region, the unit region 131 in FIG. 5 includes four so-called Bayer arrays each consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R, in the upper, lower, left, and right directions. The green pixels Gb and Gr are pixels having a green filter as the color filter 102, and receive light in the green wavelength band of the incident light. Similarly, the blue pixel B is a pixel that has a blue filter as the color filter 102 and receives light in the blue wavelength band, and the red pixel R is a pixel that has a red filter as the color filter 102 and receives light in the red wavelength band. Receive light.

In this embodiment, a plurality of blocks are defined so that each block includes at least one unit area 131, and pixels included in each block can be controlled using different control parameters. In other words, it is possible to obtain imaging signals with different imaging conditions between a pixel group included in one block and a pixel group included in another block. Examples of control parameters include frame rate, gain, thinning rate, number of addition rows or columns for adding pixel signals, charge accumulation time or number of accumulations, number of digitization bits (word length), and the like. The imaging device 100 can freely perform thinning not only in the row direction (X-axis direction of the imaging chip 113) but also in the column direction (Y-axis direction of the imaging chip 113). Furthermore, the control parameter may be a parameter in image processing after acquiring an image signal from a pixel.

FIG. 6 is a diagram illustrating a circuit in the unit area 131. In the example of FIG. 6, one unit area 131 is formed by nine adjacent pixels (3 pixels x 3 pixels). Note that, as described above, the number of pixels included in the unit area 131 is not limited to this, and may be less than or greater than this. Two-dimensional positions of the unit area 131 are indicated by symbols A to I.

The reset transistors of the pixels included in the unit area 131 are configured to be turned on and off individually for each pixel. In FIG. 6, a reset wiring 300 that turns on and off the reset transistor of pixel A is provided, and a reset wiring 310 that turns on and off the reset transistor of pixel B is provided separately from the reset wiring 300. Similarly, a reset wiring 320 that turns on and off the reset transistor of pixel C is provided separately from the reset wirings 300 and 310 described above. Dedicated reset wiring is also provided for the other pixels D to Pixel I to turn on and off the respective reset transistors.

The transfer transistors of the pixels included in the unit area 131 are also configured to be turned on and off individually for each pixel. In FIG. 6, a transfer wiring 302 that turns on and off the transfer transistor of pixel A, a transfer wiring 312 that turns on and off the transfer transistor of pixel B, and a transfer wiring 322 that turns on and off the transfer transistor of pixel C are provided separately. Dedicated transfer wiring for turning on and off the respective transfer transistors is also provided from the other pixels D to Pixel I.

Further, the selection transistors of the pixels included in the unit area 131 are also configured to be turned on and off individually for each pixel. In FIG. 6, a selection wiring 306 that turns on and off the selection transistor of pixel A, a selection wiring 316 that turns on and off the selection transistor of pixel B, and a selection wiring 326 that turns on and off the selection transistor of pixel C are provided separately. Dedicated selection wiring for turning on and off the respective selection transistors is also provided for the other pixels D to Pixel I.

Note that the power supply wiring 304 is commonly connected to pixels A to I included in the unit area 131. Similarly, the output wiring 308 is commonly connected to pixels A to I included in the unit area 131. Further, the power supply wiring 304 is commonly connected between a plurality of unit areas, but the output wiring 308 is provided individually for each unit area 131. Load current source 309 supplies current to output wiring 308 . The load current source 309 may be provided on the imaging chip 113 side or may be provided on the signal processing chip 111 side.

By individually turning on and off the reset transistor and transfer transistor of the unit area 131, charge including charge accumulation start time, accumulation end time, and transfer timing is independently generated from pixel A to pixel I included in the unit area 131. Accumulation can be controlled. Furthermore, by individually turning on and off the selection transistors in the unit area 131, the pixel signals from each pixel A to the pixel I can be outputted via the common output wiring 308.

Here, a so-called rolling shutter method is known in which charge accumulation is controlled in a regular order for rows and columns for pixels A to I included in the unit area 131. When pixels are selected for each row using the rolling shutter method and then a column is specified, pixel signals are output in the order of "ABCDEFGHI" in the example of FIG. 6.

By configuring the circuit with the unit area 131 as a reference in this way, the charge accumulation time can be controlled for each unit area 131. In other words, pixel signals with different frame rates can be output between the unit areas 131, respectively. In addition, by resting the unit area 131 included in another area while charge accumulation (imaging) is performed in the unit area 131 included in some areas in the imaging chip 113, it is possible to It is possible to perform imaging only and output the pixel signal. Furthermore, it is also possible to switch the area (target area for accumulation control) in which charge accumulation (imaging) is performed between frames, to sequentially perform imaging in different areas of the imaging chip 113, and to output pixel signals.

FIG. 7 is a block diagram showing the functional configuration of the image sensor 100 corresponding to the circuit illustrated in FIG. 6. The analog multiplexer 411 sequentially selects the nine PDs 104 forming the unit area 131 and outputs each pixel signal to the output wiring 308 provided corresponding to the unit area 131. The multiplexer 411 is formed on the imaging chip 113 together with the PD 104.

The pixel signal output through the multiplexer 411 is processed by a signal processing circuit 412 formed in the signal processing chip 111 and performing correlated double sampling (CDS) and analog/digital (A/D) conversion. A D conversion is performed. The A/D converted pixel signals are delivered to the demultiplexer 413 and stored in the pixel memory 414 corresponding to each pixel. Demultiplexer 413 and pixel memory 414 are formed on memory chip 112.

The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and delivers them to the subsequent image processing section. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112. Although FIG. 7 shows connections for one unit area 131, these connections actually exist for each unit area 131 and operate in parallel. However, the arithmetic circuit 415 does not need to exist for each unit area 131; for example, one arithmetic circuit 415 may process the values sequentially while sequentially referring to the values in the pixel memory 414 corresponding to each unit area 131. good.

As described above, the output wiring 308 is provided corresponding to each unit area 131. Since the image sensor 100 has an image sensor chip 113, a signal processing chip 111, and a memory chip 112 stacked, each chip can be connected in the plane direction by using electrical connections between the chips using bumps 109 for these output wirings 308. Wiring can be routed without increasing the size.

<Explanation of distance measurement>
FIG. 8 is a diagram illustrating the positions of focus detection pixels on the imaging surface of the image sensor 100. In this embodiment, focus detection pixels are arranged discretely along the X-axis direction (horizontal direction) of the imaging chip 113. In the example of FIG. 8, 15 focus detection pixel lines 60 are provided at predetermined intervals. The focus detection pixels forming the focus detection pixel line 60 output an image signal for distance measurement. In the imaging chip 113, normal imaging pixels are provided at pixel positions other than the focus detection pixel line 60. The imaging pixels output image signals for monitoring moving objects, obstacles, and the like.

FIG. 9 is an enlarged view of a region including a part of one of the focus detection pixel lines 60. In FIG. 9, a red pixel R, a green pixel G (Gb, Gr), a blue pixel B, a focus detection pixel P1, and a focus detection pixel P2 are illustrated. The red pixel R, the green pixel G (Gb, Gr), and the blue pixel B are arranged according to the Bayer arrangement rules described above.

The square-shaped areas illustrated for the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B indicate the light-receiving area of the imaging pixel. Each imaging pixel receives a light beam passing through the exit pupil of the imaging optical system. That is, the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B each have a square mask opening, and the light that passes through these mask openings reaches the light receiving part of the imaging pixel. .

Note that the shapes of the light-receiving regions (mask openings) of the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B are not limited to squares, and may be circular, for example.

The semicircular area illustrated for the focus detection pixel P1 and the focus detection pixel P2 indicates the light receiving area of the focus detection pixel. That is, the focus detection pixel P1 has a semicircular mask opening on the left side of the pixel position in FIG. 9, and light passing through this mask opening reaches the light receiving section of the focus detection pixel P1. On the other hand, the focus detection pixel P2 has a semicircular mask opening on the right side of the pixel position in FIG. 9, and light passing through this mask opening reaches the light receiving section of the focus detection pixel P2. In this way, the focus detection pixel P1 and the focus detection pixel P2 each receive a pair of light beams passing through different regions of the exit pupil of the imaging optical system.

Note that the position of the focus detection pixel line in the imaging chip 113 is not limited to the position illustrated in FIG. 8 . Furthermore, the number of focus detection pixel lines is not limited to the example shown in FIG. 8. Further, the shape of the mask opening in the focus detection pixel P1 and the focus detection pixel P2 is not limited to a semicircular shape. For example, the shape of the rectangular light receiving area (mask opening It is also possible to have a rectangular shape in which the part) is divided in the horizontal direction.

Further, the focus detection pixel line in the imaging chip 113 may be one in which focus detection pixels are arranged side by side along the Y-axis direction (vertical direction) of the imaging chip 113. An image sensor in which imaging pixels and focus detection pixels are arranged two-dimensionally as shown in FIG. 9 is well known, and detailed illustrations and explanations of these pixels will be omitted.

In the example of FIG. 9, a so-called 1PD structure, in which the focus detection pixels P1 and P2 each receive one of a pair of focus detection light beams, has been described. Instead of this, for example, as disclosed in Japanese Patent Laid-Open No. 2007-282107, a so-called 2PD structure may be used in which each focus detection pixel receives both of a pair of focus detection light beams. By adopting the 2PD structure in this way, it becomes possible to read image data also from the focus detection pixels, and the focus detection pixels do not become defective pixels.

In this embodiment, the amount of image shift ( By detecting the phase difference), the focus adjustment state (defocus amount) of the imaging optical system is calculated.

Generally, the above pair of images approaches each other in a so-called front-focus state in which the imaging optical system forms a sharp image of an object (for example, a leading vehicle) in front of the intended focal plane; In the so-called back-focus state where a sharp image is formed, they move away from each other. In a focused state in which a sharp image of the object is formed at the predetermined focal plane, the pair of images are relatively coincident. Therefore, the amount of relative positional deviation between the pair of images corresponds to the distance to the object (depth information).

The defocus amount calculation based on the phase difference is well known in the field of cameras, so a detailed explanation will be omitted. Here, since there is a one-to-one correspondence between the defocus amount and the distance to the target object, the distance from the camera to each target object can be determined by determining the defocus amount for each target object. That is, distance measurement (distance measurement) to the target object can be performed at a plurality of positions on the photographic screen. The relationship between the defocus amount and the distance to the object is prepared in advance as a mathematical formula or a lookup table, and stored in a nonvolatile memory or the like.

<Automobile control>
The control executed by the control unit 19 of the automobile 10 will be described below with reference to the flowchart of FIG. 10. It is assumed that this flowchart is started when the automobile 10 is started, for example, when the engine is started or the driving system is started. A program for executing the process according to the flowchart of FIG. 10 is stored in a storage medium such as a ROM in the control unit 19 or in the storage unit 17 of the automobile 10.

In step S1, the control unit 19 causes the image sensor 100 of the photoelectric conversion unit 15 to start capturing an image. As described above, the imaging section 5 (optical system 14, photoelectric conversion section 15) of the automobile 10 acquires images of the front and rear of the automobile 10, respectively.

In step S2, the control unit 19 communicates with other vehicles 20 (vehicles traveling in the same lane as the automobile 10, as well as vehicles traveling in the same traveling direction as the automobile 10 but in a different lane) via the communication unit 16. I do. In this embodiment, as shown in FIG. 11(a), the vehicle 73A before traveling in the same lane as the automobile 10 is a self-driving vehicle, and the vehicle 73A before traveling in the adjacent traveling lane (same traveling direction) is a self-driving vehicle. It is assumed that the vehicle 72A is a manually operated vehicle. Further, it is assumed that a vehicle 73B behind the vehicle 73B traveling in the same lane as the automobile 10 is a self-driving vehicle, and a vehicle 72B behind the vehicle 72B traveling in the adjacent lane (same traveling direction) is a manually driven vehicle.

Whether the vehicles surrounding the automobile 10 are automatically driven vehicles or manually driven vehicles is determined based on the result of communication by the communication unit 16, such as known vehicle-to-vehicle communication. Further, when an identification mark or the like is displayed on another vehicle 20, the discrimination may be made based on the imaging result by the imaging section 5 (optical system 14, photoelectric conversion section 15). The identification mark may be a predetermined mark or code displayed on the body of the vehicle, or may be one that displays identification information on a display section (not shown) provided on the roof of the vehicle or the like. Note that the control unit 19 assumes that other cars 20 that cannot be determined to be self-driving cars or manually-driving cars based on communication results (unable to communicate) or imaging results are manually-driving cars.

In step S3, the control unit 19 individually sets imaging conditions for each unit area 131 (FIG. 5) of the image sensor 100 of the photoelectric conversion unit 15. FIGS. 11(b) and 11(c) are diagrams schematically showing images of a subject (object) formed on the image sensor 100 that images the front and rear of the automobile 10, respectively. In reality, an inverted inverted image is formed, but for the sake of clarity, it is shown as an erect normal image. The white line 80a represents a dividing line on the left side of the road in the direction of travel, the white line 80b represents a boundary line between driving lanes, and the white line 80c represents a dividing line on the right side of the road.

As described above, since the vehicles 72A and 72B before and after the vehicle 10 traveling in the lane next to the vehicle 10 are manually driven vehicles, the control unit 19 sets the region including the vehicle 72A in FIG. 11(b) as the region of interest 71A. . Then, the frame rate of the unit area of the image sensor 100 corresponding to the attention area 71A is set higher than the frame rate of the normal area (for example, 60 fps), and the thinning rate is set to 0 to 20%, which is lower than that of the normal area.

Similarly, the control unit 19 sets the area including the vehicle 72B in FIG. 11(c) as the attention area 71B. Then, the frame rate of the unit area of the image sensor 100 corresponding to the attention area 71B is set higher than the frame rate of the normal area (for example, 60 fps), and the thinning rate is set to 0 to 20%, which is lower than that of the normal area.
The control unit 19 further changes the setting of this thinning rate depending on the moving speed of the automobile 10 or the relative moving speed between the automobile 10 and another vehicle 20. For example, the thinning rate is changed to lower as the relative moving speed becomes faster.

Note that FIG. 11(b) is an example in which all areas except the attention area 71A are normal areas, but the area surrounding the self-driving vehicle 73A is the quasi-attention area 74A, and the attention area 71A and the quasi-attention area 74A are The area excluding the above may be set as a normal area. Further, although FIG. 11(c) is an example in which all regions except the attention region 71B are normal regions, the region surrounding the vehicle 73B, which is an automatic driving vehicle, is the quasi-attention region 74B, and the attention region 71B and the quasi-attention region 74B The area excluding the above may be set as a normal area. Note that the control unit 19 may set the imaging conditions for the semi-attention regions 74A and 74B to be different between fully automatic operation and semi-automatic operation. In this case, the control unit 19 may set the frame rate of the image sensor 100 in the case of semi-automatic operation to be higher than the frame rate of the image sensor 100 in the case of fully automatic operation. Further, the control unit 19 may set the imaging area as a normal area in the case of fully automatic driving.

The control unit 19 sets the frame rate of the unit area of the image sensor 100 that corresponds to the semi-attention areas 74A and 74B to be lower than the frame rate of the normal area for the image sensor 100 that images the front and rear of the vehicle 10, respectively. (for example, 30 fps), and the thinning rate is set to about 30 to 60%.

Further, in addition to the above-mentioned regions of interest 71A and 71B, the control unit 19 may set an area of interest that includes a white line on the road. Then, the frame rate of the unit area of the image sensor 100 corresponding to the region of interest is set higher than the frame rate of the normal area (for example, 60 fps) for the image sensor 100 that images the front and rear of the vehicle 10, respectively, and the thinning rate 0 to 20%.
In this way, by differentiating the imaging conditions for each unit area of the image sensor 100 when capturing images of a manually driven vehicle and an autonomous vehicle, the image sensor 100 can be used efficiently, reducing power consumption and heat generation. can be suppressed.

In step S4 of FIG. 10, the control unit 19 determines whether communication with the traffic light 40 is possible via the communication unit 16, that is, whether or not the vehicle is approaching the traffic light 40 (intersection). If communication with the traffic light 40 is not possible (outside the communicable area), the control unit 19 makes a negative determination in step S4 and proceeds to step S7. On the other hand, if communication with the traffic light 40 is possible (within the communicable area), the control unit 19 makes an affirmative determination in step S4 and proceeds to step S5.

In step S5, the control unit 19 receives information based on the image acquired by the photoelectric conversion unit 44 of the traffic light 40 (vehicle traffic light 40a or pedestrian traffic light 40b). For example, the control unit 19 receives information regarding a person from the pedestrian traffic light 40b when the automobile 10 turns left (when vehicles turn right in areas where vehicles drive on the right, such as in the United States). In this case, the control unit 46 of the traffic light 40 determines the presence or absence of a pedestrian based on the image acquired by the photoelectric conversion unit 44, and the control unit 19 receives information regarding the pedestrian determined by the control unit 46.
Note that the control unit 19 of the automobile 10 may receive the image data acquired by the photoelectric conversion unit 44 of the traffic light 40, and determine the presence or absence of a pedestrian based on the received image data.

When the automobile 10 makes a right turn (when the vehicle turns left in an area where vehicles drive on the right side), the control unit 19 receives information from the automobile traffic light 40a, such as whether a straight vehicle in the oncoming lane is a self-driving vehicle or a manual vehicle. do. Further, the control unit 19 receives information regarding traffic signal switching (for example, signal switching information such as a change from a green light to a red light after several seconds) from the automobile traffic light 40a.

In step S6, the control unit 19 sets imaging conditions for the image sensor 100 of the photoelectric conversion unit 15 based on the information obtained in step S5. For example, when the car 10 turns left at an intersection, the control unit 19 may set the frame rate of the unit area of the image sensor 100 corresponding to the left side of the shooting screen to be higher than the frame rate of the unit area corresponding to the right side of the shooting screen. , the frame rate is changed in accordance with the speed of the automobile 10 and the moving speed of the pedestrian (for example, 4 km/h).

For example, when the car 10, which was traveling at 50 km/h, decelerates to about 10 km/h to make a left turn, the frame rate of the unit area corresponding to the left side of the photographic screen is lowered compared to before the deceleration. Further, when the car 10 turns left, the control unit 19 sets the imaging conditions of the image sensor 100 depending on whether a pedestrian crossing the sidewalk approaches the car 10 or moves away from the car 10. do. That is, the control unit 19 increases the frame rate of the unit area of the image sensor 100 that corresponds to a person approaching the car 10 and lowers the thinning rate to detect a pedestrian moving away from the car 10 (particularly when the car 10 is scheduled to pass). The frame rate of the unit area of the image sensor 100 corresponding to the person who has already crossed the crosswalk is lowered, and the thinning rate is increased.

Further, when the car 10 turns right at an intersection, the control unit 19 controls the frame rate of the unit area corresponding to the right side of the shooting screen if the other car 20 coming straight from the oncoming lane is a manually driven car. is set relatively higher than the frame rate of the unit area corresponding to the left side of the shooting screen, and the thinning rate is lowered. Furthermore, if a pedestrian crossing the crosswalk at the right turn destination is approaching the car 10, the frame rate of the unit area of the image sensor 100 corresponding to the approaching pedestrian is further increased to further increase the thinning rate. make low.

When the automobile 10 makes a right turn or a left turn, the control unit 19 predicts the imaging area to be focused on (in which pedestrians, etc. may be captured) according to the operation state of the turn signal switch and the amount of operation of the steering wheel. It is also possible to change the settings of the imaging conditions by predicting the imaging area.

Further, the control unit 19 receives information regarding the switching of traffic signals from the automobile traffic light 40a, and changes the imaging conditions of the image sensor 100 based on the information. For example, the control unit 19 receives information regarding the switching of a traffic signal from a green light to a red light after a few seconds from the automobile traffic light 40a, and when the automobile 10 decelerates, the control unit 19 receives a frame of the image sensor 100 that images the front. The rate is controlled to be lower than before deceleration or the thinning rate is increased. On the other hand, the control unit 19 controls the imaging condition of the imaging device 100 for imaging the rear to be maintained as it is.
Note that when the car 10 decelerates, it is expected that a following car will approach the car 10, so control is performed to increase the frame rate of the image sensor 100 that images the rear than before the deceleration or to lower the thinning rate. You may do so.
When the vehicle 10 changes its speed, the control unit 19 may predict the speed change according to the amount of operation of the brake or accelerator (the amount of pedal depression) and change the settings of the imaging conditions.

In step S7, the control unit 19 determines whether the engine (or the driving system) is on. If the engine (or the driving system) is on, the control unit 19 makes an affirmative determination in step S7 and repeats the processing from step S2 onwards. If the engine (or the driving system) is off, the control unit 19 makes a negative determination in step S7 and ends the process according to this flowchart.

<Traffic light control>
Next, the control executed by the control unit 46 of the traffic light 40 will be described with reference to the flowchart of FIG. 12. A program for executing the process according to the flowchart in FIG. 12 is stored in a storage medium such as a ROM in the control unit 46 or in the storage unit 45.

The control unit 46 determines whether a traffic signal has been received from the traffic signal generation device 30 in step S10. When the control unit 46 receives the traffic signal from the traffic signal generation device 30, it makes an affirmative determination in step S10 and proceeds to step S11. If the control unit 46 has not received the traffic signal from the traffic signal generation device 30, it makes a negative determination in step S10 and waits for reception.

The control unit 46 controls the display of the display unit 42 in step S11. For example, the control unit 46 performs control to switch the signal light display on the display unit 42 from red to blue in accordance with the traffic signal received from the traffic signal generation device 30.

The control unit 46 communicates with one or more vehicles or other traffic lights 40 in step S12. Vehicles with which the control unit 46 communicates can include the automobile 10 and another vehicle 20 that includes the communication unit 21 .
Note that the communication target vehicle may be a vehicle within a predetermined range from the intersection or the traffic light 40, or may be a vehicle that can communicate via an information providing system (not shown) installed on the road.

The control unit 46 acquires information from the vehicle to be communicated about the mode of movement, such as whether the vehicle or vehicles around the vehicle are self-driving cars or manually driving cars. Furthermore, the control unit 46 acquires information regarding the driving status of the vehicle or vehicles around the vehicle from the vehicle to be communicated with. For example, the control unit 19 or 25 of the communication target vehicle predicts a course change (right turn or left turn) at an intersection based on the state of a turn signal switch for operating a turn signal. The control unit 46 acquires right turn prediction information or left turn prediction information predicted by the communication target vehicle from the communication target vehicle as information regarding the driving situation.

Note that the control unit 46 may determine whether the vehicle is an automatically driven vehicle or a manually driven vehicle based on the result of communication with the communication target vehicle. Further, in the case where an identification mark or the like is displayed on the vehicle, the control unit 46 may make the determination based on the imaging result by the imaging unit (optical system 43, photoelectric conversion unit 44). Furthermore, the control unit 46 may determine whether the vehicle changes course (right turn or left turn) based on the operating state of the turn signal of the vehicle based on the imaging result by the imaging unit (optical system 43, photoelectric conversion unit 44). The determination may be made based on whether the vehicle is on a left-turn lane or a right-turn lane.

The control unit 46 also communicates with other traffic lights 40, including the traffic lights 40a for cars and the traffic lights 40b for pedestrians, and can acquire information on traffic conditions at intersections and the like. The control unit 46 further communicates with the traffic signal generation device 30 related to traffic signal generation of each traffic signal, as necessary, and acquires information about the display status of the traffic signal.

In step S13, the control unit 46 sets imaging conditions for the image sensor 100 of the photoelectric conversion unit 44 based on the display status of the signal light on the display unit 42 and the information acquired in step S12. Details of setting the imaging conditions in the image sensor 100 of the photoelectric conversion unit 44 will be described later.

In step S14, the control unit 46 causes the imaging unit (optical system 43, photoelectric conversion unit 44) to perform imaging under the imaging conditions set in step S13.

In step S15, the control unit 46 transmits the image data acquired in step S14 to the automobile 10, another car 20, another traffic light 40, etc. via the communication unit 41. In addition, the control unit 46 performs image processing or image analysis based on the image data and extracts information, such as the direction and speed of the vehicle, and identifies the operating state of the turn signal. Information on the driving lane of the vehicle is also transmitted in the same way.

Furthermore, the control unit 46 recognizes and recognizes the estimated driving lane of the communication target vehicle (car 10 and other cars 20) and objects (including vehicles and pedestrians) that may become obstacles based on the analyzed information. A message may be generated based on the result, and the message may be transmitted from the communication unit 41 to the communication target vehicle. The messages include, for example, "A motorcycle is coming from behind." "A pedestrian is crossing the street." "An oncoming vehicle is going straight." The control unit 46 repeatedly executes the processes from step S10 to step S15.

<Setting imaging conditions at traffic lights>
FIG. 13 shows control of imaging conditions in the image sensor 100 of the imaging unit 50-1 installed integrally with or near the automotive traffic light 40a when the automotive traffic light 40a is displaying a green light. FIG.

In FIG. 13, an automobile traffic light 40a, an imaging unit 50-1, and a traffic signal generation device 30-1 for a driving lane A are shown. Other traffic lights and the like at the intersection are not shown. The control unit 46 of the automobile traffic light 40a sets the range of the driving lane in which the vehicle is moving due to the green light, within the range of the imaging area 70 of the imaging unit 50-1, as the region of interest 71 (the shaded area ).

The control unit 46 controls the unit area of the image sensor 100 corresponding to the area of interest 71 by increasing the frame rate or reducing the thinning rate compared to unit areas other than the area of interest 71. Further, when a vehicle 72 that is a manually operated vehicle that is not automatically driven is present in the region of interest 71 in FIG. 13, the control unit 46 sets an area surrounding the vehicle 72 as a region of special interest 75. Then, the frame rate of the unit area of the image sensor 100 corresponding to the special attention area 75 is made higher than that of the unit area corresponding to the attention area 71. The control unit 46 also controls the thinning rate of the unit area of the image sensor 100 corresponding to the special attention area 75 to be lower than the thinning rate of the unit area corresponding to the attention area 71.

In addition, the control unit 46 also sets the thinning rate lower than the unit area corresponding to the attention area 71 for the area including the vehicle 76 and the vehicle 77 that are temporarily stopped for turning right or left in the attention area 71. By doing so, it is possible to capture images with high resolution.

FIG. 14 shows the imaging conditions for the image sensor 100 of the imaging unit 50-1 installed integrally with or near the automotive traffic light 40a when the automotive traffic light 40a is displaying a red light. FIG. 3 is a diagram showing an example of control.

In FIG. 14, an automobile traffic light 40a, an imaging unit 50-1, and a traffic signal generation device 30-1 for a driving lane A are shown. Other traffic lights and the like at the intersection are not shown. The control unit 46 sets, within the range of the imaging area 70 of the imaging unit 50-1, a crosswalk into which a pedestrian 90 can enter and its vicinity as an attention area 71 (shaded area).

The control unit 46 controls the unit area of the image sensor 100 corresponding to the area of interest 71 by increasing the frame rate or reducing the thinning rate compared to unit areas other than the area of interest 71. Furthermore, when the control unit 46 recognizes the pedestrian 90, it sets a predetermined area including the pedestrian 90 as the special attention area 75. Then, the frame rate of the unit area of the image sensor 100 corresponding to the special attention area 75 is made higher than that of the unit area corresponding to the attention area 71. The control unit 46 also controls the thinning rate of the unit area of the image sensor 100 corresponding to the special attention area 75 to be lower than the thinning rate of the unit area corresponding to the attention area 71.

FIG. 15 is a diagram showing an example of controlling the imaging conditions of the imaging unit 50-2 installed integrally with or near the pedestrian traffic light 40b. In FIG. 15, a pedestrian traffic light 40b, an imaging section 50-2, and a traffic signal generation device 30-2 are shown. Other traffic lights and the like at the intersection are not shown. The control unit 46 of the pedestrian traffic light 40b sets a crosswalk into which the pedestrian 90 can enter and its vicinity as an attention area 71 within the range of the imaging area 70 of the imaging unit 50-2 (the shaded area ).

The control unit 46 controls the unit area of the image sensor 100 corresponding to the area of interest 71 by increasing the frame rate or reducing the thinning rate compared to unit areas other than the area of interest 71. Furthermore, when the control unit 46 recognizes the pedestrian 90, it sets a predetermined area including the pedestrian 90 as the special attention area 75. Then, the frame rate of the unit area of the image sensor 100 corresponding to the special attention area 75 is made higher than that of the unit area corresponding to the attention area 71. The control unit 46 also controls the thinning rate of the unit area of the image sensor 100 corresponding to the special attention area 75 to be lower than the thinning rate of the unit area corresponding to the attention area 71.

FIG. 16(a) is a diagram illustrating a scene imaged by the imaging unit 50-2 installed at the pedestrian traffic light 40b. FIG. 16(b) is a diagram illustrating setting of imaging conditions based on a subject recognition result using image data acquired by the imaging unit 50-2. In FIG. 16(a), an imaging unit 50-2 having the above-described imaging device 100 is installed at a traffic light 40b for pedestrians. According to the image sensor 100 according to the present embodiment, it is possible to measure movement not only in the vertical and horizontal directions but also in the depth direction, so it is possible to measure the speed Vo at which the pedestrian 90, who is the subject, moves. .

In FIG. 16(b), the control unit 46 sets an area including the pedestrian 90 crossing the crosswalk as the attention area 71 within the imaging area 70 of the imaging unit 50-2. Then, the control unit 46 makes the imaging conditions different for the unit area of the image sensor 100 corresponding to the area of interest 71 and for the unit areas other than the area of interest 71. At this time, the control unit 46 changes the imaging conditions depending on the speed Vo of the pedestrian 90.

For example, when the absolute value of the speed Vo of the pedestrian 90 is large, the control unit 46 controls the unit area of the image sensor 100 corresponding to the area of interest 71 including the pedestrian 90 to have a frame rate higher than that of the unit area other than the area of interest 71. control by increasing the rate or decreasing the thinning rate.

Further, the control unit 46 may change the imaging conditions for the region of interest 71 based on the positional relationship between the pedestrian 90 and nearby vehicles or buildings. For example, if a vehicle operating its blinker to turn right or left at an intersection exists in the imaging area 70, there is a possibility that the vehicle will enter a crosswalk. , the area closer to the vehicle is set as the special attention area. Then, the frame rate of the unit area of the image sensor 100 corresponding to the special attention area is made higher than that of the unit area corresponding to the attention area 71. The control unit 46 also controls the thinning rate of the unit area of the image sensor 100 corresponding to the special attention area to be lower than the thinning rate of the unit area corresponding to the attention area 71.

Further, in the region of interest 71, the control unit 46 may also vary the imaging conditions between the plurality of pixels or regions 78P indicated by diagonal lines and the plurality of pixels or regions 78S not indicated by diagonal lines. In the example of FIG. 16(b), the imaging conditions are different between pixels or regions that are vertically and horizontally adjacent in a checkered pattern, but the imaging conditions are not limited to this.

Further, when the control unit 46 recognizes objects such as other pedestrians and bicycles within the range of the imaging region 70, the control section 46 may add regions each including these plural objects to the region of interest 71. . In the plurality of regions of interest 71, the imaging conditions may be made different between the plurality of pixels or regions 78P indicated by diagonal lines and the plurality of pixels or regions 78S not indicated by diagonal lines.

FIG. 17 is a diagram illustrating an example of a scene imaged by the imaging unit 50-1 installed in the automobile traffic light 40a at an intersection. In FIG. 17, a traffic light 40a for an automobile includes a display section 42a having an indicator light indicating whether the vehicle can go straight, turn left, or turn right. An example of setting imaging conditions based on a subject recognition result using image data captured by the imaging unit 50-1 will be described.

FIG. 17 shows a state in which the display section 42a of the traffic light 40a lights up indicator lights indicating that it is possible to go straight and that it is possible to turn left, and the vehicle is waiting for a right turn. The control unit 46 of the automobile traffic light 40a changes the frame rate of the unit area of the image sensor 100 corresponding to the driving lane A through which vehicles going straight or turning left pass, and the driving lane B through which vehicles going straight ahead to the other driving lanes. The frame rate is controlled to be higher than the frame rate of the unit area corresponding to C, or the thinning rate is lowered. In other words, the control unit 46 controls by lowering the frame rate of the unit area corresponding to the driving lane C or setting a higher thinning rate.
Note that lowering the frame rate also includes setting not to perform imaging in that unit area.

In controlling the imaging conditions of the image sensor 100 in the automobile 10 or the traffic light 40 (traffic light 40a for automobiles, traffic light 40b for pedestrians) according to the embodiment described above, the imaging is performed almost simultaneously with the switching timing of the display on the display unit 42. The conditions may be changed, or the imaging conditions may be changed at fixed time intervals from the display switching timing. Alternatively, for a certain period of time immediately after the display is switched, the imaging conditions may be changed so that both the region of interest set before the display is switched and the region of interest that should be set after the display is switched are set as the region of interest.

According to the embodiment described above, the following effects can be obtained.
(1) The imaging device of the automobile 10 (or traffic light 40) includes an imaging unit 5 (or imaging unit 50) equipped with an imaging device 100 that can set imaging conditions for a plurality of areas, and the driving of other surrounding cars 20. The control unit 19 (or control unit 41) is configured to set imaging conditions for a plurality of unit areas based on a method. Thereby, imaging conditions suitable for the driving methods of surrounding cars 20 can be set for the imaging device 100.

(2) Since the control unit 19 (or the control unit 41) sets different imaging conditions for each area for imaging other cars 20 with different driving methods, Different imaging conditions can be set between regions to be imaged.

(3) The driving methods of the other cars 20 are an automatic driving method and a manual driving method, and the control unit 19 (or control unit 41) has an area for imaging the car 20 in the automatic driving method and an area for imaging the car 20 in the manual driving method. Since different imaging conditions are set for the area for imaging the car 20, the area for imaging the automatically driven car 20 and the area for imaging the manually operated car 20 are different for the image sensor 100. Imaging conditions can be set.

(4) The control unit 19 (or the control unit 41) sets the frame rate of the area in which the manually driven car 20 is imaged to be higher than the frame rate in the area in which the automatically driven car 20 is imaged. As a result, the frequency of capturing images of the manually operated car 20 is higher than the frequency of capturing images of the automatically operated vehicle 20, so that the degree of attention to the manually operated car 20 can be increased. That is, it becomes possible to quickly and accurately obtain information regarding the unpredictable behavior of the manually driven vehicle 20.

(5) The control unit 19 (or the control unit 41) sets the pixel thinning rate in the area where the manual driving car 20 is imaged to be lower than the pixel thinning rate in the area where the automatic driving car 20 is imaged. do. As a result, the amount of information about the manually driven car 20 can be made larger than the amount of information about the automatically driven car 20. That is, it becomes possible to obtain more accurate information regarding the unpredictable behavior of the manually operated vehicle 20.

(6) Since the control unit 19 (or control unit 41) is provided to acquire information regarding the movement methods of other surrounding cars 20, for example, even when other surrounding cars 20 are replaced, the latest information obtained Based on the information, imaging conditions can be set for the imaging device 100.

(7) Since the control unit 19 (or the control unit 41) acquires information through communication with other vehicles 20, the imaging conditions of the image sensor 100 are appropriately adjusted based on new information obtained through communication. Can be set to .

(8) Since the control unit 19 (or the control unit 41) acquires information by imaging the other vehicle 20 by the imaging unit 5 (or the imaging unit 50), even in a situation where communication is not possible, the control unit 19 (or the control unit 41) acquires information based on new information. Imaging conditions for the image sensor 100 can be appropriately set.

(9) The control unit 19 acquires information from a traffic light 40 different from that of other cars 20. Thereby, even in a situation where communication with another vehicle 20 is not possible, the imaging conditions of the imaging device 100 can be appropriately set based on new information.

(10) The control unit 19 (or the control unit 41) acquires information indicating whether the other vehicle 20 is a self-driving vehicle or a manually driven vehicle, and determines an area for imaging the self-driving vehicle and an area for imaging the manually driven vehicle. Set different imaging conditions. Thereby, imaging conditions can be appropriately set for each unit region on the imaging surface of the image sensor 100 so that, for example, manually driven cars attract more attention than self-driving cars. In particular, by increasing the frame rate and lowering the pixel thinning rate for manually driven vehicles, it is possible to quickly and accurately obtain information about the unpredictable behavior of manually driven vehicles.

(11) Since the automobile 10 is equipped with the imaging devices described in (1) to (10) above, the imaging conditions for the imaging device can be appropriately set in accordance with the movement methods of other cars 20 around the automobile 10.

(12) The control unit 19 (or the control unit 41) of the automobile 10 changes the imaging conditions of the image sensor 100 according to the operation of the steering wheel, turn signal switch, accelerator, brake, etc. in the automobile 10. Thereby, imaging conditions can be appropriately set for each unit region on the imaging surface of the imaging device 100 in response to changes in course, speed, etc. of the automobile 10.

(13) The traffic light 40 uses an imaging unit 50 equipped with an image sensor 100 that can independently set imaging conditions for a plurality of regions, and information regarding the movement of the automobile 10 and other vehicles 20 to capture a plurality of regions. It also includes a control section 46 that sets imaging conditions. Thereby, imaging conditions can be set for the imaging device 100 according to the traffic conditions of the automobile 10 and other vehicles 20 moving through an intersection or the like.

(14) Information regarding movement of the car 10 and other cars 20 includes a signal that allows movement of the car 10 and other cars 20, and a signal that does not allow movement of the car 10 and other cars 20, and controls The unit 46 sets imaging conditions based on each signal. Thereby, the imaging conditions of the image sensor 100 can be set appropriately for each case where the car 10 and other cars 20 are moving and when the car 10 and other cars 20 are not moving.

(15) The signals that permit movement of the automobile 10 and other vehicles 20 include a signal that permits going straight, a signal that permits a left turn, and a signal that permits a right turn, and the control unit 46 controls each of the signals. Imaging conditions are now set based on. Thereby, when the car 10 and the other car 20 are going straight, turning left, and turning right, the imaging conditions of the image sensor 100 can be set appropriately.

(16) Since the control unit 46 changes the imaging conditions in accordance with the switching of information regarding the movement of the automobile 10 and other vehicles 20, the imaging conditions of the imaging device 100 can be appropriately changed at the timing of switching of the above-mentioned signals. I can do it.

(17) Since the traffic light 40 includes a control unit 46 that acquires information regarding the movement of surrounding cars 10 and other cars 20, for example, even when other surrounding cars 20 are replaced, the traffic light 40 is based on the latest information obtained. Accordingly, imaging conditions can be set for the imaging device 100.

(18) Since the control unit 46 acquires information through communication with the automobile 10 and other vehicles 20, it is possible to appropriately set the imaging conditions of the image sensor 100 based on new information obtained through communication. .

(19) Since the control unit 46 acquires information by imaging the car 10 and other cars 20 by the imaging unit 50, even in a situation where communication is not possible, the control unit 46 appropriately adjusts the imaging conditions of the imaging device 100 based on new information. Can be set.

(20) The control unit 46 acquires information indicating whether the vehicle 10 and other vehicles 20 are automatic driving vehicles or manual driving vehicles as movement-related information, and provides an area for imaging automatic driving vehicles; Different imaging conditions are set for the region in which images are taken of manually operated vehicles. Thereby, imaging conditions can be appropriately set for each unit area on the imaging surface of the image sensor 100 so that, for example, a manually driven vehicle attracts more attention than an automatically driven vehicle. In particular, by increasing the frame rate and lowering the pixel thinning rate for manually driven vehicles, it is possible to quickly and accurately obtain information about the unpredictable behavior of manually driven vehicles.

(21) The control unit 46 acquires information indicating the course changes of the automobile 10 and other cars 20, and changes the area in which the imaging conditions are set based on the course changes of the automobile 10 and other cars 20. Thereby, the imaging conditions of the image sensor 100 can be appropriately changed at the timing of changing course.

(22) Since the car 10 is provided with a communication unit 41 that communicates with other cars 10 and other cars 20 different from the other cars 20, for example, information on other cars 20 around the traffic light 40 is transmitted to the car 10. can be sent to.

(23) The automobile 10 and the traffic light 40 measure the speed at which the object to be imaged moves, and change the imaging conditions of the image sensor 100 according to the magnitude and direction of the speed. Therefore, imaging conditions can be appropriately set for each area on the imaging surface of the imaging device 100, depending on the movement status of the object.

(24) The automobile 10 acquires information such as how many seconds later the traffic light will change, and reflects the information on the driving situation of the automobile 10. Therefore, it is possible to realize smooth operation that takes signal switching into consideration in advance.

(25) For a certain period of time immediately after the signal changes, the automobile 10 or the traffic light 40 includes both the area of interest set in the previous signal and the area of interest to be set in the signal after the change as an area of interest. , sets the imaging conditions of the imaging device 100. Thereby, the imaging conditions of the image sensor 100 can be appropriately set for the transient state during signal switching.

(26) With the imaging system 1 including the automobile 10 and the traffic light 40, it is possible to realize a more organized transportation system based on information acquisition through accurate and efficient imaging and communication.

In the above-described embodiment, the imaging unit 5 and the imaging unit 50 are controlled by the control performed by the control unit 19 of the automobile 10 or the control unit 46 of the traffic light 40, respectively. However, part of the control of the imaging unit 5 and the imaging unit 50 may be performed by a control circuit (such as a CPU) inside the imaging unit.

Furthermore, part of the processing performed by the control unit 46 of the traffic light 40 may be performed by the control unit 34 of the signal information generation device 30. The imaging unit 50 such as a camera does not necessarily have to be attached to the traffic light 40, and may be installed depending on the traffic signal or traffic situation at an intersection or the like.

Further, in the above-described embodiment, the display section and audio playback section of the navigation system 13 are used to display messages, but a separate display and playback device may be used. Furthermore, a display/playback device including a HUD (Head Up Display) that projects information onto the windshield of the automobile 10 and a speaker that plays audio information may be used.

The following modifications are also within the scope of the invention, and it is also possible to combine one or more of the modifications with the embodiments described above.

(Modification 1)
The control unit 46 of the traffic light 40 acquires information regarding the ratio of self-driving vehicles on the road or at the intersection through imaging by the imaging unit 50 or communication by the communication unit 41, and changes the imaging conditions according to the ratio. Good too. For example, the control unit 46 controls the thinning rate to be higher during a time period when the proportion of self-driving cars is high compared to when the proportion of self-driving cars is low. As a result, power consumption can be saved, and more efficient imaging can be performed.

(Modification 2)
The control unit 19 of the automobile 10 or the control unit 46 of the traffic light 40 may identify signs such as a novice driver sign or an elderly driver sign in addition to identifying whether the vehicle is an automatically driven vehicle or a manually driven vehicle by imaging. . Then, different imaging conditions are set for the unit area of the image sensor 100 corresponding to a novice driver's car and an elderly driver's car. For example, a vehicle on which a novice driver sign is displayed may be imaged at a frame rate higher than that of the unit area of the image sensor 100 corresponding to a manually driven vehicle. Thereby, imaging conditions can be appropriately set for each unit region on the imaging surface of the image sensor 100, depending on the object.

(Modification 3)
In the above description, distance measurement and detection of surrounding moving bodies and obstacles were performed by imaging using the image sensor 100, but a radar (not shown) may also be used. This makes it possible to obtain more reliable traffic information by taking advantage of the characteristics of the image sensor 100 and the radar.

Although various embodiments and modifications have been described above, the present invention is not limited to these embodiments. A mode in which the configurations shown in the embodiments and modified examples are used in combination is also included within the scope of the present invention. Other embodiments considered within the technical spirit of the present invention are also included within the scope of the present invention.

1...Imaging system 5, 50 (50-1, 50-2)...Imaging unit 10...Cars 19, 46...Control unit 20...Other cars 30...Traffic signal generation device 40...Traffic light 40a...Car signal 40b... Pedestrian traffic lights 42...display section 70...imaging areas 71, 71A, 71B...attention areas 72, 72A, 72B, 73A, 73B, 76, 77...vehicles 74A, 74B...quasi-attention area 75...special attention area 90... Pedestrian 100...imaging device 113...imaging chip

Claims (21)

A vehicle equipped with a driving assistance device,
The driving support device includes:
an imaging unit that captures an image of the exterior of the vehicle;
a distance measuring unit that measures a distance to an object existing outside the vehicle;
a positioning unit that measures the current position of the vehicle;
a control unit that generates data for providing driving support for the vehicle based on information from the imaging unit, the ranging unit, and the positioning unit;
a receiving unit that receives information transmitted from another vehicle imaged by the imaging unit;
has
The control unit determines whether the other vehicle is a vehicle running in automatic operation based on information from the reception unit, and sets imaging conditions for the imaging unit according to the determined result. Change frame rate and thinning rate ,
vehicle. Among the information from the imaging unit, the control unit converts information originating from the other vehicle that is determined to be a vehicle that is not running in automatic operation into a vehicle that is running in automatic operation. generating the data by using more heavily than information originating from the determined other vehicle;
The vehicle according to claim 1. The control unit determines that the other vehicle, from which the receiving unit cannot receive information, is a vehicle that is not running automatically.
The vehicle according to claim 1 or 2. The control unit sets an area in which it is determined that the other vehicle is traveling in automatic operation as an area of interest, and selects information originating from an area other than the area of interest from information originating from the area of interest. Generate the data by using
The vehicle according to claim 2. The control unit sets a first attention area and a second attention area different from the first attention area as the attention area, and transmits information originating from the second attention area to the first attention area. generating the data by using more heavily than the original information ;
The vehicle according to claim 4. The control unit sets the first region of interest in a region of the other vehicle that is performing the first automatic driving by control, and the user performs at least part of the control of the first automatic driving. setting the second region of interest in the region of the other vehicle that performs second automatic driving;
The vehicle according to claim 5. The control unit may have a different amount of information between a first automatic operation in which the other vehicle is operated by control and a second automatic operation in which a user performs at least a part of the control of the first automatic operation. generating the data with the information of;
The vehicle according to claim 2. The control unit transmits information originating from the other vehicle determined to be the vehicle running in the first automatic operation to the vehicle determined to be the vehicle running in the second automatic operation. generating the data by using information more heavily than information originating from the other vehicle;
The vehicle according to claim 7. A vehicle equipped with a driving assistance device,
The driving support device includes:
an imaging unit that captures an image of the exterior of the vehicle;
a distance measuring unit that measures a distance to an object existing outside the vehicle;
a positioning unit that measures the current position of the vehicle;
a control unit that generates data for providing driving support for the vehicle based on information from the imaging unit, the ranging unit, and the positioning unit;
has
The control unit determines whether the other vehicle is a vehicle running in automatic operation based on information from the imaging unit, and sets imaging conditions for the imaging unit according to the determined result. Change frame rate and thinning rate ,
vehicle. Among the information from the imaging unit, the control unit converts information originating from the other vehicle that is determined not to be a vehicle running in automatic operation to indicate that the vehicle is a vehicle running in automatic operation. generating the data by using more heavily than information originating from the determined other vehicle;
The vehicle according to claim 9. The control unit determines whether or not the other vehicle is a vehicle that is running automatically, based on a result of imaging predetermined identification information.
The vehicle according to claim 9 or 10. The control unit determines whether or not the other vehicle is a vehicle that is running automatically, based on the imaging result of the identification information of the body of the vehicle.
The vehicle according to claim 11. The control unit determines whether the other vehicle is a vehicle that is running automatically, based on the imaging result of the identification information displayed on the body of the vehicle.
Vehicle according to claim 12. If the control unit cannot determine that the other vehicle is a vehicle running in automatic mode based on the imaging result of the imaging unit, the control unit determines that the other vehicle is a vehicle running in manual mode.
The vehicle according to claim 9. The control unit determines a region in which it is determined that the other vehicle is traveling in automatic operation as a region of interest, and selects information originating from a region other than the region of interest from information originating from the region of interest. Generate the data by using
The vehicle according to claim 9. The control unit sets a first attention area and a second attention area different from the first attention area as the attention area, and transmits information originating from the second attention area to the first attention area. generating the data by using more heavily than the original information ;
The vehicle according to claim 15. The control unit sets the first region of interest in a region of the other vehicle that is performing the first automatic driving by control, and the user performs at least part of the control of the first automatic driving. setting the second region of interest in the region of the other vehicle that performs second automatic driving;
Vehicle according to claim 16. The control unit may have a different amount of information between a first automatic operation in which the other vehicle is operated by control and a second automatic operation in which a user performs at least a part of the control of the first automatic operation. generating the data with the information of ;
The vehicle according to claim 9. The control unit transmits information originating from the other vehicle determined to be the vehicle running in the first automatic operation to the vehicle determined to be the vehicle running in the second automatic operation. generating the data by using information more heavily than information originating from the other vehicle;
Vehicle according to claim 18. an input section into which information obtained by capturing an image of the outside of the vehicle is input;
a distance measuring unit that measures a distance to an object existing outside the vehicle;
a positioning unit that measures the current position of the vehicle;
a control unit that generates data for providing driving support for the vehicle based on information from the input unit, the ranging unit, and the positioning unit;
a receiving unit that receives information transmitted from another vehicle imaged by the imaging unit;
has
The control unit determines whether the other vehicle is a vehicle running in automatic operation based on information from the reception unit, and sets imaging conditions for the imaging unit according to the determined result. Change frame rate and thinning rate ,
Driving support equipment. an input section into which information obtained by capturing an image of the outside of the vehicle is input;
a distance measuring unit that measures a distance to an object existing outside the vehicle;
a positioning unit that measures the current position of the vehicle;
a control unit that generates data for providing driving support for the vehicle based on information from the input unit, the ranging unit, and the positioning unit;
has
The control unit determines whether the other vehicle is a vehicle that is running automatically based on information from the imaging unit, and determines a frame that is an imaging condition of the imaging unit according to the determined result. change the rate and thinning rate ,
Driving support equipment.

Images