JP2023171067A - Distance measuring system, distance measuring device, and distance measuring method - Google Patents

Abstract

To provide a distance measuring system capable of easily simplifying a configuration.SOLUTION: A distance measuring system 500 comprises: a camera portion 480 that outputs GS image data corresponding to global shutter operation and RS image data corresponding to rolling shutter operation by performing the global shutter operation and the rolling shutter operation on a distance metering object; and a distance measuring device 1. The distance measuring device 1 includes: a distortion detection portion 510 that detects the magnitude of distortion of a shape of the distance metering object indicated by the RS image data with respect to a shape of the distance metering object indicated by the GS image data; and a distance derivation portion 503 that acquires the moving speed of the camera portion 480 and derives a distance from the camera portion 480 to the distance metering object on the basis of the moving speed and the magnitude of distortion.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to a distance measuring device and the like.

Conventionally, a distance detection device using stereo images has been proposed as a distance measurement system that measures the distance to a distance measurement target (see, for example, Patent Document 1). This distance detection device includes two cameras. Then, the distance detection device determines the distance measurement by determining the positional shift of the image of the distance measurement target displayed in the two images (i.e., stereo images) obtained by imaging with those cameras. Detect the distance to the object. Note that the object to be measured is a subject imaged by a camera.

Patent No. 3452075

However, the distance detection device disclosed in Patent Document 1 has a problem in that the configuration may become complicated.

Therefore, the present disclosure provides a distance measuring system and the like whose configuration can be easily simplified.

A distance measurement system according to an aspect of the present disclosure performs a global shutter operation and a rolling shutter operation on an object to be measured, thereby obtaining first data corresponding to the global shutter operation and first data corresponding to the rolling shutter operation. and a distance measuring device, the distance measuring device outputs the second data indicated by the second data with respect to the shape of the object to be measured indicated by the first data. a distortion detection section that detects the magnitude of distortion in the shape of the distance object; and a moving speed of the camera section; based on the moving speed and the magnitude of the distortion detected by the distortion detection section, and a distance derivation section that derives a distance from the camera section to the object to be measured.

Note that these comprehensive or specific aspects may be realized by an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM. and a recording medium may be used in any combination. Further, the recording medium may be a non-temporary recording medium.

The ranging system of the present disclosure can easily simplify the configuration.

Further advantages and advantages of one aspect of the disclosure will become apparent from the specification and drawings. Such advantages and/or effects may be provided by each of the embodiments and features described in the specification and drawings, but not necessarily all to obtain one or more of the same features. There's no need.

FIG. 1 is a diagram illustrating an example of a vehicle equipped with a ranging system according to the first embodiment. FIG. 2 is a block diagram illustrating an example of the configuration of the ranging system according to the first embodiment. FIG. 3 is a block diagram showing an example of the configuration of the camera section in the first embodiment. FIG. 4 is a diagram illustrating an example of a schematic configuration of a pixel array included in the imaging device according to the first embodiment. FIG. 5 is a diagram for explaining an example of the operation of the imaging device in the first embodiment. FIG. 6 is a diagram illustrating an example of the position and imaging range of a camera unit attached to a vehicle in the first embodiment. FIG. 7 is a diagram illustrating an example of a situation in which distance is measured by the distance measuring system according to the first embodiment. FIG. 8 is a diagram showing another example of a situation in which distance is measured by the distance measuring system according to the first embodiment. FIG. 9 is a diagram illustrating an example of rolling shutter distortion in the first embodiment. FIG. 10 is a diagram for explaining in detail an example of edge pairing by the angle detection unit in the first embodiment. FIG. 11 is a diagram for explaining detection of an angle between edges by the angle detection unit in the first embodiment. FIG. 12 is a diagram illustrating an example of a table used in the distance derivation unit in the first embodiment. FIG. 13 is a flowchart illustrating an example of the processing operation of the ranging system in the first embodiment. FIG. 14 is a diagram illustrating an example of the relationship between the configuration of the angle detection section, the distance derivation section, the shutter control section, and the vehicle speed detection section in Modification 1 of Embodiment 1. FIG. 15 is a diagram illustrating an example of a table used in the distance derivation unit in Modification 1 of Embodiment 1. FIG. 16 is a block diagram illustrating an example of the configuration of a distance measuring system in Modification 2 of Embodiment 1. FIG. 17 is a diagram for explaining the relationship between the angle between edges and the shutter parameter in the second modification of the first embodiment. FIG. 18 is a diagram schematically showing an example of the timing at which each of the GS image data and the RS image data is acquired in the third modification of the first embodiment. FIG. 19 is a schematic diagram showing an exemplary circuit configuration of the imaging device according to the second embodiment. FIG. 20 is a schematic cross-sectional view showing an exemplary device structure of a unit pixel cell of the imaging device according to the second embodiment. FIG. 21 is a schematic plan view showing the relationship between a unit pixel cell and a counter electrode of the imaging device according to the second embodiment. FIG. 22 is a graph showing an example of the photocurrent characteristics of the photoelectric conversion layer of the imaging device according to the second embodiment. FIG. 23 is a schematic plan view showing the relationship between a unit pixel cell and a counter electrode of the imaging device according to Embodiment 2, and an example of the configuration of a voltage supply circuit. FIG. 24 is a diagram for explaining an example of the operation of the imaging device according to the second embodiment.

(Findings that formed the basis of this disclosure)
The present inventor has found that the following problem occurs with the distance detection device of Patent Document 1 described in the "Background Art" section.

The distance detecting device disclosed in Patent Document 1 includes a plurality of cameras for capturing an image of an object to be measured in order to detect a distance using a stereo method. Therefore, the configuration of the distance detection device becomes complicated by the number of cameras. Further, if the distance detection device is equipped with only one camera, it is necessary to cause that camera to take images multiple times. That is, the distance detection device causes the camera located at the first position to take an image of the object to be measured, and then moves the camera to the second position. Then, the distance detection device causes the camera located at the second position to take an image of the object to be measured. Thereby, two images can be acquired as stereo images, similar to the case where the distance detection device is equipped with two cameras. However, in this case, since the two images are captured at different timings, it may be difficult to accurately detect the distance to the object, and furthermore, it may be difficult to accurately detect the distance to the object. Delays may occur.

Therefore, a distance measurement system according to an aspect of the present disclosure performs a global shutter operation and a rolling shutter operation on an object to be measured, thereby obtaining first data corresponding to the global shutter operation and the rolling shutter operation. a camera unit that outputs second data corresponding to a distortion detection unit that detects the magnitude of distortion in the shape of the distance measurement target; and a movement speed of the camera unit, and based on the movement speed and the magnitude of the distortion detected by the distortion detection unit. and a distance derivation section that derives a distance from the camera section to the object to be measured. For example, the larger the distortion, the shorter the distance will be derived, and conversely, the smaller the distortion, the longer the distance will be derived.

As a result, the distance to the object to be measured is derived using one camera unit, making it easier to simplify the configuration of the ranging system compared to devices equipped with multiple cameras to obtain stereo images, for example. can be achieved.

Further, the camera section may perform the global shutter operation and the rolling shutter operation simultaneously.

As a result, two types of shutter operations are performed at the same time, which reduces the time required to derive the distance compared to the case where imaging is performed multiple times at different timings to obtain a stereo image. delay can be suppressed. Furthermore, the accuracy of distance measurement can be improved.

Further, the distortion detection unit detects one or more linear first edges from the shape of the distance measurement target indicated by the first data, and detects one or more linear first edges of the distance measurement target indicated by the second data. a straight line detection unit that detects one or more linear second edges based on the shape; one first edge of the one or more linear first edges; and one or more linear first edges; an angle detection unit that associates any one of the two edges with a second edge and detects an angle between the associated first edge and the second edge as the magnitude of the distortion; You may prepare.

Thereby, the magnitude of the distortion can be appropriately detected, and the accuracy of distance measurement can be improved.

Further, the distance measuring device may further include a shutter control section that controls at least one of the global shutter operation and the rolling shutter operation of the camera section based on the moving speed.

Thereby, the influence of the moving speed of the camera unit on at least one of the first data and the second data can be suppressed. As a result, for example, even if the moving speed of the camera unit is too fast or too slow, the magnitude of the detected distortion can be kept within an appropriate range. Therefore, by using the shutter parameter indicating the content of control by the shutter control unit and the magnitude of distortion within an appropriate range, it is possible to improve the accuracy of distance measurement.

The distance measuring device further includes a shutter control unit that controls at least one of the global shutter operation and the rolling shutter operation of the camera unit based on the magnitude of the distortion detected by the distortion detection unit. You may prepare.

As a result, at least one of the two types of shutter operations to be performed next is controlled based on the magnitude of distortion detected in response to the two types of shutter operations. Feedback control can be executed. As a result, for example, the magnitude of detected distortion can be kept within an appropriate range. Therefore, by using the shutter parameter indicating the content of control by the shutter control unit and the magnitude of distortion within an appropriate range, it is possible to improve the accuracy of distance measurement.

Further, the shutter control unit changes a frame period when controlling the global shutter operation, and changes at least one of a frame period and a horizontal period when controlling the rolling shutter operation. The horizontal period may be a period from the start of an exposure period of one row of the pixel array included in the camera unit until the start of exposure of the next row. For example, when controlling the global shutter operation, the slower the moving speed, the longer the frame period, and when controlling the rolling shutter operation, changing the frame period to a longer period when the moving speed is lower. The slower the frame period, the longer at least one of the frame period and the horizontal period may be changed.

For example, when the moving speed of the camera section is fast, the distortion tends to become large, and conversely, when the moving speed of the camera section is slow, the distortion tends to become small. In contrast, in the ranging system according to one aspect of the present disclosure, the slower the moving speed of the camera unit, the longer the horizontal period is changed, for example. Therefore, when the moving speed of the camera unit is slow, distortion can be prevented from becoming too small. As a result, by using shutter parameters such as the horizontal period, it is possible to suppress errors included in the magnitude of detected distortion and improve the accuracy of distance measurement. Alternatively, in the ranging system according to one aspect of the present disclosure, the slower the moving speed of the camera unit, the longer the frame period is changed. Therefore, when the moving speed of the camera unit is slow and two types of shutter operations are repeatedly executed, it is possible that a plurality of first data that are similar to each other and a plurality of second data that are similar to each other are output in a short period of time. It can be suppressed. This makes it possible to reduce the amount of data, reduce the memory storage capacity, and reduce the processing load.

Further, when controlling the global shutter operation, the shutter control unit may control a time interval at which the first data is sequentially output. For example, when controlling the global shutter operation, the shutter control unit may lengthen the time interval as the movement speed is slower.

For example, if the moving speed of the camera unit is slow and the first data is repeatedly output in a short period, it is difficult to make a difference in the content indicated by the first data, and many Data may be output from the camera unit. However, in the distance measurement system according to one aspect of the present disclosure, the slower the moving speed of the camera unit, the longer the period, which is the time interval at which the first data is sequentially output, is set. The output of data with low contribution can be suppressed. As a result, it is possible to reduce the amount of data, the memory storage capacity, and the processing load. Note that the above-mentioned time interval corresponds to a frame period.

Note that the distance measuring device included in the distance measuring system includes each component other than the above-mentioned camera section, and acquires the first data and second data from the camera section. Similar effects can be achieved.

Hereinafter, embodiments will be specifically described with reference to the drawings.

Note that the embodiments described below are all inclusive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are examples, and do not limit the present disclosure. Further, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the most significant concept will be described as arbitrary constituent elements.

Furthermore, each figure is a schematic diagram and is not necessarily strictly illustrated. Moreover, in each figure, the same reference numerals are attached to the same constituent members.

(Embodiment 1)
[Distance measurement system configuration]
FIG. 1 is a diagram showing an example of a vehicle equipped with a ranging system according to the present embodiment.

Vehicle 1000 includes a distance measurement system 500 and a vehicle speed detection section 600. Vehicle speed detection section 600 detects the speed of vehicle 1000 as a vehicle speed, and outputs vehicle speed information indicating the detected vehicle speed to distance measuring system 500. For example, vehicle speed detection section 600 detects the vehicle speed based on a pulse signal output according to the rotation of the wheels of vehicle 1000. Note that the vehicle speed detection unit 600 may detect the vehicle speed not only by the method based on the pulse signal but also by any other method. Further, vehicle speed detection section 600 may detect vehicle speed by acquiring vehicle speed information from an ECU (Electronic Control Unit) that is included in vehicle 1000 and handles vehicle speed. The distance measurement system 500 includes a camera unit 480 that captures an image of a distance measurement target. Then, the distance measurement system 500 measures the distance from the camera unit 480 to the object to be measured using the image data obtained by the camera unit 480 and the vehicle speed information output from the vehicle speed detection unit 600. Outputs distance measurement information indicating.

FIG. 2 is a block diagram showing an example of the configuration of the ranging system 500.

The distance measurement system 500 includes a camera section 480 and a distance measurement device 1. The distance measuring device 1 includes a distortion detection section 510, a distance derivation section 503, and a shutter control section 504.

The shutter control unit 504 acquires vehicle speed information output from the vehicle speed detection unit 600, and sets shutter parameters of the camera unit 480 according to the vehicle speed indicated by the vehicle speed information. Furthermore, the shutter control section 504 outputs shutter information indicating the set shutter parameters to the distance derivation section 503. The shutter parameter can be said to be a parameter for defining the shutter speed, and is, for example, a frame period, a horizontal period, etc. The frame period is a period for generating one frame, and is a period including an exposure period and a non-exposure period. The horizontal period is a period from the start of the exposure period of one row of the pixel array included in the camera section 480 until the start of exposure of the next row. Note that the frame period and the horizontal period are also expressed as one frame period and one horizontal period, or 1V and 1H, respectively. For example, the shutter control unit 504 sets at least one of the frame period and the horizontal period to be long so that the slower the vehicle speed, the slower the shutter speed. Conversely, the shutter control unit 504 sets at least one of the frame period and the horizontal period to be short so that the faster the vehicle speed is, the faster the shutter speed is. Note that the horizontal period is also called line scanning time.

Camera section 480 includes a pixel array including a plurality of pixels arranged in a matrix, and outputs GS image data and RS image data to distortion detection section 510 for each frame period. GS image data is data obtained by global shutter operation, and is also called first data. RS image data is data obtained by rolling shutter operation, and is also called second data. Further, when obtaining the image data, the camera section 480 performs a global shutter operation and a rolling shutter operation according to the shutter parameters set by the shutter control section 504.

In this way, the camera unit 480 in the present embodiment performs the global shutter operation and the rolling shutter operation on the object to be measured, thereby obtaining the first data corresponding to the global shutter operation and the rolling shutter operation. and second data corresponding to the second data. Furthermore, camera unit 480 in this embodiment simultaneously performs a global shutter operation and a rolling shutter operation. In other words, two types of shutter operations are performed in one frame period.

Distortion detection section 510 acquires GS image data and RS image data from camera section 480. Then, the distortion detection unit 510 detects the magnitude of distortion in the shape of the object to be measured from the image data. Specifically, the distortion is a distortion of the shape of the distance measurement object shown in the RS image data with respect to the shape of the distance measurement object shown in the GS image data. In other words, the distortion detection unit 510 detects the magnitude of distortion in the shape of the distance measurement target indicated by the second data with respect to the shape of the distance measurement target indicated by the first data. Note that such distortion is also called rolling shutter distortion. Further, the distortion detection section 510 includes a straight line detection section 501 and an angle detection section 502.

The straight line detection unit 501 acquires GS image data and RS image data from the camera unit 480, and detects one or more edges included in the image shown in the image data for each of the GS image data and RS image data. To detect. These detected edges are straight edges. Then, the straight line detection unit 501 outputs edge information indicating the plurality of straight edges to the angle detection unit 502. The edge information is, for example, information indicating detected edges in each of the GS image data and the RS image data.

The angle detection unit 502 acquires edge information from the straight line detection unit 501, and identifies two edges that correspond to each other in the GS image data and the RS image data from a plurality of edges indicated by the edge information. These two edges are edges of the same part of the same distance measurement target that is shown in each of the GS image data and the RS image data. The angle detection unit 502 then detects the angle between the two identified straight edges. This angle is an example of rolling shutter distortion. The angle detection unit 502 then outputs angle information indicating the angle between the edges to the distance derivation unit 503.

Distance derivation section 503 acquires angle information, shutter information, and vehicle speed information from angle detection section 502, shutter control section 504, and vehicle speed detection section 600. The distance deriving unit 503 then derives the distance to the object to be measured based on the angle information, shutter information, and vehicle speed information. In other words, the distance deriving unit 503 calculates the distance to the object to be measured based on the angle detected by the angle detecting unit 502, the shutter parameter set by the shutter control unit 504, and the vehicle speed information output from the vehicle speed detecting unit 600. Derive the distance. As a result, the distance from the camera section 480 to the object to be measured is measured.

Note that although the distance measuring device 1 includes the shutter control section 504 in this embodiment, it does not need to include the shutter control section 504. That is, the shutter parameters may be fixed without being controlled. In such a case, the distance derivation unit 503 may derive the distance based on at least the angle information and the vehicle speed information. Furthermore, since the camera section 480 is fixed to the vehicle 1000, it can be said that the vehicle speed is the moving speed of the camera section 480. Therefore, the distance deriving unit 503 in this embodiment acquires the moving speed of the camera unit 480, and uses the distance measuring unit 503 from the camera unit 480 based on the moving speed and the magnitude of the distortion detected by the distortion detecting unit 510. Derive the distance to the object.

Note that the number of camera units 480 included in the distance measuring system 500 in this embodiment is not limited to one, and may be two or more.

FIG. 3 is a block diagram showing an example of the configuration of camera section 480 in this embodiment. As shown in FIG. 3, the camera section 480 includes an optical system 410, an imaging device 100, and an image forming circuit 430.

The optical system 410 includes an aperture, an image stabilization lens, a zoom lens, a focus lens, and the like. The number of lenses included in the optical system 410 is determined as appropriate depending on the required functions. Note that the optical system 410 does not need to include an image stabilization lens, and may include a normal lens other than the image stabilization lens.

The imaging device 100 has the above-described pixel array. An image of the object to be measured is projected onto this pixel array via an optical system 410. The imaging device 100 outputs a signal corresponding to the image to the image forming circuit 430 under the control of the shutter control unit 504.

The image forming circuit 430 forms an image based on the output of the imaging device 100. The image forming circuit 430 may be, for example, a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or the like. Image forming circuit 430 may include memory.

In the example shown in FIG. 3, the image forming circuit 430 includes an output buffer 440. The image forming circuit 430 outputs image data representing the formed image to the outside of the camera section 480 via the output buffer 440. The image data output from the image forming circuit 430 is GS image data and RS image data. Note that the format of these image data may be, for example, RAW data or a 12-bit width signal. Further, the image data output from the image forming circuit 430 may be, for example, an H. The data may be compressed in accordance with the H.264 standard.

FIG. 4 is a diagram illustrating an example of a schematic configuration of a pixel array included in the imaging device 100.

The pixel array PA includes a plurality of pixels 10 arranged in a matrix. Each of these plurality of pixels 10, also called a unit pixel cell, converts light received from outside the camera unit 480 via the optical system 410 into electric charges during the exposure period, and accumulates the accumulated electric charges. Outputs the corresponding signal. For example, the odd numbered rows included in the pixel array PA perform a rolling shutter operation, and the even numbered rows included in the pixel array PA perform a global shutter operation. Further, the rolling shutter operation and the global shutter operation are executed simultaneously.

FIG. 5 is a diagram for explaining an example of the operation of the imaging device 100. Note that GS_0, GS_1, GS_2, and GS_3 in FIG. 5 indicate the operations of the GS_0th row, GS_1st row, GS_2nd row, and GS_3rd row in the pixel array PA, respectively. Further, RS_0, RS_1, RS_2, and RS_3 in FIG. 5 indicate the operations of the RS_0th row, RS_1th row, RS_2nd row, and RS_3rd row in the pixel array PA, respectively.

For example, the even-numbered rows GS_0, GS_1, GS_2, and GS_3 perform a global shutter operation. Specifically, from the GS_0th row to the GS_3th row, an exposure period and a non-exposure period are formed for each frame period. Furthermore, in the GS_0th row to the GS_3th row, the exposure periods are formed at the same timing, and the non-exposure periods are formed at the same timing. For example, in the frame period from time t0 to time t8, the non-exposure period is the period from time t0 to time t4, and the exposure period is the period from time t4 to time t8. The camera unit 480 outputs GS image data by executing such a global shutter operation from the GS_0 line to the GS_3 line.

On the other hand, the RS_0th row, RS_1st row, RS_2nd row, and RS_3rd row, which are odd-numbered rows, perform a rolling shutter operation. Specifically, in the RS_0th row to the RS_3th row, an exposure period and a non-exposure period are formed for each frame period, which is the same as in the case of the global shutter operation. Here, in the RS_0th row to the RS_3th row, the exposure periods are formed at mutually different timings, and the non-exposure periods are formed at mutually different timings. For example, the exposure period in the RS_0th row starts from time t1, and the exposure period in the RS_1th row starts from time t2, delayed by 1H period compared to the RS_0th row. Similarly, the exposure period in the RS_2 row is delayed by 1H period compared to its RS_1 row and starts at time t3, and the exposure period in the RS_3 row is delayed by 1H period compared to its RS_2 row and starts at time t4. will be started. The camera unit 480 outputs RS image data by executing such a rolling shutter operation from the RS_0th line to the RS_3rd line.

In the example shown in FIG. 5, four lines from GS_0 to GS_3 perform a global shutter operation, and four lines from RS_0 to RS_3 perform a rolling shutter operation. The number of lines is not limited to four, but may be five or more.

FIG. 6 is a diagram illustrating an example of the position and imaging range of camera unit 480 attached to vehicle 1000. Note that (a) of FIG. 6 shows a state in which the vehicle 1000 is viewed from vertically above, and (b) in FIG. 6 shows a state in which the vehicle 1000 is seen from the front.

Vehicle 1000 includes four camera units 480, for example, as shown in FIG. 6(a). In other words, in the example of FIG. 6, the ranging system 500 includes four camera units 480. In this case, the shutter control unit 504 sets shutter parameters for each of the four camera units 480. Further, the distortion detection unit 510 detects rolling shutter distortion for each of the four camera units 480 based on the GS image data and RS image data output from the camera unit 480. Note that the vehicle 1000 may be equipped with four ranging systems 500, each of which is equipped with one camera unit 480. As a result, for each of the four camera units 480, the distance from that camera unit 480 to the distance measurement target imaged by that camera unit 480 is derived.

The four camera units 480 are arranged in the vehicle 1000 so as to be able to take images of the front, rear, left side, and right side of the vehicle 1000, for example, as shown in FIG. 6(a). At this time, these four camera sections 480 are attached to the vehicle 1000 so that distortion is likely to occur in the RS image data output from each of the four camera sections 480 as the vehicle 1000 travels. For example, the four camera sections 480 are attached to the vehicle 1000 so that each row of the pixel array PA included in each of the four camera sections 480 is along the direction of movement of the image of the subject projected onto the pixel array PA. The direction of movement of the image of the subject is the direction of movement of camera unit 480 when the subject is stationary. Furthermore, since the camera unit 480 is fixed to the vehicle 1000, it can be said that the direction of movement of the image of the subject is the traveling direction or the turning direction of the vehicle 1000. Therefore, the four camera sections 480 are arranged on the vehicle 1000 such that each row of the pixel array PA included in each of the four camera sections 480 is along the direction of movement of the camera sections 480, that is, the traveling direction or the turning direction of the vehicle 1000. can be attached to. In a specific example, four camera units 480 are attached so that each row of the pixel array PA is aligned in the horizontal direction.

More specifically, as shown in FIG. 6(b), among the four camera units 480, two camera units 480 attached to the right and left sides of the vehicle 1000 are configured to move forward or backward when the vehicle 1000 moves forward or backward. , moves in the front and rear direction together with the vehicle 1000. In this case, the images of the subject projected onto the respective pixel arrays PA of the two camera sections 480 move in the horizontal direction or in the front-back direction. Therefore, the two camera units 480 are attached to the vehicle 1000 so that each row of the pixel array PA is along the horizontal direction or the front-rear direction.

Furthermore, each of the four camera units 480 is attached to the vehicle 1000 so that the optical axis of the camera faces in the horizontal direction or in the direction of the road surface.

Note that in each row included in the pixel array PA of the camera unit 480, when line scanning is performed such that exposure of each pixel is started from one end of the row toward the other end, the direction of the line scanning is unified. You can leave it there. For example, in the two camera units 480 attached to the right and left sides of the vehicle 1000, the direction of line scanning may be unified to the front of the vehicle 1000. That is, in each row of the pixel array PA, exposure of each pixel is started from the front pixel to the rear pixel. Alternatively, the direction of line scanning may be unified to the rear of vehicle 1000. That is, in each row of the pixel array PA, exposure of each pixel is started from the rear pixel to the front pixel. Thereby, it is possible to match the tendency of the error of rolling shutter distortion in each row, and it is possible to reduce the error in the derived distance.

[Operation of ranging system]
FIG. 7 is a diagram illustrating an example of a situation in which distance is measured by distance measuring system 500.

A vehicle 1000 equipped with a ranging system 500 travels in a parking lot, for example. At this time, when another vehicle 1001 already parked in the parking lot is imaged by the camera unit 480, the distance measuring system 500 measures the distance to the vehicle 1001. Alternatively, when another vehicle 1002 already parked in the parking lot is imaged by the camera unit 480, the distance measuring system 500 measures the distance to the vehicle 1002. Specifically, distance measuring system 500 measures the distance to each characteristic portion E in vehicle 1001 or the distance to each characteristic portion E in vehicle 1002. In this case, each characteristic portion E of the vehicle 1001, each characteristic portion E of the vehicle 1002, etc. are treated as objects to be measured.

The characteristic portion E described above is a portion that is imaged by the camera unit 480 as, for example, a linear edge along one direction. The edge projected onto pixel array PA moves along the direction of each row of pixel array PA as vehicle 1000 runs or turns, and spans multiple rows of pixel array PA. Therefore, rolling shutter distortion may occur in the characteristic portion E.

FIG. 8 is a diagram illustrating another example of a situation in which distance is measured by distance measuring system 500. Note that FIG. 8 shows a state in which the vehicle 1000 and the truck 1003 are viewed from vertically above.

For example, as shown in FIG. 8, vehicle 1000 passes on the right side of stopped truck 1003. At this time, the camera unit 480 of the ranging system 500 mounted on the vehicle 1000 images the right side of the truck 1003. Then, the camera unit 480 captures an image of the track 1003 and outputs GS image data and RS image data in which each characteristic portion E of the track 1003 is displayed.

FIG. 9 is a diagram showing an example of rolling shutter distortion.

The camera unit 480 of the ranging system 500 outputs the GS image data shown in FIG. 9(a) and the RS image data shown in FIG. 9(b) by capturing images in the situation shown in FIG. 8, for example.

The straight line detection unit 501 of the distortion detection unit 510 performs edge detection from each of the GS image data and RS image. Specifically, the straight line detection unit 501 detects a straight edge that crosses a plurality of rows of the pixel array PA. That is, edges that are not parallel to each row of the pixel array PA are detected. For example, as in the example of FIG. 9, the straight line detection unit 501 detects a straight edge Eg1, which is an image of the characteristic portion E of the track 1003, from the GS image data, and detects a straight edge Eg1, which is an image of another characteristic portion E, from the GS image data. An edge Eg2 of the shape is detected. Further, the straight line detection unit 501 detects a straight edge Er1, which is an image of the characteristic portion E of the track 1003, and a straight edge Er2, which is an image of another characteristic portion E, from the RS image data. The horizontal direction of the image indicated by each of the GS image data and the RS image data is parallel to each row of the pixel array PA. Therefore, each of the above-mentioned detected edges Eg1, Eg2, Er1, and Er2 is not parallel to each row of the pixel array PA. Such edge detection may be performed by machine learning such as deep learning, or may be performed using other known techniques.

In this way, the straight line detection unit 501 in this embodiment detects the linear edges Eg1 and Eg2 from the shape of the distance measurement target indicated by the GS image data, and detects the straight edges Eg1 and Eg2 from the shape of the distance measurement target indicated by the RS image data. Linear edges Er1 and Er2 are detected from the shape.

The angle detection unit 502 associates edges included in the GS image data with edges included in the RS image data. For example, the edge Eg1 shown in the GS image data and the edge Er1 shown in the RS image data are images of the characteristic portion E at the rear end of the same track 1003. Therefore, the angle detection unit 502 associates the edge Eg1 and the edge Er1. Similarly, the edge Eg2 shown in the GS image data and the edge Er2 shown in the RS image data are images of the characteristic portion E located in front of the same track 1003. Therefore, the angle detection unit 502 associates the edge Eg2 and the edge Er2. This kind of edge correspondence is also called edge pairing. Further, a set of two edges that are associated with each other is also called a pair.

Next, the angle detection unit 502 detects the angle between the associated edges, as shown in FIG. 9(c). FIG. 9(c) is a diagram in which the GS image data and the RS image data are superimposed to clearly show the angle between the edges. Specifically, the angle detection unit 502 detects the angle θ1 between the edge Eg1 and the edge Er1, and detects the angle θ2 between the edge Eg2 and the edge Er2.

In this way, the angle detection unit 502 in this embodiment detects one of the straight edges Eg1 and Eg2 (for example, the edge Eg1) and one of the straight edges Er1 and Er2. 2 edges (for example, edge Er1), and the angle between the associated edge Eg1 and edge Er1 is detected as the magnitude of distortion. Thereby, the magnitude of distortion can be appropriately detected.

The distance deriving unit 503 corresponds to the edge Eg1 and the edge Er1 based on the vehicle speed of the vehicle 1000 when the GS image data and the RS image data are output, the detected angle θ1, and the shutter parameters at that time. The distance of the feature part E is derived. Specifically, the distance is the distance from the camera section 480 to the characteristic portion E at the rear end of the track 1003. Similarly, the distance deriving unit 503 calculates the edge Eg2 and the edge Er2 based on the vehicle speed of the vehicle 1000 when the GS image data and the RS image data are output, the detected angle θ2, and the shutter parameters at that time. The distance of the feature portion E corresponding to is derived. Specifically, the distance is the distance from the camera unit 480 to the characteristic portion E located in front of the truck 1003.

In deriving the distance, distance deriving section 503 uses the vehicle speed indicated in the vehicle speed information output from vehicle speed detecting section 600 as the vehicle speed of vehicle 1000. Further, the distance derivation unit 503 derives the distance using a table. The table shows angles, vehicle speeds, shutter parameters, and distances in association with each other. Such a table may be held in the distance deriving unit 503 or may be stored in a memory included in the distance measuring system 500 or the distance measuring device 1. Specifically, the distance derivation unit 503 refers to the table and correlates the vehicle speed of the vehicle 1000 when the two types of image data described above were output, the detected angle θ1, and the shutter parameters at that time. Determine from the table the distance that is The distance deriving unit 503 then derives the specified distance as the distance of the characteristic portion E at the rear end of the track 1003. The distance derivation unit 503 derives the distance of the characteristic portion E in front of the track 1003 by performing similar processing on the detected angle θ2.

Note that although the distance derivation unit 503 in this embodiment uses a table to derive the distance, a function may be used instead of the table. The function is a function for calculating the distance from the angle, vehicle speed, and shutter parameter. Alternatively, machine learning models may be used. This machine learning model is a model that has been trained to output distance in response to inputs of angle, vehicle speed, and shutter parameter. For example, the distance deriving unit 503 inputs the vehicle speed of the vehicle 1000 when the two types of image data described above, the detected angle, and the shutter parameters at that time into the machine learning model. , derive the distance of the feature E.

The processing of the angle detection section 502, distance derivation section 503, and shutter control section 504 will be described in detail below.

[Processing of angle detection unit]
FIG. 10 is a diagram for explaining in detail an example of edge pairing by the angle detection unit 502. Note that FIG. 10 is a diagram in which the respective images of GS image data and RS image data are superimposed to clearly show the angle between the edges.

For example, as shown in FIG. 10, the straight line detection unit 501 detects edges Eg3 and Eg4 from the GS image data, and detects edges Er3 and Er4 from the RS image data. In this case, the angle detection unit 502 searches the RS image data for the edge closest to the edge Eg3 of the GS image data in the line n direction of the image data. Note that the line n direction of image data is a direction along the rows of the pixel array PA. It can also be said that line n is a line of an image obtained by the (2N-1)th row and the 2Nth row in the pixel array PA. Note that N is an integer greater than or equal to 1, the (2N-1)th row is a row in which a rolling shutter operation is performed, and the 2Nth row is a row in which a global shutter operation is performed.

Specifically, the angle detection unit 502 searches the RS image data for the edge closest to each edge included in the GS image data. For example, when searching for the edge closest to edge Eg3 included in the GS image data, the angle detection unit 502 first identifies a point Dg3 where line n and edge Eg3 intersect. Further, the angle detection unit 502 identifies a point Dr3 where the line n intersects with an edge Er3 included in the RS image data. Further, the angle detection unit 502 specifies a point Dr4 where the line n intersects with an edge Er4 included in the RS image data. Next, the angle detection unit 502 measures a distance L3 from point Dg3 to point Dr3 and a distance L4 from point Dg3 to point Dr4. The angle detection unit 502 then compares the distance L3 and the distance L4, and if the distance L3 is shorter than the distance L4, adds 1 to the evaluation value of the edge Er3 corresponding to the distance L3. Conversely, if the distance L4 is shorter than the distance L3, the angle detection unit 502 adds 1 to the evaluation value of the edge Er4 corresponding to the distance L4. Note that an evaluation value is set for each edge included in the RS image data, and the initial value of the evaluation value is 0, for example.

The angle detection unit 502 moves the line n by one pixel in a direction perpendicular to the line n. For example, the angle detection unit 502 moves the line n in the direction of the arrow shown in FIG. Then, the angle detection unit 502 updates the evaluation value of each edge included in the RS image data by repeating the above-described process every time the line n moves by one pixel. Then, the angle detection unit 502 evaluates each edge included in the RS image data when the above processing is performed on all points included in the edge Eg3 of the GS image data as the line n moves. Compare values. That is, the angle detection unit 502 compares the evaluation value of the edge Er3 and the evaluation value of the edge Er4 included in the RS image data. As a result, the angle detection unit 502 selects the edge with the highest evaluation value, and pairs the selected edge with edge Eg3 of the GS image data. In the example shown in FIG. 10, the angle detection unit 502 selects the edge Er3 of the RS image data and pairs the edge Er3 with the edge Eg3.

Note that in the example shown in FIG. 10, the number of edges included in the RS image data is two, but it may be three or more or one.

Furthermore, if the edge that is the image of the characteristic portion E shown in the GS image data is associated with the edge that is the image of the characteristic portion E shown in the RS image data, the edge pairing method can be used. is not limited to the example shown in FIG. For example, a correlation value between an edge of GS image data and each edge of RS image data may be calculated, and an edge having the highest correlation value may be paired with an edge of GS image data. In this case, for example, the angle detection unit 502 divides each image of GS image data and RS image data into a plurality of units, and calculates a correlation value between each unit of GS image data and each unit of GS image data. do. Then, for each unit indicating a part of the edge of the GS image data, the angle detection unit 502 identifies a unit having the highest correlation value with that unit from the RS image data. As a result, the edge included in the GS image data is paired with the edges indicated by the identified several units. Furthermore, as described above, edge pairing may be performed by machine learning such as deep learning, or may be performed using other known techniques.

FIG. 11 is a diagram for explaining detection of an angle between edges by the angle detection unit 502. Note that FIG. 11 is a diagram in which the respective images of GS image data and RS image data are superimposed to clearly show the angle between the edges.

For example, the angle detection unit 502 detects the angle θ between the paired edge Eg3 and edge Er3. In the example shown in FIG. 11, the edge Eg3 is arranged along the direction perpendicular to the line n. In this case, the angle detection unit 502 forms a right triangle having the points D3, Dg3, and Dr3 as vertices, where the edges Eg3 and Er3 intersect, respectively, and calculates the angle θ of the right triangle. That is, the angle detection unit 502 calculates the angle θ using the length a and the length b of the two right-angled sides of the right triangle, and θ=atan (a/b). Note that the length a is the distance from point Dg3 to point Dr3, which is the width of the right triangle, and the length b is the distance from point Dg3 to point D3, which is the height of the right triangle. .

Note that in the examples shown in FIGS. 10 and 11, one end of each of the two paired edges is at the same position. That is, in the example shown in FIG. 11, one end of edge Eg3 and one end of edge Er3 are both located at point D3. However, these ends may be located at separate locations. In this case, the angle detection unit 502 moves one of the edges, rearranges one end of each of the two edges at the same position, and detects the angle between the edges in the same manner as the example shown in FIG. You may.

Here, the angle θ between the edges depends not only on the distance to the object to be measured, but also on the speed of the vehicle 1000, that is, the moving speed of the camera section 480, and the shutter parameter set by the shutter control section 504. different. Furthermore, if the angle θ is too small or too large, the angle θ may include many errors. As a result, the distance to the object to be measured based on the angle θ may also include many errors. Therefore, the shutter control unit 504 adjusts the shutter parameters based on the vehicle speed indicated by the vehicle speed information output from the vehicle speed detection unit 600 so that the angle θ falls within an appropriate range suitable for deriving the distance. For example, the shutter control unit 504 adjusts the line scanning time (ie, 1H or horizontal period) used for rolling shutter operation based on the vehicle speed. That is, the faster the vehicle speed is, the shutter control unit 504 adjusts the line scanning time to be shorter, and the slower the vehicle speed is, the shutter control unit 504 adjusts the line scanning time to be longer. Thereby, the angle θ can be kept within an appropriate range. Furthermore, when two types of shutter operations that are performed simultaneously are periodically performed, if the vehicle speed is slow and the frame period is short, multiple pieces of GS image data that are similar to each other will be acquired; RS image data is likely to be acquired. That is, while a large amount of image data is acquired in a short period of time, it is difficult for useful changes to occur in the distances repeatedly derived from the image data. Therefore, the shutter control unit 504 may lengthen each frame period of the global shutter operation and the rolling shutter operation as the vehicle speed is lower.

That is, shutter control section 504 in this embodiment controls at least one of the global shutter operation and rolling shutter operation of camera section 480 based on the moving speed of camera section 480. This makes it possible to suppress the influence of the moving speed of the camera unit 480 on at least one of the GS image data and the RS image data. As a result, for example, even if the moving speed of the camera section 480 is too fast or too slow, the magnitude of the detected distortion can be kept within an appropriate range. Therefore, by using the shutter parameter indicating the content of control by the shutter control unit 504 and the magnitude of distortion within an appropriate range, it is possible to improve the accuracy of distance measurement.

Specifically, the shutter control unit 504 changes the frame period when controlling the global shutter operation, and changes at least one of the frame period and the horizontal period when controlling the rolling shutter operation. do. More specifically, when controlling the global shutter operation, the shutter control unit 504 changes the frame period to a longer period as the movement speed is slower, and when controlling the rolling shutter operation, changes the frame period to a longer period as the movement speed becomes slower. The slower the time, the longer at least one of the frame period and the horizontal period is changed. For example, if the moving speed is faster than the first threshold, the frame period and horizontal period are changed to shorter periods, and if the moving speed is slower than the second threshold (<first threshold), the frame period and horizontal period are changed to shorter periods. The period is changed to a longer period.

As described above, when the vehicle speed, that is, the moving speed of the camera section 480 is fast, the distortion tends to increase, and conversely, when the moving speed of the camera section 480 is slow, the distortion tends to become small. In contrast, in the ranging system 500 and the ranging device 1 according to the present embodiment, the slower the moving speed of the camera section 480, the longer the horizontal period is changed, for example. Therefore, when the moving speed of the camera unit is slow, distortion can be prevented from becoming too small. As a result, by using shutter parameters such as the horizontal period, it is possible to suppress errors included in the magnitude of detected distortion and improve the accuracy of distance measurement. Alternatively, in the ranging system 500 and the ranging device 1 according to one aspect of the present disclosure, the slower the moving speed of the camera unit 480, the longer the frame period is changed. Therefore, when the moving speed of the camera unit 480 is slow and two types of shutter operations are repeatedly performed, a plurality of first data that are similar to each other and a plurality of second data that are similar to each other are output in a short period of time. can be suppressed. This makes it possible to reduce the amount of data, reduce the memory storage capacity, and reduce the processing load.

[Processing of distance derivation part]
As described above, the distance deriving unit 503 derives the distance to the object to be measured using the table.

FIG. 12 is a diagram showing an example of a table used in the distance deriving unit 503.

For example, as shown in FIG. 12, table T1 shows, for each combination of angle, vehicle speed, and shutter parameter, the distance corresponding to that combination. The angle is the angle between the edges, and the vehicle speed is the speed of the vehicle 1000. The shutter parameters are parameters used for imaging by the camera unit 480. For example, table T1 indicates distance "L01" for a combination of angle "θ01", vehicle speed "v01", and shutter parameter "p01". Table T1 also shows distance "L02" for the combination of angle "θ02", vehicle speed "v02", and shutter parameter "p02".

The distance derivation unit 503 derives the distance to the object to be measured by referring to such table T1. Specifically, the distance derivation unit 503 first calculates the angle θ detected by the angle detection unit 502, the vehicle speed v indicated by the vehicle speed information output from the vehicle speed detection unit 600, and the angle θ set by the shutter control unit 504. The shutter parameter p is obtained. The distance deriving unit 503 then refers to the table T1 and finds from the table T1 the closest combination of angle θ, vehicle speed v, and shutter parameter p. For example, the distance deriving unit 503 treats the combination of the angle θ, the vehicle speed v, and the shutter parameter p as a vector of (θ, v, p), and similarly treats the combination included in the table T1 as a vector. Then, the distance deriving unit 503 finds a combination having the closest inter-vector distance to the vector (θ, v, p) from the table T1. The distance deriving unit 503 derives the distance associated with the found combination in the table T1 as the distance from the camera unit 480 to the object to be measured.

Note that if the shutter parameter is uniquely determined for the vehicle speed under the control of the shutter control unit 504, the shutter parameter may not be included in the combinations shown in the table T1.

[Processing flow]
FIG. 13 is a flowchart illustrating an example of the processing operation of the ranging system 500.

The shutter control unit 504 of the ranging system 500 acquires vehicle speed information from the vehicle speed detection unit 600 (step S1). Then, the shutter control unit 504 sets shutter parameters for the camera unit 480 based on the vehicle speed indicated by the vehicle speed information (step S2). For example, the shutter control unit 504 lengthens the shutter parameters 1H and 1V as the vehicle speed is lower, and conversely shortens the shutter parameters 1H and 1V as the vehicle speed increases.

The camera unit 480 captures an image based on the shutter parameters set in step S2. That is, the camera unit 480 performs a global shutter operation (ie, GS) and a rolling shutter operation (ie, RS) according to the shutter parameters. As a result, the camera unit 480 acquires GS image data and RS image data and outputs these image data (step S3).

Next, the straight line detection unit 501 detects one or more edges from each of the GS image data and RS image data output in step S3 (step S4). That is, the straight line detection unit 501 detects one or more straight edges from the GS image data, and one or more straight edges from the RS image data. The angle detection unit 502 then creates a pair of edges by associating an edge corresponding to one characteristic portion E included in the GS image data with an edge corresponding to that one characteristic portion E included in the RS image data. is determined (step S5). If a plurality of characteristic portions E are displayed as edges in each of the GS image data and the RS image data, the angle detection unit 502 determines a pair for each of the plurality of characteristic portions E.

Next, for each pair, the angle detection unit 502 detects the angle between the two edges included in the pair (step S6). Then, the distance deriving unit 503 calculates the distance of the characteristic portion E based on the angle detected in step S6, the shutter parameter set in step S2, and the vehicle speed indicated by the vehicle speed information acquired in step S1. (Step S7). Note that when the shutter parameter is fixed regardless of the vehicle speed, or when the shutter parameter is uniquely determined from the vehicle speed, the distance derivation unit 503 derives the distance based on the angle and the vehicle speed. .

In this manner, in this embodiment, the distance to the object to be measured is derived by one camera unit 480 that performs a global shutter operation and a rolling shutter operation. Therefore, the configuration of the ranging system 500 can be easily simplified compared to, for example, a device equipped with a plurality of cameras to obtain stereo images. In addition, since two types of shutter operations are performed simultaneously, the time required to derive the distance can be shortened compared to the case where imaging is performed multiple times at different timings to obtain a stereo image. Delays can be suppressed. Furthermore, the accuracy of distance measurement can be improved.

(Modification 1)
In the first embodiment, the distance to the object to be ranged is derived according to the combination of angle, vehicle speed, and shutter parameter. In this modification, the distance to the object to be ranged is derived according to a combination including not only the angle, vehicle speed, and shutter parameter but also the pixel position. In this case, since the distance is derived in consideration of the pixel position, the influence of distortion of the lens of the camera unit 480 can be suppressed, and a more accurate distance can be derived. Note that the pixel position is the position of the pixel 10 included in the pixel array PA.

FIG. 14 is a diagram showing an example of the relationship between the configuration of the angle detection section, the distance derivation section 503, the shutter control section 504, and the vehicle speed detection section 600 in this modification.

The angle detection section 502a according to this modification includes a pairing processing section 521, an angle processing section 522, and a pixel position detection section 523.

The pairing processing section 521 executes some of the processing of the angle detection section 502 according to the above embodiment. Part of that process is edge pairing. When pairing two edges, the pairing processing unit 521 outputs pair information indicating a pair of the paired edges to the angle processing unit 522 and the pixel position detection unit 523. Note that the pair information may indicate a plurality of pairs.

The angle processing section 522 executes some of the processing of the angle detection section 502 according to the above embodiment. Part of the process is detecting angles between edges. That is, upon acquiring the pair information from the pairing processing unit 521, the angle processing unit 522 detects the angle between two edges included in the pair indicated by the pair information. Then, the angle processing unit 522 outputs angle information indicating the angle to the distance deriving unit 503.

When the pixel position detection unit 523 acquires the pair information from the pairing processing unit 521, the pixel position detection unit 523 detects, for example, the midpoint of the edge included in the RS image data among the two edges included in the pair indicated by the pair information. Detect a certain pixel position. Then, the pixel position detection unit 523 outputs information indicating the pixel position to the distance derivation unit 503. Note that the pixel position detection unit 523 may detect, for example, the pixel position of the midpoint of the edge included in the GS image data among the two edges. Alternatively, the pixel position detection unit 523 may specify the midpoint of each of the two edges, and detect the midpoint of a line segment connecting the midpoints as the pixel position. Note that the pixel position indicates the pixel in the image indicated by the GS image data or the RS image data, that is, the position of the pixel 10 of the pixel array PA. For example, the pixel position is expressed as a coordinate position in a coordinate space, which has an origin at the upper left corner of the image (i.e., X=0, Y=0), and an axis along the horizontal direction of the image, X and the Y-axis, which is an axis along the vertical direction of the image.

The distance derivation unit 503 uses a table to derive the distance to the object to be measured.

FIG. 15 is a diagram showing an example of a table used in the distance deriving unit 503 in this modification.

For example, as shown in FIG. 15, the table T2 according to this modification shows, for each combination of angle, vehicle speed, shutter parameter, and pixel position, the distance corresponding to that combination. The angle, vehicle speed, and shutter parameters are the same as in the example of FIG. 12 of the first embodiment.

For example, table T2 indicates distance "L01" for a combination of angle "θ01", vehicle speed "v01", shutter parameter "p01", and pixel position "x01, y01". Table T2 also shows distance "L02" for the combination of angle "θ02", vehicle speed "v02", shutter parameter "p02", and pixel position "x02, y02".

The distance derivation unit 503 derives the distance to the object to be measured by referring to such table T2. Specifically, the distance derivation unit 503 first calculates the angle θ detected by the angle detection unit 502, the vehicle speed v indicated by the vehicle speed information output from the vehicle speed detection unit 600, and the angle θ set by the shutter control unit 504. The shutter parameter p and the pixel position d detected by the pixel position detection unit 523 are acquired. The distance deriving unit 503 then refers to the table T2 and finds from the table T2 the closest combination of angle θ, vehicle speed v, shutter parameter p, and pixel position d. For example, the distance deriving unit 503 treats the combination of angle θ, vehicle speed v, shutter parameter p, and pixel position d as a vector of (θ, v, p, d), and similarly the combinations included in table T2 are vectors. treated as Then, the distance deriving unit 503 finds a combination having the closest inter-vector distance to the vector (θ, v, p, d) from the table T2. The distance deriving unit 503 derives the distance associated with the found combination in the table T2 as the distance from the camera unit 480 to the object to be measured.

Note that in the above example, by using the pixel position to derive the distance, it is possible to suppress the influence of lens distortion and derive an appropriate distance. On the other hand, the influence of lens distortion may be suppressed in advance. In other words, the angle between the edges may be detected by taking lens distortion into consideration. In this case, it is possible to suppress the influence of lens distortion and derive an appropriate distance without directly using the pixel position to derive the distance.

Further, as in the example shown in FIG. 12, if the shutter parameters are uniquely determined for the vehicle speed by the control of the shutter control unit 504, the combinations shown in the table T2 do not include the shutter parameters. You don't have to.

(Modification 2)
In the first embodiment, shutter control unit 504 sets shutter parameters based on the vehicle speed of vehicle 1000. On the other hand, the shutter control unit according to this modification sets the shutter parameter based on the angle between the edges detected by the angle detection unit 502 instead of the vehicle speed.

FIG. 16 is a block diagram showing an example of the configuration of a ranging system according to this modification.

A distance measuring system 500a according to this modification includes a camera section 480 and a distance measuring device 1a. The distance measuring device 1a includes a distortion detection section 510, a distance derivation section 503, and a shutter control section 504a. That is, the distance measuring system 500a according to this modification includes a shutter control section 504a instead of the shutter control section 504 of the distance measuring system 500 according to the first embodiment.

The shutter control unit 504a acquires angle information from the angle detection unit 502 of the distortion detection unit 510 without acquiring vehicle speed information from the vehicle speed detection unit 600. The shutter control unit 504a then sets the shutter parameters of the camera unit 480 based on the angle between the edges indicated by the angle information. Further, the shutter control unit 504a outputs shutter information indicating the set shutter parameters to the distance derivation unit 503.

FIG. 17 is a diagram for explaining the relationship between the angle between edges and the shutter parameter. Note that (a) and (b) of FIG. 17 are diagrams in which the respective images of GS image data and RS image data are superimposed to clearly show the angle between the edges.

For example, when the vehicle speed of the vehicle 1000 is high, the angle information output by the angle detection unit 502 tends to indicate a large angle as shown in FIG. 17(a). In other words, the angle between the edge Eg included in the GS image data and the edge Er included in the RS image data appears large. In such a case, the shutter control unit 504a sets the line scanning time indicated by the shutter parameter to be shorter based on the angle information.

On the other hand, when the vehicle speed of vehicle 1000 is slow, the angle information output by angle detection section 502 tends to indicate a small angle, as shown in FIG. 17(b). In other words, the angle between the edge Eg included in the GS image data and the edge Er included in the RS image data appears small. In such a case, the shutter control unit 504a sets the line scanning time indicated by the shutter parameter to be longer based on the angle information.

Specifically, the shutter control unit 504a identifies the maximum angle among the angles between the plurality of edges indicated by the angle information. Then, when the maximum angle is larger than the first threshold value, the shutter control unit 504a shortens the current line scanning time by Δt time. On the other hand, if the maximum angle is smaller than the second threshold, the shutter control unit 504a changes the current line scanning time to be longer by Δt time. Note that the second threshold is a smaller angle than the first threshold.

For example, when a small angle between edges is detected, the detected angle may include a relatively large error. As a result, the distance derived from such an angle may also include a large error. However, in this modification, by adjusting the shutter parameters as described above, the angle used to derive the distance can be kept within an appropriate range, so errors in the distance derived from that angle can be avoided. Can be suppressed.

In this way, the shutter control unit 504a in this modification controls at least one of the global shutter operation and the rolling shutter operation of the camera unit 480 based on the magnitude of distortion detected by the distortion detection unit 510 instead of the vehicle speed. Control. As a result, at least one of the two types of shutter operations to be performed next is controlled based on the magnitude of distortion detected according to the two types of shutter operations, so that the shutter control unit 504a can It is possible to execute feedback control. As a result, for example, the magnitude of detected distortion can be kept within an appropriate range. Therefore, by using the shutter parameter indicating the content of control by the shutter control unit 504a and the magnitude of distortion within an appropriate range, it is possible to improve the accuracy of distance measurement. In other words, in this modification, it is possible to achieve the same effect as in the first embodiment, in which the shutter parameters are set according to the vehicle speed.

Note that in this modification, the distortion detection section 510 includes the angle detection section 502 in the first embodiment, but may also include the angle detection section 502a of the first modification. In this case, the distance derivation unit 503 derives the distance using table T2 shown in FIG. 15.

(Modification 3)
In the first embodiment, camera unit 480 simultaneously acquires GS image data and RS image data for each frame period. On the other hand, the camera unit 480 according to this modification executes a process of thinning out the acquired GS image data.

FIG. 18 is a diagram schematically showing an example of the timing at which each of GS image data and RS image data is acquired.

For example, the camera unit 480 according to the first embodiment acquires GS image data and RS image data at times t01, t02, and t03, respectively, as shown in FIG. 18(a). Here, when the vehicle speed of vehicle 1000 is slow, almost no change is seen in the images displayed in the plurality of continuously acquired GS image data. Therefore, as shown in FIG. 18(b), the camera unit 480 according to this modification acquires GS image data and RS image data at times t01 and t03, and acquires RS image data at times t02 and t04, respectively. Get only the data. In this way, the camera unit 480 thins out the acquired GS image data. As a result, the frame rate of GS image data is set lower than the frame rate of RS image data. For example, the shutter control unit 504 may increase the number of GS image data to be thinned out, that is, the number of GS image data that is not acquired, as the speed of the vehicle 1000 is lower. In other words, the shutter control unit 504 may lower the frame rate of the GS image data. Thereby, the transmission band used for outputting data from the camera section 480 can be kept low. As a result, it is possible to reduce the cost of the ranging system. Note that the lower the frame rate is, the longer the frame period is, and the longer the time interval at which GS image data is sequentially output.

In this manner, when controlling the global shutter operation, the shutter control unit 504 in this modification controls the time interval at which the first data is sequentially output. Specifically, when controlling the global shutter operation, the shutter control unit 504 makes the time interval longer as the movement speed is slower. For example, if the moving speed of the camera unit 480 is slow and GS image data is repeatedly output in a short cycle, it is difficult for differences to occur in the content shown by the GS image data, and the contribution to distance measurement is low. data may be output from the camera unit 480. However, in the ranging system 500 in this modified example, the slower the moving speed of the camera unit 480, the longer the period, which is the time interval at which GS image data is sequentially output, is set, so the above-mentioned distance measurement It is possible to suppress the output of data with low contribution. As a result, it is possible to reduce the amount of data, the memory storage capacity, and the processing load. Note that the above-mentioned time interval corresponds to a frame period.

(Other variations)
In Embodiment 1 and Modifications 1 to 3 thereof, the global shutter operation and the rolling shutter operation are executed simultaneously. However, as long as the accuracy of distance measurement is maintained and the delay time falls within a predetermined range, the timings at which these shutter operations are performed may be shifted.

Furthermore, in Embodiment 1 and Modifications 1 to 3 thereof, straight edges are detected, but if the magnitude of distortion can be detected, the shape of the edge is not limited to a straight line, and other shapes can be used. There may be. Furthermore, in the first embodiment and its first to third variations, the angle between edges is detected as the magnitude of distortion, but a parameter other than the angle may be used to detect the magnitude of distortion.

Further, in Embodiment 1 and its modifications 1 to 3, the lengths of the exposure periods of the global shutter operation and the rolling shutter operation are different, but the lengths of these exposure periods may be equal.

Further, in Embodiment 1 and Modifications 1 to 3 thereof, the distance measurement systems 500 and 500a are attached to the vehicle 1000, but the distance measurement system of the present disclosure is not limited to a vehicle, and can be applied to other moving bodies or It may be attached to a device or the like. For example, it may be installed in industrial equipment, robots, drones, etc.

Furthermore, when the camera unit 480 of the distance measuring system 500, 500a is not measuring distance, it may take an image by performing only one of the global shutter operation and the rolling shutter operation. For example, when the vehicle 1000 performs automatic parking, the camera unit 480 performs a global shutter operation and a rolling shutter operation to measure distance, and when the vehicle 1000 performs normal driving, it performs one of these operations. Do only one. Further, the shutter control unit 504 determines the mode of the vehicle 1000, and based on the determination result, switches whether to perform a global shutter operation, a rolling shutter operation, or only one of these operations. Good too. The mode of vehicle 1000 is a mode in which automatic parking is performed (that is, automatic parking mode) or a mode in which normal driving is performed (that is, normal driving mode). In this case, when the mode is switched to automatic parking mode by the driver of vehicle 1000, shutter control section 504 causes camera section 480 to perform a global shutter operation and a rolling shutter operation. Alternatively, when the reverse range is selected by the shift lever or select lever of vehicle 1000, shutter control section 504 causes camera section 480 to perform a global shutter operation and a rolling shutter operation.

Further, when the camera unit 480 executes only one of the global shutter operation and the rolling shutter operation, the camera unit 480 selectively performs one of the global shutter operation and the rolling shutter operation depending on the situation. You may switch. For example, if the speed of vehicle 1000 is high, camera unit 480 may perform a global shutter operation to suppress distortion of the displayed image. Furthermore, when the surroundings of the vehicle 1000 are darker than a predetermined brightness, for example at night, the camera section 480 may perform a rolling shutter operation, thereby suppressing noise occurring in the displayed image. Can be done.

Furthermore, although the camera unit 480 in the first embodiment and its first to third variations is an image sensor that outputs image data, it may be an event camera. An event camera is also called an event-based camera or an event-based sensor, and outputs a combination of pixel position and time information as data instead of image data. At least one of a global shutter operation and a rolling shutter operation may be performed by the event camera.

(Embodiment 2)
In this embodiment, an example of a specific aspect of the imaging device 100 included in the camera section 480 according to the first embodiment and its modification will be described in detail.

[Circuit configuration of imaging device]
First, the circuit configuration of the imaging device according to Embodiment 2 will be described using FIG. 19. FIG. 19 is a schematic diagram showing an exemplary circuit configuration of the imaging device according to this embodiment.

The imaging device 100 shown in FIG. 19 has a pixel array PA including a plurality of unit pixel cells 10 arranged in a matrix. FIG. 19 schematically shows an example in which four unit pixel cells 10 are arranged in a matrix of 2 rows and 2 columns. Needless to say, the number and arrangement of unit pixel cells 10 in the imaging device 100 are not limited to the example shown in FIG. 19.

Each unit pixel cell 10 is an example of a pixel included in the imaging device 100, and includes a photoelectric conversion section 13 and a signal detection circuit 14. As will be described later with reference to the drawings, the photoelectric conversion unit 13 has a photoelectric conversion layer sandwiched between two electrodes facing each other, and generates a signal by receiving incident light. The entire photoelectric conversion section 13 does not need to be an independent element for each unit pixel cell 10, and for example, a portion of the photoelectric conversion section 13 may span a plurality of unit pixel cells 10. Details of the specific structure of the photoelectric conversion section 13 will be described later.

The signal detection circuit 14 is a circuit that detects the signal generated by the photoelectric conversion section 13. In this example, signal detection circuit 14 includes a signal detection transistor 24 and an address transistor 26. Signal detection transistor 24 and address transistor 26 are typically field effect transistors (FETs). Here, an N-channel MOSFET (Metal Oxide Semiconductor Field Effect Transistor) is illustrated as the signal detection transistor 24 and the address transistor 26.

As schematically shown in FIG. 19, the control terminal (gate here) of the signal detection transistor 24 has an electrical connection with the photoelectric conversion section 13. Signal charges (specifically, holes or electrons) generated by the photoelectric conversion section 13 are accumulated in the charge storage node 41 between the gate of the signal detection transistor 24 and the photoelectric conversion section 13 . Charge storage node 41 is also called a floating diffusion node.

The photoelectric conversion section 13 of each unit pixel cell 10 further has a connection with a sensitivity control line 42. In the configuration illustrated in FIG. 19, the sensitivity control line 42 is connected to the voltage supply circuit 32 included in the imaging device 100.

The voltage supply circuit 32 can individually supply at least two types of voltages to the photoelectric conversion unit 13 for each row via the sensitivity control line 42 during operation of the imaging device 100. The voltage supply circuit 32 is not limited to a specific power supply circuit, and may be a circuit that generates a predetermined voltage, or a circuit that converts a voltage supplied from another power source into a predetermined voltage. good. As will be described in detail later, in one or more predetermined rows, the voltage supplied from the voltage supply circuit 32 to the photoelectric conversion unit 13 is switched between a plurality of different voltages, so that the photoelectric conversion unit 13 The start and end of accumulation of signal charges from the charge accumulation node 41 to the charge accumulation node 41 is controlled. In other words, switching between the exposure period and the non-exposure period is performed by switching the voltage supplied from the voltage supply circuit 32 to the photoelectric conversion unit 13 in one or more predetermined rows. Note that one or more predetermined rows are rows used for a global shutter operation described later, and one or more other rows are rows used for a rolling shutter operation described later. The voltage supply circuit 32 continuously supplies one of the two types of voltages to each of the other one or more rows described above during operation of the imaging device 100. An example of the operation of the imaging device 100 will be described later.

Each unit pixel cell 10 has a connection with a power supply line 40 that supplies power supply voltage VDD. As shown in the figure, the input terminal (for example, the drain) of the signal detection transistor 24 is connected to the power supply line 40. Since the power supply line 40 functions as a source follower power supply, the signal detection transistor 24 amplifies and outputs the signal generated by the photoelectric conversion section 13.

An input terminal (here, the drain) of the address transistor 26 is connected to an output terminal (here, the source) of the signal detection transistor 24 . The output terminal (source here) of the address transistor 26 is connected to one of a plurality of vertical signal lines 47 arranged for each column of the pixel array PA. The control terminal (gate here) of the address transistor 26 is connected to the address control line 46, and by controlling the potential of the address control line 46, the output of the signal detection transistor 24 is sent to the corresponding vertical signal line 47. It can be read out selectively.

In the illustrated example, the address control line 46 is connected to the vertical scanning circuit 36. The vertical scanning circuit 36 is also called a row scanning circuit. The vertical scanning circuit 36 selects a plurality of unit pixel cells 10 arranged in each row on a row-by-row basis by applying a predetermined voltage to the address control line 46. As a result, the signal of the selected unit pixel cell 10 is read out.

The vertical signal line 47 is an example of an output signal line into which signals from a plurality of unit pixel cells 10 are input, and is a main signal line that transmits pixel signals from the pixel array PA to peripheral circuits. A column signal processing circuit 37 is connected to the vertical signal line 47 . The column signal processing circuit 37 is also called a row signal accumulation circuit. The column signal processing circuit 37 performs noise suppression signal processing typified by correlated double sampling (CDS), analog-to-digital conversion (AD conversion), and the like. As illustrated, the column signal processing circuit 37 is provided corresponding to each column of unit pixel cells 10 in the pixel array PA.

A horizontal signal readout circuit 38 is connected to these column signal processing circuits 37 . The horizontal signal readout circuit 38 is also called a column scanning circuit. The horizontal signal reading circuit 38 sequentially reads signals from the plurality of column signal processing circuits 37 to the horizontal common signal line 49.

In the configuration illustrated in FIG. 19, the unit pixel cell 10 has a reset transistor 28. Reset transistor 28, like signal detection transistor 24 and address transistor 26, can be a field effect transistor, for example. In the following, unless otherwise specified, an example in which an N-channel MOSFET is used as the reset transistor 28 will be described. As illustrated, reset transistor 28 is connected between charge storage node 41 and reset voltage line 44 that supplies reset voltage Vr. A control terminal (gate in this case) of the reset transistor 28 is connected to a reset control line 48 . By controlling the potential of the reset control line 48, the potential of the charge storage node 41 can be reset to the reset voltage Vr. In this example, reset control line 48 is connected to vertical scanning circuit 36. Therefore, by applying a predetermined voltage to the reset control line 48 by the vertical scanning circuit 36, it is possible to reset the plurality of unit pixel cells 10 arranged in each row row by row.

In this example, a reset voltage line 44 that supplies reset voltage Vr to reset transistor 28 is connected to reset voltage source 34 . The reset voltage source 34 is also called a reset voltage supply circuit. The reset voltage source 34 only needs to have a configuration that can supply a predetermined reset voltage Vr to the reset voltage line 44 during operation of the imaging device 100, and similarly to the voltage supply circuit 32 described above, it can be used for a specific power supply circuit. Not limited. Each of voltage supply circuit 32 and reset voltage source 34 may be part of a single voltage supply circuit or may be independent and separate voltage supply circuits. Note that one or both of the voltage supply circuit 32 and the reset voltage source 34 may be part of the vertical scanning circuit 36. Alternatively, the sensitivity control voltage from the voltage supply circuit 32 and/or the reset voltage Vr from the reset voltage source 34 may be supplied to each unit pixel cell 10 via the vertical scanning circuit 36.

When holes are used as signal charges, it is also possible to use the ground of the signal detection circuit 14 as the reset voltage Vr. In this case, the voltage supply circuit (not shown in FIG. 19) that supplies the ground to each unit pixel cell 10 and the reset voltage source 34 can be shared. Furthermore, since the power supply line 40 and the reset voltage line 44 can be shared, the wiring in the pixel array PA can be simplified. However, using different voltages for the reset voltage Vr and the ground of the signal detection circuit 14 enables more flexible control of the imaging device 100.

The imaging device 100 includes a control circuit 39 that controls the voltage supply circuit 32 and the vertical scanning circuit 36. The control circuit 39 controls the voltage supply circuit 32 and the vertical scanning circuit 36 to cause the pixel array PA to simultaneously perform a global shutter operation and a rolling shutter operation in one imaging operation. In other words, the control circuit 39 causes the unit pixel cells 10 included in each of one or more predetermined rows to perform a global shutter operation in one imaging operation. While the global shutter operation is being performed, the control circuit 39 causes the unit pixel cells 10 included in each of the other one or more rows to perform a rolling shutter operation.

[Device structure of unit pixel cell]
Next, the device structure of the unit pixel cell 10 will be explained using FIGS. 20 and 21.

FIG. 20 is a schematic cross-sectional view showing an exemplary device structure of the unit pixel cell 10 of the imaging device 100 according to the present embodiment. FIG. 21 is a schematic plan view showing the relationship between the unit pixel cell 10 and the counter electrode 12 of the imaging device 100 according to the present embodiment.

In the configuration illustrated in FIG. 20, the above-described signal detection transistor 24, address transistor 26, and reset transistor 28 are formed on the semiconductor substrate 20. The semiconductor substrate 20 is not limited to a substrate whose entirety is a semiconductor. The semiconductor substrate 20 may be an insulating substrate provided with a semiconductor layer on the surface on which the photosensitive region is formed. Here, an example in which a P-type silicon (Si) substrate is used as the semiconductor substrate 20 will be described.

The semiconductor substrate 20 has impurity regions (here, N-type regions) 26s, 24s, 24d, 28d, and 28s, and an element isolation region 20t for electrical isolation between the unit pixel cells 10. Here, the element isolation region 20t is also provided between the impurity region 24d and the impurity region 28d. The element isolation region 20t is formed, for example, by implanting acceptor ions under predetermined implantation conditions.

Impurity regions 26s, 24s, 24d, 28d, and 28s are typically diffusion layers formed within semiconductor substrate 20. As schematically shown in FIG. 20, signal detection transistor 24 includes impurity regions 24s and 24d and a gate electrode 24g. The gate electrode 24g is, for example, a polysilicon electrode. The impurity region 24s functions as, for example, a source region of the signal detection transistor 24. The impurity region 24d functions as, for example, a drain region of the signal detection transistor 24. A channel region of the signal detection transistor 24 is formed between the impurity region 24s and the impurity region 24d.

Similarly, address transistor 26 includes impurity regions 26s and 24s and a gate electrode 26g connected to address control line 46 shown in FIG. 19. The gate electrode 26g is, for example, a polysilicon electrode. In this example, the signal detection transistor 24 and the address transistor 26 are electrically connected to each other by sharing an impurity region 24s. The impurity region 26s functions as, for example, a source region of the address transistor 26. Impurity region 26s has a connection with vertical signal line 47 shown in FIG.

Reset transistor 28 includes impurity regions 28d and 28s, and a gate electrode 28g connected to reset control line 48 shown in FIG. 19. The gate electrode 28g is, for example, a polysilicon electrode. The impurity region 28s functions as, for example, a source region of the reset transistor 28. Impurity region 28s has a connection with reset voltage line 44 shown in FIG. 19.

An interlayer insulating layer 50 is arranged on the semiconductor substrate 20 so as to cover the signal detection transistor 24, address transistor 26, and reset transistor 28. The interlayer insulating layer 50 is formed using an insulating material such as silicon oxide, silicon nitride, or tetraethyl orthosilicate (TEOS). As shown in FIG. 20, a wiring layer 56 may be placed in the interlayer insulating layer 50. The wiring layer 56 is typically made of metal such as copper, and may include wiring such as the above-mentioned vertical signal line 47 as a part thereof, for example. The number of insulating layers in the interlayer insulating layer 50 and the number of layers included in the wiring layer 56 arranged in the interlayer insulating layer 50 can be set arbitrarily, and are not limited to the example shown in FIG. 20.

The above-described photoelectric conversion section 13 is arranged on the interlayer insulating layer 50. In other words, in this embodiment, a plurality of unit pixel cells 10 forming the pixel array PA are formed on the semiconductor substrate 20. A plurality of unit pixel cells 10 arranged two-dimensionally on the semiconductor substrate 20 form a photosensitive region. The photosensitive area is also called a pixel area. The distance between two adjacent unit pixel cells 10 (ie, pixel pitch) may be, for example, about 2 μm.

The photoelectric conversion unit 13 includes a pixel electrode 11, a counter electrode 12, and a photoelectric conversion layer 15 disposed between them. In this example, the counter electrode 12 and the photoelectric conversion layer 15 are formed across a plurality of unit pixel cells 10.

The pixel electrode 11 is provided for each unit pixel cell 10, and is spatially separated from the pixel electrode 11 of the other adjacent unit pixel cell 10, so that it is electrically isolated from the pixel electrode 11 of the other unit pixel cell 10. are separated.

By controlling the potential of the counter electrode 12 with respect to the potential of the pixel electrode 11, one of the holes and electrons among the hole-electron pairs generated in the photoelectric conversion layer 15 by photoelectric conversion is transferred by the pixel electrode 11. can be collected. For example, when holes are used as signal charges, holes can be selectively collected by the pixel electrode 11 by setting the potential of the counter electrode 12 higher than that of the pixel electrode 11. Below, a case where holes are used as signal charges will be exemplified. Of course, it is also possible to use electrons as signal charges.

By applying an appropriate bias voltage between the counter electrode 12 and the pixel electrode 11, the pixel electrode 11 facing the counter electrode 12 absorbs the positive and negative charges generated by photoelectric conversion in the photoelectric conversion layer 15. Collect one. The pixel electrode 11 is formed of a metal such as aluminum or copper, a metal nitride, or polysilicon doped with impurities to provide conductivity.

The pixel electrode 11 may be a light-shielding electrode. For example, by forming a TaN electrode with a thickness of 100 nm as the pixel electrode 11, sufficient light shielding properties can be achieved. By making the pixel electrode 11 a light-shielding electrode, it is possible to suppress light that has passed through the photoelectric conversion layer 15 from entering the channel region or impurity region of the transistor formed in the semiconductor substrate 20. Note that the transistor formed on the semiconductor substrate 20 is, for example, at least one of the signal detection transistor 24, the address transistor 26, and the reset transistor 28.

A light shielding film may be formed within the interlayer insulating layer 50 using the above-mentioned wiring layer 56. By suppressing the incidence of light into the channel region of the transistor formed in the semiconductor substrate 20, it is possible to suppress a shift in characteristics of the transistor, such as a change in threshold voltage. Further, by suppressing the incidence of light into the impurity region formed in the semiconductor substrate 20, it is possible to suppress the incorporation of noise due to unintended photoelectric conversion in the impurity region. In this way, suppressing the incidence of light onto the semiconductor substrate 20 contributes to improving the reliability of the imaging device 100.

As schematically shown in FIG. 20, the pixel electrode 11 is connected to the gate electrode 24g of the signal detection transistor 24 via a plug 52, a wiring 53, and a contact plug 54. In other words, the gate of the signal detection transistor 24 has an electrical connection with the pixel electrode 11. The plug 52 and the wiring 53 are made of metal such as copper. The plug 52, the wiring 53, and the contact plug 54 constitute at least a part of the charge storage node 41 (see FIG. 19) between the signal detection transistor 24 and the photoelectric conversion section 13. The wiring 53 may be part of the wiring layer 56. Furthermore, the pixel electrode 11 is also connected to the impurity region 28d via a plug 52, a wiring 53, and a contact plug 55. In the configuration illustrated in FIG. 20, the gate electrode 24g, the plug 52, the wiring 53, the contact plugs 54 and 55 of the signal detection transistor 24, and the impurity region 28d, which is one of the source region and the drain region of the reset transistor 28, are connected to the pixel. It functions as a charge accumulation region that accumulates signal charges collected by the electrode 11.

As signal charges are collected by the pixel electrode 11, a voltage corresponding to the amount of signal charges accumulated in the charge accumulation region is applied to the gate of the signal detection transistor 24. Signal detection transistor 24 amplifies this voltage. The voltage amplified by the signal detection transistor 24 is selectively read out as a signal voltage via the address transistor 26.

The counter electrode 12 is typically a transparent electrode formed from a transparent conductive material. The counter electrode 12 is arranged on the side of the photoelectric conversion layer 15 on which light is incident. Therefore, the light that has passed through the counter electrode 12 is incident on the photoelectric conversion layer 15 . Note that the light detected by the imaging device 100 is not limited to light within the wavelength range of visible light (for example, 380 nm or more and 780 nm or less). "Transparent" in this specification means that at least a portion of the light in the wavelength range to be detected is transmitted, and it is not essential that the light be transmitted over the entire wavelength range of visible light. In this specification, all electromagnetic waves including infrared rays and ultraviolet rays are expressed as "light" for convenience. For the counter electrode 12, for example, transparent conductive oxide (TCO) such as ITO, IZO, AZO, FTO, SnO ₂ , TiO ₂ , ZnO ₂ or the like can be used.

In this embodiment, as shown in FIG. 21, the counter electrodes 12 are continuous between a plurality of unit pixel cells 10 within the same pixel block 10b, and separated between different pixel blocks 10b. Here, each pixel block 10b is composed of a plurality of unit pixel cells 10 belonging to the same row. In other words, the plurality of unit pixel cells 10 constitute a plurality of pixel blocks 10b for each row.

The counter electrode 12 has a plurality of electrode pieces 12b in one-to-one correspondence with the plurality of pixel blocks 10b. The plurality of electrode pieces 12b are provided for each row and are separated from each other. Each electrode piece 12b has a rectangular shape that is elongated in the row direction. For example, the electrode piece 12b covers each pixel electrode 11 of a plurality of unit pixel cells 10 belonging to the same row. An insulating layer may be arranged between adjacent electrode pieces 12b.

As described with reference to FIG. 19, the counter electrode 12 has a connection with the sensitivity control line 42 connected to the voltage supply circuit 32. The sensitivity control line 42 is provided for each electrode piece 12b. Therefore, for each electrode piece 12b, a desired magnitude of sensitivity control voltage is applied from the voltage supply circuit 32 via the corresponding sensitivity control line 42 to the plurality of unit pixel cells 10 belonging to the corresponding pixel block 10b. It is possible to apply the

As will be described in detail later, the voltage supply circuit 32 supplies different voltages to each electrode piece 12b of the counter electrode 12 during the exposure period and the non-exposure period. In this specification, the "exposure period" refers to a period for accumulating signal charges, which are either positive or negative charges generated by photoelectric conversion, in a charge accumulation region, and may also be referred to as a "charge accumulation period." good. Furthermore, in this specification, a period during which the imaging device is in operation and other than the exposure period is referred to as a "non-exposure period." Note that the "non-exposure period" is not limited to a period in which light is blocked from entering the photoelectric conversion unit 13, but may include a period in which the photoelectric conversion unit 13 is irradiated with light. Furthermore, the "non-exposure period" includes a period in which signal charges are unintentionally accumulated in the charge accumulation region due to the occurrence of parasitic sensitivity.

The photoelectric conversion layer 15 is located between the pixel electrode 11 and the counter electrode 12, and converts light into signal charges. Specifically, the photoelectric conversion layer 15 receives incident light and generates hole-electron pairs. Either the holes or electrons generated are signal charges.

The photoelectric conversion layer 15 is made of, for example, an organic material. The organic material is, for example, an organic semiconductor material. As the organic semiconductor material, for example, a material containing tinnaphthalocyanine having an absorption wavelength in the near-infrared region can be used, but the material is not limited thereto. As the photoelectric conversion layer 15, one or more types of photoelectric conversion materials having an absorption wavelength in a desired wavelength range can be used.

For example, the photoelectric conversion layer 15 may include a p-type semiconductor layer, an n-type semiconductor layer, and a mixed layer located between the p-type semiconductor layer and the n-type semiconductor layer. The p-type semiconductor layer is formed using, for example, a donor organic semiconductor material. The n-type semiconductor layer is formed using, for example, an acceptor organic semiconductor material. The mixed layer is, for example, a bulk heterojunction structure layer of a p-type semiconductor and an n-type semiconductor. The bulk heterojunction layer is described in detail in, for example, Patent Document 4 (Japanese Patent No. 5,553,727), and the disclosure content of Patent Document 4 is fully incorporated herein for reference.

Further, the photoelectric conversion layer 15 may include one or more functional layers other than the layer formed using the photoelectric conversion material. For example, the photoelectric conversion layer 15 may include at least one of a hole blocking layer, an electron blocking layer, a hole transporting layer, and an electron transporting layer as a functional layer.

Note that, similarly to the counter electrode 12, the photoelectric conversion layer 15 may be provided separately for each predetermined region. For example, the photoelectric conversion layer 15 may be provided separately for each unit pixel cell 10 or for each electrode piece 12b.

[Photocurrent characteristics in photoelectric conversion layer]
Next, the photocurrent characteristics in the photoelectric conversion layer will be explained using FIG. 22.

FIG. 22 is a graph showing an example of the photocurrent characteristics of the photoelectric conversion layer 15 of the imaging device 100 according to the present embodiment. In FIG. 22, the thick solid line graph shows an exemplary current-voltage characteristic (IV characteristic) of the photoelectric conversion layer 15 in a state where it is irradiated with light. Note that in FIG. 22, an example of the IV characteristic in a state where no light is irradiated is also shown by a thick broken line.

FIG. 22 shows changes in current density between the two main surfaces of the photoelectric conversion layer 15 when the bias voltage applied between the two main surfaces is changed under constant illuminance. In this specification, forward direction and reverse direction in bias voltage are defined as follows. When the photoelectric conversion layer 15 has a junction structure of a layered p-type semiconductor and a layered n-type semiconductor, a bias voltage is applied in the forward direction such that the potential of the p-type semiconductor layer is higher than that of the n-type semiconductor layer. is defined as the bias voltage of On the other hand, a bias voltage such that the potential of the p-type semiconductor layer is lower than that of the n-type semiconductor layer is defined as a reverse bias voltage.

Even when an organic semiconductor material is used, the forward direction and reverse direction can be defined in the same way as when an inorganic semiconductor material is used. When the photoelectric conversion layer 15 has a bulk heterojunction structure, as shown in the above-mentioned Patent Document 4, one of the two main surfaces of the bulk heterojunction structure facing the electrode has more than an n-type semiconductor. More p-type semiconductors appear on the other surface, and more n-type semiconductors than p-type semiconductors appear on the other surface. Therefore, the bias voltage is set in the forward direction so that the potential on the main surface side where more p-type semiconductor appears than n-type semiconductor is higher than the potential on the main surface side where more n-type semiconductor appears than p-type semiconductor. Defined as bias voltage.

As shown in FIG. 22, the photocurrent characteristics of the photoelectric conversion layer 15 are roughly characterized by three voltage ranges. The first voltage range is a reverse bias voltage range, and is a voltage range in which the absolute value of the output current density increases as the reverse bias voltage increases. The first voltage range can be said to be a voltage range in which the photocurrent increases as the bias voltage applied between the main surfaces of the photoelectric conversion layer 15 increases. The second voltage range is a forward bias voltage range, and is a voltage range in which the output current density increases as the forward bias voltage increases. That is, the second voltage range is a voltage range in which the forward current increases as the bias voltage applied between the main surfaces of the photoelectric conversion layer 15 increases. The third voltage range is a voltage range between the first voltage range and the second voltage range.

The first voltage range, the second voltage range, and the third voltage range can be distinguished by the slope of the graph of the photocurrent characteristics when linear vertical and horizontal axes are used. For reference, in FIG. 22, the average slope of the graph in each of the first voltage range and the second voltage range is shown by a broken line L1 and a broken line L2, respectively. As illustrated in FIG. 22, the rate of change in output current density with respect to increase in bias voltage in the first voltage range, second voltage range, and third voltage range is different from each other. The third voltage range is defined as a voltage range in which the rate of change of the output current density with respect to the bias voltage is smaller than the rate of change in the first voltage range and the rate of change in the second voltage range. Alternatively, the third voltage range may be determined based on the position of the rise (fall) in the graph showing the IV characteristic. The third voltage range is typically greater than -1V and less than +1V. In the third voltage range, even if the bias voltage is changed, the current density between the main surfaces of the photoelectric conversion layer 15 hardly changes. As illustrated in FIG. 22, in the third voltage range, the absolute value of the current density is typically 100 μA/cm ² or less.

[Example of configuration of voltage supply circuit 32 and operation of imaging device 100]
FIG. 23 is a schematic plan view showing the relationship between the unit pixel cell 10 and the counter electrode 12 of the imaging device 100 according to the present embodiment, and an example of the configuration of the voltage supply circuit 32.

The imaging device 100 according to this embodiment includes a plurality of pixel blocks 10b, as shown in FIG. 21. Here, the plurality of pixel blocks 10b include, as shown in FIG. 23, a plurality of first pixel blocks 10bg and a plurality of second pixel blocks 10br. The plurality of first pixel blocks 10bg are used for global shutter operation and correspond to one or more predetermined rows described above. The plurality of second pixel blocks 10br are used for rolling shutter operation and correspond to the other one or more rows described above. Each of the plurality of first pixel blocks 10bg and each of the plurality of second pixel blocks 10br are arranged alternately, as shown in FIG. 23. That is, one second pixel block 10br is arranged between two adjacent first pixel blocks 10bg, and one first pixel block 10bg is arranged between two adjacent second pixel blocks 10br. There is. In other words, along the direction in which the plurality of pixel blocks 10b are arranged, the first pixel blocks 10bg are arranged in even numbers, and the second pixel blocks 10br are arranged in odd numbers.

The voltage supply circuit 32 includes a first voltage supply circuit 32G for global shutter operation and a second voltage supply circuit 32R for rolling shutter operation. The first voltage supply circuit 32G is connected to each electrode piece 12b of the plurality of first pixel blocks 10bg via a first sensitivity control line 42g of the sensitivity control lines 42. In other words, the voltage supply circuit 32 is electrically connected to the counter electrode 12 corresponding to each of the plurality of first pixel blocks 10bg.

Such a first voltage supply circuit 32G controls each unit pixel cell 10 belonging to the plurality of first pixel blocks 10bg so that the plurality of first pixel blocks 10bg performs a global shutter operation according to the control by the control circuit 39. supply voltage to. In other words, the control circuit 39 causes the voltage supply circuit 32 to perform a global shutter operation for the plurality of first pixel blocks 10bg. In the global shutter operation, an exposure period is formed by applying a first voltage between the counter electrode 12 and the pixel electrode 11 at the same timing to each of the plurality of first pixel blocks 10bg, and This is an operation in which a non-exposure period is formed by applying a second voltage between the pixel electrode 11 and the pixel electrode 11 at the same timing. The first voltage is, for example, a voltage included in the first voltage range in FIG. 22. The second voltage is, for example, a voltage included in the third voltage range in FIG. 22. Therefore, in each of the plurality of first pixel blocks 10bg, when the first voltage is applied between the counter electrode 12 and the pixel electrode 11, signal charges move from the photoelectric conversion layer to the pixel electrode. In other words, an exposure period is formed. Furthermore, when the second voltage is applied between the counter electrode 12 and the pixel electrode 11, movement of signal charges from the photoelectric conversion layer 15 to the pixel electrode 11 is suppressed. In other words, a non-exposure period is formed.

The second voltage supply circuit 32R is connected to each electrode piece 12b of the plurality of second pixel blocks 10br via a second sensitivity control line 42r of the sensitivity control lines 42. In other words, the voltage supply circuit 32 is electrically connected to the counter electrode 12 corresponding to each of the plurality of second pixel blocks 10br. Such a second voltage supply circuit 32R supplies a voltage to each unit pixel cell 10 belonging to the plurality of second pixel blocks 10br so that the plurality of second pixel blocks 10br perform a rolling shutter operation. For example, the second voltage supply circuit 32R continuously applies the first voltage between the opposing electrode 12 and the pixel electrode 11 to each of the plurality of second pixel blocks 10br. In a state where such a first voltage is continuously applied, the control circuit 39 performs a rolling shutter operation on each of the plurality of second pixel blocks 10br by controlling the vertical scanning circuit 36, for example. let

Next, an example of the operation of the imaging device 100 will be described using FIG. 24 while referring to FIGS. 19 to 23 as appropriate. For the sake of simplicity, in the following description, an example of the operation will be described when the total number of rows of unit pixel cells 10 included in the pixel array PA is 8 rows. The total 8 rows are 4 rows from GS_0 row to GS_3 row, each corresponding to the first pixel block 10bg, and 4 rows from RS_0 row to RS_3 row, each corresponding to the second pixel block 10br. It is the sum total. Note that the GS_0th row, the GS_1st row, the GS_2nd row, and the GS_3rd row are even-numbered rows. Furthermore, the RS_0th row, the RS_1st row, the RS_2nd row, and the RS_3rd row are odd-numbered rows, respectively. That is, these rows are arranged in the order of RS_0th row, GS_0th row, RS_1st row, GS_1st row, RS_2nd row, GS_2nd row, RS_3rd row, and GS_3rd row.

FIG. 24 is a diagram for explaining an example of the operation of the imaging device 100 according to this embodiment. FIG. 24 shows the falling or rising timing of the synchronization signal, the temporal change in the magnitude of the bias voltage applied to the photoelectric conversion layer 15, and the timing of reset and exposure in each row of the pixel array PA. ing.

More specifically, the top graph in FIG. 24 shows the timing of the fall or rise of the vertical synchronization signal Vss. The second graph from the top shows the timing of the fall or rise of the horizontal synchronization signal Hss. The pulse interval of the horizontal synchronization signal Hss is one horizontal period represented by 1H. The pulse interval of the vertical synchronization signal Vss is one vertical period expressed by 1V. One vertical period corresponds to one frame period.

The graph indicated by ITO_R below these graphs shows the time of the bias voltage Vb applied from the second voltage supply circuit 32R to the electrode piece 12b of each second pixel block 10br via the second sensitivity control line 42r. This shows an example of a change. Furthermore, the graph represented by ITO_G is an example of a temporal change in the bias voltage Vb applied from the first voltage supply circuit 32G to the electrode piece 12b of each first pixel block 10bg via the first sensitivity control line 42g. It shows. ITO_R can be considered as an electrode piece 12b provided for each second pixel block 10br. ITO_G can be considered as an electrode piece 12b provided for each first pixel block 10bg.

Further below, a chart represented by GS_0 to GS_3 schematically shows the timing of reset and exposure in each first pixel block 10bg of the pixel array PA. Further below, a chart represented by RS_0 to RS_3 schematically shows the timing of reset and exposure in each second pixel block 10br of the pixel array PA. Specifically, GS_0 to GS_3 each represent the operation of the unit pixel cell 10 belonging to the first pixel block 10bg corresponding to ITO_G. RS_0 to RS_3 each represent the operation of the unit pixel cell 10 belonging to the second pixel block 10br corresponding to ITO_R. For example, GS_0 indicates the operation of a plurality of unit pixel cells 10 belonging to the GS_0th row, which is an even numbered row of the pixel array PA. Similarly to GS_0, GS_1, GS_2, and GS_3 also show the operations of a plurality of unit pixel cells 10 belonging to the even-numbered rows of the pixel array PA, ie, the GS_1-th row, the GS_2-th row, and the GS_3-th row. The operations of the plurality of unit pixel cells 10 belonging to these rows are controlled by changes in the voltage indicated by ITO_G. RS_0 indicates the operation of a plurality of unit pixel cells 10 belonging to the RS_0th row, which is an odd-numbered row of the pixel array PA. Similarly to RS_0, RS_1, RS_2, and RS_3 also show the operations of the plurality of unit pixel cells 10 belonging to the odd-numbered rows of the pixel array PA, ie, the RS_1, RS_2, and RS_3 rows. The operations of the plurality of unit pixel cells 10 belonging to these rows are controlled by the voltage indicated by ITO_R.

Each chart represented by GS_0 to GS_3 and RS_0 to RS_3 represents the content of the operation depending on the presence or absence and type of shading within the rectangular frame. Specifically, a white rectangle with no shading indicates an exposed state. That is, the period occupied by the white-painted rectangle (hereinafter simply referred to as "white-painted period") is the exposure period of the unit pixel cell 10 belonging to the corresponding row. A rectangle with diagonal hatching and a rectangle with dot hatching both indicate that they are not in an exposed state. That is, the period occupied by the rectangle with diagonal hatching or the rectangle with dot hatching is the non-exposure period of the unit pixel cell 10 belonging to the corresponding row. Of these, the period occupied by the rectangle shaded with dots (hereinafter simply referred to as "dot period") is the period during which signal readout, reset, and reset readout of the unit pixel cells 10 belonging to the corresponding row are performed. It is. In other words, the dot period is the total period of a readout period for reading signals and a reset period for resetting the charge storage region and reading after reset.

In this embodiment, the control circuit 39 controls the voltage supply circuit 32 and the vertical scanning circuit 36 to cause the pixel array PA to simultaneously perform a global shutter operation and a rolling shutter operation for each frame period. . In the global shutter operation, the timing of switching from the exposure period to the non-exposure period is the same in each of the first pixel blocks 10bg from the GS_0 row to the GS_3 row, and the timing of switching from the non-exposure period to the exposure period is the same. are the same. On the other hand, in the rolling shutter operation, the timing of switching from the exposure period to the non-exposure period and the timing of switching from the non-exposure period to the exposure period are different in each second pixel block 10br from the RS_0 row to the RS_3 row. .

Specifically, in the global shutter operation, the first voltage supply circuit 32G forms an exposure period by simultaneously applying the first voltage to the counter electrode 12 of each first pixel block 10bg. The first voltage is, for example, a voltage included in the first voltage range in FIG. 22. Further, the first voltage supply circuit 32G forms a non-exposure period by simultaneously applying the second voltage to the opposing electrode 12 of the first pixel block 10bg. The second voltage is, for example, a voltage included in the third voltage range in FIG. 22.

On the other hand, in the rolling shutter operation, the second voltage supply circuit 32R continuously applies the first voltage to the opposing electrode 12 of each second pixel block 10br during one frame period. Then, the vertical scanning circuit 36 performs signal readout, reset, and reset readout processing, thereby forming an exposure period and a non-exposure period. Further, the vertical scanning circuit 36 performs the processing at different timings for each second pixel block 10br. As a result, in each second pixel block 10br, the timing of switching from the exposure period to the non-exposure period and the timing of switching from the non-exposure period to the exposure period are different.

Hereinafter, the global shutter operation in the GS_0th row to the GS_3th row will be described in detail first.

In acquiring an image, first, the charge storage region of each unit pixel cell 10 included in each first pixel block 10bg in the pixel array PA is reset. For example, as shown in FIG. 24, resetting of the plurality of unit pixel cells 10 belonging to the GS_0th row is started at time t0 based on the vertical synchronization signal Vss. Note that if exposure has been performed in the immediately previous frame, the pixel signal is read out before the reset, and the post-reset signal (that is, the reset signal) is read out after the reset as well.

In resetting the unit pixel cell 10 belonging to the GS_0th row, by controlling the potential of the address control line 46 of the GS_0th row, the address transistor 26 whose gate is connected to the address control line 46 is turned on. Furthermore, by controlling the potential of the reset control line 48 in the GS_0 row, the reset transistor 28 whose gate is connected to the reset control line 48 is turned on. Thereby, charge storage node 41 and reset voltage line 44 are connected, and reset voltage Vr is supplied to the charge storage region. That is, the potentials of the gate electrode 24g of the signal detection transistor 24 and the pixel electrode 11 of the photoelectric conversion section 13 are reset to the reset voltage Vr. Thereafter, the reset pixel signal is read out from the unit pixel cell 10 in the GS_0th row via the vertical signal line 47. The pixel signal obtained at this time is a pixel signal corresponding to the magnitude of the reset voltage Vr. After reading the pixel signal, the reset transistor 28 and address transistor 26 are turned off.

In this embodiment, as schematically shown in FIG. 24, pixels belonging to each row from the GS_0th row to the GS_3th row are sequentially reset row by row in accordance with the horizontal synchronization signal Hss. Hereinafter, the interval between pulses of the horizontal synchronizing signal Hss, in other words, the period from when a certain row is selected until when the next row is selected may be referred to as a "1H period." In this example, the period from time t0 to time t1 corresponds to the 1H period.

Here, in the period from time t0 to time t4, the voltage V3 such that the potential difference between the pixel electrode 11 and the counter electrode 12 falls within the above-mentioned third voltage range is applied to the counter electrode 12 of each first pixel block 10bg. is applied to. That is, the voltage V3 is applied from the first voltage supply circuit 32G to the electrode pieces 12b of the counter electrode 12 in each row from GS_0th row to GS_3th row. In other words, during the period from time t0 to start time t4 of the exposure period, the photoelectric conversion layer 15 of the photoelectric conversion unit 13 is in a state where a bias voltage in the third voltage range is applied.

When a bias voltage in the third voltage range is applied to the photoelectric conversion layer 15, almost no signal charge moves from the photoelectric conversion layer 15 to the charge storage region. This is because when a bias voltage in the third voltage range is applied to the photoelectric conversion layer 15, most of the positive and negative charges generated by light irradiation are quickly recombined and collected by the pixel electrode 11. It is presumed that this is because it disappears before it disappears. Therefore, in a state where a bias voltage in the third voltage range is applied to the photoelectric conversion layer 15, even if light is incident on the photoelectric conversion layer 15, almost no signal charge is accumulated in the charge accumulation region. Therefore, occurrence of unintended sensitivity during periods other than the exposure period is suppressed. Note that unintended sensitivity is also called parasitic sensitivity. In this way, by setting the bias voltage to the photoelectric conversion layer 15 in the third voltage range, the sensitivity can be quickly reduced to 0. When the bias voltage is in the third voltage range, signal charges are not collected by the pixel electrode 11, so the state is the same as in no exposure. The period in which the bias voltage is in the third voltage range is a non-exposure period, as shown in FIG. Note that the voltage V3 for applying the bias voltage in the third voltage range is, for example, 0V, but is not limited to 0V. Voltage V3 is an example of a second voltage, and is a voltage for forming a non-exposure period.

In the GS_0th row to the GS_3th row, an exposure period is started at time t4 when the voltage applied to the electrode piece 12b corresponding to ITO_G is switched to a voltage Ve different from the voltage V3. The exposure period is started by the voltage supply circuit 32 switching the voltage applied to the electrode piece 12b of the counter electrode 12 to a voltage Ve different from the voltage V3. The voltage Ve is an example of a first voltage, and is a voltage for forming an exposure period. The voltage Ve is, for example, a voltage such that the potential difference between the pixel electrode 11 and the counter electrode 12 falls within the above-described first voltage range. The voltage Ve is, for example, about 10V. By applying the voltage Ve to the counter electrode 12, signal charges (holes in this example) in the photoelectric conversion layer 15 are collected by the pixel electrode 11 and accumulated in the charge accumulation region. That is, signal charges are accumulated in the charge accumulation node 41.

The exposure period ends when the first voltage supply circuit 32G switches the voltage applied to the electrode piece 12b corresponding to ITO_G to the voltage V3 again at time t8. In this way, in the GS_0th row to GS_3th row in this embodiment, the voltage applied to the electrode piece 12b of the counter electrode 12 is switched between the voltage V3 and the voltage Ve, so that the exposure period and the non-exposure period are changed. can be switched. Furthermore, as can be seen from FIG. 24, in this embodiment, resets are sequentially performed at different timings in the GS_0th row to GS_3th row, and the exposure period and non-exposure period are switched at the same timing.

Based on the horizontal synchronization signal Hss, signals from the unit pixel cells 10 belonging to each row of the pixel array PA are read out. In this example, from time t8, signals from the unit pixel cells 10 belonging to each of the rows GS_0 to GS_3 are sequentially read out row by row. Hereinafter, the period from when a unit pixel cell 10 belonging to a certain row is selected to when a unit pixel cell 10 belonging to that row is selected again may be referred to as a "1V period."

In this example, the period from time t0 to time t8 corresponds to a 1V period. When reading signals from the unit pixel cells 10 belonging to the GS_0th row after the exposure period ends, the address transistor 26 of the GS_0th row is turned on. As a result, a pixel signal corresponding to the amount of charge accumulated in the charge accumulation region during the exposure period is output to the vertical signal line 47. Following reading of the pixel signal, the reset transistor 28 is turned on to reset the unit pixel cell 10. After the reset, the reset transistor 28 is turned off again. Then, the signal after turning off the reset transistor 28 (ie, the reset signal) is read out. After reading the reset signal, the address transistor 26 is turned off. By taking the difference between the pixel signal from the unit pixel cell 10 belonging to each row of the pixel array PA and the reset signal, a signal from which fixed noise has been removed can be obtained.

In this manner, in this embodiment, the exposure period and the non-exposure period are switched by switching the voltage applied to the counter electrodes 12 in the GS_0 to GS_3 rows between the voltage V3 and the voltage Ve. As can be seen from FIG. 24, the start and end of the exposure period in rows GS_0 to GS_3 are performed simultaneously in all pixels included in the plurality of first pixel blocks 10bg. As a result, a global shutter operation is performed in the imaging device 100. Furthermore, the exposure period can be freely selected from a 1H period to approximately a 1V period (specifically, a period of 1V-1H), excluding the period during which a read operation is performed.

On the other hand, a rolling shutter operation is performed in the RS_0th row to the RS_3th row. That is, as shown in the graph of ITO_R, the second voltage supply circuit 32R always applies the voltage Ve to the counter electrode 12 in each row from the RS_0th row to the RS_3th row. Thereby, signal charges (holes in this example) in the photoelectric conversion layer 15 are collected by the pixel electrode 11 and accumulated in the charge accumulation region. That is, signal charges are accumulated in the charge accumulation node 41. This signal charge accumulation is performed during periods excluding signal readout, reset, and reset readout. In the rolling shutter operation, the period during which signal charges are accumulated is the exposure period, and the other period, that is, the period during which signal readout, reset, and reset readout are performed, is the non-exposure period. In the RS_0th row to the RS_3th row in this embodiment, non-exposure periods start and end at mutually different timings from time t0 to time t4. Specifically, in the RS_0th row, a non-exposure period is formed from time t0 to time t1, and an exposure period starts from time t1. In the RS_1 row, a non-exposure period is formed from time t1 to time t2, delayed by 1H period compared to the RS_0 row, and the exposure period starts from time t2. Similarly, in the RS_2 row, a non-exposure period is formed from time t2 to time t3, delayed by 1H period compared to the RS_1 row, and the exposure period starts from time t3. In the RS_3 row, a non-exposure period is formed from time t3 to time t4, delayed by 1H period compared to the RS_2 row, and the exposure period starts from time t4. In this way, in the RS_0th row to the RS_3th row, the exposure period and the non-exposure period are performed at different timings. That is, a rolling shutter operation is performed.

In this way, in this embodiment, by dividing the counter electrode 12 into two groups and controlling each group independently, it is possible to simultaneously obtain an image based on the rolling shutter operation and an image based on the global shutter operation. It is. Furthermore, each of the plurality of unit pixel cells 10 includes a pixel electrode 11, a counter electrode 12, a photoelectric conversion layer 15, and a charge storage region, and also has a dedicated circuit for global shutter operation and a dedicated circuit for rolling shutter operation. It does not have any circuits etc. Therefore, the configuration of those unit pixel cells 10 can be simplified. As a result, a plurality of images with different shutter operations can be obtained using a plurality of simply configured unit pixel cells 10, respectively.

Further, in this embodiment, the counter electrode 12 is continuous between a plurality of unit pixel cells 10 in the same pixel block 10b among the plurality of pixel blocks 10b, and is separated between different pixel blocks 10b. The voltage supply circuit 32 is electrically connected to the counter electrode 12 corresponding to each of the plurality of first pixel blocks 10bg. Thereby, the voltage supply circuit 32 switches the potential of the counter electrode 12 for each of the plurality of first pixel blocks 10bg, so that the counter electrode 12 of each unit pixel cell 10 included in the first pixel block 10bg and the pixel The voltage between the electrode 11 can be switched between a first voltage and a second voltage. As a result, it is possible to easily match the timing of switching between the exposure period and the non-exposure period performed in each of the plurality of first pixel blocks 10bg.

Further, in the present embodiment, in each of the plurality of first pixel blocks 10bg, when the first voltage is applied between the counter electrode 12 and the pixel electrode 11, signal charges are transferred from the photoelectric conversion layer 15 to the pixel electrode. Move to 11. When the second voltage is applied between the counter electrode 12 and the pixel electrode 11, movement of signal charges from the photoelectric conversion layer 15 to the pixel electrode 11 is suppressed. As a result, when the first voltage is applied between the counter electrode 12 and the pixel electrode 11, an exposure period is formed, and when the second voltage is applied between the counter electrode 12 and the pixel electrode 11, an exposure period is formed. , a non-exposure period is formed. Therefore, by switching the voltage between the counter electrode 12 and the pixel electrode 11, it is possible to appropriately switch between the exposure period and the non-exposure period.

Furthermore, in this embodiment, each of the plurality of pixel blocks 10b is made up of a plurality of unit pixel cells 10 belonging to the same one row. This allows the shutter operation to be controlled independently for each row. Furthermore, the image obtained by the global shutter operation and the image obtained by the rolling shutter operation can be configured on a line-by-line basis, and high image quality can be achieved.

Further, in this embodiment, each of the plurality of first pixel blocks 10bg and each of the plurality of second pixel blocks 10br are arranged alternately. Thereby, it is possible to suppress positional deviation between the image obtained by the global shutter operation and the image obtained by the rolling shutter operation.

In the example shown in FIG. 24, the period of each dot in the GS_0th row and the RS_0th row is the same, and both are periods from time t0 to time t1. Similarly, the dot periods in the GS_1-th row and the RS_1-th row are also the same, and both are the period from time t1 to time t2. The period of each dot in the GS_2 row and the RS_2 row is also the same, and the period of each dot in the GS_3 row and the RS_3 row is also the same. In other words, the period of dots in a row corresponding to one first pixel block 10bg is the same as the period of dots in a row corresponding to one second pixel block 10br.

However, in the configuration of the imaging device 100 shown in FIG. 19, signal readout, reset, and reset readout are not performed over the entire dot period in the first pixel block 10bg and the second pixel block 10br. For example, signal readout, reset, and reset readout may be performed sequentially for each of the first pixel block 10bg and the second pixel block 10br within one dot period. For example, signal readout, reset, and reset readout may be performed first on the first pixel block 10bg, and then signal readout, reset, and reset readout may be performed on the second pixel block 10br.

Furthermore, in the imaging device 100 shown in FIG. 19, one vertical signal line 47 and one column signal processing circuit 37 are arranged for each column of the pixel array PA, but two vertical signal lines 47 and two A column signal processing circuit 37 may also be arranged. In this case, one of the two vertical signal lines 47 and one of the two column signal processing circuits 37 are used for the first pixel block 10bg. The other of the two vertical signal lines 47 and the other of the two column signal processing circuits 37 are used for the second pixel block 10br. Therefore, in this case, signal readout, reset, and reset readout for the first pixel block 10bg and signal readout, reset, and reset readout for the second pixel block 10br may be performed simultaneously.

Furthermore, in the rolling shutter operation in this embodiment, the exposure period starts at the end of the dot period. However, by using an electronic shutter, the exposure period may be started after a predetermined period has elapsed from the end of the dot period. The electronic shutter is a reset operation that determines the start point of the exposure period, and is the same operation as the reset performed during the above-mentioned dot period. By using this electronic shutter, it is possible to adjust the exposure period of each row in which the rolling shutter operation is performed. As a result, the length of the exposure period for the global shutter operation and the length of the exposure period for the rolling shutter operation can be easily matched.

(Other aspects, etc.)
Although the ranging system and the like according to one or more aspects of the present disclosure have been described above based on each embodiment and each modification, the present disclosure is limited to those embodiments and modifications. It's not a thing. Unless departing from the spirit of the present disclosure, the present disclosure may include various modifications that occur to those skilled in the art to the above-described embodiments or modified examples. Further, the present disclosure may also include a configuration constructed by combining components in a plurality of mutually different embodiments or modifications.

Note that the following cases are also included in the present disclosure.

(1) At least one of the above devices is specifically a computer system consisting of a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), hard disk unit, display unit, keyboard, mouse, etc. be. A computer program is stored in the RAM or hard disk unit. The at least one device described above achieves its functions by the microprocessor operating according to a computer program. Here, a computer program is configured by combining a plurality of instruction codes indicating instructions to a computer in order to achieve a predetermined function.

(2) A part or all of the components constituting at least one of the devices described above may be composed of one system LSI (Large Scale Integration). A system LSI is a super-multifunctional LSI manufactured by integrating multiple components onto a single chip, and specifically, it is a computer system that includes a microprocessor, ROM, RAM, etc. . A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating according to a computer program.

(3) Some or all of the components constituting at least one of the devices described above may be comprised of an IC card or a single module that is detachable from the device. An IC card or module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or module may include the above-mentioned super multifunctional LSI. An IC card or module achieves its functions by a microprocessor operating according to a computer program. This IC card or this module may be tamper resistant.

(4) The present disclosure may be the method described above. Furthermore, it may be a computer program that implements these methods using a computer, or it may be a digital signal formed from a computer program.

Further, the present disclosure describes how to store a computer program or a digital signal in a computer-readable recording medium, such as a flexible disk, a hard disk, a CD (Compact Disc)-ROM, a DVD, a DVD-ROM, a DVD-RAM, and a BD (Blu-ray). (registered trademark) Disc), semiconductor memory, etc. Further, it may be a digital signal recorded on these recording media.

Further, the present disclosure may be applied to transmitting a computer program or a digital signal via a telecommunication line, a wireless or wired communication line, a network typified by the Internet, data broadcasting, or the like.

Alternatively, the program or digital signal may be implemented by another independent computer system by recording the program or digital signal on a recording medium and transferring it, or by transferring the program or digital signal via a network or the like.

The present disclosure can be applied to, for example, a device or system that is mounted on a vehicle and measures the distance to obstacles around the vehicle.

1, 1a Distance measuring device 10 pixel (unit pixel cell)
10b Pixel block 10bg First pixel block 10br Second pixel block 11 Pixel electrode 12 Opposing electrode 12b Electrode piece 13 Photoelectric conversion section 14 Signal detection circuit 15 Photoelectric conversion layer 20 Semiconductor substrate 20t Element isolation region 24 Signal detection transistor 24d, 24s, 26s , 28d, 28s Impurity regions 24g, 26g, 28g Gate electrode 26 Address transistor 28 Reset transistor 32 Voltage supply circuit 32G First voltage supply circuit 32R Second voltage supply circuit 34 Reset voltage source 36 Vertical scanning circuit 37 Column signal processing circuit 38 Horizontal Signal readout circuit 39 Control circuit 40 Power line 41 Charge storage node 42 Sensitivity control line 42g First sensitivity control line 42r Second sensitivity control line 44 Reset voltage line 46 Address control line 47 Vertical signal line 48 Reset control line 49 Horizontal common signal line 50 Interlayer insulating layer 52 Plug 53 Wiring 54, 55 Contact plug 56 Wiring layer 100 Imaging device 410 Optical system 430 Image forming circuit 440 Output buffer 480 Camera section 500, 500a Distance measuring system 501 Straight line detection section 502, 502a Angle detection section 503 Distance Derivation section 504, 504a Shutter control section 510 Distortion detection section 521 Pairing processing section 522 Angle processing section 523 Pixel position detection section 600 Vehicle speed detection section 1000, 1001, 1002 Vehicle 1003 Truck D3, Dg3, Dg4, Dr3, Dr4 Point E Features Part Eg, Eg1, Eg2, Eg3, Eg4, Er, Er1, Er2, Er3, Er4 Edge T1, T2 Table

Claims (18)

a camera unit that outputs first data according to the global shutter operation and second data according to the rolling shutter operation by performing a global shutter operation and a rolling shutter operation on a distance measurement target;
Equipped with a distance measuring device,
The distance measuring device is
a distortion detection unit that detects the magnitude of distortion of the shape of the distance measurement target indicated by the second data with respect to the shape of the distance measurement target indicated by the first data;
a distance deriving unit that acquires a moving speed of the camera unit and derives a distance from the camera unit to the object to be measured based on the moving speed and the magnitude of the distortion detected by the distortion detecting unit; and
Ranging system. The camera unit simultaneously performs the global shutter operation and the rolling shutter operation.
The ranging system according to claim 1. The distortion detection section is
One or more linear first edges are detected from the shape of the object to be measured as indicated by the first data, and one or more linear first edges are detected from the shape of the object to be measured as shown in the second data. a straight line detection unit that detects a second edge of
any one first edge of the one or more linear first edges and any one second edge of the one or more linear second edges are associated, and an angle detection unit that detects an angle between the first edge and the second edge as the magnitude of the distortion;
The ranging system according to claim 1. The distance measuring device further includes:
comprising a shutter control unit that controls at least one of the global shutter operation and the rolling shutter operation of the camera unit based on the movement speed;
The ranging system according to claim 1. The distance measuring device further includes:
comprising a shutter control section that controls at least one of the global shutter operation and the rolling shutter operation of the camera section based on the magnitude of the distortion detected by the distortion detection section;
The ranging system according to claim 1. The shutter control section includes:
When controlling the global shutter operation, change the frame period,
When controlling the rolling shutter operation, changing at least one of a frame period and a horizontal period,
The horizontal period is a period from the start of the exposure period of one row of the pixel array included in the camera unit until the start of exposure of the next row.
The ranging system according to claim 4. The shutter control section includes:
When controlling the global shutter operation, the slower the moving speed, the longer the frame period is changed,
When controlling the rolling shutter operation, the slower the moving speed, the longer at least one of the frame period and the horizontal period is changed;
The ranging system according to claim 6. The shutter control section includes:
When controlling the global shutter operation, controlling a time interval at which the first data is sequentially output;
The ranging system according to claim 4. The shutter control section includes:
When controlling the global shutter operation, the slower the movement speed, the longer the time interval;
The ranging system according to claim 8. When the camera section performs a global shutter operation and a rolling shutter operation on the object to be measured, first data corresponding to the global shutter operation and first data corresponding to the rolling shutter operation are output from the camera section. a distortion detection unit that detects the magnitude of distortion of the shape of the distance measurement target indicated by the second data with respect to the shape of the distance measurement target indicated by the first data;
a distance deriving unit that acquires a moving speed of the camera unit and derives a distance from the camera unit to the object to be measured based on the moving speed and the magnitude of the distortion detected by the distortion detecting unit; and,
A distance measuring device equipped with. The distortion detection section is
One or more linear first edges are detected from the shape of the object to be measured as indicated by the first data, and one or more linear first edges are detected from the shape of the object to be measured as shown in the second data. a straight line detection unit that detects a second edge of
any one first edge of the one or more linear first edges and any one second edge of the one or more linear second edges are associated, and an angle detection unit that detects an angle between the first edge and the second edge as the magnitude of the distortion;
The distance measuring device according to claim 10. The distance measuring device further includes:
comprising a shutter control unit that controls at least one of the global shutter operation and the rolling shutter operation of the camera unit based on the movement speed;
The distance measuring device according to claim 10. The distance measuring device further includes:
comprising a shutter control section that controls at least one of the global shutter operation and the rolling shutter operation of the camera section based on the magnitude of the distortion detected by the distortion detection section;
The distance measuring device according to claim 10. The shutter control section includes:
When controlling the global shutter operation, change the frame period,
When controlling the rolling shutter operation, changing at least one of a frame period and a horizontal period,
The horizontal period is a period from the start of the exposure period of one row of the pixel array included in the camera unit until the start of exposure of the next row.
The distance measuring device according to claim 12. The shutter control section includes:
When controlling the global shutter operation, the slower the moving speed, the longer the frame period is changed,
When controlling the rolling shutter operation, the slower the moving speed, the longer at least one of the frame period and the horizontal period is changed;
The distance measuring device according to claim 14. The shutter control section includes:
When controlling the global shutter operation, controlling a time interval at which the first data is sequentially output;
The distance measuring device according to claim 12. The shutter control section includes:
When controlling the global shutter operation, the slower the movement speed, the longer the time interval;
The distance measuring device according to claim 16. The camera unit performs a global shutter operation and a rolling shutter operation on the object to be measured, thereby outputting first data corresponding to the global shutter operation and second data corresponding to the rolling shutter operation. ,
detecting the magnitude of distortion of the shape of the distance measurement object indicated by the second data with respect to the shape of the distance measurement object indicated by the first data;
Obtaining the moving speed of the camera section,
Deriving a distance from the camera unit to the object to be measured based on the acquired moving speed and the detected magnitude of the distortion;
Distance measurement method.

Images