JP2019201301A - Imaging system - Google Patents

Abstract

To favorably avoid a reduction in accuracy of distance measurement with respect to a traveling direction of a vehicle.SOLUTION: A first imaging apparatus and a second imaging apparatus mounted on a vehicle are provided. The first imaging apparatus has a first angle of view. The second imaging apparatus has a second angle of view smaller than the first angle of view. The second imaging apparatus is driven by a drive unit so as to move an imaging range of the second imaging apparatus. The drive unit is controlled by a control unit on the basis of information related to a progress of the vehicle.SELECTED DRAWING: Figure 4

Description

  The present technology relates to an imaging system, and more particularly, to an imaging system including an imaging device mounted on a vehicle.

  Ranging in a conventional in-vehicle stereo camera is realized from a parallax image of the same type of twin-lens camera near the upper center of the windshield of the vehicle. In this in-vehicle stereo camera, since the camera is fixed to the vehicle, the imaging range of the camera is always fixed to the vehicle. Depending on the driving conditions, there is a difference between the front direction of the vehicle and the traveling direction of the vehicle, and there is a concern that the accuracy of distance measurement with respect to the traveling direction of the vehicle is lowered.

  For example, Patent Document 1 proposes a mobile system that controls the line-of-sight direction of a stereo camera in accordance with a change in the traveling direction of a vehicle. In this mobile system, the entire stereo camera is rotationally controlled, and there is a mechanical problem that the movable range becomes large.

Japanese Patent Laid-Open No. 2006-131367

  An object of the present technology is to satisfactorily avoid a decrease in distance measurement accuracy in the traveling direction of the vehicle.

The concept of this technology is
A first imaging device mounted on a vehicle and having a first angle of view;
A second imaging device mounted on the vehicle and having a second angle of view smaller than the first angle of view;
A drive unit that drives the second imaging device to move the imaging range of the second imaging device;
The imaging system includes a control unit that controls the drive unit based on information related to the progress of the vehicle.

  The present technology includes a first imaging device and a second imaging device mounted on a vehicle. The first imaging device has a first angle of view. The second imaging device has a second field angle smaller than the first field angle. The second imaging device is driven by the drive unit so as to move the imaging range of the second imaging device. For example, the drive unit may be configured to rotationally drive the second imaging device so as to move the imaging range of the second imaging device. Further, for example, the drive unit may drive the second imaging device in a range where the imaging range of the second imaging device does not exceed the imaging range of the first imaging device. For example, the drive unit may be configured to be able to drive the second imaging device in the horizontal direction and the vertical direction.

  The drive unit is controlled by the control unit based on information related to the progress of the vehicle. For example, the information may be curve information. For example, the information may be map information. Further, for example, the information may be given from a vehicle control unit mounted on the vehicle. Further, for example, the information may be given from a sensor mounted on the vehicle. Further, for example, the information may be information indicating an angle of the traveling direction of the vehicle with respect to the front direction of the vehicle.

  For example, a parallax determination unit that obtains parallax information in the imaging range of the second imaging device based on image data from the first imaging device and image data from the second imaging device may be further provided. Good. In this case, for example, a distance calculation unit that obtains distance information in the imaging range of the second imaging device based on the parallax information may be further provided. In this case, for example, an image processing unit that obtains high-resolution image data by combining image data from the first imaging device and image data from the second imaging device so as to overlap based on parallax information. May be further provided.

  As described above, in the present technology, the second imaging device having the second field angle smaller than the first field angle is rotationally driven based on the information related to the traveling of the vehicle. Therefore, it is possible to satisfactorily avoid a decrease in distance measurement accuracy in the traveling direction of the vehicle without causing a mechanical problem that the range of motion becomes large.

  According to the present technology, it is possible to satisfactorily avoid a decrease in distance measurement accuracy with respect to the traveling direction of the vehicle. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

1 is a block diagram illustrating a configuration example of an imaging system as a first embodiment. FIG. It is a figure which shows an example of the correspondence of imaging range AL of a 1st imaging device, and imaging range AR of a 2nd imaging device. It is a figure shown about the conventional vehicle-mounted stereo camera. It is a figure shown about the vehicle-mounted stereo camera of embodiment. It is a block diagram which shows the structural example of an image process part. It is a figure which shows a general camera model. It is a figure which shows the camera model of the conventional vehicle-mounted stereo camera. It is a figure which shows the camera model of the vehicle-mounted stereo camera of embodiment. It is a figure for demonstrating the process of an image deformation | transformation. It is a figure which shows schematically the interpolation process for obtaining high resolution image data.

Hereinafter, modes for carrying out the invention (hereinafter referred to as “embodiments”) will be described. The description will be given in the following order.
1. Embodiment 2. FIG. Modified example

<1. Embodiment>
[Imaging system]
FIG. 1 shows a configuration example of an imaging system 100 as an embodiment. The imaging system 100 includes a first imaging device 101, a second imaging device 102, a drive unit 103, an image processing unit 104, a vehicle control unit (ECU: Electronic Control Unit) 105, a steering 106, A brake 107 and a display unit 108 are provided.

  The first imaging device 101 is a wide-angle camera having a first angle of view, and the first imaging device is a narrow-angle camera having a second angle of view that is smaller than the first angle of view. The first imaging device (wide-angle camera) 101 and the second imaging device (narrow-angle camera) 102 differ only in the angle of view, and use the same image sensor (common pixel pitch). The first imaging device 101 and the second imaging device 102 constitute an in-vehicle stereo camera 111 mounted on the vehicle. The first imaging device 101 is fixed to the vehicle so as to capture the front direction of the vehicle. On the other hand, the second imaging device 102 is capable of imaging not only in the front direction of the vehicle but also in a direction shifted from the front direction, for example, in the horizontal direction and the vertical direction, in this embodiment, in the horizontal direction. It is attached to the vehicle so as to be rotatable.

  The drive unit 103 is configured by a motor or the like, and rotates the second imaging device 102 so as to move the imaging range of the second imaging device 102 in the left-right direction under the control of the vehicle control unit 105. . In this case, the optical axis (gaze direction) of the first imaging device 101 is in the front direction of the vehicle, whereas the optical axis (gaze direction) of the second imaging device 102 follows the traveling direction of the vehicle. To be. In this case, the drive unit 103 rotationally drives the second imaging device 102 so that the imaging range of the second imaging device 102 does not exceed the imaging range of the first imaging device 101.

  FIG. 2 shows an example of a correspondence relationship between the imaging range AL of the first imaging device (wide angle camera) 101 and the imaging range AR of the second imaging device (narrow angle camera) 102. As illustrated, the imaging range AR is included in the imaging range AL. Whereas the imaging range AL is fixed, the imaging range AR moves in the horizontal direction (left-right direction) indicated by the arrows when the second imaging device 102 is rotationally driven. Note that the position of the illustrated imaging range AR indicates a case where the traveling direction of the vehicle matches the front direction of the vehicle.

  In this case, the vehicle control unit 105 controls the drive unit 103 based on, for example, information related to the progress of the vehicle. For example, this information includes curve information obtained by recognizing a white road line from captured image data, map information from a navigation system (not shown), or a vehicle (for example, a steering 106 or a sensor ( Information indicating the angle of the traveling direction of the vehicle with respect to the front direction of the vehicle given by a radar, a lidar, etc.).

  3A and 3B show a conventional in-vehicle stereo camera 10. The in-vehicle stereo camera 10 includes a left eye camera 11 and a right eye camera 12 having the same angle of view. Since this in-vehicle stereo camera 10 is fixed to the vehicle 40 so that the optical axis (line-of-sight direction) of the camera coincides with the front direction of the vehicle 40, the imaging range of the camera is always fixed with respect to the vehicle 40. It is in. FIG. 3A shows a straight traveling state. In this state, the front direction of the vehicle 40 and the traveling direction of the vehicle 40 match. In contrast, FIG. 3B shows a right curve state. In this state, the traveling direction of the vehicle 40 deviates from the front direction of the vehicle 40.

  FIGS. 4A and 4B show the in-vehicle stereo camera 111 of this embodiment. The first imaging device (wide-angle camera) 101 is fixed to the vehicle 40 so that the optical axis (gaze direction) of the camera coincides with the front direction of the vehicle 40, but the second imaging device (narrow-angle camera). Via 102, it is attached to the vehicle 40 so as to be rotatable in the horizontal direction (left-right direction), and its optical axis follows the traveling direction of the vehicle 40.

  FIG. 4A shows a straight traveling state. In this state, the front direction of the vehicle coincides with the traveling direction of the vehicle. Therefore, the optical axis of the second imaging device (narrow angle camera) 102 matches the front direction of the vehicle in the same manner as the optical axis of the first imaging device (wide angle camera) 101. On the other hand, FIG. 4B shows a right curve state. In this state, the traveling direction of the vehicle deviates from the front direction of the vehicle, and the optical axis (gaze direction) of the second imaging device (narrow-angle camera) 102 matches the traveling direction of the vehicle 40.

  Returning to FIG. 1, the image processing unit 104 performs parallax in the imaging range of the second imaging device 102 based on the image data VL from the first imaging device 101 and the image data VR from the second imaging device 102. get information. In this case, parallax information is obtained by obtaining the corresponding pixels of the first imaging device 101, for example, in units of subpixels in the imaging range of the second imaging device 102 by image matching processing. Then, the image processing unit 104 calculates a distance (depth) for each pixel in the imaging range of the second imaging device 102 based on the parallax information, obtains a depth map, and supplies the depth map to the vehicle control unit 105.

  Further, the image processing unit 104 combines the image data VL from the first imaging device 101 and the image data VR from the second imaging device 102 so as to overlap each other based on the parallax information, so that the second High-resolution image data VH in the imaging range of the imaging device 102 is obtained. Then, the image processing unit 104 performs object recognition processing on the high-resolution image data VH and supplies the recognition result to the vehicle control unit 105. This object recognition processing can include the above-described white line recognition of roads in addition to vehicles, people, road signs, traffic lights, and the like. As described above, in the object recognition process using the high-resolution image data, it is possible to accurately recognize an object existing far in the traveling direction.

  In addition, the image processing unit 104 performs object recognition processing on the image data VL from the first imaging device 101 and supplies the recognition result to the vehicle control unit 105. As described above, in the object recognition processing using the image data VL from the first image pickup apparatus 101, object recognition is performed from picked-up image data with a wide field of view, and a vehicle or a person suddenly coming out from the side is recognized. It becomes possible.

  The vehicle control unit 105 controls the steering 106 and the brake 107 based on the depth map and the object recognition result supplied from the image processing unit 104, and also controls warning display on the display unit 108.

  FIG. 5 shows a configuration example of the image processing unit 104. The image processing unit 104 includes a parallax determination unit 141, a distance calculation unit 142, an image deformation unit 143, an image interpolation unit 144, and object recognition units 145 and 146.

  The parallax determination unit 141 is supplied with the image data VL obtained by the first imaging device (wide angle camera) 101 and the image data VR obtained by the second imaging device (narrow angle camera) 102. Is done. For example, the parallax determination unit 141 processes the image data VR to obtain image data VR ′ having the same resolution as the image data VL.

  The parallax determination unit 141 performs image matching processing in a state where the image based on the image data VR ′ is arranged at the corresponding position of the image based on the image data VL, and obtains parallax information on each pixel of the image data VR ′. For example, when the position of the target pixel of the image data VR ′ is (u2, v2) and the corresponding pixel position of the image data VL is (u1, v1), these (u2, v2), (u1, v1) ) Constitutes disparity information of the target pixel of the image data VR ′.

  The distance calculation unit 142 calculates a distance (depth) for each pixel in the imaging range of the second imaging device 102 based on the above-described parallax information, obtains a depth map, and supplies the depth map to the vehicle control unit 105. To do.

  FIG. 6 shows a general camera model. If the point P of the three-dimensional coordinates (X, Y, Z) is reflected at the point p (u, v) on the image, the following formula (1) is established. Here, K is called a camera matrix or an internal parameter matrix of the camera, (cx, cy) is a principal point (usually the image center), and f is a focal length expressed in units of pixels. [R | T], which is a translation-rotation homogeneous transformation matrix, is called an external parameter matrix. Λ is a scale factor.

  FIG. 7 shows a camera model of a conventional in-vehicle stereo camera 10 (see FIG. 3). Assuming that the point P of the three-dimensional coordinates (X, Y, Z) appears on the point p1 (u1, v1) on the image by the left-eye camera 11, the following equation (2) is established. If the point P of the three-dimensional coordinates (X, Y, Z) is reflected on the point p2 (u2, v2) on the image by the right eye camera 12, the following formula (3) is established. Here, B is the baseline length.

In this case, the parallax d is expressed by the following formula (4), and the distance Z is calculated by the following formula (5).
d = u2−u1 (4)
Z = Bf / d (5)

  FIG. 8 shows a camera model of the in-vehicle stereo camera 111 (see FIG. 4) according to the embodiment. Assuming that the point P of the three-dimensional coordinates (X, Y, Z) appears in the point p1 (u1, v1) on the image by the first imaging device 101, the following equation (6) is established. If the point P of the three-dimensional coordinates (X, Y, Z) is reflected at the point p2 (u2, v2) on the image by the second imaging device 102, the following formula (7) is established. In this case, (u2, v2) is converted by a rotation matrix representing the rotation θ around the Y axis expressed by the following formula (8).

In this case, the distance Z is calculated by the following formula (9). In other words, the distance calculation unit 142 calculates the formula (9) for each pixel in the imaging range of the second imaging device 102 based on the above-described parallax information (u2, v2) and (u1, v1). It will be.

  Returning to FIG. 5, the image deforming unit 143 corresponds the image data VL obtained by the first imaging device (wide-angle camera) 101 and the image data VR obtained by the second imaging device (narrow-angle camera) 102. Deformation processing is performed on the image data VL so that the portions overlap. In this case, the image transformation unit 143 grasps the pixel of the image data VL corresponding to each pixel of the image data VR ′ based on the parallax information obtained by the parallax determination unit 141, and uses the pixel of the image data VL as the image data. It moves based on parallax information so that it may overlap with the corresponding pixel of VR. As a result, the image deforming unit 143 obtains deformed image data VLa corresponding to the imaging range of the second imaging device (narrow angle camera) 102.

  For example, as shown in FIG. 9, a point (pixel) P21 in the imaging range AR of the second imaging device (narrow angle camera) 102 is a point (in the imaging range AL of the first imaging device (wide angle camera) 101) ( When it is determined that (pixel) P11 corresponds, the point (pixel) P11 in the imaging range AL is moved based on the parallax information so as to overlap the point (pixel) P21 in the imaging range AR. Similarly, as shown in FIG. 9, a point (pixel) P22 in the imaging range AR of the second imaging device (narrow-angle camera) is a point in the imaging range AL of the first imaging device (wide-angle camera). When it is determined that the (pixel) P12 corresponds, the point (pixel) P12 in the imaging range AL is moved based on the parallax information so as to overlap the point (pixel) P22 in the imaging range AR.

  Returning to FIG. 5, the image interpolation unit 144 performs an interpolation process using the image data VR and the deformed image data VLa to obtain high-resolution image data VH. FIG. 10 schematically shows the interpolation process. FIG. 10A shows an example of a correspondence relationship between the respective pixels of the image data VL and the image data VR before being deformed by the image deforming unit 143. As described above, the first image pickup device (wide-angle camera) 101 and the second image pickup device (narrow-angle camera) 102 are different from each other only in the angle of view and are composed of the same image sensor (common pixel pitch). Accordingly, in this case, since the image data VL relates to wide-angle imaging while the image data VR relates to narrow-angle imaging, the pixel interval dR of the pixel data VR is equal to the pixel interval dL of the image data VL. It is shorter.

  FIG. 10B illustrates an example of a correspondence relationship between each pixel of the image data VLa and the image data VR that has been subjected to the deformation process by the image deformation unit 143, and a pixel of the high-resolution image data VH obtained by the interpolation process. Show. In the illustrated example, a state in which the pixel P1 of the image data VL is deformed so as to overlap the pixel P2 of the corresponding image data VR is illustrated. Each pixel of the high-resolution image data VH is generated with a pixel interval dH shorter than the pixel interval dR using the pixels of the image data VLa and the image data VR.

  Returning to FIG. 5, the object recognition unit 145 performs object recognition based on the high-resolution image data VH corresponding to the imaging range of the second imaging device (narrow angle camera) 102 obtained by the image interpolation unit 144. The recognition result is supplied to the vehicle control unit 105. In this case, the object recognition unit 145 recognizes an object such as a vehicle, a pedestrian, a road sign, or a traffic signal using a known object recognition technology. As described above, in the object recognition process using the high-resolution image data VH, it is possible to accurately recognize an object that exists far away.

  The object recognition unit 146 performs object recognition based on the image data VL obtained by the first imaging device (wide angle camera) 101 and supplies the recognition result to the vehicle control unit 105. In this case, the object recognizing unit 146 recognizes an object such as a vehicle, a pedestrian, a road sign, or a traffic light using a known object recognition technique. Thus, in the object recognition process using the image data VL, object recognition is performed from image data with a wide field of view, and it is possible to recognize a vehicle or a person suddenly coming out from the side.

  As described above, in the imaging system 100 shown in FIG. 1, the second imaging device (narrow-angle camera) 102 constituting the in-vehicle stereo camera 11 is rotationally driven based on information relating to the traveling of the vehicle. . Therefore, it is possible to satisfactorily avoid a decrease in distance measurement accuracy in the traveling direction of the vehicle without causing a mechanical problem that the range of motion becomes large.

  In the imaging system 100 shown in FIG. 1, only the second imaging device (narrow angle camera) 102 that constitutes the in-vehicle stereo camera 111 is rotationally driven based on the information related to the progress of the vehicle. The first imaging device (wide-angle camera) 101 to be configured is fixed to the vehicle so as to capture the front direction of the vehicle. Therefore, it is possible to avoid the occurrence of a visual field shift due to the follow-up direction tracking.

  In the imaging system 100 shown in FIG. 1, the image data VL from the first imaging device (wide-angle camera) 101 and the image data VR from the second imaging device (narrow-angle camera) 102 are based on the parallax information. The high resolution image data VH in the imaging range of the second imaging device 102 is obtained by combining the corresponding portions so as to overlap each other. For this reason, it is possible to accurately recognize an object existing far in the traveling direction.

<2. Modification>
In the above-described embodiment, the present technology is applied to a car, and the second imaging device (narrow angle camera) 102 is driven to rotate in the horizontal direction, that is, the example driven to rotate around the Y axis. Indicated. For this purpose, the point p2 (u2, v2) on the image by the second imaging device 102 is coordinate-transformed with a rotation matrix (see equation (8)) representing the rotation θ around the Y axis.

  However, the imaging system of the present technology can be mounted on a moving body such as a drone in addition to a car. In this case, it is necessary to consider the rotation θ around the X axis and the rotation θ around the Z axis in addition to the rotation θ around the Y axis. In that case, the point p2 (u2, v2) on the image by the second imaging device 102 is set around the X axis, the Y axis, and the Z axis represented by the following formulas (10), (11), and (12). The present invention can be applied in the same manner as the coordinate transformation with the rotation matrix indicating the rotation around the axis.

  Moreover, although the preferred embodiment of this indication was described in detail, referring an accompanying drawing, the technical scope of this indication is not limited to this example. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

Moreover, this technique can also take the following structures.
(1) a first imaging device mounted on a vehicle and having a first angle of view;
A second imaging device mounted on the vehicle and having a second angle of view smaller than the first angle of view;
A drive unit that rotationally drives the second imaging device to move the imaging range of the second imaging device;
An imaging system comprising: a control unit that controls the drive unit based on information relating to the progress of the vehicle.
(2) The imaging system according to (1), wherein the driving unit rotationally drives the second imaging device so as to move an imaging range of the second imaging device.
(3) The driving unit drives the second imaging device in a range in which an imaging range of the second imaging device does not exceed an imaging range of the first imaging device. (1) or (2) The imaging system described.
(4) The imaging system according to any one of (1) to (3), wherein the driving unit is configured to be able to drive the second imaging device in a horizontal direction and a vertical direction.
(5) The imaging system according to any one of (1) to (4), wherein the information is curve information.
(6) The imaging system according to any one of (1) to (4), wherein the information is map information.
(7)
The imaging system according to any one of (1) to (4), wherein the information is information indicating an angle of a traveling direction of the vehicle with respect to a front direction of the vehicle.
(8) The imaging system according to any one of (1) to (7), wherein the information is given from a vehicle control unit mounted on the vehicle.
(9) The imaging system according to any one of (1) to (7), wherein the information is given from a sensor mounted on the vehicle.
(10) The apparatus further includes a parallax determination unit that obtains parallax information in an imaging range of the second imaging device based on image data from the first imaging device and image data from the second imaging device. The imaging system according to any one of 1) to (9).
(11) The imaging system according to (10), further including a distance calculation unit that obtains distance information in an imaging range of the second imaging device based on the parallax information.
(12) An image processing unit that obtains high-resolution image data by combining corresponding image data from the first imaging device and image data from the second imaging device so as to overlap based on the parallax information The imaging system according to (10) or (11).

DESCRIPTION OF SYMBOLS 100 ... Camera system 101 ... 1st imaging device (wide angle camera)
102: Second imaging device (narrow angle camera)
DESCRIPTION OF SYMBOLS 103 ... Drive part 104 ... Image processing part 105 ... Vehicle control part 106 ... Steering 107 ... Brake 108 ... Display part 111 ... Car-mounted stereo camera

Claims (12)

A first imaging device mounted on a vehicle and having a first angle of view;
A second imaging device mounted on the vehicle and having a second angle of view smaller than the first angle of view;
A drive unit that drives the second imaging device to move the imaging range of the second imaging device;
An imaging system comprising: a control unit that controls the drive unit based on information relating to the progress of the vehicle. The imaging system according to claim 1, wherein the driving unit rotationally drives the second imaging device so as to move an imaging range of the second imaging device. The imaging system according to claim 1, wherein the driving unit drives the second imaging device in a range in which an imaging range of the second imaging device does not exceed an imaging range of the first imaging device. The imaging system according to claim 1, wherein the driving unit is configured to be able to drive the second imaging device in a horizontal direction and a vertical direction. The imaging system according to claim 1, wherein the information is curve information. The imaging system according to claim 1, wherein the information is map information. The imaging system according to claim 1, wherein the information is information indicating an angle of a traveling direction of the vehicle with respect to a front direction of the vehicle. The imaging system according to claim 1, wherein the information is given from a vehicle control unit mounted on the vehicle. The imaging system according to claim 1, wherein the information is given from a sensor mounted on the vehicle. The parallax determination unit that obtains parallax information in an imaging range of the second imaging device based on image data from the first imaging device and image data from the second imaging device. Imaging system. The imaging system according to claim 10, further comprising a distance calculation unit that obtains distance information in an imaging range of the second imaging device based on the parallax information. An image processing unit that obtains high-resolution image data by combining image data from the first imaging device and image data from the second imaging device so as to overlap based on the parallax information is further provided. The imaging system according to 10.

Images