JP2022143387A - Imaging device and program - Google Patents

Abstract

To reduce the deviation of scanning timing occurring in left and right imaging units having different angles of view.SOLUTION: An imaging device includes a first imaging unit having a rolling shutter type imaging element, a second imaging unit having a rolling shutter type imaging element with a resolution different from that of the first imaging unit, and a delay instructing unit that delays the scanning start timing of either the first imaging unit or the second imaging unit by a delay time which is the deviation of the scanning completion timing of the first imaging unit and the second imaging unit in an arbitrary region in the first captured image captured by the first imaging unit and an arbitrary region in the second captured image captured by the second imaging unit.SELECTED DRAWING: Figure 11

Description

The present invention relates to an imaging device and a program.

Conventionally, a stereo camera in which two cameras are arranged on the left and right sides is mounted on a vehicle such as an automobile and used to measure distance information from the vehicle to an object. In addition to distance information, stereo cameras are used to recognize various objects around the vehicle. Such a stereo camera contributes to improvement of safety during driving assistance of a vehicle.

It should be noted that object recognition can be performed using only an image captured by a single camera without distance information. Therefore, it has been considered to configure the stereo camera with cameras having different angles of view such as a wide-angle camera and a narrow-angle camera. According to such a stereo camera, it is possible to perform object recognition in a wide range with the wide-angle camera, and to measure the distance to the object in the portion where the imaging ranges of the narrow-angle camera and the wide-angle camera overlap.

In Patent Document 1, a delay time of vertical timing for correcting the deviation is calculated from the amount of deviation of the trimming position of the left and right images, and a vertical timing adjuster is instructed to adjust the deviation in the vertical direction. A technique is disclosed for adjusting so as to approach zero. By adjusting in this way, it is possible to obtain a high-quality stereoscopic image without deviation between the left and right cameras.

However, when the rolling shutter method is used in a stereo camera mounted with cameras having different angles of view, as in conventional cameras, there is a problem that the scanning timings of the left and right cameras of the stereo camera do not match. .

More specifically, the rolling shutter method is a method of sequentially scanning pixels in the sensor. For this reason, when the angles of view (the amount of pixels to be scanned) of the left and right cameras are different, even if the scanning is started at the same time, a timing shift occurs during the scanning. If a timing shift occurs during scanning in this way, the same object will be imaged at different timings on the left and right sides, and an error will occur in the distance information.

SUMMARY OF THE INVENTION The present invention has been made in view of the above, and it is an object of the present invention to reduce the deviation in scanning timing that occurs in left and right imaging units having different angles of view.

In order to solve the above-described problems and achieve the object, the present invention provides a first imaging unit having a rolling shutter imaging device, and a rolling shutter imaging device having a resolution different from that of the first imaging unit. 2 imaging units, and the first imaging unit and the a delay instructing unit that delays the scanning start timing of either the first imaging unit or the second imaging unit by a delay time that is a difference in scanning completion timing with the second imaging unit. Characterized by

ADVANTAGE OF THE INVENTION According to this invention, it is effective in the ability to reduce the deviation of the scanning timing which generate|occur|produces in the left-right imaging part from which an angle of view differs.

FIG. 1 is a schematic plan view showing the configuration of an imaging device according to a first embodiment; FIG. FIG. 2 is a schematic plan view showing the configuration of the first imaging section and the second imaging section. FIG. 3 is a block diagram showing the hardware configuration of the control system of the imaging apparatus. FIG. 4 is a diagram illustrating an example of magnification adjustment and trimming in the correction unit. FIG. 5 is a diagram showing an example of distance measurement by the imaging device. FIG. 6 is a diagram showing an example of horizontal angles of view of the first imaging unit and the second imaging unit when the Z axis is arranged as the optical axis direction. FIG. 7 is a diagram showing an example of the vertical angle of view of the first imaging unit and the second imaging unit when the Z axis is arranged along the optical axis. FIG. 8 is a diagram exemplifying the difference in scanning timing between the imaging element of the first imaging section and the imaging element of the second imaging section. FIG. 9 is a diagram exemplifying the difference in scanning timing between the lines before and after the subject. FIG. 10 is a diagram showing a case where the scanning timings for the subject are shifted between the first imaging section and the second imaging section. FIG. 11 is a diagram showing a case where the difference in scanning timing for the object between the first imaging unit and the second imaging unit is brought close to zero. FIG. 12 is a diagram showing an example in which scanning timing is delayed. FIG. 13 is a block diagram of a hardware configuration of a control system of an imaging apparatus according to the second embodiment; FIG. 14 is a diagram exemplifying a vanishing point. FIG. 15 is a flow chart showing the flow of processing for reflecting the delay time from the vanishing point. FIG. 16 is a diagram showing an example in which the imaging device of the embodiment is provided for an object detection device for a vehicle. FIG. 17 is a diagram showing a schematic configuration of an automobile object detection device to which the imaging device of the embodiment is applied. FIG. 18 is a diagram showing a configuration example of an automobile object detection device to which the imaging device of the embodiment is applied. FIG. 19 is a top view of an aircraft provided with an imaging device according to an embodiment. FIG. 20 is a front view of an aircraft provided with an imaging device according to an embodiment.

Embodiments of an imaging device and a program will be described in detail below with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a schematic plan view showing the configuration of an imaging device 10 according to the first embodiment, and FIG. 2 is a schematic plan view showing the configurations of a first imaging section 1 and a second imaging section 2. As shown in FIG. A stereo imaging apparatus 10 has a first imaging section 1 and a second imaging section 2 as stereo cameras for imaging a subject.

The first imaging unit 1 and the second imaging unit 2 are arranged with a base line length B (for example, 60 mm) as a predetermined interval. The image pickup apparatus 10 can image the same subject with the first image pickup section 1 and the second image pickup section 2 by synchronizing the scanning timing.

As shown in FIG. 2, the first imaging section 1 and the second imaging section 2 are provided with optical lenses 8 (8a, 8b) and imaging elements 9 (9a, 9b). The first imaging unit 1 and the second imaging unit 2 are integrally fixed in a state where they are held with a baseline length B therebetween by a stay member 12 .

The optical lens 8a is, for example, a lens with a focal length f of 2.5 mm. The optical lens 8b is, for example, a lens with a focal length f of 5 mm.

The first imaging unit 1 is a wide-angle camera. The second imaging unit 2 is a narrow-angle camera. That is, the first imaging unit 1 and the second imaging unit 2 have different angles of view and different focal lengths due to different resolutions.

The imaging device 9a of the first imaging unit 1 is a CMOS image sensor of a rolling shutter type with high resolution. The image sensor 9a has a pixel pitch pp of 3 μm. The rolling shutter method is a method of sequentially scanning pixels in the sensor.

The imaging device 9b of the second imaging unit 2 is a rolling shutter low-resolution CMOS image sensor. The image sensor 9b has a pixel pitch pp of 3 μm.

Next, the control system will be explained.

The imaging device 10 includes a control device such as a CPU (Central Processing Unit), a storage device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), and an external storage device such as an HDD for storing control programs. It has a hardware configuration using a normal computer.

The CPU of the imaging device 10 functions as the storage unit 3, the correction unit 6, the parallax calculation unit 7, and the delay instruction unit 5 as shown in FIG. 3 by executing the control program.

Note that the control program executed by the imaging apparatus 10 of this embodiment may be stored in a computer connected to a network such as the Internet, and may be provided by being downloaded via the network. Also, the control program executed by the imaging apparatus 10 of this embodiment may be provided or distributed via a network such as the Internet.

Also, the control program executed by the imaging apparatus 10 of the present embodiment may be configured to be pre-installed in a ROM or the like and provided.

FIG. 3 is a block diagram showing the hardware configuration of the control system of the imaging device 10. As shown in FIG. As shown in FIG. 3 , the imaging device 10 includes a storage section 3 , a correction section 6 , a parallax calculation section 7 and a delay instruction section 5 .

Storage unit 3 stores captured image 11 , corrected image 14 , parallax image 15 , and delay time 17 .

Here, the captured image captured by the first imaging unit 1 is referred to as a "first captured image". A captured image captured by the second imaging unit 2 is referred to as a "second captured image". Also, when the first captured image and the second captured image are not distinguished, they are simply referred to as "captured images". The captured image 11 is the first captured image or the second captured image.

The correction unit 6 performs magnification adjustment and trimming so that the first captured image and the second captured image appear to be images captured at the same focal length, and adjusts the position of the first imaging unit 1 (second imaging unit 2). A coordinate deviation of the first captured image (second captured image) caused by the deviation is corrected.

As described above, the first imaging unit 1, which is a wide-angle camera, and the second imaging unit 2, which is a narrow-angle camera, have different angles of view and different focal lengths due to different resolutions. Therefore, it is necessary to adjust the focal length in order to obtain the distance to the subject, which will be described later. The correction unit 6 performs magnification adjustment and trimming to match the focal length.

FIG. 4 is a diagram showing an example of magnification adjustment and trimming in the corrector 6. As shown in FIG. In FIG. 4, a first captured image 11-1 is an image captured by the first imaging unit 1, which is a wide-angle camera, and a second captured image 11-1 is an image captured by the second imaging unit 2, which is a narrow-angle camera. 2.

In FIG. 4, the focal length of the first imaging section 1 is f ₁ , and the focal length of the second imaging section 2 is f ₂ . It is known that the magnification is obtained by the focal length ratio, and the magnification m of the second captured image 11-2 with respect to the first captured image 11-1 is (f ₂ /f ₁ ). Therefore, the correction unit 6 enlarges the first captured image 11-1 by m times and trims it to the same sizes X ₁ ' and Y ₁ ' as the sizes X ₂ and Y ₂ of the second captured image 11-2. , a first corrected image 14-1 having the same magnification and image size as those of the second captured image 11-2.

In this embodiment, as described above, the focal length f1 of the optical lens 8a of the first imaging section ₁ is 2.5 mm, and the focal length f2 of the optical lens 8b of the second imaging section ₂ is 5 mm. In such a case, for example, the focal length of the captured image is apparently adjusted as follows by adjusting the magnification and trimming.
Apparent focal length of the image captured by the first imaging unit 1: 5 mm
Apparent focal length of the image captured by the second imaging unit 2: 5 mm

The parallax calculator 7 calculates parallax from the first corrected image obtained by correcting the first captured image by the corrector 6 and the second captured image. Then, the parallax calculator 7 uses the parallax to generate a parallax image 15 including parallax information of the coordinates of the captured image 11 . A specific parallax calculation method is the same as that of a well-known parallel coordinate stereo camera.

The parallax calculation unit 7 calculates the correlation between the image area around the feature points extracted from the reference imaged image and the corresponding candidate area on the imaged image to be compared while shifting the corresponding candidate area by one pixel unit. By interpolating the correlation value of with a parabola or the like, the parallax is calculated with an accuracy of a pixel unit or less. The parallax calculator 7 of the present embodiment generates a parallax image 15 with an accuracy of 1/10 pixel or less.

Here, the principle of distance measurement in the imaging device 10 using the first imaging unit 1 and the second imaging unit 2 will be described.

Here, FIG. 5 is a diagram showing an example of distance measurement by the imaging device 10. In FIG. As shown in FIG. 5, the first imaging unit 1 has a focal length f of the optical lens 8a, an optical center O ₀ of the optical lens 8a, an imaging surface S ₀ of the imaging device 9a, and an optical axis direction of the Z axis. The second imaging unit 2 has a focal length f of the optical lens 8b, an optical center O ₁ of the optical lens 8b, an imaging surface S ₁ of the imaging element 9b, and the Z axis as the optical axis direction.

The first imaging unit 1 and the second imaging unit 2 are arranged parallel to the X-axis and separated by a base line length B (for example, 60 mm).

It is assumed that the subject A is located at a distance d in the optical axis direction from the optical center O0 of the first imaging unit ₁ .

The first imaging unit 1 forms an image of the subject A at P ₀ which is the intersection of the straight line AO ₀ and the imaging surface S ₀ . On the other hand, the second imaging unit ₂ forms _an image of the subject _A at P1, which is the intersection of the straight line AO1 and the imaging surface S1.

Here, the intersection of a straight line passing through the optical center O ₁ of the second imaging unit 2 and parallel to the straight line AO ₀ and the imaging surface S ₁ is defined as P ₀ ′. Also, let p be the distance between P ₀ ′ and P ₁ . The parallax calculator 7 calculates, as a parallax, the distance p, which is the amount of positional deviation between the images of the same subject A captured by the first imaging unit 1 and the second imaging unit 2 . Then, the parallax calculator 7 uses the parallax to generate a parallax image 15 including parallax information of the coordinates of the captured image 11 .

As shown in FIG. 5, triangle AO ₀ -O ₁ and triangle O ₁ -P ₀ ′-P ₁ are similar. Therefore, the distance d to the subject A can be obtained from the base line length B, the focal length f, and the parallax p, as in the following formula.
d=B×f/p

The imaging device 10 periodically and repeatedly performs imaging, correction, and parallax calculation of the captured image by the above-described units (the first imaging unit 1, the second imaging unit 2, the storage unit 3, the correction unit 6, and the parallax calculation unit 7). to continue generating the parallax image 15 .

By the way, as in the present embodiment, in the imaging apparatus 10 equipped with the first imaging unit 1 and the second imaging unit 2 having different angles of view (the amount of pixels to be scanned), when the rolling shutter method is adopted as the shutter method, Even if the scanning is started at the same time, there is a problem that the first image pickup unit 1 and the second image pickup unit 2 are out of timing during the scanning. If a timing shift occurs during scanning in this way, the same subject will be imaged at different timings on the left and right sides, resulting in an error in distance information.

Here, an example in which an error occurs in distance information when a rolling shutter method is adopted as a shutter method in the imaging apparatus 10 equipped with the first imaging unit 1 and the second imaging unit 2 having different angles of view will be described.

Here, FIG. 6 is a diagram showing an example of the horizontal angle of view of the first imaging section 1 and the second imaging section 2 when the Z axis is arranged as the optical axis direction. As shown in FIG. 6, when the Z-axis is arranged as the optical axis direction, the horizontal angle of view of the first imaging section 1, which is a wide-angle camera, is represented by reference numeral 1a, and the horizontal angle of view of the second imaging section 2, which is a narrow-angle camera. The angle of view is denoted by reference numeral 2a.

FIG. 7 is a diagram showing an example of the vertical angle of view of the first imaging section 1 and the second imaging section 2 when the Z axis is arranged along the optical axis. As shown in FIG. 7, when the Z-axis is arranged as the optical axis direction, the vertical angle of view of the first imaging section 1, which is a wide-angle camera, is represented by reference numeral 1b, and the vertical angle of view of the second imaging section 2, which is a narrow-angle camera. The angle of view is denoted by reference numeral 2b.

Further, FIG. 8 is a diagram illustrating the difference in scanning timing between the imaging device 9a of the first imaging unit 1 and the imaging device 9b of the second imaging unit 2. As shown in FIG. FIG. 8 shows the difference in scanning timing with respect to the subject A when the scanning timings of the imaging device 9a of the first imaging unit 1, which is a wide-angle camera, and the imaging device 9b of the second imaging unit 2, which is a narrow-angle camera, are matched. It explains Δtv.

The rolling shutter imaging device 9 (9a, 9b) scans an image line by line from the top to the bottom of the screen. As shown in FIG. 8, the vertical line scanned up to the object A by the imaging device 9a of the first imaging unit 1 is v1, and the vertical line scanned up to the object A by the imaging device 9b of the second imaging unit 2 is v2. As shown in FIG. 8, the number of vertical lines scanned up to the subject A is v1>v2.

Further, as shown in FIG. 8, the scanning time per line in the imaging device 9a of the first imaging unit 1 is th1, and the scanning time per line in the imaging device 9b of the second imaging unit 2 is th2. Therefore, the scanning time tv1 up to the object A in the imaging device 9a of the first imaging unit 1 is represented by (th1×v1), and the scanning time tv2 up to the object A in the imaging device 9b of the second imaging unit 2 is (th2×v1). v2).

That is, as shown in FIG. 8, when the scanning time th1 per line in the imaging device 9a of the first imaging unit 1 and the scanning time th2 per line in the imaging device 9b of the second imaging unit 2 are different, the subject A , there is a time difference .DELTA.tv as shown in the following equation (1).
Δtv=tv1−tv2=(th1×v1)−(th2×v2) (1)

In the following, a case where th1<th2 is satisfied will be described as an example.

FIG. 9 is a diagram exemplifying the difference in scanning timing between lines before and after subject A. In FIG. FIG. 9 shows scanning of the subject A in the front and rear lines when the scanning timing of the imaging device 9a of the first imaging unit 1, which is a wide-angle camera, and the imaging device 9b of the second imaging unit 2, which is a narrow-angle camera, are matched. This explains the difference in timing. As shown in FIG. 9, the scanning time th1 per line in the imaging device 9a of the first imaging unit 1 and the scanning time th2 per line in the imaging device 9b of the second imaging unit 2 have a relationship of th1<th2. Therefore, the time difference Δtv in the scanning timing of the subject A becomes smaller each time the scanning is advanced.

Here, a problem that occurs when the scanning timings of the first imaging unit 1 and the second imaging unit 2 are different from each other will be described.

FIG. 10 is a diagram showing a case where the scanning timings for the subject A are shifted between the first imaging unit 1 and the second imaging unit 2. In FIG. FIG. 10 shows an imaging plane S′ ₀ of the first imaging unit 1 and an imaging plane S′ ₁ of the second imaging unit 2 , during the time Δth after the first imaging unit 1 has captured the object A. When the object is moved leftward by a distance g, the second imaging unit ₂ images the object A2 after the movement. Then, as shown in FIG. 10, as a result, the parallax becomes large, and the distance d' of the position of the subject A', which is closer than the actual distance of the subject A, is obtained.

As described above, the apparent focal length f of the optical lens 8 (8a, 8b) after trimming to match the focal length is 5 mm, and the distance between the first imaging unit 1 and the second imaging unit 2 is Assume that the base line length B is 60 mm and the pixel pitch pp is 3 μm.

Further, the scanning time th1 per line of the first imaging unit 1 is 20 μs, the number of lines tv1 scanned up to the object A is 480, the scanning time th2 per line of the second imaging unit 2 is 50 μs, and the object A is scanned. It is assumed that the number of lines tv2 is 240.

The scanning timing difference Δtv for the subject A is −2.4 ms from the above equation (1).

Here, it is assumed that the distance d to the subject A is 30 m, and the subject A is moving to the left at a speed of 60 km/h.

The distance g that the subject A moves in 2.4 ms after the first imaging unit 1 scans the subject A and before the second imaging unit 2 captures an image of the subject A is
g=60km/h×2.4ms=0.04m
becomes.

Here, let G be the parallax increment due to movement by the distance g. Also, triangle AA ₂ -O ₁ and triangle O ₁ -P ₁ -P ₁ ′ are similar. Therefore, the parallax increment G is about 2.2 pixels according to the following formula.
G=(((f×g)/d))/pp

By the way, the _{parallax value} p for the distance 30 m to the original subject _A is P ₌ ((B _× f )/d)/pp is about 3.3 pixels. Therefore, the parallax value p' is about 5.5 pixels.

Here, as shown in FIG. 10, the triangle A'-O ₀ -O ₁ and the triangle O ₁ -P ₀ '-P ₁ are similar. Therefore, d'=B*f/(p'*pp), and the distance d' to the subject A' is approximately 18 m.

That is, the imaging device 10 measures the subject A, which is actually at a distance of 30 m, as if it were at a distance of 18 m. Assuming that the own vehicle is running and the braking distance is 20 m, the object detection device of the automobile provided with the imaging device 10 predicts that there will be an object within the braking distance from this result alone, and that a collision will occur. That is, the control system may determine that an operation such as emergency braking or avoidance is necessary.

However, in the object detection device for automobiles, a margin is usually provided such that multiple times of collision determination are required before control is started. Therefore, although erroneous control does not necessarily occur in the above situation, it is naturally desirable that the stereo camera module itself is highly reliable. In this way, the combination of the first imaging unit 1, which is a wide-angle camera, and the second imaging unit 2, which is a narrow-angle camera, can cause the problem that the subject appears to be moving.

Therefore, in the present embodiment, the imaging device 10 adjusts the scanning timing. Specifically, as shown in FIG. 3 , the imaging device 10 includes a delay instructing section 5 . Further, as shown in FIG. 3 , the imaging device 10 stores the delay time 17 in the storage section 3 .

Although details will be described later, the delay instruction unit 5 reads the delay time 17 from the storage unit 3 and instructs the imaging device 10 to delay the scanning start timings of the first imaging unit 1 and the second imaging unit 2 .

Here, FIG. 11 is a diagram showing a case where the difference in scanning timing with respect to the object A between the first imaging unit 1 and the second imaging unit 2 is brought close to zero. In FIG. 11, when the difference in scanning timing of the subject A between the imaging device 9a of the first imaging unit 1, which is a wide-angle camera, and the imaging device 9b of the second imaging unit 2, which is a narrow-angle camera, is brought close to zero, The delay time Δtv of the scanning start timing will be explained.

As shown in FIG. 8, when scanning is started at the same scanning timing, the time required to scan the subject A is:
tv1=th1×v1
In the imaging element 9b of the second imaging unit 2,
tv2=th2×v2
becomes. Then, the difference Δtv in scanning timing with respect to the object A between the first imaging unit 1 and the second imaging unit 2 is
Δtv=tv1−tv2=(th1×v1)−(th2×v2)
It turns out that

Therefore, as shown in FIG. 11, by shifting the scanning start timings of the imaging device 9a of the first imaging unit 1 and the imaging device 9b of the second imaging unit 2 by Δtv, the scanning timing deviation of the subject A can be reduced. can be done.

That is, the imaging apparatus 10 stores the scanning timing difference Δtv with respect to the subject A between the first imaging unit 1 and the second imaging unit 2 in the storage unit 3 as the delay time 17 .

Then, the delay instruction unit 5 of the imaging device 10 reads out the delay time 17 from the storage unit 3 and instructs the delay of the scanning timing of the first imaging unit 1 and the second imaging unit 2 .

FIG. 12 is a diagram showing an example in which scanning timing is delayed. As shown in FIG. 12 , the delay instructing unit 5 of the imaging device 10 delays the scanning timing shift Δtv, which is the delay time 17 read from the storage unit 3, to the image sensor 9a of the first imaging unit 1, which is a wide-angle camera. The scanning timing is delayed so that the difference between the scanning timing and the scanning timing of the imaging element 9b of the second imaging section 2 of the narrow-angle camera approaches zero.

According to the example shown in FIG. 12, the scanning time th1 per line in the imaging device 9a of the first imaging unit 1 and the scanning time th2 per line in the imaging device 9b of the second imaging unit 2 satisfy th1<th2. Because of the relationship, the difference in scanning timing increases as scanning progresses. However, according to the example shown in FIG. 12, the scanning timing deviation of the lines before and after the object A is reduced compared to the example shown in FIG.

As described above, according to the present embodiment, at the scanning start timing of the wide-angle camera (first imaging unit 1) and the narrow-angle camera (second imaging unit 2) having imaging elements with different resolutions, the left and right imaging elements are adjusted in advance. Based on the difference in scanning completion timing, one of the imaging elements is delayed by the time difference between the scanning completion timings so that the left and right imaging elements scan at the same timing in an arbitrary area (area where the same subject exists). Change the scan start trigger of the image sensor. As a result, it is possible to reduce the deviation in scanning timing that occurs in the left and right imaging units having different angles of view.

In the above description, the case of the subject A was described as an example, but it goes without saying that the target for reducing the scanning timing deviation may be adjusted to any region.

In the above description, the case where th1<th2 holds is explained as an example. By shifting the scanning start timing of the imaging element 9b of the imaging unit 2 by Δtv, the deviation of the scanning timing of the subject A can be reduced.

(Second embodiment)
Next, a second embodiment will be described. In the first embodiment, an area in which the same subject exists is applied as an arbitrary area in the captured image. This embodiment differs from the first embodiment in that it applies an area in which there is a vanishing point where the directions of movement converge. Hereinafter, in the description of the second embodiment, the description of the same portions as those of the first embodiment will be omitted, and the portions that differ from the first embodiment will be described.

FIG. 13 is a block diagram showing the hardware configuration of the control system of the imaging device 10 according to the second embodiment. As shown in FIG. 13, the imaging device 10 further includes a vanishing point detection unit 4 that detects vanishing points and the like, compared to the imaging device 10 shown in FIG.

Here, the vanishing point will be explained.

FIG. 14 is a diagram exemplifying a vanishing point. The example shown in FIG. 14 shows a case in which the first imaging unit 1, which is a wide-angle camera, and the second imaging unit 2, which is a narrow-angle camera, are arranged with the optical axis direction having a depression angle with respect to the Z axis. be. In this case, the vertical angle of view of the first imaging section 1, which is a wide-angle camera, is denoted by 1c, and the vertical angle of view of the second imaging section 2, which is a narrow-angle camera, is denoted by 2c.

In vehicles with a high vehicle height, such as trucks, the camera mounting position is high, and the reflection of the road surface on the near side is reduced. be done.

Normally, collision determination in a vehicle control system is performed with respect to an object in the traveling direction. For this reason, as shown in FIG. 14, it is important to match the scanning timing with respect to the subject A imaged in the traveling direction D of the imaging device 10 .

As shown in FIG. 14, the subject A appearing in the traveling direction D of the imaging device 10 appears at the same position as the vanishing point where the moving directions of the stationary objects converge on the captured image. Therefore, the imaging apparatus 10 should adjust the scanning timing to the vanishing point on the captured image.

The vanishing point detection unit 4 reads the first captured image 11-1 captured by the first imaging unit 1 and the second captured image 11-2 captured by the second imaging unit 2, and detects the vanishing point. As a method for detecting a vanishing point from a captured image, for example, a method described in the following literature is known.
Reference: So Yamamoto and Kunihiko Kaneko (2011). Parallelization of vanishing point trajectory estimation processing from moving camera image database by MapReduce, 3-4.https://db-event.jpn.org/deim2011/proceedings/pdf/ e9-2.pdf

Then, the vanishing point detection unit 4 writes the delay time at which the scanning timings of the vanishing points on the captured images of the first imaging unit 1 and the second imaging unit 2 match to the storage unit 3 as the delay time 17 .

In the above description, the vanishing point on the captured image is used to detect the traveling direction of the vehicle. However, the present invention is not limited to this. You may make it detect.

Next, the process of reflecting the delay time from the vanishing point by the vanishing point detector 4 will be described in detail.

FIG. 15 is a flow chart showing the flow of processing for reflecting the delay time from the vanishing point. As shown in FIG. 15, the vanishing point detection unit 4 reads out the first captured image 11-1 captured by the first imaging unit 1 and detects a vanishing point F1 (step S1).

Next, the vanishing point detection unit 4 reads out the second captured image 11-2 captured by the second imaging unit 2, and detects the vanishing point F2 (step S2).

Note that the detection of the vanishing point F1 of the first imaging unit 1 and the detection of the vanishing point F2 of the second imaging unit 2 may be processed in parallel.

Next, the vanishing point detector 4 calculates the delay time Δtv of the scanning timing from the vertical line number v1 of the vanishing point F1 and the vertical line number v2 of the vanishing point F2 (step S3).
Δtv=tv1−tv2=(th1×v1)−(th2×v2)

Then, the vanishing point detection unit 4 writes the delay time Δtv of the scanning timing into the storage unit 3 as the delay time 17 (step S4).

As described above, according to the present embodiment, at the scanning start timing of the wide-angle camera (first imaging unit 1) and the narrow-angle camera (second imaging unit 2) having imaging elements with different resolutions, the left and right imaging elements are adjusted in advance. Based on the difference in scan completion timing, one of the image sensors is delayed by the amount of the time difference between the scan completion timings, and the left and right image sensors scan an arbitrary area (an area with a vanishing point where the moving direction of a stationary object converges). The scanning start trigger of the image sensor is changed so that the timing becomes the same. As a result, it is possible to reduce the deviation in scanning timing that occurs in the left and right imaging units having different angles of view.

(Example)
(Example of application to automobiles)
Next, the imaging device 10 of each of the embodiments described above can be installed in a mobile device such as a robot device, an automobile, an aircraft, or a ship. An application example in which the imaging device 10 is applied to a moving object will be described below.

First, FIG. 16 is a diagram showing an example in which the imaging device 10 of the embodiment is provided for an object detection device for an automobile. As shown in FIG. 16, the object detection device including the imaging device 10 is installed on the windshield of an automobile or the like, and detects pedestrians, road signs, road surfaces, preceding vehicles, guardrails, etc. in a predetermined imaging range in front of the traveling direction. is imaged. The imaging device 10 has the configuration of a stereo camera as described above, and captures two images of the left-eye field of view and the right-eye field of view.

FIG. 17 is a diagram showing a schematic configuration of an object detection device. The object detection device outputs two images captured by the imaging device 10 to a vehicle ECU (Engine Control Unit) 50 . The vehicle ECU 50 analyzes the parallax image based on each image from the imaging device 10, and performs control such as engine control, braking control, lane keeping assist, steering assist, etc. of the automobile based on the analysis results.

FIG. 18 is a diagram showing the configuration of an object detection device. As shown in FIG. 18 , an imaging device 10 including a first imaging section 1 and a second imaging section 2 is connected to an image processing device 30 . The first imaging section 1 and the second imaging section 2 are provided with lenses 8a and 8b, imaging elements 9a and 9b, and sensor controllers 23a and 23b, respectively. The sensor controller 23 performs, for example, exposure control, image readout control, communication with external circuits, image data transmission control, and the like for each of the imaging elements 9a and 9b.

The image processing device 30 is provided in the vehicle ECU 50 shown in FIG. 17, for example. The image processing device 30 includes, for example, a data bus line 300 , a serial bus line 302 , a CPU (Central Processing Unit) 304 , an FPGA (Field-Programmable Gate Array) 306 , a ROM (Read Only Memory) 308 , a RAM (Random Access Memory) 310 . , a serial IF (Interface) 312 and a data IF (Interface) 314 .

The imaging device 10 is connected to the image processing device 30 via a data bus line 300 and a serial bus line 302 . The CPU 304 controls the overall operation of the image processing apparatus 30 and executes image processing and image recognition processing. The captured images captured by the image sensors 9 a and 9 b of the first imaging section 1 and the second imaging section 2 are written to the RAM 310 of the image processing device 30 via the data bus line 300 . Sensor exposure value change control data, image read parameter change control data, various setting data, and the like from the CPU 304 or FPGA 306 are transmitted and received via the serial bus line 302 .

The FPGA 306 performs processing that requires real-time processing on the image data stored in the RAM 310, such as gamma correction, distortion correction (parallelization of left and right images), and parallax calculation by block matching to generate the parallax image. Generate and write to RAM 310 again. Note that the parallax image is information in which the vertical position, horizontal position, and depth direction position of an object are associated with each other.

The CPU 304 controls the sensor controllers 23 a and 23 b of the imaging device 10 and the overall control of the image processing device 30 . The CPU 304 acquires, for example, CAN (Controller Area Network) information of the own vehicle as parameters (vehicle speed, acceleration, steering angle, yaw rate, etc.) via the data IF 314 .

Vehicle control data, which is recognition data of objects such as preceding vehicles, people, guardrails, road surfaces, etc., is supplied to the vehicle ECU 50 via the serial IF 312, and is provided as a control function of the vehicle ECU 50, such as an automatic braking system or a driving assist system. etc. The automatic braking system performs brake control of the automobile. In addition, the driving assist system performs driving lane keeping assist, steering assist, and the like of the automobile.

(Example of application to an aircraft)
Next, FIGS. 19 and 20 are diagrams showing an installation example of the imaging device 10 of the embodiment with respect to an aircraft. FIG. 19 shows a top view of the flying object provided with the imaging device of the embodiment, and FIG. 20 shows a front view of the flying object provided with the imaging device of the embodiment. As shown in FIGS. 19 and 20, the flying object is a remotely controllable flying object having a frame 101, a driving device 102, rotors 103 and legs 104. FIG. The aircraft also has an imaging device 10 that captures a still image or a moving image of an object, which is a subject.

For example, as shown in FIG. 19, the frame 101 is arranged such that a web portion 101a and flange portions 101b arranged at both ends of the web portion 101a form an H shape in plan view. Note that the shape of the frame 101 is not limited to this. For example, the frame 101 may be configured to have four arms extending radially from the center of the flying vehicle and provided in an X shape so as to form mutually perpendicular angles. Also, the frame 101 may have a structure in which arms radially extend from the center in the number corresponding to the number of the driving devices 102 and the rotor blades 103 .

The driving device 102 is a motor that rotates the rotor blades 103 . As an example, drive device 102 may be a brushless motor. The driving device 102 is attached to the tip of each of the two flange portions 101 b and connected to the rotating shaft of the rotor blade 103 . Each drive device 102 is controlled independently of each other by the control device 100 .

The rotor blades 103 are attached to the respective driving devices 102 and rotate when the driving devices 102 are driven. The rotation speed of the rotor blades 103 is adjusted by independently controlling the drive of each drive device 102 . Note that FIG. 19 illustrates the flying object as having four rotor blades 103, but if the number of rotor blades 103 is three or more, the flying object can fly stably. Furthermore, it is preferable to provide an even number of rotor blades 103, such as four, six, or eight. 19 and 20, the rotor blades 103 are not limited to a form in which a plurality of rotor blades 103 are arranged substantially on the same plane, and a plurality of rotor blades arranged substantially on the same plane. A configuration in which the blades 103 are laminated in multiple stages may also be used.

The leg portion 104 is connected to the bottom surface of the frame 101 at a substantially central position. The leg portion 104 has an inverted V-shaped lower portion to protect the flying object from impact when the flying object lands on the ground.

The control device 100 is provided in the web portion 101a of the frame 101, for example. The control device 100 controls each driving device 102 and the like according to signals from various sensors provided on the flying object according to purposes and operation signals transmitted from the outside.

In addition, the control device 100 independently controls the rotation speed of each rotor blade 103 by independently controlling the driving of each drive device 102, and controls the ascent, descent, forward movement, and retreat of the aircraft, as well as the yaw axis ( Z-axis), and moves into the air to control actions such as hovering (stop flight). In addition, the control device 100 tilts the attitude of the flying object to the left and right to control the movement of the flying object in the left and right direction in the same manner as forward movement and backward movement.

The imaging device 10 is provided on the web portion 101a of the frame 101 via a spacer that absorbs vibrations transmitted through the web portion 101a. This can reduce the influence of the vibration of the frame 101 and the web portion 101a on the measurement of the distance to the structure. Also, as will be described later, the supporting member (housing) of the imaging device 10 is formed of a carbon fiber reinforced plastic member (CFRP: Carbon Fiber Reinforced Plastic) in which the fiber direction is adjusted.

As a result, the imaging device 10 reduces the effects of external loads such as bending, twisting, and elongation, which are applied by direct external factors such as wind and contact. For this reason, the imaging device 10 can measure the measurement target (object) with high accuracy by using the spacer and the CFRP in which the directions of the fibers are adjusted.

Such an imaging device 10 images the surroundings of an aircraft. As an example, the imaging device 10 is a so-called stereo camera device having an imaging device 9 . The imaging device 10 is fixed to the web portion 101 a of the frame 101 . The imaging device 10 is provided on the web portion 101a so that the optical axis is horizontal during horizontal flight in which the aircraft is parallel to the ground. Further, the imaging device 10 is provided on the web portion 101a so that the imaging direction coincides with the advancing direction of the aircraft.

By providing a plurality of imaging devices 10, it may be possible to capture an omnidirectional (360°) panoramic image around the aircraft (omnidirectional camera device).

As described above, the imaging device 10 of the embodiment can be installed in a mobile object such as an automobile or an aircraft. It is also applicable to industrial robots and the like. In either case, the same effect as described above can be obtained.

Such novel embodiments can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. The embodiments and modifications of the embodiments are included in the scope and gist of the invention, as well as the invention described in the claims and the scope of equivalents thereof.

1 first imaging unit 2 second imaging unit 4 vanishing point detection unit 5 delay instruction unit 10 imaging device

Japanese Patent No. 5683025

Claims (8)

a first imaging unit having a rolling shutter type imaging element;
a second imaging unit having a rolling shutter imaging device with a resolution different from that of the first imaging unit;
the first imaging unit and the second imaging unit in an arbitrary area in the first captured image captured by the first imaging unit and an arbitrary area in the second captured image captured by the second imaging unit; a delay instruction unit that delays the scanning start timing of either the first imaging unit or the second imaging unit by a delay time that is the deviation of the scanning completion timing;
An imaging device comprising: An arbitrary region in the first captured image and an arbitrary region in the second captured image are regions in which the same subject exists,
2. The imaging device according to claim 1, wherein: The arbitrary region in the first captured image and the arbitrary region in the second captured image are regions in which there exists a vanishing point where the moving direction of the stationary object converges on the first captured image and the second captured image. is
2. The imaging device according to claim 1, wherein: A vanishing point detection unit that detects a vanishing point where the moving direction of the stationary object converges on the first captured image and the second captured image,
4. The imaging device according to claim 3, characterized in that: a first imaging unit having a rolling shutter type imaging element;
a second imaging unit having a rolling shutter imaging device with a resolution different from that of the first imaging unit;
a computer that controls an imaging device comprising
The first imaging unit and the second imaging unit in an arbitrary region in the first captured image captured by the first imaging unit and an arbitrary region in the second captured image captured by the second imaging unit A program for functioning as delay instructing means for delaying the scanning start timing of either the first imaging section or the second imaging section by a delay time, which is the difference in scanning completion timing between the first imaging section and the second imaging section. An arbitrary region in the first captured image and an arbitrary region in the second captured image are regions in which the same subject exists,
6. The program according to claim 5, characterized by: The arbitrary region in the first captured image and the arbitrary region in the second captured image are regions in which there exists a vanishing point where the moving direction of the stationary object converges on the first captured image and the second captured image. is
6. The program according to claim 5, characterized by: causing the computer to function as vanishing point detection means for detecting a vanishing point at which the moving direction of the stationary object converges on the first captured image and the second captured image;
8. The program according to claim 7, characterized by:

Images