JP7206602B2 - Imaging device, vehicle, and imaging method - Google Patents

Description

The present invention relates to an imaging device, a vehicle, and an imaging method.

Space recognition and environment recognition using three-dimensional information that can be obtained using a stereo camera or the like has become popular in applications such as in-vehicle use and monitoring. In a stereo camera or the like, parallax, which is a relative deviation amount of pixel groups between images, is calculated from a pixel group in one reference image and a pixel group in the other image. The distance to the subject is measured by triangulation based on the calculated parallax.

Stereo cameras have a trade-off relationship between the wide field of view and the distance measurement resolution, and achieving both is one of the challenges. As a stereo camera that has a wide field of view and performs high-precision distance measurement, for example, a technology for measuring distance with a stereo camera that has a camera with a low spatial resolution but a wide field of view and a camera with a narrow field of view but a high spatial resolution is disclosed. (See Patent Document 1, for example).

However, in an imaging device that has a plurality of cameras that acquire images with different spatial resolutions, such as a stereo camera in Patent Document 1, and that measures the distance to a subject, one of the cameras has a low spatial resolution. However, there is a problem that the spatial resolution of the range image to be processed becomes low.

The present invention has been made in view of the above points, and provides an imaging apparatus that has a plurality of cameras that acquire images with different spatial resolutions and measures the distance to an object to form a distance image. An object of the present invention is to improve the spatial resolution of an image.

An imaging device according to an aspect of the disclosed technology includes a camera that acquires a first image having a first spatial resolution in a predetermined field of view, and a second image having a second spatial resolution that is lower than the first spatial resolution. and a camera that obtains an image of is arranged at a predetermined interval in a predetermined arrangement direction, and a distance image is formed from the distance to the subject measured based on the first image and the second image. wherein the range image has the second spatial resolution in the array direction and the first spatial resolution in a direction crossing the array direction. .

According to an embodiment of the present invention, in an imaging apparatus that has a plurality of cameras that acquire images with different spatial resolutions and that measures the distance to a subject and forms a range image, the spatial resolution of the range image is improved. can be done.

It is a figure showing an example of composition of an imaging device of a 1st embodiment. 4A and 4B are diagrams illustrating an example of an image acquired by the imaging device of the first embodiment; FIG. 3 is a diagram showing functional blocks of components of the image processing board of the first embodiment; FIG. It is a figure explaining an example of the process by the image area extraction part of 1st Embodiment. It is a figure explaining an example of the process by the pixel number matching part of 1st Embodiment. It is a figure explaining the outline|summary of the detection method of parallax. It is a figure explaining an example of a cost curve. It is a figure explaining an example of the process by the parallax detection part of 1st Embodiment. 7 is a flowchart illustrating an example of processing by a parallax detection unit according to the first embodiment; It is a figure explaining an example of the distance image of 1st Embodiment. 4 is a flow chart showing an example of processing by the image processing board of the first embodiment; It is a figure which shows an example of a structure of the vehicle which mounts the imaging device of 2nd Embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In each drawing, the same components are denoted by the same reference numerals, and redundant description may be omitted.

[First Embodiment]
FIG. 1 is a diagram showing an example of the configuration of an imaging device according to this embodiment.

The imaging device 100 has a camera 1 , a camera 2 and an image processing board 10 .

The camera ₁ has a lens 31-1, an image sensor 32-1, and _an image sensor controller _33-1 . The camera ₂ has a lens 31-2, an image sensor _32-2 , and an image sensor controller _33-2 . In the following description, the lenses 31 ₁ and 31 ₂ are simply referred to as the lens 31, the image sensors 32 ₁ and 32 ₂ are simply referred to as the image sensor 32, and the image sensor controllers 33 ₁ to 33 _N are simply referred to as the image sensor controller 33. There is

The camera ₁ captures an image of the subject 50 through the lens 31-1 with the image sensor _32-1 and outputs it. Similarly, the camera ₂ captures an image of the object 50 through the lens 31-2 with the image sensor _32-2 and outputs it.

_The focal length f1 of the lens 31-1 of the camera ₁ is different from the focal length f2 of the lens 31-2 of the camera ₂ , and the focal length _f2 is shorter _than the focal length f1. As a result, the optical magnification of camera 2 is lower than that of camera 1 . Since the optical magnification is low, camera 2 acquires an image with a wider field of view than camera 1 . On the other hand, the spatial resolution of the image by camera 2 is lower than that of camera 1 . The field of view and spatial resolution of such images will be discussed in greater detail below.

The image sensors 32-1 and 32-2 provided in the cameras ₁ and ₂ have the same specifications. Similarly, the image sensor controller 33-1 and the image sensor controller _33-2 provided in the camera ₁ have the same specifications.

The image sensor 32 is, for example, an imaging device, and a CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) can be used as the imaging device.

The cameras 1 and 2 are arranged in a direction intersecting the optical axis of the lens 31 and fixed on the base plate. _The optical axis of the lens 31-1 and the optical axis of the lens 31-2 _are substantially parallel. The distance between the optical axis of the lens 31-1 provided in the camera ₁ and the optical axis of the lens 31-2 provided in the camera ₂ in the direction in which the cameras 1 and 2 are arranged is referred to as a base line length B. FIG.

The direction in which the cameras 1 and 2 are arranged is an example of the "arrangement direction", and the baseline length B is an example of the "predetermined interval". The direction crossing the direction in which the cameras 1 and 2 are arranged is an example of the "direction crossing the arrangement direction." The array direction corresponds to the horizontal direction of the images captured by cameras 1 and 2 , and the direction crossing the array direction corresponds to the vertical direction of the images captured by cameras 1 and 2 .

The image sensor controller 33 performs exposure control of the image sensor 32, readout control of image data, communication with external circuits, transmission of image data, and the like. The image data is luminance image data.

Camera 1 and camera 2 are connected to image processing board 10 via data bus 20 and serial bus 21, respectively.

The image processing board 10 includes a CPU (Central Processing Unit) 11, an FPGA (Field-Programmable Gate Array) 12, a RAM (Random Access Memory) 13, a ROM (Read Only Memory) 14, and a serial IF (Interface) 15. , data IF16. These are interconnected via a data bus 20 and a serial bus 21 .

The CPU 11 comprehensively controls the operation of the image processing board 10 . The CPU 11 also executes image processing, image recognition processing, and the like for images captured by the cameras 1 and 2 . The CPU 11 executes the programs stored in the ROM 14 or the like using the RAM 13 as a work area (work area), thereby executing the above-described control and processing, and realizing various functions described later. Some or all of the functions of the CPU 11 may be realized by hardware based on wired logic.

Image data from cameras 1 and 2 are transferred from the image sensor 32 to the RAM 13 of the image processing board 10 through the data bus 20 .

The FPGA 12 executes processing that requires real-time processing on the image data stored in the RAM 13 . Processing that requires real-time processing includes, for example, image processing such as gamma correction and distortion correction (parallelization of left and right images). The FPGA 12 also performs parallax calculation by block matching on the image data from the cameras 1 and 2 to generate parallax images. The generated parallax image is written back to the RAM 13 .

The CPU 11 and FPGA 12 change sensor exposure control values, change image reading parameters, and transmit/receive various setting data to/from the image sensor controller 33 via the serial bus 21 .

The serial IF 15 is a connection interface that sequentially transmits digital data bit by bit or uses such a system. The data IF 16 is an interface for connecting the image processing board 10 and an external device such as a PC (Personal Computer).

FIG. 2 is a diagram showing an example of images acquired by cameras 1 and 2. As shown in FIG. (a) shows the subject in the real environment, (b) shows the image 1 _Im acquired by the camera 1, and (c) shows the image 2 _Im acquired by the camera 2. FIG. The image of the subject 50 is shown as subject image 50 _Im1 in camera 1 and as subject image 50 _Im2 in camera 2 .

As shown in FIG. 2, image 2 _Im captured by camera 2 has a wider field of view than image 1 _Im captured by camera 1, and a portion of the field of view of image 2 _Im has a larger field of view than image 1 _Im . Includes field of view. A parallax corresponding to the distance to the subject and the base length B is included between the image 1 _Im and the image 2 _Im . In addition, below, the distance to an object may be called object distance.

On the other hand, the subject image 50 _Im2 is imaged smaller than the subject image 50 _Im1 . This means that the subject image 50 _Im2 has fewer pixels than the subject image 50 _Im1 , which is an image of the same subject 50, and has a lower spatial resolution. That is, image 2 _Im has a lower spatial resolution than image 1 _Im .

The spatial resolution is a value obtained by dividing the pixel size of the image sensor 32 by the optical magnification of the lens 31 . The optical magnification is approximately the focal length of the lens 31 divided by the subject distance. Therefore, in this embodiment, the difference in spatial resolution between Image 1 _Im and Image 2 _Im is determined by the difference in the focal lengths of lens 31 - ₁ and lens 31 - ₂ . However, it is not limited to this. The image sensor 32-1 of the camera ₁ and the image sensor _32-2 of the camera 2 may have different specifications such as pixel size or number of pixels so as to have different spatial resolutions. Also, the spatial resolutions of the cameras 1 and 2 may be differentiated by combining the specifications of the lens 31 and the specifications of the image sensor 32 .

Note that the term "pixel pitch" may be used in this embodiment, but the term "pixel pitch" is used to express spatial resolution obtained by dividing the "pixel size" by the optical magnification. On the other hand, when the term "pixel size" is used, it simply refers to the pixel size of the image sensor 32 without considering the optical magnification.

Camera 1 is an example of "a camera that acquires a first image having a first spatial resolution", Image 1 _Im is an example of a "first image", and the spatial resolution of Image 1 _Im is a "first 1 spatial resolution”. Further, the camera 2 is an example of "a camera that acquires a second image having a second spatial resolution lower than the first spatial resolution", the image 2 _Im is an example of a "second image", and the image A spatial resolution of 2 _Im is an example of a "second spatial resolution."

In the following, an example will be described in which the spatial resolution of image 1 _Im is twice the spatial resolution of image 2 _Im . This double means double in the horizontal direction and double in the vertical direction.

In the imaging apparatus 100 of the present embodiment, the image 1 _Im and the image 2 _Im are acquired, and the image processing board 10 executes predetermined processing to form a distance image. A distance image is an image formed by two-dimensionally arranging pixels or groups of pixels whose distances have been measured.

FIG. 3 is a diagram showing functional blocks of components of the image processing board 10 included in the imaging apparatus 100 of the present embodiment. Each functional block shown in FIG. 3 is conceptual, and does not necessarily need to be physically configured as shown. All or part of each functional block can be functionally or physically distributed and combined in arbitrary units. All or any part of each processing function performed in each functional block can be realized by the program executed by the above-described CPU 11, or can be realized as hardware by wired logic. The image processing board 10 is an example of the "processing means".

The image processing board 10 has an image region extracting section 25 , a pixel number adjusting section 26 , a parallax detecting section 27 and a distance image forming section 28 .

The image area extraction unit 25 extracts an image area corresponding to the field of view of the image 1 _Im acquired by the camera 1 from the image 2 _Im acquired by the camera 2 . Note that the image area extracting unit 25 is an example of "image area extracting means".

FIG. 4 is a diagram for explaining an example of extraction of the image area as described above by the image area extraction unit 25 of this embodiment. (a) shows image 1 _Im acquired by camera 1 and (b) shows image 2 _Im acquired by camera 2 . An image area 2 _area indicated by a dashed line in (b) is an image area corresponding to the field of view of the image 1 _Im . (c) shows an image area 22 that is the result of extracting such an image area 2 _area from the image 2 _Im .

When the imaging apparatus 100 is assembled, the relative positional relationship between the cameras 1 and 2 is adjusted, and the image area corresponding to the field of view of the image 1 _Im is determined as a predetermined area of the image 2 _Im . Therefore, the image area 22 can be extracted by cutting out a predetermined area of the image 2 _Im .

Returning to FIG. 3, the pixel number matching unit 26 performs a process of matching the number of pixels of the image area 22 to the number of pixels of the image 2 _Im . It should be noted that the pixel number matching unit 26 is an example of the "pixel number matching means".

FIG. 5 is a diagram illustrating an example of processing for matching the number of pixels by the pixel number matching unit 26 of this embodiment. In FIG. 5, (a) shows the image 1 _Im , and (b) shows the pixel number matching image 22a after the pixel number matching process is executed. The pixel number adjustment image 22a in (b) is obtained by enlarging the image area 22 by two times in the horizontal direction and by two times in the vertical direction. As a result of the enlargement, the number of pixels forming the image 1 _Im and the total number of pixels forming the pixel number adjustment image 22a are the same.

Focusing on the pixels forming the pixel number matching image 22a, the brightness of the pixel 22p forming the image area 22 is copied to the adjacent pixels in the horizontal, vertical and diagonal directions, and is shown by the dashed line in (b) and displayed in gray. A pixel 22ap filled with is composed of four pixels 22p. When such processing is executed for all the pixels forming the image area 22, the pixel count image 22a is obtained by expanding the image area 22 by two times in the horizontal direction and by two times in the vertical direction. become an image. Since the camera 1 and the camera 2 use the image sensor 32 of the same specification, the pixel 1p of the image 1 _Im shown in (a) and the pixel 22p of the pixel number adjustment image 22a shown in (b) are the same. pixel size.

The pixel pitch obtained by dividing the pixel size of the pixel 1p by the optical magnification as described above represents the "first spatial resolution" and is an example of the "first pixel pitch representing the first spatial resolution". On the other hand, the pixel pitch obtained by dividing the pixel size of the pixel 22ap (the pixel indicating the area painted in gray) by the optical magnification represents the "second spatial resolution" and the "second pixel pitch representing the second spatial resolution". is an example of The pixel number adjustment image 22a is an example of the "pixel number adjustment image".

Returning to FIG. 3, the parallax detection unit 27 detects parallax between the image 1 _Im and the pixel count image 22a. Note that the parallax detection unit 27 is an example of a “parallax detection unit”.

Here, an overview of the parallax detection method will be described with reference to FIGS. FIG. 6 is a diagram showing an image 41 _Im and an image 42 _Im having parallax. The image 41 _Im and the image 42 _Im each include a subject image 51 _Im . Due to parallax, the position of the subject image 51 _Im in the image 42 _Im is displaced from the position of the subject image 51 _Im in the image 41 _Im by the parallax d in the horizontal direction of the image, that is, in the arrangement direction.

For example, in FIG. 6, the image area around the subject image 51 _Im in the image 41 _Im is shifted rightward in the figure, that is, in the arrangement direction by one pixel, and the subject image 50 of the image 41 _Im and the image 42 _Im is shifted at each shift. Compute the difference of the image region around _Im . The image area difference is a process of obtaining the luminance difference between two images for each pixel constituting the image area and calculating the sum or average of the luminance differences of all pixels in the image area. When the subject image 51 _Im is shifted by the parallax d, such a difference in image regions is minimized.

FIG. 7 is a diagram for explaining the change in the difference of the image regions accompanying the shift of the subject image 51 _Im . When the image area around the subject image 51 _Im in the image 41 _Im is shifted, the difference between the image areas is sometimes minimized as shown in FIG. The shift value at this time is the detected value of the parallax d.

In order to obtain the minimum value of the difference in the image area, it is necessary to shift the image area around the subject image 50 _Im within a certain range so that the shift value that minimizes the difference is included. The range is called a “parallax search range”. The range indicated by "SW" in FIGS. 6 and 7 is the "parallax search range".

The image region difference is an example of the “similarity evaluation value”. Below, the “similarity evaluation value” may be referred to as “cost”. Also, the change in the difference of the image area due to the shift may be referred to as a "cost curve". The process of acquiring such a cost curve is an example of "parallax search process".

To simplify the explanation above, an example of shifting the image area around the subject image 51 _Im was shown _. A difference from a minute block of the same size in _Im may be obtained. Micro blocks include blocks of 1×1 pixels. According to this, the object distance can be calculated for each minute block. A distance image can be formed by two-dimensionally arranging minute blocks for which subject distances have been calculated to form an image. Note that the minute block is an example of "a pixel group of part or all of an image". A process of evaluating similarity using minute blocks is a so-called block matching process.

Although an example of a difference between image regions is shown as the "similarity evaluation value", any known similarity evaluation value may be used as the "similarity evaluation value" for block matching.

In the above, an example of calculating the "similarity evaluation value", that is, the cost between the image 42 _Im while shifting the image 41 _Im has been described. An image for calculating the cost between is referred to as a comparative image. In the above example, image 41 _Im is the reference image and image 42 _Im is the comparison image.

Since the image area is shifted by one pixel, the parallax detection value is obtained as an integer value of the number of pixels. However, it is also possible to calculate the parallax in decimal units by, for example, approximating the change in cost associated with the shift to a polynomial and calculating the minimum value using the polynomial. In this way, by obtaining the parallax detection value in decimal units, that is, in the order of sub-pixels, it is possible to improve the measurement resolution of the subject distance.

FIG. 8 is a diagram illustrating an example of processing by the parallax detection unit 27 of this embodiment. (a) shows the image 1 _Im , and (b) shows the pixel count image 22a. The X direction indicated by the arrow in the figure is the horizontal direction of the image, and the Y direction is the vertical direction of the image. Therefore, the X direction corresponds to the arrangement direction, and the Y direction corresponds to the direction crossing the arrangement direction.

Pixels 1p(i,j), 1p(i+3,j), and 1p(i,j+1) in (a) denote pixels of image 1 _Im , where i is the X coordinate and j is the Y coordinate in parenthesis. showing the coordinates. The pixel 1p(i+3,j) is shifted from the pixel 1p(i,j) by 3 pixels in the X direction, and the pixel 1p(i,j+1) is shifted from the pixel 1p(i,j) in the Y direction. It means that the position is shifted by 1 pixel.

Pixels 22ap (m, n) and 22ap (m+1, n) in (b) also indicate pixels of the pixel number adjustment image 22a, similarly to (a). Y coordinates are indicated. The pixel 22ap(m+1,n) is shifted by one pixel in the X direction from the pixel 22ap(m,n).

In this embodiment, the pixel count image 22a is used as a reference image, and the image 1 _Im is used as a comparison image.

In (c), the pixel 22ap(m, n) of the pixel number matching image 22a, which is the reference image, is added to the upper left image area of the image 1 _Im in order to calculate the cost with the image 1 _Im , which is the comparison image. shows an example applied to The minute block in this case has a block size of 1×1 pixels. A minute block is an example of "a portion of a pixel-count adjusted image".

In (c), the average luminance of pixels 1p(i, j), 1p(i+1, j), 1p(i, j+1), and 1p(i+1, j+1) and the luminance of pixel 22ap(m, n) Calculate the cost between the values.

Next, (d) shows an example in which the pixel 22ap(m, n) is shifted in the X direction with respect to (c). In this case, the cost is the average luminance of pixels 1p(i+1,j), 1p(i+2,j), 1p(i+1,j+1), and 1p(i+2,j+1) and the luminance of pixel 22ap(m,n) It is calculated between the brightness value. The parallax detection unit 27 calculates the cost curve while shifting in the X direction in this manner. That is, the parallax detection unit 27 executes parallax search processing.

Next, (e) shows an example in which the pixel 22ap(m,n) is shifted in the Y direction by a pixel 1p with respect to (c). In this case, the cost is the average luminance of pixels 1p(i,j+1), 1p(i+1,j+1), 1p(i,j+2), and 1p(i+1,j+2) and the average luminance of pixel 22ap(m,n). It is calculated between the brightness value.

Next, (f) shows an example in which the pixel 22ap(m, n) is shifted in the X direction with respect to (e). In this case, the cost is the average luminance of pixels 1p(i+1,j+1), 1p(i+2,j+1), 1p(i+1,j+2), and 1p(i+2,j+2) and the average luminance of pixel 22ap(m,n). It is calculated between the brightness value. In this manner, the parallax detection unit 27 calculates the cost curve while shifting in the X direction in the region shifted by 1p pixels in the Y direction. That is, the parallax detection unit 27 executes parallax search processing in a region shifted by 1p pixels in the Y direction.

As described above, the parallax detection unit 27 of the present embodiment shifts the minute blocks of the pixel number matching image 22a, which is the reference image, by the pixel pitch of the pixels 1p, and for each shift, the small blocks are configured by the pixel pitch of the pixels 22ap. Calculate the cost for the image That is, parallax search processing is executed. Further, the parallax detection unit 27 performs parallax search processing by shifting the pixel pitch of the pixel 1p in the Y direction.

As described above, pixels 1p have a first pixel pitch representing a first spatial resolution and pixels 22ap have a second pixel pitch representing a second spatial resolution. Therefore, the parallax detection unit 27 of the present embodiment shifts a part or all of the pixel groups of the pixel number adjustment image 22a in the arrangement direction by the first pixel pitch, and calculates the cost for each shift. processing is running.

Further, the parallax detection unit 27 performs parallax search processing while shifting a part or all of the pixel groups of the pixel number adjustment image 22a by the first pixel pitch in the direction intersecting with the arrangement direction.

Furthermore, in the parallax search process, the parallax detection unit 27 calculates the cost for the image configured with the second pixel pitch.

By calculating the cost while shifting the minute blocks of the reference image, a cost curve as shown in FIG. 7 is obtained for each minute block. The parallax is calculated from the shift amount when the cost is minimized.

In the parallax search process, the parallax search range (the number of pixels) may be optimized each time the parallax search process is executed so that the shift that provides the minimum cost value is included in the parallax search range. be. In the present embodiment, the parallax search processing of FIGS. 8C and 8D and the parallax search processing of FIGS. In the comparison image, the target pixels in FIGS. 8C and 8D and the target pixels in FIGS. The pixels are the same, and the parallax search ranges are considered to be substantially the same. Therefore, in the case of the example shown in FIG. 8, the search range of the parallax search process for the rows where the shift amount in the Y direction is an odd number as shown in FIGS. The parallax search range is optimized by determining from the parallax calculated in the even rows (±1 rows in the Y direction) as shown in FIG. 8D, and the processing is efficient.

In the above description, an example is given in which the average value of the brightness of four pixels 1p(i, j), 1p(i+1, j), 1p(i, j+1), and 1p(i+1, j+1) is used to calculate the cost. Although shown, it is not limited to this. For example, the sum of the luminances of four pixels may be used, or the luminance of one of the four pixels may be used as the representative value of the luminances of the four pixels.

FIG. 9 is a flowchart showing an example of processing by the parallax detection unit 27 of this embodiment.

First, the parallax detection unit 27 shifts the minute blocks of the pixel number adjustment image 22a, which is the reference image, and calculates the cost between the image 1 _Im , which is the comparison image, and the cost curve (step S1001). .

Next, from the calculated cost curve, the parallax detection unit 27 obtains the integer value of the shift when the cost is minimized (step S1003). In this embodiment, this shift can be determined at the first pixel pitch.

Next, the parallax detection unit 27 performs sub-pixel estimation based on the cost curve and the integer value of the shift (step S1005). That is, sub-pixel estimation is performed using the equiangular straight line method, the parabolic fitting method, or the like from the integer value and the cost value of the shift when the cost is the minimum and before and after that time. An EEC (sub-pixel estimation error reduction method; Estimation Error Cancel method) method, an equiangular straight line EEC method or a parabolic EEC method combining this with the equiangular straight line method, or a parabolic fitting method may also be used.

Let Cost _min be the minimum cost value in a predetermined pixel of interest, and _{Cost min+1} and Cost _min−1 be the cost values in the vicinity thereof. can be calculated.

d_para=(Cost _min−1 −Cost _min+1 )/{2×(Cost _min−1 −2×Cost _min +Cost _min+1 )} (1)
When shifting the pixel number adjustment image 22a, the parallax detection unit 27 calculates the cost curve while shifting the area to be the minute block when shifting the minute blocks of the pixel number adjustment image 22a. The parallax detector 27 calculates a cost curve for all minute blocks forming the pixel count image 22a, performs sub-pixel estimation, and calculates parallax.

To perform the parallax search process at a first pixel pitch representing a first spatial resolution, parallax is detected and sub-pixel estimated at the first pixel pitch.

Returning to FIG. 3, the distance image forming unit 28 calculates the object distance D using the following equation (2) based on the parallax d and the baseline length B thus detected.

D=(B×f ₂ )/(d×Δ) (2)
where f ₂ is the focal length of the lens ₃₁₂ and Δ is the pixel size of the image sensor 32 . The distance image forming unit 28 is an example of "distance image forming means".

In the present embodiment, since the parallax search process is performed at the first pixel pitch, the distance resolution is the distance measurement resolution when measured at the first spatial resolution. That is, the image acquired by the camera ₂ is an image acquired using a lens with a focal length of f2. However, by executing the parallax search process described above, it is possible to obtain a measurement resolution equivalent to that obtained by measuring a distance using an image acquired using _a lens with a focal length f1 that is _twice f2.

The distance image forming unit 28 replaces the distance with pixel brightness, for example, and arranges the pixel brightness representing the distance of each minute block two-dimensionally to form and output a distance image.

FIG. 10 is a diagram illustrating an example of a distance image according to this embodiment. Pixels 22ap indicated by dashed lines indicate a second pixel pitch representing a second spatial resolution. Pixels 23 hatched with horizontal lines and pixels 24 hatched with oblique lines are pixels for which the parallax is calculated and the distance is calculated by the formula (2).

In this embodiment, the parallax search process is executed by shifting the pixels 1p in the Y direction, that is, by the first pixel pitch. Therefore, as illustrated, pixels 23 and 24 have half the pixel pitch of pixel 22ap in the Y direction, that is, the first pixel pitch. Therefore, the range image has a first spatial resolution in the Y direction, that is, a direction intersecting with the arrangement direction, and the spatial resolution is improved with respect to the second spatial resolution.

Further, the calculated parallax may not only be used as it is, but may be calculated using polynomial fitting or the like.

In the above description, the j+1 line in FIG. 8(a) is parallax-searched with the n-row line in FIG. 8(b). A parallax value may be obtained using a weighted average or the like. By doing so, more accurate parallax calculation can be performed when the spatial resolution difference is doubled or more.

FIG. 11 is a flow chart showing an example of processing by the image processing board 10 of this embodiment.

First, camera 1 and camera 2 each acquire an image of a subject (step S1201).

Next, the image area extraction unit 25 extracts an image area 2 _area corresponding to the field of view of the image 1 _Im acquired by the camera 1 from the image 2 _Im acquired by the camera 2 to acquire an image area 22 . The image area extraction unit 25 outputs the image area 22 to the pixel count adjustment unit 26 (step S1203).

Next, the pixel number matching unit 26 performs a process of matching the number of pixels of the image area 22 with the number of pixels of the image 2 _Im . The pixel number adjustment unit 26 outputs the image 1 _Im and the pixel number adjustment image 22a to the parallax detection unit 27 (step S1205).

Next, the parallax detection unit 27 detects parallax between the image 1 _Im and the pixel number adjusted image 22a. The parallax detection unit 27 outputs the detected parallax to the distance image formation unit 28 (step S1207).

Based on the detected parallax d and the baseline length B, the distance image forming unit 28 calculates the subject distance D using the equation (2). The distance image forming unit 28 replaces the distance with pixel brightness, for example, arranges the replaced pixels two-dimensionally to form a distance image, and outputs the image (step S1209).

Thus, the image processing board 10 can output a distance image based on the images acquired by the cameras 1 and 2. FIG.

As described above, according to this embodiment, in an imaging device that has a plurality of cameras that acquire images with different spatial resolutions and that measures the distance to a subject, the spatial resolution of the acquired range image is improved. can be made

Further, according to the present embodiment, since parallax search processing is performed at the first pixel pitch, distance measurement resolution equivalent to the case of measurement at the first spatial resolution can be obtained. In other words, the image acquired by the camera ₂ is an image acquired using a lens with a focal length of f2, but by executing the parallax search process, it has _a focal length of f1 that is _twice f2. A stereo camera or the like using a lens provides the same measurement resolution as distance measurement.

In this embodiment, a second image, that is, an image with a wide field of view, may be output together with the distance image. This makes it possible to use the luminance image with a wide field of view together with the distance image. For example, in the scene shown in FIG. 2(a), the distance image including the subject 50, which is the vehicle traveling ahead on the road, is output, and the luminance image shown in FIG. 2(c) is output. This makes it possible to recognize obstacles such as plants on the side of the road as well as the distance to the vehicle traveling ahead. It can be used to ensure the safety of vehicles.

[Second embodiment]
Next, an example of the image forming apparatus according to the second embodiment will be explained using FIG. In addition, in the first embodiment, the description of the same components as those of the already described embodiment may be omitted.

FIG. 12 is a diagram showing an example of the configuration of a vehicle equipped with the imaging device of this embodiment.

FIG. 12 is a schematic diagram showing an example in which the imaging device 100 is installed in the rearview mirror 401 of the vehicle 400. As shown in FIG. As shown in FIG. 12, the imaging device 100 is installed on the front side of the rearview mirror 401 of the vehicle 400, that is, on the traveling direction side of the vehicle 400 indicated by the black arrow in the figure. Vehicle 400 is driven by driver 402 . The imaging device 100 acquires an image facing the traveling direction of the vehicle 400 . Note that the position where the imaging device 100 is installed is not limited to the above, and may be any position.

A distance image acquired by the imaging device 100 is transferred to an in-vehicle controller of the vehicle 400 . The in-vehicle controller recognizes a preceding vehicle in front of the vehicle 400 from the distance image, for example, and controls the vehicle 400 to brake when the distance to the preceding vehicle becomes shorter than a specified value. As a result, for example, it is possible to ensure the safety of driving the vehicle 400 .

When the imaging device 100 of the present embodiment is installed in the vehicle 400 , some or all of the functions of the image processing board 10 may be realized by an in-vehicle controller of the vehicle 400 . Also, part of the functions of the on-vehicle controller of the vehicle 400 may be realized by the image processing board 10 .

Other effects are the same as those described in the first embodiment.

Although the image forming apparatus and the image forming method according to the embodiments have been described above, the present invention is not limited to the above embodiments, and various modifications and improvements are possible within the scope of the present invention.

1, 2 Camera 10 Image processing board (an example of processing means)
11 CPUs
12 FPGAs
13 RAM
14 ROMs
15 serial interface
16 data interface
20 data bus 21 serial bus 22a pixel count image 25 image area extractor (an example of image area extractor)
26 pixel number matching unit (an example of pixel number matching means)
27 parallax detection unit (an example of parallax detection means)
28 distance image forming unit (an example of distance image forming means)
31 lens 32 image sensor 33 image sensor controller 50 subject 100 imaging device (an example of imaging device)
400 vehicle (an example of a vehicle)
401 rearview mirror 402 driver B baseline length D object distance SW search parallax range

Japanese Patent No. 2998791

Claims (7)

A camera that acquires a first image having a first spatial resolution in a predetermined field of view and a camera that acquires a second image having a second spatial resolution lower than the first spatial resolution in a predetermined array. are arranged at predetermined intervals in the direction,
An imaging device that forms a distance image from a distance to a subject measured based on the first image and the second image,
The distance image is
having the second spatial resolution in the array direction;
An imaging device having the first spatial resolution in a direction intersecting with the arrangement direction. processing means for forming the distance image based on the first image and the second image;
The processing means
image region extracting means for extracting an image region corresponding to the field of view of the first image in the second image;
a pixel number matching means for forming a pixel number matching image in which the number of pixels of the image area is matched with the number of pixels of the first image;
parallax detection means for detecting a parallax in the arrangement direction between the pixel number adjusted image and the first image;
2. The imaging apparatus according to claim 1, further comprising distance image forming means for forming the distance image from the distance calculated based on the parallax and the distance. The first image is configured with a first pixel pitch representing the first spatial resolution,
The second image is configured with a second pixel pitch representing the second spatial resolution,
The parallax detection means shifts a part or all of the pixel groups of the pixel number adjusted image by the first pixel pitch in the arrangement direction, and for each shift, the pixel number adjusted image and the first pixel group are shifted. Execute parallax search processing for calculating an evaluation value of image similarity,
The parallax detection means performs the parallax search process while shifting the pixel group of the pixel number adjusted image by the first pixel pitch in a direction intersecting with the arrangement direction,
The evaluation value is calculated for an image configured with the second pixel pitch,
3. The imaging apparatus according to claim 2, wherein the imaging apparatus has the first spatial resolution in a direction intersecting with the arrangement direction. 4. The imaging apparatus according to any one of claims 1 to 3, wherein the second image and the distance image are output. A vehicle comprising the imaging device according to any one of claims 1 to 4. A camera that acquires a first image having a first spatial resolution in a predetermined field of view and a camera that acquires a second image having a second spatial resolution lower than the first spatial resolution in a predetermined array. are arranged at predetermined intervals in the direction,
An imaging method for forming a distance image from a distance to a subject measured based on the first image and the second image,
an image region extracting step of extracting an image region corresponding to the field of view of the first image in the second image;
a pixel number matching step of forming a pixel number matching image in which the number of pixels of the image area is matched to the number of pixels of the first image;
a parallax detection step of detecting a parallax in the arrangement direction between the pixel number adjustment image and the first image;
a distance image forming step of forming a distance image from a distance calculated based on the parallax and the distance;
The distance image is
having the second spatial resolution in the array direction;
An imaging method, wherein the first spatial resolution is obtained in a direction intersecting with the arrangement direction. The first image is configured with a first pixel pitch representing the first spatial resolution,
The second image is configured with a second pixel pitch representing the second spatial resolution,
The parallax detection means shifts a part or all of the pixel groups of the pixel number adjusted image by the first pixel pitch in the arrangement direction, and for each shift, the pixel number adjusted image and the first pixel group are shifted. Execute parallax search processing for calculating an evaluation value of image similarity,
The parallax detection means performs the parallax search process while shifting the pixel group of the pixel number adjusted image vertically by the first pixel pitch in a direction intersecting the array direction, and performing the second parallax search process. 3. The parallax of the corresponding position is detected from a plurality of evaluation values calculated by vertically shifting the pixels of the image configured with the pixel pitch of the first pixel pitch. The imaging device described.

Images