JP2020009351A - Image processing device, image processing method, and program - Google Patents

Abstract

To detect an object, even when a subject distance is large.SOLUTION: An image processing device (13) for processing an output signal from an imaging element (11) in which an imaging pixel and a range-finding pixel capable of range-finding by an image-pickup-surface phase difference system are arranged in a predetermined arrangement pattern includes first generation means (130) for generating a first image on the basis of the output signal of the imaging pixel, range-finding means (132) for generating distance information on the basis of the output signal of the range-finding pixel, setting means (133) for setting an area farther than a predetermined distance on the basis of the distance information for the first image, second generation means (130) for generating a second image having resolution equal to or higher than resolution corresponding to the arrangement pattern of the imaging pixel on the basis of the output signals of the imaging pixel and the range-finding pixel for the set area, and detection means (134) for detecting the area of a predetermined object from one of the first image and the second image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing technique when an image acquired by an imaging device is used for object detection.

  2. Description of the Related Art There is known a technology in which a moving body such as an automobile or a robot mounts an imaging device and a distance measuring device, recognizes a surrounding environment, and moves autonomously. Specifically, an image obtained from an imaging device installed on a moving object is analyzed to detect a specific object (a car, a pedestrian, or the like). Next, distance information from the detected object and the distance measuring device installed on the moving body, and a planned moving path of the moving body are comprehensively determined to determine the possibility of collision with the object. Then, an action plan of how to behave, such as stopping or avoiding the moving body itself, is created, and the behavior of the moving body itself is controlled. When these technologies are installed in a vehicle, they are referred to as driving assistance, ADAS (advanced driving assistance system), automatic driving, and the like in order to assist the driver in driving.

  In this series of processing flows, there are several methods for detecting a specific object such as a car or a pedestrian. For example, there is a feature point-based method in which a feature amount is extracted from an image acquired by an imaging device, and a determination is performed using a classifier learned in advance based on the feature amount. Examples of the feature amount include a Harr-Like feature and a HOG (Histograms of Oriented Gradients) feature. As the identification method, there are AdaBoost, SVM (Support Vector Machine), and the like. Other than this method, there is a neural network method in which a feature extraction and a classifier are directly acquired by deep learning using a convolutional neural network. In either method, the position and size of an object to be detected in an image are designated by a rectangle and used for learning. However, the size of the object reflected in the image changes according to the relative distance between the object and the imaging device. When the distance between the object and the imaging device is short, the object appears largely in the image. In order to cope with such a case, hierarchical processing is generally performed in which a plurality of images with gradually reduced resolution are generated from the obtained images, object detection is performed on each image, and the results are integrated. I have. For example, Patent Literature 1 discloses a technique for detecting an object more efficiently by controlling how to lower the resolution using distance information from a distance measuring device.

  Patent Document 2 discloses an imaging device having a distance measurement function that can acquire a distance from an imaging device to a subject (referred to as a subject distance) at a plurality of pixel positions simultaneously with a captured image. In particular, the distance measurement function is realized by arranging a plurality of distance measurement pixels, each of which can detect a subject distance by a phase difference method, on the image plane of the image sensor.

JP 2014-142202 A JP 2017-163538 A

  In the case of an imaging apparatus including the above-described imaging element in which the imaging element and the distance measurement pixel are arranged, object detection is performed using an image generated from the imaging pixel. However, since the distance measuring pixels are also arranged in the image sensor, the resolution of the generated image is reduced, and for example, a small object cannot be detected due to a long subject distance.

  Therefore, an object of the present invention is to make it possible to detect an object even when the subject distance is long.

  The present invention is an image processing apparatus for processing an output signal from an imaging element in which imaging pixels and ranging pixels capable of measuring a distance by an imaging surface phase difference method are arranged in a predetermined arrangement pattern, First generating means for generating a first image based on the output signal of the distance measuring device, distance measuring means for generating distance information based on the output signal of the distance measuring pixel, and the distance information for the first image. Setting means for setting an area that is farther than a predetermined distance based on the setting, and for the area set by the setting means, the arrangement of the imaging pixels based on the output signals of the imaging pixels and the ranging pixels. A second generation unit configured to generate a second image having a resolution equal to or higher than the resolution according to the pattern, and detecting an area of a predetermined object from one of the first image and the second image And detecting means. That.

  According to the present invention, an object can be detected even when the subject distance is long.

FIG. 2 is an explanatory diagram of a configuration of an imaging device including an image processing device and an imaging element. FIG. 3 is an explanatory diagram illustrating a relationship between an imaging optical system, a distance measurement pixel, and a parallax amount and a defocus amount. 9 is a flowchart illustrating a flow of image processing. FIG. 9 is an explanatory diagram of a process of generating an image used for object detection. It is explanatory drawing of setting of the area | region which should be expanded. It is explanatory drawing of the pixel arrangement | positioning shifted by half pixel. It is a figure showing an example of application to a driving support system.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The image processing apparatus according to the present embodiment is an apparatus that processes an output signal from an imaging element in which imaging pixels and ranging pixels that can perform ranging by the imaging surface phase difference method are arranged in a predetermined arrangement pattern. It has a function of detecting a predetermined object from an image generated based on output signals of the elements. In the drawings used in the following description, parts performing the same configuration or processing are denoted by the same reference numerals even if the figure numbers are different.

  FIG. 1A is a diagram illustrating a schematic configuration example when the image processing device (image processing unit 13) of the present embodiment is applied to the imaging device 1. The imaging apparatus 1 includes an imaging optical system 10, an imaging element 11, a control unit 12, an image processing unit 13, a storage unit 14, an input unit 15, a display unit 16, and a communication unit 17.

  The imaging optical system 10 is a photographing lens of the imaging device 1 and has a function of forming a light image of a subject on the imaging element 11. The imaging optical system 10 includes a plurality of lens groups (not shown), and has an exit pupil 101 at a position separated from the image sensor 11 by a predetermined distance. Note that, with respect to the x-axis, y-axis, and z-axis depicted in the drawings of the present embodiment, the z-axis is an axis parallel to the optical axis 102 of the imaging optical system 10, and the x-axis and the y-axis are perpendicular to each other. It is assumed to be an axis and an axis perpendicular to the optical axis (z axis).

  The image sensor 11 is a CMOS (complementary metal oxide semiconductor) or CCD (charge-coupled device), and is an image sensor having not only the function of an image pixel but also the function of a distance measurement pixel by an image plane phase difference method. is there. That is, the imaging element 11 photoelectrically converts the subject image formed on the imaging surface by the imaging optical system 10, and generates an image signal based on the subject image and distance information from the imaging device 1 to the subject. Details of the imaging pixel and the distance measurement pixel will be described later.

  The control unit 12 controls each unit of the imaging device 1. For example, the control unit 12 performs automatic focusing by auto focus (AF), changes the focus position, changes the F-number (aperture), captures an image, the storage unit 14, the input unit 15, the display unit 16, the communication unit 17, and the like. Control.

The image processing unit 13 includes an image generation unit 130, a memory 131, a distance generation unit 132, an area setting unit 133, an object detection unit 134, and an information integration unit 135.
The image generation unit 130 performs various signal processing such as noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, and color correction on the image pickup signal supplied from the image pickup device 11 to obtain an image for viewing and an object. Generate an image to be used for detection. Note that the ornamental image is generated based on an imaging signal from an imaging pixel of the imaging element 11. A detailed description regarding generation of an image used for object detection will be described later. Then, the image data output from the image generation unit 130 is temporarily stored in the memory 131. Of the image data stored in the memory 131, data of an image used for object detection is sent to an object detection unit 134 described later, and data of an ornamental image is used for display on the display unit 16 or transmitted to another device. And so on. Note that image data used for object detection may also be transmitted to another device or the like.

The distance generation unit 132 generates a distance image representing the distribution of the distance information by using a signal (distance information) acquired by the distance measurement pixels included in the image sensor 11 as described later.
The area setting unit 133 sets an area to be enlarged based on the image generated by the image generation unit 130 or the distance image generated by the distance generation unit 132, as described later.
The object detection unit 134 detects an area of a specific object from an image used for object detection generated by the image generation unit 130, and specifies the position and size of the detected object area in the image, as described later. I do.
The information integration unit 135 integrates information from the distance generation unit 132 and the object detection unit 134 as described later, and based on the information, determines how far the object detected by the object detection unit 134 To determine if it exists. The information integration unit 135 may comprehensively determine the detected object and the distance information, the planned moving path of the moving object, and the like, and may determine whether the detected object becomes an obstacle.

  Each unit of the image processing unit 13 can be configured by hardware using a logic circuit. As another form, the image processing unit 13 may be configured by, for example, a central processing unit (CPU) and a memory that stores an arithmetic processing program. In this case, the processing of each unit described above is realized by a software configuration in which the CPU executes an arithmetic processing program in the memory. Further, part of the image processing unit 13 may be realized by a hardware configuration, and the rest may be realized by a software configuration.

The storage unit 14 is a non-volatile storage medium that stores data and intermediate data acquired by the imaging device 1, parameter data used by the imaging device 1, and the like. As the storage unit 14, any storage medium that can read and write at high speed and has a large capacity may be used. As the storage unit 14, for example, a flash memory or the like can be used.
The input unit 15 is an interface that is operated by a user to input information or change settings to the imaging device 1. As the input unit 15, for example, a dial, a button, a switch, a touch panel, or the like can be used.
The display unit 16 displays an image for confirming the composition at the time of shooting, various setting screens, message information, an object detection result, and the like. The display unit 16 is a display device including a liquid crystal display and an organic EL.
The communication unit 17 transmits the captured image and the distance information generated by the image processing unit 13, the information of the detected object, the obstacle determination information when the information integration unit 135 determines the obstacle, and the like to another device. And a function of receiving information transmitted from other devices.

Next, the configuration and function of the image sensor 11 included in the image pickup apparatus 1 of the present embodiment will be described in detail with reference to FIGS.
As described above, the image sensor 11 of the present embodiment includes the imaging pixels and the distance measurement pixels based on the imaging surface phase difference method, and converts the subject image formed on the imaging surface via the imaging optical system 10 into a photoelectric image. It has a function of converting and generating an image signal and distance information based on a subject image.

  FIG. 1B is a schematic xy cross-sectional view of the image sensor 11. As shown in FIG. 1B, the imaging element 11 is configured by arranging a plurality of pixel groups 110 of 4 rows × 4 columns. The pixel group 110 includes R pixels, which are imaging pixels corresponding to red (R), G and B pixels of imaging pixels corresponding to green (G) and blue (B), and imaging corresponding to white (W). Each pixel is composed of W pixels and M pixels, which are distance measurement pixels according to the imaging surface phase difference distance measurement method, arranged for each row. Therefore, from the pixel group 110, an image signal including three colors of R, G, and B from the R, G, and B pixels and white information from the W pixel, and distance information from the M pixel are output. You.

  FIG. 1D schematically shows respective pass wavelength bands of an R color filter provided for an R pixel, a G color filter provided for a G pixel, and a B color filter provided for a B pixel. . In FIG. 1D, a pass band 1121 represents an R color filter, a pass band 1122 represents a G color filter, and a pass band 1123 represents a pass wavelength band of a B color filter. For the W pixel, a color filter may not be used, or a filter having a wide pass wavelength band 1124 as shown in FIG. 1E including, for example, R, G, and B pass wavelength bands may be used. Is also good. By using the signal from the W pixel, it is possible to obtain an image with improved sensitivity and high S / N.

Next, the ranging pixels (M) of the image sensor 11 will be described.
FIG. 1C is a diagram schematically showing a cross section taken along line II ′ of one pixel of the distance measurement pixel M in the pixel group 110 of FIG. 1B. As shown in FIG. 1C, one ranging pixel M includes a light guide layer 113 and a light receiving layer 114. The light guide layer 113 includes a microlens 111 for efficiently guiding a light beam incident on the distance measurement pixel to the two photoelectric conversion units, a filter 112 for passing light in a predetermined wavelength band, and an image reading and pixel (not shown). It is composed of driving wires and the like. The filter 112 is, for example, a filter having a wide pass wavelength band like the W pixel, but this filter is not necessarily required. The light receiving layer 114 includes two photoelectric conversion units, a photoelectric conversion unit 115 and a photoelectric conversion unit 116. Note that wirings for image reading and pixel driving (not shown) may be provided in the light receiving layer 114.

Next, the operation of the photoelectric conversion unit 115 and the photoelectric conversion unit 116 included in the ranging pixel M of the image sensor 11 according to the present embodiment, and the light beams received by the two photoelectric conversion units will be described with reference to FIGS. 2 (B).
FIG. 2A shows an exit pupil 101 of the imaging optical system 10 and a light beam received by the photoelectric conversion unit 115 of the pixel in the image sensor 11 (the light beam is indicated by a dotted line in the figure). FIG. FIG. 2B is a schematic diagram showing a light beam similarly received by the photoelectric conversion unit 116.

  The microlenses 111 shown in FIGS. 2A and 2B are arranged such that the exit pupil 101 and the light receiving layer 114 have an optically conjugate relationship. The light beam that has passed through the exit pupil 101 of the imaging optical system 10 is condensed by the microlens 111 and guided to the photoelectric conversion unit 115 or 116. At this time, the photoelectric conversion unit 115 and the photoelectric conversion unit 116 mainly receive light beams that have passed through different pupil regions, as shown in FIGS. 2A and 2B, respectively. The photoelectric conversion unit 115 receives the light beam that has passed through the pupil region 210, and the photoelectric conversion unit 116 receives the light beam that has passed through the pupil region 220.

  The plurality of photoelectric conversion units 115 included in the image sensor 11 mainly receive the light beam that has passed through the pupil region 210, and the output signals of the plurality of photoelectric conversion units 115 are based on the light beam that has passed through the pupil region 210. The image signal S1 is obtained. Similarly, the plurality of photoelectric conversion units 116 mainly receive the light beam that has passed through the pupil region 220, and the output signals of the plurality of photoelectric conversion units 116 determine the image signal based on the light beam that has passed through the pupil region 220. S2 will be obtained. In addition, from the image signal S1, the intensity distribution of the light image formed on the image sensor 11 can be obtained by the light flux that has passed through the pupil region 210, and similarly, the image signal S2 has passed through the pupil region 220 from the image signal S2. An intensity distribution of a light image formed on the image sensor 11 can be obtained by the light beam.

  Here, the relative positional shift amount (so-called parallax amount) between the image signal S1 and the image signal S2 is a value corresponding to the defocus amount. The relationship between the amount of parallax and the amount of defocus will be described with reference to FIGS. 2C, 2D, and 2E. FIG. 2C, FIG. 2D, and FIG. 2E are diagrams schematically illustrating the imaging element 11 and the imaging optical system 10 of the present embodiment.

In FIGS. 2C to 2D, it is assumed that the light beam 211 is a light beam passing through the pupil region 210 and the light beam 221 is a light beam passing through the pupil region 220.
FIG. 2C shows a state at the time of focusing, and the light beam 211 and the light beam 221 are converged on the image sensor 11. At this time, the amount of parallax between the image signal S1 formed by the light beam 211 and the image signal S2 formed by the light beam 221 becomes zero.
FIG. 2D shows a state where the image is defocused in the negative direction of the z-axis on the image side. At this time, the amount of parallax between the image signal S1 formed by the light beam 211 and the image signal S2 formed by the light beam 221 does not become 0 but has a negative value.
FIG. 2E shows a state where the image is defocused in the positive direction of the z-axis on the image side. At this time, the amount of parallax between the image signal S1 formed by the light beam 211 and the image signal S2 formed by the light beam 221 has a positive value.
From the comparison between FIG. 2D and FIG. 2E, it can be seen that the direction of the positional shift is switched according to the sign of the defocus amount. In addition, it can be seen that a positional shift occurs according to the imaging relationship (geometric relationship) of the imaging optical system 10 according to the defocus amount. The amount of parallax representing the displacement between the image signal S1 and the image signal S2 can be detected by a matching method for obtaining a correlation based on an area using a matching area and a reference area, which will be described later.

  Next, using the flowchart of FIG. 3A, the image processing unit 13 of this embodiment generates an image used for object detection from the start of data acquisition, and outputs a result of object detection using the image. The process flow until the information is integrated will be described. In the following description, each processing step S301 to S307 in the flowchart of FIG. 3A is abbreviated as S301 to S307, and the same applies to other flowcharts described later.

  Here, in the image processing unit 13 of the present embodiment, the image generation unit 130 can generate a first object detection image and a second object detection image as images used for object detection. Although the details will be described later, the first object detection image is an image generated based on an imaging signal from an imaging pixel of the imaging element 11 and has an image having a resolution equivalent to a resolution according to an arrangement pattern of the imaging element. Is generated as In particular, the second object detection image corresponds to the area set by the area setting unit 133 to be enlarged because the distance (subject distance) from the imaging apparatus 1 is long and the object is small, according to the arrangement pattern of the imaging pixels. Is generated as an enlarged image having a resolution equal to or higher than that of the In the following description, the first object detection image is simply referred to as an object detection image, and the second object detection image is particularly referred to as an enlarged object detection image.

First, in S301 of FIG. 3A, the imaging device 1 performs shooting in accordance with the set focal position, aperture, exposure time, and the like, and outputs an output signal from the imaging device 11 to the image processing unit 13. Are recorded in the memory 131 at the same time.
Next, in S302, the image processing unit 13 generates an object detection image (a first object detection image corresponding to the resolution of the image sensor 11) and a distance image.

The generation processing of the object detection image in S302 will be described with reference to the flowchart in FIG. The object detection image is generated by the image generation unit 130.
As shown in FIG. 1B described above, the pixel array of the image sensor 11 includes a row in which each image pixel including R, G, B, and W pixels is arranged, and a distance measurement of the pixel M. It is divided into rows in which pixels are arranged. As shown in FIG. 4, the number of pixels of the image sensor 11 is Ws × Hs pixels in which the horizontal direction (x-axis direction) is Ws pixels and the vertical direction (y-axis direction) is Hs pixels.

  First, in step S311, the image generation unit 130 generates a Bayer array image 411 of Wc × Hc pixels using the signals of the R, G, and B pixels of the image sensor 11. At this time, the number of pixels in the horizontal direction is Wc = Ws / 2, and the number of pixels in the vertical direction is Hc. The number of pixels is Hc = Ws / 2. Further, based on the Bayer array image 411, the image generating unit 130 generates an image having a resolution equivalent to the resolution according to the arrangement pattern of the R, G, and B pixels, that is, the number of rows of the imaging pixels. Demosaicing is performed to generate R, G, and B color images of the same number of lines as.

Next, in step S312, the image generation unit 130 performs a process of scaling the image of each of the R, G, and B colors after the process of step S311 to detect an object in which each of the R, G, and B colors includes Wc × Hc pixels. The image for use 412 is generated.
As described above, the image generation unit 130 according to the present embodiment generates an image having the same number of rows as the number of imaging pixels arranged in the imaging element 11 and scales the image to obtain an object detection image. 412 is generated.

Here, the method not using the signal of the W pixel has been described, but the signal of the W pixel may be used for generating the object detection image. Specifically, a luminance image having Ws × Hc pixels is generated as an object detection image by using a signal of W pixels.
There is no particular limitation on the demosaicing method. For example, the pixel values of the R pixel, the G pixel, and the B pixel may be interpolated so as to improve the SN ratio using the signal of the W pixel.

  The image generation unit 130 performs processing such as noise removal, luminance signal conversion, aberration correction, white balance adjustment, and color correction on the object detection image in addition to the above-described processing. Is stored in the memory 131.

Next, the distance image generation processing in S302 of FIG. 3A will be described. The distance image is generated by the distance generation unit 132.
The image signal 413 read from the M pixel which is the distance measurement pixel of the image sensor 11 shown in FIG. 4 is obtained by converting the image signal S1 from the photoelectric conversion unit 115 and the image signal S2 from the photoelectric conversion unit 116 described above. Signal. The distance generator 132 generates a distance image 415 (distance image D) based on the image signal S1 and the image signal S2.

FIG. 3C is a detailed flowchart of the distance image generation process.
In step S321, the distance generation unit 132 performs a light amount correction process on the image signal S1 and the image signal S2. At the peripheral angle of view of the imaging optical system 10, the shapes of the pupil region 210 and the pupil region 220 are different due to vignetting (vignetting), and as a result, the light amount balance between the image signal S1 and the image signal S2 is lost. Sometimes. For this reason, in S321, the distance generation unit 132 performs the light amount correction processing on the image signals S1 and S2 using the light amount correction values stored in the memory 131 in advance.

  In the next step S322, the distance generation unit 132 performs a process for reducing noise caused by the image sensor 11. Specifically, the distance generation unit 132 performs a filtering process on the image signal S1 and the image signal S2 to reduce noise included in the image signal S1 and the image signal S2. Here, generally, the higher the spatial frequency, the lower the S / N ratio and the relatively large the noise component. Therefore, the distance generation unit 132 performs a filtering process using a low-pass filter in which the pass rate decreases as the frequency increases. Note that the light amount correction in S321 often does not become as designed due to a manufacturing error of the imaging optical system 10, or the like. For this reason, at the time of the noise reduction process in S322, it is desirable to use a bandpass filter that cuts off the DC component and has a low transmittance of the high-frequency component.

  Next, in step S323, the distance generation unit 132 calculates the amount of parallax based on the image signals S1 and S2 after the processing in steps S321 and S322 described above. Specifically, first, the distance generation unit 132 sets a point of interest corresponding to the representative pixel information Isp in the image signal S1, and sets a collation area centered on the point of interest. The matching area is, for example, a rectangular area in which one side around the point of interest has a predetermined number of pixels. Further, the distance generation unit 132 sets a reference point in the image signal S2, and sets a reference area centered on the reference point. The reference area has the same size and shape as the matching area. Next, the distance generation unit 132 calculates the degree of correlation between the image signal S1 included in the matching area and the image signal S2 included in the reference area while sequentially moving the reference points, and focuses on the point having the highest degree of correlation. The corresponding points correspond to the points. The relative displacement between the point of interest and the corresponding point is the amount of parallax at the point of interest. By calculating the amount of parallax while sequentially changing the noted point according to the representative pixel information Isp, it is possible to calculate the amount of parallax at a plurality of pixel positions. As a method of calculating the degree of correlation, NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), or the like can be used. Then, the distance generation unit 132 obtains the distance image 415 by setting the position at which the amount of parallax is calculated to be similar to the position of the object detection image 412.

  Further, the distance generation unit 132 calculates the amount of parallax by using the predetermined conversion coefficient and calculating the following expression (1), thereby obtaining the amount of defocus from the image sensor 11 to the focal point of the imaging optical system 10. Convert to In the equation (1), K is a predetermined conversion coefficient, ΔL is a defocus amount, and d is a parallax amount.

  ΔL = K × d Equation (1)

  Further, as shown in Expression (2), the distance generation unit 132 converts the defocus amount ΔL into an object distance using a lens formula in geometrical optics. In the equation (2), Da is the distance from the object plane to the principal point of the imaging optical system 10, Db is the distance from the principal point of the imaging optical system 10 to the image plane, and F is the focal length of the imaging optical system 10. It is.

  1 / Da + 1 / Db = 1 / F Equation (2)

  In this equation (2), the value of the distance Db can be calculated from the defocus amount ΔL, and the focal length F is obtained by the imaging optical system 10, so that the distance Da to the object plane can be calculated.

The description returns to the flowchart of FIG. After S302 described above, the image processing unit 13 advances the processing to S303.
In step S303, the image processing unit 13 sets an area to be enlarged using the distance image generated in step S302. The area to be enlarged is a distant area, and the area to be enlarged is set by the area setting unit 133.
Hereinafter, setting of an area to be enlarged will be described with reference to FIG.

In FIG. 5, an image 510 shows an example of the object detection image 412 generated in S302 described above, and an image 511 shows an example of the distance image 415 also generated in S302. It is assumed that the object detection image 510 includes an object 502 at a long distance and an object 501 at a short distance.
Here, the size corresponding to the minimum model capable of detecting the object is represented by Wmin pixel, the size of the real model to be detected is represented by Wm, the focal length of the imaging optical system 10 is represented by F, and the pixel pitch of the image sensor 11 is represented by P. And In this case, the first distance Dmin from the imaging device 1 to the object that can be detected as the minimum model can be expressed by Expression (3).

  Dmin = Wm × F / (Wmin × P) Equation (3)

In other words, an object (for example, the object 502) located at a distance longer than the first distance Dmin is smaller than the minimum model Wmin pixel in the image captured by the imaging device 1, and can be detected as it is. Means you can't.
Further, for example, among the distances from the imaging device 1 to the object, a distant object or the like that is equal to or more than a predetermined second distance (Dmax or more) cannot be resolved in the captured image of the imaging device 1 in the first place. It is assumed that an object cannot be detected even if the image region in which the image is displayed is enlarged, for example. In other words, in other words, if the distance from the imaging device 1 to the object is less than the second predetermined distance (less than Dmax), for example, if the captured image is enlarged, the object can be detected.

  Based on the above, the area setting unit 133 specifies an area in the distance image that falls within the distance range Drg (Drg = Dmin-Dmax) based on the first distance Dmin and the second distance Dmax. Then, the area setting unit 133 sets an area to be enlarged in the object detection image 510 based on the area specified as being within the distance range Drg from the distance image. In the example of FIG. 5, a region 504 surrounded by a dotted line in the distance image 511 represents a region specified as being within the distance range Drg, and similarly, a region 504 surrounded by a dotted line in the object detection image 510. Indicates an area set to be enlarged. That is, the region setting unit 133 excludes a distant region that is equal to or greater than the second distance Dmax, such as a distant sky region, and corresponds to a distance range Drg that is less than the predetermined second distance Dmax and is greater than the first distance Dmin. The region to be enlarged is set as the region 504 to be enlarged. Accordingly, if the area 504 set in the object detection image 510 is enlarged, for example, it is possible to detect an object appearing in the enlarged image (512) in the subsequent object detection processing. . Further, in the present embodiment, since the area to be enlarged is limited, the load of the subsequent object detection processing can be reduced and the used memory capacity can be reduced as compared with the case where the entire object detection image 510 is enlarged. It is also possible to do.

Description will be returned to the flowchart of FIG. After S303, the image processing unit 13 proceeds to S304.
In S304, the image processing unit 13 determines whether there is a distant area to be enlarged, that is, whether there is an area set to be enlarged by the area setting unit 133. This determination is made, for example, by the image generation unit 130. If the image generation unit 130 determines in S304 that there is no distant region to be enlarged, the process proceeds to S306, while if it determines that there is a distant region to be enlarged, the process proceeds to S305.

  In step S305, the image generation unit 130 generates an enlarged image of the area set to be enlarged in step S303, and then proceeds to step S306. In the example of FIG. 5, an enlarged image (512) of the region 504 set to be enlarged is generated in the object detection image 510. In the present embodiment, the enlarged image is an enlarged object detection image 512, that is, a second object detection image generated as an image having a resolution equal to or higher than the resolution of the imaging pixels of the imaging element 11. In the case of the present embodiment, since the imaging element 11 has imaging pixels and ranging pixels arranged for each row, the image generation unit 130 determines the number of rows equal to or greater than the number of imaging pixels arranged in the imaging element 11. Is generated as an enlarged object detection image.

  As described above, in the present embodiment, in the enlarged image generation processing in S305, in the object detection image generated in S302, the area 504 set to be enlarged in S303 is set to the resolution of the image sensor 11 or higher. This is a process that expands to. For example, the upper left coordinate of the rectangle in the region 504 set in S303 is (xc, yc), and the size of the rectangular region is Wt × Ht pixels. The image generation unit 130 calculates the corresponding position and size on the pixel array of the image sensor 11 based on the coordinates and size of the rectangle of the area 504. In the case of the pixel array of the image sensor 11 of the present embodiment, the size of the rectangular area on the pixel array of the image sensor 11 may be twice the value of Wt × Ht. For example, the upper left coordinate of the rectangular area on the pixel array of the image sensor 11 is (xs, ys), and the size is Wts × Hts pixels. That is, Wts = Wt × 2 and Hts = Ht × 2. Then, the image generation unit 130 generates an enlarged object detection image 512 from the object detection image 510 based on the information on the rectangular area thus obtained.

Hereinafter, the process of generating an enlarged object detection image in which a rectangular area is enlarged will be described in detail with reference to FIG.
In FIG. 4, an image 416 corresponds to an image obtained by cutting out a rectangular area of Wts × Hts pixels at upper left coordinates (xs, ys) in the pixel array of the image sensor 11.

  Further, it is assumed that the signal of the M pixel, which is the distance measuring pixel of the image sensor 11, is handled as a luminance signal of M = S1 + S2 obtained by adding the image signal S1 and the image signal S2 described above. At this time, since the M pixel and the W pixel are both obtained as signals that have not been color-separated by the color filter, they can be similarly treated as luminance signals. Therefore, the image generation unit 130 generates the luminance image 417 using the signals of the W pixels and the M pixels. Specifically, the image generation unit 130 determines, for each pixel position of the R pixel, the G pixel, and the B pixel, pixels interpolated by the pixel values of the W pixel and the M pixel (underline, as in an image 417 in FIG. 4). The luminance image 417 is generated by using a W pixel with a.

  Next, the image generation unit 130 cuts out the area (the area 504 in FIG. 5) set to be enlarged in S303 from the object detection image 412 in FIG. 4 and enlarges the area to the same size as the luminance image 417. To generate the enlarged object detection image 418 of FIG.

  Further, the image generation unit 130 corrects the enlarged object detection image 418 using the luminance image 417. Here, the luminance image 417 has a higher resolution than the image for object detection 412 because the luminance image 417 is an image having the resolution originally possessed by the image sensor 11. Therefore, the image generation unit 130 first calculates the color ratio at the same position as the pixel of the luminance image 417. That is, the image generation unit 130 acquires the luminance at the position where the R pixel, the G pixel, and the B pixel were originally located from the luminance image 417, and calculates the color ratio corresponding to the luminance at the position. Then, the image generation unit 130 multiplies the enlarged object detection image 418 of each of R, G, and B by the color ratio calculated at that position. As a result, a clear enlarged object detection image whose resolution has been corrected can be obtained.

  Note that, in order to detect a distant object, an image enlarged stepwise may be generated using the enlarged object detection image 418 as an original image. Also, here, an example in which the enlarged object detection image 418 is generated as an R, G, B color image has been described. However, if the subsequent object detection process also supports a luminance image, only the luminance image is generated. Just do it. As described above, it is possible to generate an enlarged object detection image with high resolution and high S / N.

Description will be returned to the flowchart of FIG.
After proceeding to S306 after S305, the image processing unit 13 performs an object detection process. The object detection processing is performed in the object detection unit 134.
Here, the object detection unit 134 uses the image for object detection generated in S302 or the image for enlarged object detection generated in S305 when it is determined that there is a distant region to be enlarged in 304, Perform object detection. As for the object detection, for example, as described above, there is no particular limitation such as a method based on a feature point or a method using a neural network, and any method may be used. Then, the object detection unit 134 outputs, as the object detection result, rectangular area information indicating at which position in the image the size of the object exists. Note that the rectangular area detected from the enlarged object detection image is finally output as rectangular area information matched to the size of the object detection image in consideration of the enlargement ratio at the time of the image enlargement processing. .

  Next, in step S307, the image processing unit 13 integrates the distance image generated in step S302 and the information on the object detection result output in step S306. The information integration process is performed in the information integration unit 135. The information integration unit 135 sets a rectangular area based on the rectangular area information detected in S306 for the distance image generated in S302, and sets a distance from the imaging device 1 to the object based on the distance information in the rectangular area. I do. Note that the distance to be set is not particularly limited, such as using a value of distance information near the center in the rectangular area or a median of each distance information in the rectangular area, and various methods may be used. it can. Thereby, it is possible to output at which distance the detected object is located from the imaging device 1. Then, the information integration unit 135 sends the result of the information integration to another device or the like via the communication unit 17. As another device that receives the integrated information result, for example, an action planning device 71 mounted on the vehicle 700 in FIG. Then, the action planning device 71 determines the possibility of collision with the object based on the acquired integrated information result, and for example, formulates an action plan of the own vehicle.

As described above, in the present embodiment, the image processing unit 13 outputs the signal from the ranging pixel in addition to the signal of the imaging pixel of the imaging element 11 in which the imaging pixel and the ranging pixel are arranged in a predetermined pattern. By using also, an object that is far from the imaging device 1 can be detected.
In the above description, the area to be enlarged is set using only the distance of the distance image, but is not limited thereto. For example, the object detection image 510 in FIG. 5 may be analyzed to detect, for example, a travel path, and an area on the travel path and at a long distance may be set as an area to be enlarged. By making such settings, it is possible to reduce the amount of use of the memory 131 that stores the image for enlarged object detection and to reduce the time required for the object detection processing using the image for enlarged object detection.

  Further, in the above-described embodiment, an example in which the pixel arrangement pattern on the image sensor 11 is a square array is used. However, the present invention is not limited to this. For example, it is assumed that the image pixel row and the distance measurement pixel row are alternately shifted by half a pixel. Arrangement pattern may be used. This example is shown in FIG. The image sensor 61 in FIG. 6 has an arrangement in which each row of imaging pixels and each row of distance measurement pixels is one set, and each set is shifted by half a pixel in the horizontal direction (x-axis direction) (so-called staggered arrangement). Pixel arrangement pattern. In the case of such a staggered arrangement, parallax resolution is improved as a distance measurement pixel, and a more accurate distance measurement can be performed than in the case of a square arrangement. In this case, the resolution of the object detection image and the enlarged object detection image is also improved in the horizontal direction.

When a pattern like the image sensor 61 of FIG. 6 is used, in the flowchart of FIG. 3A, a part of the processing of S302 and S305 is different from the processing described above. Hereinafter, only different processing portions will be described.
When an image for object detection is generated from a signal of the image sensor 61, the resolution of the image for object detection is half the resolution of the image sensor 61. In this case, in S302, the image generation unit 130 does not generate an image by a pixel shift process of performing interpolation based on a shift amount of a half pixel, and performs R, G, and R pixel arrangement similar to the Bayer array image 411 in FIG. The object detection image 412 of B is generated.

  When generating an image for detecting an enlarged object, the image generating unit 130 in FIG. 6 first uses the signal of the W pixel of the image sensor 61 and the signal obtained by adding the image signal S1 and the image signal S2 of the M pixels in S305. A luminance image 612 as shown is generated. Specifically, the image generation unit 130 considers that a pixel having a resolution twice as large as the number of pixels Ws in the horizontal direction of the image sensor 61 is shifted by half a pixel in the horizontal direction, W pixels and M pixels are arranged as a set of pixel rows. Since there are no pixel signals at the positions of the underlined W pixel and M pixel in the drawing, the image generation unit 130 calculates the pixels at those nonexistent positions by interpolation from the luminance signals of the peripheral pixels. To generate a luminance image 612. As described above, the enlarged object detection image is generated by using the pixel shift process of performing the interpolation based on the shift amount of the half pixel.

  After that, the image generation unit 130 generates an enlarged object detection image by enlarging the area set to be enlarged in the object detection image so as to match the luminance image 612, as in S305 described above. As described above, by correcting the color information of the enlarged object detection image based on the luminance image 612, it is also possible to generate an enlarged object detection image having twice the resolution of the image sensor 61. As described above, the object detection unit 134 may use the luminance image 612 as the enlarged object detection image, or may perform object detection by further enlarging the enlarged object detection image. As described above, it is possible to detect an object having a farther object distance or a smaller object than in the above-described example.

FIG. 7 schematically illustrates a case where the imaging apparatus 1 according to the present embodiment is mounted on a mobile object such as a car (vehicle 700), and an advanced driving support system or the like that recognizes the surrounding environment and moves autonomously is realized. FIG. 3 is a diagram showing a typical configuration example.
As shown in FIG. 7, the vehicle 700 includes an imaging device 1, an action planning device 71, an alarm device 72, a control device 73, and a vehicle information acquisition device 74.

In the vehicle 700 of FIG. 7, the imaging device 1 is an imaging device having the image processing device according to the present embodiment described above, and analyzes an image acquired as described above to analyze a specific object (such as a car or a pedestrian). ) Is detected.
The vehicle information acquisition device 74 acquires at least one of the vehicle speed (speed), the yaw rate, and the steering angle of the vehicle 700 as vehicle information.
The action planning device 71 comprehensively determines the captured image from the imaging device 1, the ranging image, information on the detected object, the vehicle information from the vehicle information acquisition device 74, and the like, and determines the planned movement route of the vehicle 700, Recognize the surrounding environment and formulate an action plan such as acceleration / deceleration and steering angle of the vehicle.

The control device 73 acquires the action plan from the action plan device 71, and controls the steering wheel, the brake, the accelerator, and the like so that the acceleration, deceleration, steering angle, and the like determined by the action plan are obtained.
Further, for example, when there is a high possibility of a collision with a vehicle ahead, the alarm device 72 issues an alarm such as a sound, or displays an alarm on a display device such as a car navigation system or a head-up display, so that the driver Warning.

  In the above-described embodiment, an example in which an imaging device including an image processing device is mounted on a vehicle has been described. However, the image processing device according to the present embodiment includes various mobile terminals such as a smartphone and a tablet terminal having a camera function, and a monitoring camera. , An industrial camera or the like.

  In the present embodiment, the present invention can also be realized by supplying a storage or recording medium storing program codes of software for realizing the functions of the above-described embodiments to an image processing apparatus. A computer (or CPU, MPU, or the like) of the calculation unit reads out the program code stored in the storage medium and executes the above-described function. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the program and the storage medium storing the program constitute the present invention. By installing the program of the present invention in a computer of an imaging apparatus including a predetermined imaging optical system, a predetermined imaging unit, and a computer, the imaging apparatus can detect a distance with high accuracy. The computer of the present invention can be distributed via the Internet in addition to the storage medium.

  Each of the above-described embodiments is merely an example of a specific embodiment for carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features.

  1: imaging device, 10: imaging optical system, 11: imaging device, 12: control unit, 13: image processing unit, 130: image generation unit, 131: distance generation unit, 132: memory, 133: area setting unit, 134: Object detection unit, 135: Information integration unit, 14: Storage unit, 15: Input unit, 16: Display unit, 17: Communication unit

Claims (16)

An image processing apparatus for processing an output signal from an imaging element in which imaging pixels and ranging pixels capable of measuring a distance by an imaging surface phase difference method are arranged in a predetermined arrangement pattern,
First generating means for generating a first image based on an output signal of the imaging pixel;
Distance measuring means for generating distance information based on an output signal of the distance measuring pixel;
Setting means for setting an area that is farther than a predetermined distance based on the distance information, for the first image;
A second image generating a second image having a resolution equal to or higher than the resolution according to the arrangement pattern of the imaging pixels based on the output signals of the imaging pixels and the ranging pixels for the area set by the setting unit; Means for generating
Detecting means for detecting an area of a predetermined object from any one of the first image and the second image;
An image processing apparatus comprising: The arrangement pattern of the imaging device is a pattern in which the imaging pixels and the ranging pixels are arranged in each row,
The method according to claim 1, wherein the second generation unit generates the second image having a number of rows equal to or greater than the number of rows of the imaging pixels for the area set by the setting unit. The image processing apparatus according to any one of the preceding claims.   The image processing apparatus according to claim 1, wherein the first generation unit generates the first image having a resolution equivalent to a resolution according to an arrangement pattern of the imaging pixels. The arrangement pattern of the imaging device is a pattern in which the imaging pixels and the ranging pixels are arranged in each row,
The method according to claim 3, wherein the first generation unit generates an image having a number of rows equal to the number of rows of the imaging pixels, and scales the image to generate the first image. The image processing apparatus according to any one of the preceding claims. The arrangement pattern of the imaging element, the imaging pixels and the ranging pixels are arranged by a half pixel for each set as a set of the rows of the imaging pixels and the rows of the ranging pixels,
5. The image processing apparatus according to claim 1, wherein the second generation unit generates an enlarged image by performing interpolation according to the shift of the half pixel as the second image. 6. Image processing device.   The image processing apparatus according to claim 5, wherein the first generation unit generates the first image without performing interpolation according to a shift of the half pixel. The second generation means includes:
A luminance signal is generated by adding a plurality of signals read from the ranging pixels of the imaging element, and a luminance image having a resolution equal to or higher than the resolution by the imaging pixels is generated,
For the area set by the setting unit, an image enlarged so as to have the same number of pixels as the luminance image is corrected using the luminance image to generate the second image. The image processing apparatus according to claim 1.   The second generation unit calculates a color ratio corresponding to luminance for each pixel of the luminance image, and multiplies the color ratio calculated for each pixel by a corresponding pixel of the enlarged image, The image processing apparatus according to claim 7, wherein the correction is performed.   The image processing apparatus according to claim 1, wherein the second generation unit generates the second image including a luminance image.   The setting means may set, as the area farther than the predetermined distance, an area that falls within a distance range from a first distance to a second distance farther than the first distance. The image processing device according to claim 1. The first distance is a distance set based on a minimum model of an object that can be detected from the first image,
The image processing apparatus according to claim 10, wherein the second distance is a distance set based on a minimum model of an object that can be detected from the second image.   The apparatus according to claim 1, wherein the setting unit sets, for an area of the traveling road included in the first image, the area that is farther than a predetermined distance based on the distance information. The image processing apparatus according to claim 1. An imaging element in which imaging pixels and ranging pixels capable of measuring a distance by an imaging surface phase difference method are arranged in a predetermined arrangement pattern,
An image processing apparatus according to any one of claims 1 to 12,
An imaging device comprising: An imaging device according to claim 13,
Using the detection result of the object by the imaging device, an action planning device that generates an action plan of the moving object,
A control device that controls the operation of the moving object based on the action plan,
A support device comprising: An image processing method for processing an output signal from an imaging element in which imaging pixels and ranging pixels capable of measuring a distance by an imaging surface phase difference method are arranged in a predetermined arrangement pattern,
A first generation step of generating a first image based on an output signal of the imaging pixel;
A ranging step of generating distance information based on an output signal of the ranging pixel;
A setting step of setting an area that is farther than a predetermined distance based on the distance information, for the first image;
A second image generating a second image having a resolution equal to or higher than the resolution according to the arrangement pattern of the imaging pixels based on the output signals of the imaging pixels and the ranging pixels for the area set in the setting step; Generation process,
A detecting step of detecting a region of a predetermined object from any one of the first image and the second image;
An image processing method comprising:   A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.