JP2011232975A - Image processor, surrounding monitoring system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional image processing technology for searching corresponding points accurately and rapidly.SOLUTION: An imaging part 2 comprises a multimode imaging control part 201 for switching photoelectric conversion characteristics between linear conversion characteristics and logarithmic conversion characteristics according to an incident light amount. An image processor inputs stereo image signals captured by the imaging part 2, and determines spatial correspondence among a plurality of images contained in the stereo image signals so as to acquire three-dimensional image information of an imaging object. For areas in captured images where photoelectrical conversion with linear conversion characteristics has been performed, a first corresponding point searching method offering high accuracy is applied so as to allow acquisition of the three-dimensional image information; and for areas in the images where photoelectrical conversion with logarithmic conversion characteristics has been performed, a second corresponding point searching method offering rapidity is applied so as to allow acquisition of the three-dimensional image information. By using a corresponding point search method with a different characteristic according to an image area type, both accuracy and rapidity are ensured.

Description

  The present invention relates to an image processing technique, and in particular, to a corresponding point search technique for stereo images.

  In recent years, a self-propelled system such as a car or robot can be safely driven by installing a camera in the system to detect an object that may be present in the front or the vicinity and that may cause a collision. Development of a system that automatically determines whether or not there is a possibility is in progress.

  In determining the possibility of collision with the target object, it is possible to apply a method that is calculated by a single camera, but when the moving speed is low (relative speed is low) or the target is far away May not be detected with high accuracy. Further, it is necessary to discriminate and extract the type of target object (vehicle type / person), and the processing becomes complicated.

  As a method for solving these problems, a distance information is calculated by a stereo camera, and a collision possibility with a target object is determined based on the calculated distance information (for example, Patent Document 1), object detection / periphery Monitoring methods (for example, Patent Document 1 and Patent Document 2) have been proposed. The advantage of the analysis of the possibility of collision using these distance information is that the position of the object in the time direction can be extracted with high accuracy, and the possibility of collision with high accuracy from the moving speed and distance of the object. Can be determined.

  In addition, when calculating distance information from a stereo camera, corresponding point search accuracy is important, but various corresponding point search methods have been proposed (for example, Patent Documents 3 to 6).

  On the other hand, in an imaging device such as a stereo camera, with the aim of widening the dynamic range, the output of an electrical signal linearly converted with respect to the incident light amount and an electric signal converted logarithmically with respect to the incident light amount are output. Possible image sensors (hereinafter referred to as sensors having linear-log characteristics) have been proposed, but distance measuring systems have also been proposed as correction methods for sensors having linear-log characteristics (for example, patent documents). 7).

JP 2006-134035 A JP 2006-54504 A JP 2008-89402 A JP 2009-146296 A JP 2008-216126 A JP 2008-236276 A JP 2008-046004 A

  For sensor output with linear-log characteristics with a wide dynamic range, for example, it is possible to search for corresponding points after performing image correction on areas that include overexposed areas due to the influence of oncoming vehicles and other lights. This is thought to lead to improved accuracy.

  However, in the sensor output having the Linear-Log characteristic, in the logarithmic conversion characteristic region, there is a characteristic that the luminance difference is small and the fine change is crushed, so the accuracy of the corresponding point search by the frequency characteristic is lowered. In addition, since the inflection point varies from pixel to pixel due to a change in conditions such as temperature, the corresponding point search accuracy decreases in the vicinity of the inflection point.

  Therefore, in the conventional technique, it is difficult to search for corresponding points with high accuracy in an image including an overexposed region due to complicated devices and processing.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing technique capable of performing corresponding point search that achieves both high accuracy and high speed.

  In order to solve the above-mentioned problem, the invention of claim 1 provides a stereo image signal picked up using a stereo image pickup means whose photoelectric conversion characteristics are switched between linear conversion characteristics and logarithmic conversion characteristics in accordance with the amount of incident light. An image processing apparatus that inputs and determines a spatial correspondence between a plurality of images included in the stereo image signal to obtain three-dimensional image information of the imaging target, (a) of the stereo image signals, For the linear region as the image region obtained by the linear transformation characteristic, the corresponding point search between the plurality of images is performed by applying the first corresponding point search method, and the image region obtained by the logarithmic transformation characteristic is obtained as the image region. In the logarithmic region, based on a corresponding point search result by the corresponding point searching means, (b) a corresponding point searching means for applying a second corresponding point search method to search for corresponding points between the plurality of images, Obtain 3D image information of imaging object And a method for providing a search result with higher accuracy than the second corresponding point search method is employed as the first corresponding point search method. On the other hand, as the second corresponding point search method, A method of providing a search result at a higher speed than the first corresponding point search method is employed.

  The invention according to claim 2 is the image processing device according to claim 1, wherein the image processing device determines whether each image area of the stereo image is based on a feature amount of each image area of the stereo image. The image forming apparatus further includes a characteristic determination unit that determines whether the region is a linear region or a logarithmic region, and the corresponding point search unit performs the first and second for each image region based on a determination result of the characteristic determination unit. One of the corresponding point search methods is selected and applied.

  The invention according to claim 3 is the image processing apparatus according to claim 2, wherein the stereo imaging unit is configured to perform imaging in a linear imaging mode in which photoelectric conversion is performed with linear conversion characteristics regardless of incident light amount, and incident light amount. And a multi-mode imaging control unit that periodically switches and executes imaging in the composite imaging mode in which the photoelectric conversion characteristic changes between the linear conversion characteristic and the logarithmic conversion characteristic according to The determination as to whether each image area of the stereo image obtained in the composite imaging mode is a linear area or a logarithmic area is performed based on the stereo image obtained in the linear imaging mode.

  According to a fourth aspect of the present invention, in the image processing apparatus according to the third aspect, the characteristic determining unit detects the overexposed region in the image obtained in the linear imaging mode, and the composite imaging Means for identifying a corresponding area corresponding to the overexposed area in the image obtained in the mode, and the corresponding point search means for the corresponding area among the images obtained in the composite imaging mode. A corresponding point search is performed by a second corresponding point search method, and a corresponding point search is performed by the first corresponding point search method for the remaining region.

The invention according to claim 5 is the image processing apparatus according to claim 3 or 4, wherein
The linear imaging mode and the composite imaging mode are mode switching with a time difference of the same imaging device, and the corresponding point search means extrapolates temporally the imaging results of two or more times in the linear imaging mode. Thus, the location of the corresponding area in the composite imaging mode is estimated.

  The invention according to claim 6 is the image processing apparatus according to claim 1, wherein the corresponding point search determining means partially performs the corresponding point search by the first corresponding point search method, Means for obtaining an index value indicating the reliability of corresponding points obtained by partial corresponding point search, and when the index value is equal to or greater than a predetermined threshold, the first corresponding point search method is continued. And means for switching to the second corresponding point search method when the index value is less than the threshold value.

  The invention according to claim 7 is the image processing apparatus according to any one of claims 1 to 6, wherein the first corresponding point search method is based on frequency characteristics of the plurality of images. It is a method for searching for corresponding points with sub-pixel accuracy.

  The invention according to claim 8 is the image processing apparatus according to any one of claims 1 to 6, wherein the second corresponding point search method is based on a correlation between luminance values of the plurality of images. On the basis of this, it is a method for searching for corresponding points.

  The invention of claim 9 is the image processing apparatus according to any one of claims 1 to 6, wherein the second corresponding point search method is based on frequency characteristics of the plurality of images. It is a method for searching for corresponding points with pixel accuracy.

  According to a tenth aspect of the present invention, based on the image processing device according to any one of the first to ninth aspects and three-dimensional image information of the photographic subject, the photographic subject satisfies a predetermined condition. A target object determining unit that determines whether or not to perform, and a warning unit that issues a warning when the target object determining unit determines that the photographing target object satisfies the predetermined condition. This is a surrounding monitoring system.

  According to an eleventh aspect of the present invention, based on the image processing device according to any one of the third to fifth aspects and the three-dimensional image information of the photographic subject, the photographic subject satisfies a predetermined condition. Object determining means for determining whether or not to perform, and warning means for giving a warning when the object determining means determines that the photographing object satisfies the predetermined condition, and the stereo image In the case where the object to be imaged in is localized only in one of the linear area and the logarithmic area, the multi-mode imaging control means is configured to select one of the linear imaging mode and the composite imaging mode. The periphery monitoring system is characterized in that the imaging ratio in one imaging mode corresponding to the localized region is increased.

  The invention according to claim 12 is the periphery monitoring system according to claim 10 or 11, wherein the warning means displays an image obtained by imaging in the composite imaging mode on a predetermined screen display means. The display control means to be provided is provided.

  The invention of claim 13 is a program that causes the computer to function as the image processing apparatus according to any one of claims 1 to 9 by being installed and executed in the computer. .

  The invention according to claim 14 is the program according to claim 13, wherein the computer is executed by the computer to cause the computer to perform the periphery monitoring system according to any one of claims 10 to 12. It functions as an image processing apparatus for use.

  According to the first to fourteenth aspects of the present invention, high-accuracy corresponding point search processing and high-speed correspondence can be performed for each image area of a stereo image in accordance with the linear transformation characteristic and the logarithmic transformation characteristic. It is possible to properly use point search processing. As a result, it is possible to perform corresponding point search that achieves both high accuracy and high speed.

  According to the invention of claim 3, in order to determine whether each image area of the stereo image in the composite imaging mode is a linear area or a logarithmic area based on the stereo image in the linear imaging mode, The area determination becomes accurate.

  According to the invention of claim 4, since the overexposed area in the linear imaging mode is often a logarithmic area in the composite imaging mode, the area determination in the composite imaging mode becomes easy and accurate.

  According to the fifth aspect of the present invention, an area where the subject is moving is obtained by extrapolating temporally two or more continuous imaging results in the linear imaging mode and estimating the location of the corresponding area in the composite imaging mode. Judgment is easy and accurate.

  According to the invention of claim 6, the corresponding point search is partially performed by the first corresponding point search method, and the corresponding point search by the second corresponding point search method is performed based on the calculated reliability of the corresponding point. By determining whether or not to implement, the corresponding point search can be performed efficiently and accurately.

  According to the tenth to twelfth aspects of the present invention, it is possible to obtain a result obtained by combining accuracy and high speed in determining whether or not a subject to be photographed satisfies a predetermined condition.

  According to the eleventh aspect of the present invention, when the photographing object is localized only in one of the linear region and the logarithmic region, the specific gravity of photographing with the photoelectric conversion characteristic corresponding to the localized region. Therefore, it is possible to efficiently search for corresponding points.

  According to the twelfth aspect of the invention, since the screen display is performed with the imaging result in the composite imaging mode, it is possible to obtain an effect that is visually easy to see.

1 is a diagram illustrating a schematic configuration of an image processing system according to an embodiment. It is a figure which illustrates a Linear-Log characteristic. It is a figure explaining a mode that an inflexion point varies for every pixel. It is a figure explaining change of an image picked up, changing imaging modes. It is a figure which shows the functional structural example of the calculation control part which concerns on one Embodiment. It is a figure explaining an example of the stereo image obtained in time series. It is a figure explaining the determination method of a characteristic. It is a figure explaining the flow of POC processing. It is a figure explaining a POC function. It is a figure explaining the corresponding point search according to a characteristic area. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one Embodiment. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one Embodiment.

<1. Overview of image processing system>
FIG. 1 is a diagram showing a schematic configuration of an image processing system according to an embodiment of the present invention. The image processing system 1 is configured as a multi-viewpoint camera system. The image processing system 1 includes a two-lens stereo camera 20 as an imaging unit 2 and an image processing apparatus 3 connected to the stereo camera 20 so as to be able to transmit and receive data. Prepare.

  The stereo camera 20 includes two image pickup systems 21 and 22 each having image pickup devices 21a and 22a. The imaging systems 21 and 22 are configured to image a subject (imaging target) OB in front of the camera from different viewpoints at the same timing. Two image signals (hereinafter abbreviated as “images”) obtained by imaging at the same timing by the imaging systems 21 and 22 are transmitted to the image processing device 3 via the data line CB.

  Hereinafter, an image acquired by the imaging system 21 is referred to as a “first captured image” G1, and an image acquired by the imaging system 22 is referred to as a “second captured image” G2. That is, the first and second captured images G1 and G2 form a set of images in which the same subject OB is captured from different viewpoints.

  Here, in order to simplify the description, it is assumed that the aberration of the stereo camera 20 is corrected well. Further, it is assumed that the imaging systems 21 and 22 are spaced apart along a predetermined direction and the optical axes of the imaging systems 21 and 22 are set to be substantially parallel (preferably completely parallel). Furthermore, the imaging elements 21a and 22a of the imaging systems 21 and 22 have Linear-Log characteristics.

  The image processing device 3 is configured by an information processing device such as a personal computer (personal computer), for example, an operation unit 31 including a mouse and a keyboard, a display 32 configured by a liquid crystal display, a speaker 33, and a stereo camera. And an interface (I / F) 37 for receiving data from 20. The image processing apparatus 3 includes a storage device 34 and an arithmetic control unit 36.

  The storage device 34 is composed of, for example, a hard disk or the like, and stores first and second captured images G1 and G2 obtained by imaging with the stereo camera 20. In addition, the storage device 34 stores a program PG for performing inflection point control, which will be described later.

  The input / output unit 35 includes, for example, a portable disk drive, sets a portable storage medium such as an optical disk, and exchanges data with the arithmetic control unit 36.

  The arithmetic control unit 36 includes a CPU 36 a that functions as a processor and a memory 36 b that temporarily stores information, and comprehensively controls each unit of the image processing apparatus 3. In the arithmetic control unit 36, various functions, various information processing, and the like are realized by reading and executing the program PG in the storage unit 34. The memory 36b can store program data stored in the portable storage medium via the input / output unit 35. This stored program can be appropriately reflected in the operation of the image processing apparatus 3.

  The calculation control unit 36 calculates the distance to the object to be photographed based on information obtained by a corresponding point search operation described later, and visually outputs an image of the specific subject OB on the display 32. Note that, if it is determined that there is a danger such as a collision based on the distance information calculated by the arithmetic control unit 36, a warning is given by the speaker 33.

<1-1. General properties and assumptions of Linear-Log characteristics>
As preparation for explaining the details of the inflection point control of the image pickup devices 21a and 22a in this embodiment, the general properties of the Linear-Log characteristics that are the premise of this embodiment and the circumstances that accompany it, that is, the conventional technology Explain the circumstances that were happening.

  In the following, an image sensor with Linear-Log characteristics is abbreviated as “LL characteristic sensor”. In the LL characteristic sensor, photocurrent generated in each photodiode corresponding to the pixel of the image sensor is converted into an integration circuit through a MOSFET. A given output signal is obtained. Linear conversion characteristics (Linear characteristics) at low illuminance, an output signal proportional to the time integral value (incident light intensity) of the photocurrent generated by the photodiode is obtained, and logarithmic conversion characteristics (Log characteristics) at high illuminance. An output voltage that is logarithmically dependent on the time integral value (incident light quantity) of the photocurrent generated in step S1 is obtained. The inflection points as the boundaries (critical points) can be moved by changing the difference between the gate voltage of the MOSFET during non-imaging and the gate voltage of the MOSFET during imaging. The inflection point control signal is a signal for variably setting such a gate voltage difference.

  FIG. 2 is a diagram illustrating an image signal from the stereo camera 20. As shown in FIG. 2, in the sensor having only the linear conversion characteristic, the luminance is output with the linear conversion characteristic regardless of the amount of incident light. Therefore, when the output limit value MO is exceeded as shown by the broken line, an overexposed region is obtained. . On the other hand, a sensor with Linear-Log characteristics (hereinafter abbreviated as “LL characteristic sensor”) outputs luminance with linear conversion characteristics when the incident light quantity is α or less, but logarithmically when the incident light quantity is α or more. Since the output is converted into conversion characteristics, the output limit value MO is not exceeded up to a region where the amount of incident light is considerably high. At this time, the point where the incident light amount corresponds to α (incident light amount = α, output = H), that is, the point where the linear conversion characteristic is changed to the logarithmic conversion characteristic is the inflection point CP.

  On the other hand, the L-L characteristic sensor has the following problems (I) and (II).

  (I) Inflection points vary from pixel to pixel due to changes in conditions such as temperature. For this reason, there exists a subject which brings about the fall of a corresponding point search precision in the inflection point vicinity.

  FIG. 3 is a diagram illustrating how inflection points vary from pixel to pixel. As shown in FIG. 3, the position of the inflection point CP varies, and the coefficients c1, c2, and c3 in the logarithmic transformation characteristic region and the inflection point offset values d1, d2, and d3 vary. Therefore, the variation in the position (α, H) of the inflection point CP that moves from the linear transformation characteristic region to the logarithmic transformation characteristic region, and the slope and coefficient of the logarithmic transformation characteristic region due to changes in external conditions such as temperature. A phenomenon that varies from sensor to sensor occurs. This results in a decrease in corresponding point search accuracy near the inflection point.

  (II) The logarithmic conversion characteristic region has a feature that the fluctuation range of luminance corresponding to the fluctuation range of incident light quantity is small. From this, it leads to the precision fall of the corresponding point search by a frequency characteristic.

  FIG. 4 shows first and second captured images G1, G2 captured by the imaging systems 21, 22. FIG. 4A shows an image of a high-brightness object in the composite imaging mode in which the photoelectric conversion characteristic changes between the linear conversion characteristic and the logarithmic conversion characteristic in accordance with the amount of incident light, whereas FIG. b) shows an image of a high-luminance subject in a linear imaging mode in which photoelectric conversion is performed with linear conversion characteristics regardless of the amount of incident light. As shown in FIG. 4A, the first and second captured images G1 and G2 are both partially collapsed and an image of the subject OB is obtained, whereas in FIG. 4B, In both the first and second captured images G1 and G2, in the overexposed region with high illuminance, fine changes in the image of the subject OB are crushed.

  As described above, as shooting at high illuminance in which an overexposed region exists, for example, a situation such as nighttime may be overexposed by the light of the oncoming vehicle. In the on-vehicle periphery monitoring, it can be said that such an area is an important area for detecting an object having a possibility of collision. However, in a sensor having only linear conversion characteristics, the image of the overexposed region is output with a collapsed image, and therefore, a high-precision search result cannot be obtained when a corresponding point search is performed between the reference image and the reference image. . As a result, the distance information of the subject OB is erroneously calculated.

  On the other hand, in the L-L sensor, an overexposed area is eliminated by performing image correction on the overexposed area using logarithmic conversion characteristics, and the perimeter monitoring system can be enhanced (see FIG. 4A). However, since the logarithmic conversion characteristic region has a small luminance difference, a high-accuracy image cannot be obtained as compared with the linear conversion characteristic region. Also, as shown in FIG. There are also issues that vary.

  Under such a background, in the present invention, in the L-L sensor, different corresponding point search methods are used in the linear conversion characteristic region and the logarithmic conversion characteristic region.

<2. Functional configuration of image processing system>
Returning to the description of the embodiment of the present invention. The image processing apparatus 3 inputs a stereo image signal captured using the stereo camera 20 in which the photoelectric conversion characteristics are switched between the linear conversion characteristic and the logarithmic conversion characteristic in accordance with the amount of incident light, and is included in the stereo image signal. And determining a spatial correspondence between the plurality of images to obtain three-dimensional image information of the object to be imaged. Here, in particular, a functional configuration realized by the arithmetic control unit 36 for executing the corresponding point search operation will be described.

  FIG. 5 is a diagram illustrating a functional configuration of the arithmetic control unit 36. Here, although the functional configuration of the arithmetic control unit 36 is described as being realized by executing the program PG, it may be realized by a dedicated hardware configuration.

  As shown in FIG. 5, the data is received from the imaging unit 2, and the arithmetic control unit 36 has a functional configuration such as a stereo image acquisition unit 361, a characteristic determination unit 362, a corresponding point search unit 363, and distance information calculation. A target object candidate detection unit 365, a target object determination unit 366, and a warning unit 367. Hereinafter, each process will be described with reference to FIG.

<2-1. Imaging unit 2>
The imaging unit 2 is installed at a predetermined position of the automobile so that the front space of the automobile can be imaged. The imaging unit 2 has a role of acquiring a stereo image and transmitting an image signal to the image processing device 3. In addition, imaging in a linear imaging mode in which photoelectric conversion is performed with linear conversion characteristics regardless of the amount of incident light, and imaging in a composite imaging mode in which photoelectric conversion characteristics change between linear conversion characteristics and logarithmic conversion characteristics in accordance with the amount of incident light Are provided with a multi-mode imaging control unit 201 that periodically switches and executes the above. 1 corresponds to the sensors of the imaging systems 21 and 22 of the stereo camera 20. It is assumed that the imaging unit 2 is calibrated in advance and the camera parameters are known.

<2-2. Stereo Image Acquisition Unit 361>
The stereo image acquisition unit 361 acquires the image signal transmitted by the imaging unit 2. Here, in the corresponding point search operation, the first captured image G1 is used as a reference image (hereinafter also referred to as “reference image”), and the second captured image G2 is referred to (hereinafter also referred to as “reference image”). It is said. Therefore, the first captured image G1 is referred to as a standard image G1, and the second captured image G2 is referred to as a reference image G2.

  Strictly speaking, the stereo image acquisition unit 361 acquires data indicating the standard image G1 and the reference image G2, but in this specification, data indicating the standard image G1 and a standard displayed based on the data are displayed. The image G1 itself is generically referred to as “standard image G1”, and the data indicating the reference image G2 and the reference image G2 itself displayed based on the data are collectively referred to as the reference image G2.

  FIG. 6 is a diagram illustrating an example of a stereo image obtained in time series. As shown in FIG. 6, the acquired stereo image includes, for example, a reference image G1 (A0, A1, A2,...) And a reference image G2 (B0, B1, B2,. ...) and the shooting timing are synchronized with each other. Then, between the standard image G1 and the reference image G2, images having the same characteristics are acquired between the images of the corresponding frames. That is, when the standard image A0 is captured with the linear conversion characteristic, the reference image B0 is also captured with the linear conversion characteristic. When the standard image A1 is captured with the Linear-Log conversion characteristic, the reference image B0 is also captured. Images are taken with Linear-Log conversion characteristics.

Also,
・ Images captured with linear transformation characteristics are called `` linear mode images ''
・ Images captured with Linear-Log characteristics are called `` composite mode images ''
In this embodiment, the linear mode images A0, A2,... (B0, B2,...) And the composite mode images A1, A3,. The imaging frames are periodically switched so that (B1, B3,...) Are alternately acquired frame by frame at a constant time interval Δt.

  In this embodiment, each of the image pickup devices 21a and 22a is configured by an element that can pick up images in either the linear image pickup mode or the composite image pickup mode, and the linear image pickup mode and the composite image pickup mode are set for each frame based on the timing control signal. By switching to, a linear mode image and a composite mode image are acquired.

  In this regard, the weight of either frame may be increased based on the determination result of the object determination unit 366 described later. In this case, for example, the frame ratio between the linear transformation characteristic image and the Linear-Log characteristic image per unit time is set to m: n (m and n are different positive integers), and consecutive (m + n) frames are defined as one cycle. Repeat imaging.

<2-3. Characteristic determination unit 362>
In the characteristic determination unit 362, based on the feature amounts of the image areas of the stereo images G1 and G2, the image areas of the composite mode image among the stereo images G1 and G2 are
-It is a region photoelectrically converted with linear conversion characteristics (hereinafter referred to as "linear region"),
-It is a region photoelectrically converted by the logarithmic conversion characteristic (hereinafter referred to as "logarithmic region"),
Determine.

  First, a preliminary determination based on the exposure state of each image region in the linear mode images A0, A2,... (B0, B2,...) Is performed, and based on the preliminary determination, the composite mode images A1, A3,. ... (B1, B3,...) Is estimated whether each image area is a linear area or a logarithmic area. That is, the luminance is adopted as the feature amount of each image area of the stereo images G1 and G2 in this aspect.

In the composite mode images A1, A3,... (B1, B3,...), Linear transformation characteristics and logarithmic transformation characteristics are already transformed in a spatially mixed state. It is complicated to determine what photoelectric conversion characteristic is obtained. On the other hand, in the linear mode images A 0, A 2,... (B 0, B 2,...), The entire area of the image is converted with the linear conversion characteristics.
-If the image area of interest achieves the desired image expressiveness, it is determined that the area has a linear conversion characteristic and is converted by the linear conversion characteristic even in the composite mode image.
-If the image representation of the image area of interest is insufficient, the area is unsuitable for the linear transformation characteristic, so in the composite mode image, the area is transformed with the logarithmic transformation characteristic.
It can be determined.

  In general, since the imaging system and the subject are relatively moved, an error occurs when the result of area type determination in the linear mode image is applied to characteristic type determination in the composite mode image of the next frame. For example, the range of the region that is overexposed by the headlight of the oncoming vehicle in the linear mode image and the headlight range in the next composite mode image are not exactly the same. However, in imaging at a high frame rate, the amount of movement of the subject between frames that are temporally adjacent is relatively small, and such an error does not cause a practical problem.

  However, since the relative position of the subject changes with respect to the imaging system when considered over a certain time span, the result of the characteristic type determination obtained with the linear mode image of one frame is the region type determination of the next obtained composite mode image. When a new linear mode image is acquired, the characteristic determination is updated accordingly. Thereby, accumulation of errors can be prevented.

  FIG. 7 is a diagram for more specifically explaining the characteristic determination based on such a principle. Although the reference image G1 will be described here, the same applies to the reference image G2.

  In FIG. 7, linear mode images A 0, A 2,... Are obtained in the frames imaged at times T, T + Δt, T + 3Δt,. , ... are obtained.

And each area | region (each space part) of the linear mode image A0 in the time T is as follows.
An area that is not overexposed (hereinafter referred to as “appropriate exposure area”) LN0;
An overexposed region (hereinafter “overexposed region”) LG0;
It is determined whether it is either.

  Among these, the proper exposure region LN0 is a region where most of the pixels included in the region are less than the output limit value (upper limit value) MO of the image sensors 21a and 22a. In addition, since the amount of incident light is excessive in the overexposed region LG0, all or most of the pixels included in the region (pixels having a ratio equal to or greater than a predetermined threshold) are the upper limit values of the luminance outputs of the image sensors 21a and 22a. It is an area that matches the MO. Since these area sections are complementary, the proper exposure area LN0 may be detected and the others may be used as the overexposure area LG0, and conversely, the overexposure area LG0 may be detected and the other may be used as the proper exposure area LN0. Good.

  Among these, since the linear conversion functions effectively in the appropriate exposure region LN0, the image accuracy is high in both the linear mode image and the composite mode image for the region, but the overexposure region in the linear mode image. Much of the image information is lost in the region of LG0, and it can be estimated that the image accuracy is not so high even in a frame in which the same subject is captured as a composite mode image. That is, unlike the linear mode image, the logarithmic conversion functions in the composite mode image. Therefore, the image is not completely crushed even in the overexposed region LG0, and the shading on the image is expressed. However, even in the composite mode image, it can be estimated that the accuracy of the image information in the logarithmic region LG1 corresponding to the overexposed region LG0 is lower than that of the linear region LN1 corresponding to the proper exposed region LN0. .

Therefore, in the composite mode image A1 obtained at time T + Δt corresponding to the next frame, each image region is
A region that corresponds spatially to the proper exposure region LN0 determined in the linear mode image A0 at time T and is assumed to be the linear region LN1 in the composite mode image;
A region that spatially corresponds to the overexposed region LG0 determined in the linear mode image A0 at time T and is assumed to be a logarithmic region LG1 in the composite mode image A1;
Divide into

  This is achieved by creating and storing a bitmap pattern corresponding to the pixel distribution of the proper exposure region (or overexposure region) in the linear mode image A0 and applying it to the pixel coordinates of the composite mode image A1 as it is. can do.

  Thereafter, similarly, the characteristics of each region of the linear mode image A2 obtained at time T + 2Δt are determined to distinguish between the proper exposure region and the overexposure region. Then, for the composite mode image A3 obtained at the next time T + 3Δt, portions corresponding to the appropriate exposure region and the overexposure region divided by the linear mode image A2 are respectively designated as the proper exposure region and the overexposure region. To do.

By repeating the above, each image area in each composite mode image of the stereo images G1 and G2 is
The region corresponding to the appropriate exposure region of the linear mode image (“linear region”),
The region corresponding to the overexposed region of the linear mode image ("logarithmic region"),
Can be determined. That is, based on the detection of the overexposed region in the image obtained from the linear mode images A0, A2,..., The logarithmic region corresponding spatially to the overexposed region is identified in the composite imaging mode images A1, A3,. (Estimated). Along with this, the appropriate exposure region in each linear mode image and the corresponding linear region in the composite mode image are also specified (estimated).

In these,
・ "Proper exposure area" and "Linear area" correspond to each other spatially,
・ "Overexposed area" and "Logarithmic area" correspond spatially to each other,
Of these, “appropriate exposure region” and “overexposed region” are divisions from the point of whether or not the exposure is over in the linear mode image, and “linear region” and “logarithmic region” are: Because it is for distinguishing between the linear conversion characteristic and the logarithmic conversion characteristic of the image area obtained by the photoelectric conversion characteristic in the composite mode image, the difference in concept by using separate terms expressing.

<2-4. Corresponding point search unit 363>
The corresponding point search unit 363 searches for corresponding points between a plurality of images by applying the first corresponding point search method to a linear region as an image region obtained by the linear transformation characteristic in the stereo image signal. The second corresponding point search method is applied to the logarithmic area as the image area obtained by the logarithmic conversion characteristic, and corresponding point search between a plurality of images is performed. The corresponding point searching unit 363 selects and applies one of the first and second corresponding point searching methods for each image region based on the determination result of the characteristic determining unit 362.

  As the first corresponding point search method, a method of giving a search result with higher accuracy than the second corresponding point search method is adopted, and as the second corresponding point search method, the first corresponding point search method is used. A method of providing a search result at a high speed as compared with the above is adopted. Accordingly, the processing unit (hereinafter referred to as “first corresponding point search processing unit 363a”) according to the first corresponding point searching method (hereinafter referred to as “second corresponding point searching method”). A search result with higher accuracy than the search processing unit 363b ”is given. On the other hand, the second corresponding point search processing unit 363b gives a search result at a higher speed than the first corresponding point search processing unit 363a.

  Returning to FIG. 4, among the images obtained in the composite imaging mode shown in FIG. 4A, the logarithmic region corresponding to the overexposed region of the linear mode image shown in FIG. The corresponding point search processing unit 363b performs the corresponding point search, and the remaining corresponding region (linear region) is searched for by the first corresponding point search processing unit 363a.

  In the present embodiment, specifically, the first corresponding point search processing unit 363a employs sub-pixel level POC processing based on the frequency characteristics of the base image G1 and the reference image G2 (generally a plurality of images), The second corresponding point search processing unit 363b employs SAD processing based on the correlation between the luminance values of the base image G1 and the reference image G2 (generally a plurality of images similarly), or employs pixel-level POC processing. To do. However, the corresponding mode search result using the linear mode images of the base image G1 and the reference image G2 as a set of the linear mode image and the composite mode image associated by the characteristic determination unit 362, and the composite mode image The corresponding point search result using each other may be compared, the accuracy may be determined under a predetermined accuracy determination condition, and the one with higher accuracy may be employed.

  Hereinafter, an outline of the SAD method and the POC method as the corresponding point search processing will be described. Here, the acquisition time of the current frame is t, and the image acquisition time of the previous frame one frame before the current frame is t-1. Further, in the current frame, a standard window g1 (t) is set in the standard image G1, and a reference window g2 (t) is set in the reference image G2. The windows of the image one frame before the current frame are g1 (t− 1) and g2 (t-1). However, the windows g1 and g2 are not represented on the drawing.

<2-4-1. Search for corresponding points using SAD (Sum of Absolute Difference) method>
The SAD shifts the reference window g2 (t) with respect to the standard window g1 (t-1) while positioning the image in the standard window g1 and the image in the reference window g2 from the correlation value obtained by the equation (1). And the center point of the reference window g2 when the correlation shows the highest value is searched as the corresponding point gp2 of the attention point gp1. As shown in Equation (1), SAD can calculate the correlation value by subtracting the pixel values (luminance of each pixel) of the two images as they are, so the calculation amount is small and high-speed processing is possible. It has the characteristic of being.

  Here, Im1 indicates the pixel value of the image in the standard window g1, and Im2 indicates the pixel value of the image in the reference window g2. N indicates the size of the window in the horizontal direction, and M indicates the size of the window in the vertical direction.

<2-4-2. Corresponding point search using POC (Phase Only Correlation) method>
POC is a method of searching for corresponding points based on the correlation of signals whose amplitude components are suppressed (that is, the correlation of phase components) by frequency-decomposing the images in the window set as the standard image G1 and the reference image G2.

  FIG. 8 is a diagram showing the flow of the POC process, and the following processes (i) to (vi) are performed in this order.

  (i) The reference window g1 (t−1) is set so that the center is located at the measurement point set in the reference image G1, and the reference window g2 (t) is set in the reference image G2.

  (ii) Discrete Fourier transform is performed on the image in the reference window g1 (t-1) and the image in the reference window g2 (t) to obtain an image Ga and an image Gb.

  (iii) The image Ga and the image Gb subjected to discrete Fourier transform are normalized (that is, the amplitude component is made constant), and the image Ga1 and the image Gb1 are obtained.

  (iv) The normalized image Ga1 and the image Gb1 are synthesized, and the correlation image R is obtained.

  (v) The correlation image R is subjected to inverse discrete Fourier transform to obtain a POC function r. FIG. 9 is a graph showing the POC function r, which has a steep correlation peak S, and is known to have higher robustness and estimation accuracy in image matching than SAD. The height of the correlation peak S increases as the correlation between the image in the reference window g1 (t-1) and the image in the reference window g2 (t) increases. Therefore, by specifying the position of the correlation peak S, it is possible to calculate the positional deviation amount of the reference window g2 (t) with respect to the reference window g1 (t-1).

  Since the POC function r is calculated in pixel units of the reference image G1, that is, at the pixel level, the position of the correlation peak can also be obtained at the pixel level. However, the POC function is interpolated and the correlation peak is calculated at the sub-pixel level. It is also possible to estimate the position.

  Compared with pixel-level POC, sub-pixel-level POC improves processing accuracy but decreases processing speed. For this reason, when using POC at the subpixel level, a combination of using the POC at the pixel level in the logarithmic region and the POC at the subpixel level in the linear region is preferable.

  (Vi) A point on the coordinates obtained by adding the positional deviation amount to the coordinates of the center point of the reference window g2 (t) can be calculated as the corresponding point gp2 for the attention point gp1.

<2-4-3. Search for corresponding points according to characteristic area>
Hereinafter, the corresponding point search in the linear region and the logarithmic region determined by the characteristic determination unit 362 will be described with a specific example.

  FIG. 10 is a diagram for explaining a corresponding point search method according to the characteristic classification of the region. FIG. 10A shows an image signal captured in a linear imaging mode in which photoelectric conversion is performed with linear conversion characteristics regardless of the amount of incident light. In FIG. 10B, linear conversion characteristics and logarithmic conversion characteristics according to the amount of incident light. The image signal imaged in the composite imaging mode in which the photoelectric conversion characteristics change between and. The linear region corresponding to the appropriate exposure region indicated by the solid line in FIG. 10A can be searched with high accuracy by the POC processing at the subpixel level.

  On the other hand, in the composite imaging mode of FIG. 10B, the position of the output point L2 is moved although the position of the output point L1 is the same, and the overexposure region indicated by the broken line in FIG. In b), since it is a logarithmic region, the corresponding point search is enabled by SAD processing or pixel level POC processing. Further, in the vicinity of the inflection point CP positioned between the output point L1 and the output point L2, as shown in FIG. Thus, since there is a possibility that the accuracy is lowered in the vicinity of the inflection point CP, it is desirable to perform SAD processing or POC processing at the pixel level instead of POC processing at the sub-pixel level.

  FIG. 10D shows an image signal of the reference image G1 by the image sensor 21a, and FIG. 10E shows an image signal of the reference image G2 by the image sensor 22a. As described above, even when the inflection points of the image sensor 21a and the image sensor 22a are set differently, the corresponding point search can be performed with high accuracy in the range that becomes the linear region.

  From the above, it is desirable in terms of searching for corresponding points that the output is performed with the linear conversion characteristic up to the vicinity of the output limit value MO that can be output with the linear conversion characteristic, and the logarithmic conversion characteristic is output beyond that. Therefore, based on the high-precision search for corresponding points at the sub-pixel level by POC processing, the surroundings that cannot be captured in the linear imaging mode are monitored by searching for pixel-level corresponding points by SAD processing or POC processing. It is possible to increase the accuracy of the process and perform processing at high speed.

<2-5. Distance information calculation unit 364>
The distance information calculation unit 364 calculates distance information from the imaging unit 2 to the subject based on the corresponding point search result processed by the corresponding point search unit 363.

  That is, when the spatial correspondence of each part of the subject is determined in the reference image and the reference image captured at the same time, the three-dimensional coordinate value of each part of the subject with respect to the imaging unit 2 is determined by the principle of triangulation. Based on the three-dimensional image information obtained in this way, the distance from the imaging unit 2 to each part of the subject is calculated.

  Based on the distance information thus obtained and the optical flow corresponding to the temporal change amount of the position of the target point, it is also possible to discriminate between a stationary object and a moving object. This part is known, for example, in Japanese Patent Laid-Open No. 2006-134035.

<2-6. Target Object Candidate Detection Unit 365>
The target object candidate detection unit 365 in FIG. 5 detects a candidate for the imaging target based on the distance information calculated by the distance information calculation unit 364.

<2-7. Object Determination Unit 366>
The target object determination unit 366 determines whether or not the target object satisfies a predetermined condition based on the three-dimensional image information of the target object (subject) detected by the target object detection unit 365. .

  In addition, the object determination unit 366 determines that, as a result of the determination, if an object that has a possibility of collision exists in (i) the linear region, the object is determined by increasing the weight of the linear transformation characteristic image (ii) logarithm. If it exists in the area, it is possible to instruct the imaging unit 2 to increase the weight of the logarithmic conversion characteristic image and to change the setting of the acquired image. That is, when a specific subject (specific imaging target) in a stereo image is localized only in one of a linear region and a logarithmic region, the imaging unit 2 performs linear imaging mode and composite imaging. Among the modes, the imaging ratio in one imaging mode corresponding to the localized region can be increased.

  Specifically, when an image of the front in the traveling direction is taken from a traveling vehicle, an object that is in or near the range corresponding to the path space of the vehicle and has a distance from the imaging unit 2 of a predetermined distance or less. If so, it may be a collision with the vehicle.

  When the target object, that is, the collision fear object is relatively dark (for example, a pedestrian wearing black clothes), the image portion corresponding to the collision fear object is localized in the linear region. This corresponds to the determination that all or most of the image portion of the subject determined to be within a predetermined distance ahead (for example, a predetermined threshold ratio set to 80% or higher) overlaps the linear region. In this case, processing for increasing the ratio of the linear conversion characteristic is performed.

  There are two methods for the processing. The first is a change in the imaging frame ratio. For example, in the case where the linear mode image and the composite mode image are acquired at 1: 1, the frame ratio is set to 2 after the localization of the collision concern object is detected. : 1 or 3: 1 is a method of relatively increasing the acquisition frequency of the linear mode image. Until the dangerous state is avoided, all the frames may be temporarily used as linear mode images.

  The second is a method of spatially expanding the linear region in the composite mode image by raising the inflection point in the composite mode image (shifting the inflection point to the side with the larger incident light amount). In the latter case, when the object of collision moves at a relatively high speed on the screen, it jumps out from the linear region to the logarithmic region, and the image information for searching for the corresponding point suddenly deteriorates. Can be prevented.

  On the contrary, when the collision fear object is bright, for example, when a vehicle with a headlight on approaches, the image portion corresponding to the collision fear object is localized in the logarithmic region. In such a case, corresponding to the above two methods, the logarithmic region is increased by increasing the acquisition ratio (frame ratio) of the composite mode image or lowering the inflection point in the composite mode image. .

  By taking these measures, it is possible to dynamically set a photoelectric conversion characteristic suitable for obtaining information of the imaging object that is most focused on.

  Examples of such a specific target object include an object of collision in a vehicle, a human image in a security monitoring camera, and a foreign object intrusion into a production line in an industrial camera.

<2-8. Warning section 367>
The warning unit 367 issues a warning when the object determination unit 366 determines that the object to be imaged satisfies a predetermined condition. For example, based on the obtained three-dimensional image information, a moving speed and a moving direction of a moving body (for example, an oncoming vehicle) existing in the image are calculated three-dimensionally, and the moving body is placed in the course space of the own vehicle. Predict whether or not there is a possibility of intrusion. And the case where it determines with the possibility of an oncoming vehicle colliding is assumed. At this time, an image being captured may be displayed on the display 32 together with a warning sound from the speaker 33. Regarding the display on the display 32, even when there is no possibility of a collision, it may be always displayed. When the linear mode image and the composite mode image are switched between frames as in this embodiment, a linear-Log characteristic image (composite mode image) that is less crushed and visually easy to display is displayed on the display 32 or the like. Is particularly effective.

<3. Basic Operation of Image Processing Device>
FIGS. 11 and 12 are flowcharts illustrating the flow of the corresponding point search operation realized in the image processing system 1. Since the individual functions of each unit have already been described, only the overall flow will be described here.

  First, for example, this operation flow is started in response to an operation on the operation unit 31 by the user, and the process proceeds to step S1 in FIG.

  In step S1, the stereo image acquisition unit 361 acquires a stereo image composed of the standard image G1 and the reference image G2.

  In step S2, the characteristic determination unit 362 acquires image characteristic information of the standard image G1 and the reference image G2.

  In step S3, the characteristic determination unit 362 associates both the base image G1 and the reference image G2 so that each region is a linear region (appropriate exposure region) or a logarithmic region (overexposure region). Determine if there is any. If it is determined that the region is a linear region, the process proceeds to step S4. If it is determined that the region is a logarithmic region, the process proceeds to step S5.

  In step S4, the first corresponding point search processing unit 363a of the corresponding point search unit 363 is selectively activated for the region determined to be a linear region (by the characteristic determination unit 362) in step S3, and the subpixels are activated. Corresponding point search is performed by the POC processing of the level.

  In step S5, the second corresponding point search processing unit 363b of the corresponding point search unit 363 is selectively activated for the region determined to be a logarithmic region in step S3 (by the characteristic determination unit 362), and the SAD processing is performed. Alternatively, corresponding point search is performed by pixel-level POC processing.

  In step S6, the distance information calculation unit 364 calculates distance information from the corresponding point gp2 of the reference image G2 with respect to the target point gp1 of the reference image G1 processed by the corresponding point search unit 363.

  In step S <b> 7, the object candidate detection unit 365 detects a candidate for the subject OB based on the distance information obtained by the distance information calculation unit 364.

  In step S8, the object determination unit 366 determines whether or not the subject OB satisfies a predetermined condition in the three-dimensional image information of the photographing object detected by the object candidate detection unit 365.

  In step S9, when it determines with danger in step S8 (by the object determination part 366), it progresses to step S10, and when that is not right, it progresses to step S11.

  In step S10, when the warning unit 367 determines that there is a possibility of a collision by the target object determination unit 366, a warning is given. Further, a warning is displayed on the display 32 together with a warning sound from the speaker 33 (step S11).

  In step S <b> 11, display of the composite mode image on the display 32 is continued regardless of the presence / absence of danger by the object determination unit 366.

  In step S12, it is determined whether or not to continue image acquisition. If the image acquisition is to be continued, the process proceeds to step S1, and if the image acquisition is to be terminated by an instruction input by the operator, this operation flow is ended.

<4. Modification>
As mentioned above, although embodiment of this invention has been described, this invention is not limited to the said embodiment, A various deformation | transformation is possible.

  * In this embodiment, the number of image pickup units is two for simplicity of explanation. However, the number of image pickup units is not limited to two. In general, the present invention can be applied to a configuration having a plurality of image pickup units. It is.

  An image sensor dedicated to the linear imaging mode and an image sensor dedicated to the composite imaging mode may be provided separately. In this case, when discriminating between the linear region and the logarithmic region of the composite mode image using the feature amount of the linear mode image, these images are taken at the same time, and thus correspond to the frame switching time. An error can be prevented.

  * In the present embodiment, the stereo image acquisition unit 361 acquires the linear conversion characteristic image and the linear-log characteristic image by switching, but it may acquire only the linear-log characteristic image.

  That is, if the relatively bright and relatively dark parts of the acquired image do not vary much in time, know in advance which area of the acquired image is a linear area and which part is a logarithmic area. Is possible. As such an example, there is a case where the illuminance distribution due to illumination is almost known in advance, such as a surveillance camera in an elevator.

  In such a case, it is possible to determine the appropriate exposure region and the overexposure region for the linear mode image, and based on the determination, the linear region and the logarithmic region in the composite mode image are not dynamically determined, but semi-fixed. A linear region and a logarithmic region can be set in, and different corresponding point search methods can be applied to each.

  * In the present embodiment, the characteristic determination unit 362 determines the characteristic by regarding the overexposed area as the logarithmic area as a method for determining the linear area and the logarithmic area. However, the present invention is not limited to this. Can also be used.

  The first determination method is a method for predicting the position of the subject OB based on information in the time series direction. In other words, since the image processing device 3 switches between the linear imaging mode and the composite imaging mode with a time difference between the same imaging elements, the corresponding point search unit 363 takes the imaging results of two or more consecutive times in the linear imaging mode over time. By extrapolating automatically, the location of the corresponding region in the composite imaging mode is estimated.

  This will be described with reference to FIG. Information such as the self-running speed and the steering information is calculated from the linear mode images at time T and time T + 2Δt. Based on the information, the logarithmic region in the composite mode image at time T (or time T + 2Δt) is the time T + 3Δt on the assumption that the traveling parameter of the host vehicle does not change during the period between time T + 2Δt and time T + 3Δt. It is predicted (extrapolated in time) where it exists at the timing.

  By adopting such a configuration, a more precise search is possible because the factor of the moving speed is considered as compared with the case of the above-described embodiment.

  As a second determination method, instead of actually determining a linear region and a logarithmic region and switching the corresponding point search for each region, the second determination method is based on the result of a trial of one corresponding point search process. In other words, it is a method for determining which search method is to be performed entirely. For example, a part of the composite mode image is sampled, and the corresponding point search unit 363 partially performs corresponding point search by the first corresponding point search processing unit 363a and the partial corresponding point. An index value indicating the reliability of the corresponding point obtained by the search is obtained. When the index value is equal to or greater than a predetermined threshold, the processing by the first corresponding point search processing unit 363a is continued to perform the overall countermeasure for the combined mode image, while the index value is When it is less than the threshold value, the remaining area of the composite mode image is switched to processing by the second corresponding point search processing unit 363b.

  Specifically, pixel level POC processing is performed on the image acquired at time T + Δt, and switching to SAD processing is performed according to an index value indicating the reliability calculated from the corresponding point search result, or continued. Then, it is determined whether or not sub-pixel level POC processing is performed. Specifically, the above POC function is used as an index value indicating the reliability, and if the reliability is determined to be low (that is, the peak of the POC function is less than a predetermined threshold value), it is regarded as a logarithmic region.

  Sampling may be performed once for the entire composite mode image, or the composite mode image may be roughly divided into sections and sampled once for each section. For example, if the field of view is divided into the center, left and right sides, top, and bottom, and good results are obtained by sampling the top and searching for one of the corresponding points, the response for the entire top A point search method is adopted. The same is done for the center, left and right sides,.

  The second determination method is a method for indirectly estimating whether the image region is a linear region or a logarithmic region based on the quality of the corresponding point search result without performing direct characteristic determination. Therefore, although the embodiment described above is higher in accuracy, the second determination method does not need to perform characteristic determination for each region intricately along the outer shape of the subject, and the larger region. Therefore, the overall processing speed is improved. For the purpose of eliminating wasteful processing, a pixel cluster composed of a predetermined number of pixels (typically several pixels) in the logarithmically transformed region of the composite mode image is regarded as a noise region, and the corresponding point search method is switched. You may make it process by the same method as the circumference | surroundings, without being a target.

  As a third determination method, a linear transformation characteristic region and a logarithmic transformation property region are determined from position information of inflection points. Specifically, when only the Linear-Log characteristic image is acquired by the stereo image acquisition unit 361, information such as the position of the inflection point CP is acquired from the imaging unit 2, and the output value is more than a predetermined value. A logarithmic conversion characteristic region can be used when the number is large, and a linear conversion characteristic region can be used when the value is smaller than a predetermined value.

  * In this embodiment, the feature value is a luminance value. However, the present invention is not limited to this. For example, a frequency or a target detection result can be used as the feature value.

DESCRIPTION OF SYMBOLS 1 Image processing system 2 Image pick-up part 3 Image processing apparatus 20 Stereo camera 21a, 22b Image pick-up element A0, A2, ... (B0, B2, ...) Linear mode image A1, A3, ... (B1, B3, ...) Composite mode image LN0 , LN2, ... Proper exposure area LN1, LN3, ... Linear area LG0, LG2, ... Overexposure area LG1, LG3, ... Logarithmic area CP Inflection point G1 Reference image G2 Reference image

Claims (14)

A stereo image signal picked up using a stereo image pickup means whose photoelectric conversion characteristics are switched between a linear conversion characteristic and a logarithmic conversion characteristic in accordance with the amount of incident light, and input between a plurality of images included in the stereo image signal An image processing device that determines spatial correspondence to obtain three-dimensional image information of an imaging target,
(a) Of the stereo image signals,
Applying the first corresponding point search method to the linear region as the image region obtained by the linear transformation characteristic, the corresponding point search between the plurality of images is performed,
Corresponding point search means for searching corresponding points between the plurality of images by applying a second corresponding point searching method to a logarithmic region as an image region obtained by logarithmic transformation characteristics;
(b) means for obtaining three-dimensional image information of the imaging object based on the corresponding point search result by the corresponding point search means;
With
While the first corresponding point search method employs a method that gives a search result with higher accuracy than the second corresponding point search method,
As the second corresponding point search method, an image processing apparatus characterized in that a method that provides a search result at a higher speed than the first corresponding point search method is employed. The image processing apparatus according to claim 1,
The image processing apparatus is
Characteristic determining means for determining whether each image area of the stereo image is a linear area or a logarithmic area based on a feature amount of each image area of the stereo image;
Further comprising
The corresponding point search means includes
An image processing apparatus, wherein one of the first and second corresponding point search methods is selected and applied for each image region based on a determination result of the characteristic determination unit. The image processing apparatus according to claim 2,
In the stereo imaging means,
Imaging in linear imaging mode that performs photoelectric conversion with linear conversion characteristics regardless of the amount of incident light,
Multi-mode imaging control means for periodically switching and executing imaging in a composite imaging mode in which photoelectric conversion characteristics change between linear conversion characteristics and logarithmic conversion characteristics in accordance with the amount of incident light;
Further comprising
The characteristic determination means includes
Image processing for determining whether each image area of a stereo image obtained in the composite imaging mode is a linear area or a logarithmic area based on the stereo image obtained in the linear imaging mode apparatus. The image processing apparatus according to claim 3,
The characteristic determination means is
Means for detecting an overexposed region in the image obtained in the linear imaging mode;
Means for identifying a corresponding region corresponding to the overexposed region in the image obtained in the composite imaging mode;
With
The corresponding point search means includes
Of the images obtained in the composite imaging mode,
For the corresponding region, the corresponding point search is performed by the second corresponding point search method,
The remaining region is searched for corresponding points by the first corresponding point searching method. The image processing apparatus according to claim 3 or 4, wherein:
The linear imaging mode and the composite imaging mode are mode switching at the time difference of the same imaging device,
The corresponding point search means includes
An image processing apparatus, wherein the location of the corresponding region in the composite imaging mode is estimated by extrapolating temporally two or more consecutive imaging results in the linear imaging mode. The image processing apparatus according to claim 1,
The corresponding point search determining means,
Means for partially performing the corresponding point search by the first corresponding point search method and obtaining an index value indicating the reliability of the corresponding point obtained by the partial corresponding point search;
Means for continuing the first corresponding point searching method when the index value is equal to or greater than a predetermined threshold, and switching to the second corresponding point searching method when the index value is less than the threshold;
An image processing apparatus comprising: An image processing apparatus according to any one of claims 1 to 6,
The first corresponding point search method includes:
An image processing apparatus, comprising: a method for searching for corresponding points with sub-pixel accuracy based on frequency characteristics of the plurality of images. An image processing apparatus according to any one of claims 1 to 6,
The second corresponding point search method includes:
An image processing apparatus, comprising: a method for searching for corresponding points based on a correlation between luminance values of the plurality of images. An image processing apparatus according to any one of claims 1 to 6,
The second corresponding point search method includes:
An image processing apparatus, comprising: a method of searching for corresponding points with pixel accuracy based on frequency characteristics of the plurality of images. An image processing apparatus according to any one of claims 1 to 9,
Object determination means for determining whether or not the shooting object satisfies a predetermined condition based on the three-dimensional image information of the shooting object;
Warning means for giving a warning when it is determined by the object determination means that the object to be photographed satisfies the predetermined condition;
A perimeter monitoring system comprising: An image processing apparatus according to any one of claims 3 to 5,
Object determination means for determining whether or not the shooting object satisfies a predetermined condition based on the three-dimensional image information of the shooting object;
Warning means for giving a warning when it is determined by the object determination means that the object to be photographed satisfies the predetermined condition;
With
When the object to be imaged in the stereo image is localized only in one of the linear area and the logarithmic area, the multi-mode imaging control means is configured to use the linear imaging mode and the composite imaging. A periphery monitoring system characterized by increasing an imaging ratio in one imaging mode corresponding to the localized region among the modes. The perimeter monitoring system according to claim 10 or 11,
The warning means is
Display control means for displaying an image obtained by imaging in the composite photographing mode on a predetermined screen display means;
A perimeter monitoring system comprising:   A program that, when installed in a computer and executed, causes the computer to function as the image processing apparatus according to any one of claims 1 to 9. The program according to claim 13,
13. A program that causes the computer to function as the image processing apparatus for the periphery monitoring system according to claim 10 by causing the computer to execute the program.

Images