WO2020017209A1

WO2020017209A1 - Distance measurement camera

Info

Publication number: WO2020017209A1
Application number: PCT/JP2019/023661
Authority: WO
Inventors: 覚須藤
Original assignee: ミツミ電機株式会社
Priority date: 2018-07-18
Filing date: 2019-06-14
Publication date: 2020-01-23
Also published as: JP2020020775A; CN112424566B; JP7227454B2; CN112424566A

Abstract

This distance measurement camera 1 comprises: a first imaging system IS1 for acquiring a first image including a first subject image; a second imaging system IS2 for acquiring a second image including a second subject image; a size acquisition unit 3 for detecting a plurality of feature points on the first subject image in the first image, acquiring the size of the first subject image by measuring the distance between the feature points, using an epipolar line to detect a plurality of feature points on the second subject image in the second image that correspond to the plurality of feature points on the first subject image, and acquiring the size of the second subject image by measuring the distance between the feature points; and a distance calculation unit 5 for calculating the distance to the subject 100 on the basis of the image magnification ratio between the magnification of the first subject image and the magnification of the second subject image.

Description

Ranging camera

The present invention generally relates to a distance measuring camera for measuring a distance to a subject, and more specifically, is formed by at least two optical systems in which a change in magnification of a subject image according to the distance to the subject is different from each other. And a distance measuring camera for measuring a distance to a subject based on an image magnification ratio between at least two subject images.

距 Conventionally, a distance measuring camera that measures a distance to a subject by imaging the subject has been proposed. Such a distance-measuring camera includes at least an optical system for collecting light from a subject and forming a subject image, and an image sensor for converting the subject image formed by the optical system into an image. 2. Description of the Related Art A stereo camera type distance measuring camera including two pairs is known (for example, see Patent Document 1).

A stereo camera-type distance measuring camera as disclosed in Patent Literature 1 has a translational parallax between two subject images formed by two optical systems arranged to be shifted from each other in a direction perpendicular to an optical axis direction. (Parallax in a direction perpendicular to the optical axis direction) is calculated, and the distance to the subject can be calculated based on the value of the translation parallax.

In such a stereo camera-type ranging camera, if the translational parallax between the subject images is small, the distance to the subject cannot be calculated accurately. Therefore, in order to sufficiently increase the translational parallax between the subject images, it is necessary to arrange the two optical systems so as to be largely separated from each other in a direction perpendicular to the optical axis direction. This makes it difficult to reduce the size of the ranging camera.

When the subject is located at a short distance, the feature point of the subject image for calculating the translational parallax is shown in one image from the relationship of the visual field of the obtained image, but the feature point in the other image is obtained. Then, the situation that it is not reflected occurs. To avoid this situation, it is necessary to arrange the two optical systems in close proximity. However, when the two optical systems are arranged close to each other, the translational parallax between the subject images is reduced, and the accuracy of the distance measurement is reduced. For this reason, it is difficult to accurately calculate the distance to an object located at a short distance using distance measurement based on translational parallax between object images.

In order to solve such a problem, the present inventors have proposed an image magnification ratio type ranging camera that calculates a distance to a subject based on an image magnification ratio (magnification ratio) between two subject images. I have. In a distance measuring camera of the image magnification ratio method, two optical systems having different magnifications of a subject image according to the distance to the subject are used, and an image between the two subject images formed by the two optical systems is used. The distance to the subject is calculated based on the magnification ratio (ratio of magnification) (see Patent Document 2).

In such a distance-measuring camera using the image magnification ratio, the translational parallax between the subject images is not used to calculate the distance to the subject. Therefore, even if two optical systems are arranged close to each other, the distance to the subject can be reduced. Can be calculated accurately. Therefore, the size of the distance measuring camera can be reduced. Further, since the image magnification ratio between the subject images can be accurately obtained even when the subject is located at a short distance, the distance measurement camera of the image magnification ratio can measure the distance to the subject located at a short distance. The distance can be calculated accurately.

The image magnification ratio between the subject images is calculated from the ratio of the sizes of the two subject images. In order to obtain the size of the subject image, a plurality of feature points (for example, both ends in the height direction or the width direction of the distance measurement target) in the subject image in the image obtained by capturing the subject image are detected. Then, it is obtained by measuring the distance between the feature points in the image. Further, in order to obtain the image magnification ratio between the subject images, it is necessary to acquire the size of the same part of the two subject images. Therefore, after detecting a plurality of feature points of one subject image, corresponding feature point detection for detecting a plurality of feature points of the other subject image corresponding to the detected feature points of the one subject image, respectively. Processing needs to be performed.

対応 Such a corresponding feature point detection process is generally executed by searching the entire region of an image acquired by capturing the other subject image. However, the search of the entire region of the image is an operation requiring a lot of processing time, and the processing time required for the corresponding feature point detection processing becomes long. As a result, there is a problem that the processing time for calculating the distance to the subject based on the image magnification ratio between the subject images becomes long.

JP 2012-26841 A Japanese Patent Application No. 2017-241896

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described conventional problems, and has as its object to check corresponding features for detecting a plurality of feature points of the other subject image corresponding to a plurality of feature points of the one subject image, respectively. In the output processing, a distance measurement capable of reducing a processing time for calculating a distance to a subject based on an image magnification ratio between subject images by performing a search for a feature point using an epipolar line based on epipolar geometry. It is to provide a camera.

Such an object is achieved by the following (1) to (7) of the present invention.
(1) A first optical system for condensing light from a subject to form a first subject image, and a first optical system including the first subject image by capturing the first subject image A first imaging system having a first imaging element for acquiring one image;
The first optical system is arranged so as to be shifted in a direction perpendicular to the optical axis direction of the first optical system, condenses the light from the subject, and forms a second subject image. A second optical system for forming, and a second image sensor for capturing the second subject image to obtain a second image including the second subject image. Imaging system,
By detecting a plurality of feature points of the first subject image in the first image and measuring a distance between the plurality of feature points of the first subject image, Acquiring a size, detecting a plurality of feature points of the second subject image in the second image respectively corresponding to the plurality of feature points of the first subject image, A size obtaining unit for obtaining a size of the second subject image by measuring a distance between the plurality of feature points of the image;
The magnification of the first subject image and the magnification of the second subject image obtained as a ratio of the size of the first subject image acquired by the size acquiring unit to the size of the second subject image. A distance calculator for calculating the distance to the subject based on the image magnification ratio with
The size acquiring unit searches the plurality of epipolar lines in the second image corresponding to the plurality of feature points of the first subject image, respectively, to thereby obtain the second image in the second image. A distance measuring camera for detecting the plurality of feature points of a subject image.

(2) The size acquisition unit corresponds to each of the plurality of feature points of the first subject image based on a model in which characteristics and arrangement of the first imaging system and the second imaging system are considered. The ranging camera according to (1), wherein the plurality of epipolar lines in the second image are derived.

(3) The plurality of epipolar lines in the second image respectively corresponding to the plurality of feature points of the first subject image are measured in the above (2) represented by the following equation (1). Distance camera.

Here, x ₁ and y ₁ are respectively x and y coordinates in any one of the first images of the plurality of feature points of the first subject image, and x ₂ and y ₂ are respectively , X and y coordinates of feature points of the second subject image in the second image corresponding to the arbitrary one of the plurality of feature points of the first subject image, P _x and P _y Are the values in the x-axis direction and the y-axis direction of the translational parallax between the front principal point of the first optical system and the front principal point of the second optical system, respectively, and D is the A depth parallax between the first optical system and the second optical system in the optical axis direction of the first optical system or the second optical system, and PS ₁ is a pixel size of the first image sensor. in and, PS ₂ is a pixel size of the second image sensor, f ₁ is the focal of said first optical system Distance, f ₂ is the focal length of the second optical system, the exit pupil of EP ₁ is the first optical system, to the imaging position of the first object image in the case where the subject is present at infinity distance, EP ₂ from the exit pupil of the second optical system, the distance to the imaging position of the second object image when the object exists at infinity, a _FD1 is the first image sensor The distance from the front principal point of the first optical system to the subject when the first subject image is in the best focus on the imaging surface of a, and a _FD2 is the imaging surface of the second imaging device. The distance from the front principal point of the second optical system to the subject when the second subject image is the best focus.

(4) The first optical system and the second optical system may be configured such that the change in the magnification of the first subject image according to the distance to the subject is different from the magnification according to the distance from the subject. The ranging camera according to the above (1), which is configured to be different from the change in the magnification of the second subject image.

(5) The first optical system and the second optical system are configured such that a focal length of the first optical system and a focal length of the second optical system are different from each other. Thus, the change in the magnification of the first subject image according to the distance to the subject is different from the change in the magnification of the second subject image according to the distance to the subject. The distance measuring camera according to the above (4).

(6) The first optical system and the second optical system are formed by the first optical system when the subject is at infinity from the exit pupil of the first optical system. The second object formed by the second optical system when the object is at infinity from the distance to the image forming position of the first object image and the exit pupil of the second optical system The distance to the image forming position is configured to be different, whereby the change in the magnification of the first subject image according to the distance to the subject is the distance to the subject. The distance measuring camera according to the above (4) or (5), which is different from the change in the magnification of the second object image according to the following.

(7) The depth parallax in the optical axis direction of the first optical system or the second optical system is between the front principal point of the first optical system and the front principal point of the second optical system. The change in the magnification of the first subject image according to the distance to the subject is caused by the change in the magnification of the second subject image according to the distance to the subject. The distance measuring camera according to any one of the above (4) to (6), which is different from the above.

In the ranging camera of the present invention, in a corresponding feature point detection process for detecting a plurality of feature points of the other subject image corresponding to a plurality of feature points of the one subject image, an epipolar line based on epipolar geometry is used. A search for the used feature point is executed. Therefore, the processing time for calculating the distance to the subject based on the image magnification ratio between the subject images can be reduced.

FIG. 1 is a diagram for explaining the principle of distance measurement of the distance measurement camera of the present invention. FIG. 2 is a diagram for explaining the principle of distance measurement of the distance measurement camera of the present invention. FIG. 3 is an image magnification of the magnification of the first subject image formed by the first optical system shown in FIG. 2 and the magnification of the second subject image formed by the second optical system shown in FIG. 6 is a graph for explaining that a ratio changes according to a distance to a subject. FIG. 4 is an XZ plan view showing a model for deriving an epipolar line used in the distance measuring camera of the present invention. FIG. 5 is a YZ plan view showing a model for deriving an epipolar line used in the distance measuring camera of the present invention. FIG. 6 is a diagram illustrating an example of an epipolar line derived using the models illustrated in FIGS. 4 and 5. FIG. 7 is a block diagram schematically showing the distance measuring camera according to the first embodiment of the present invention. FIG. 8 is a block diagram schematically showing a distance measuring camera according to the second embodiment of the present invention. FIG. 9 is a block diagram schematically showing a distance measuring camera according to the third embodiment of the present invention. FIG. 10 is a flowchart for explaining a distance measuring method executed by the distance measuring camera of the present invention. FIG. 11 is a flowchart showing details of the corresponding feature point detection process executed in the distance measuring method shown in FIG.

First, the principle used in the distance measuring camera of the present invention for calculating the distance to the subject based on the image magnification ratio between the subject images will be described. In each of the drawings, components having the same or similar functions are denoted by the same reference numerals.

The magnification m _{OD of the} subject image formed by the optical system is calculated from the distance (subject distance) a from the front principal point (front principal plane) of the optical system to the subject and the rear principal point (rear principal plane) of the optical system. The distance b _OD to the imaging position of the subject image and the focal length f of the optical system can be expressed by the following formula (1) from the lens formula.

In addition, the size Y _OD of the subject image can be expressed by the following equation (2) from the magnification m _{OD of the} subject image and the actual size sz of the subject.

When the imaging surface of the imaging device such as a sensor is located at the imaging position of the subject image, that is, when the subject is in the best focus, the size Y _{OD of the} subject image can be expressed by the above equation (2). When the optical system has an autofocus function and always performs imaging with the best focus, the size _{YOD of the} subject image can be obtained by using the above equation (2).

However, when the optical system is a fixed focus system having no autofocus function and the imaging surface of the imaging device such as a sensor is not at the position where the subject image is formed, that is, when there is defocus, the imaging of the imaging device is performed. In order to obtain the size Y _{FD of the} subject image formed on the surface, the defocus amount, that is, the difference (shift) in the depth direction (optical axis direction) between the imaging position of the subject image and the position of the imaging surface of the image sensor is determined. Amount).

As shown in FIG. 1, the distance from the exit pupil of the optical system to the image forming position of the subject image when the subject is at infinity is defined as EP, and the subject is positioned at an arbitrary distance a from the exit pupil of the optical system. The distance from the image forming position of the subject image in the case where the object exists is _defined as EP _OD, and the distance from the exit pupil of the optical system to the imaging surface of the image sensor (Focus Distance) is defined as EP _FD . Further, the distance from the rear principal point of the optical system to the imaging position of the subject image when the subject is at an arbitrary distance a is represented by b _OD, and the distance from the rear principal point of the optical system to the imaging surface of the image sensor. _{Is defined as bFD} . In the illustrated embodiment, for simplicity of description, the optical system is schematically illustrated such that the rear principal point of the optical system is located at the center position of the optical system.

The distance b _OD from the rear principal point of the optical system to the imaging position of the subject image when the subject exists at an arbitrary distance a can be obtained from the following formula (3) from the lens formula.

Therefore, the difference Δb _OD between the focal length f and the distance b _OD can be obtained by the following equation (4).

The distance b _FD from the rear principal point of the optical system to the imaging surface of the image sensor is the distance a _FD from the front principal point of the optical system to the object when the subject image is best focused on the imaging surface of the image sensor. Can be obtained by the following equation (5) from the lens formula using

Therefore, the difference Δb _FD between the focal length f and the distance b _FD can be obtained by the following equation (6).

As is clear from FIG. 1, the intersection of the optical axis and the exit pupil of the optical system is one of the vertices, and the size of the subject image at the image forming position of the subject image when the subject exists at an arbitrary distance a. A right-angled triangle having Y _OD as one side, and a right-angled triangle having one of the vertices at the intersection of the optical axis and the exit pupil of the optical system, and having the size Y _FD of the subject image on the imaging surface of the image sensor as one side. And have a similar relationship. Therefore, from the similarity relation, EP _OD : EP _FD = Y _OD : Y _FD is established, and the size Y _FD of the subject image on the imaging surface of the imaging device can be obtained from the following equation (7).

As is apparent from the above equation (7), the size Y _FD of the subject image on the imaging surface of the imaging device is determined based on the actual size sz of the subject, the focal length f of the optical system, and the exit pupil of the optical system. , The distance EP from the exit pupil of the optical system to the subject (subject distance) a, and the optical system when the subject image is best focused on the imaging surface of the image sensor. Can be expressed as a function of the distance (focus distance) a _FD from the front principal point to the subject.

Next, as shown in FIG. 2, it is assumed that the same subject 100 is imaged using two imaging systems IS1 and IS2. The first imaging system IS1 collects light from the subject 100 and forms a first optical system OS1 that forms a first subject image, and a first subject image formed by the first optical system OS1. And a first imaging element S1 for imaging. The second imaging system IS2 condenses light from the subject 100 to form a second subject image formed by the second optical system OS2, and a second subject image formed by the second optical system OS2. A second imaging element S2 for imaging. The pixel size of the first image sensor S1 (size per one pixel) is PS _1, the pixel size of the second image pickup element S2 is PS _2.

As is clear from FIG. 2, the optical axis of the first optical system OS1 of the first imaging system IS1 is parallel to the optical axis of the second optical system OS2 of the second imaging system IS2. Do not match. The second optical system OS2 is arranged at a distance P in a direction perpendicular to the optical axis direction of the first optical system OS1.

In the illustrated configuration, the optical axis of the first optical system OS1 and the optical axis of the second optical system OS2 are parallel, but the present invention is not limited to this. For example, the first optical system OS1 and the second optical system OS1 are different from each other in that the angle of the optical axis of the first optical system OS1 (the angle parameters θ and φ in three-dimensional polar coordinates) and the angle of the optical axis of the second optical system OS2 are different from each other. Two optical systems OS2 may be arranged. However, for simplicity of description, the first optical system OS1 and the second optical system OS2 are, as shown in FIG. 2, composed of an optical axis of the first optical system OS1 and an optical axis of the second optical system OS2. Are parallel but not coincident, and are spaced apart from each other by a distance P.

The first optical system OS1 and the second optical system OS2 are fixed-focus optical systems having focal lengths f ₁ and f ₂ , respectively. When the first imaging system IS1 is configured, the position (lens position) of the first optical system OS1, that is, the separation distance between the first optical system OS1 and the first imaging element S1 is an arbitrary distance ( (Focus distance) a The first subject image of the subject 100 at the _FD1 is formed on the imaging surface of the first image sensor S1, that is, the subject 100 at the arbitrary distance a _FD1 is adjusted to be the best focus. Have been. Similarly, when the second imaging system IS2 is configured, the position (lens position) of the second optical system OS2, that is, the separation distance between the second optical system OS2 and the second imaging element S2 is arbitrary. distance (focus distance) a second object image of an object 100 in a _FD2 is formed on the imaging surface of the second image sensor S2, i.e., the object 100 at an arbitrary distance a _FD2 is best focus Has been adjusted as follows.

Further, from the exit pupil of the first optical system OS1, the distance to the imaging position of the first object image when the object 100 is present at infinity is EP _1, the exit pupil of the second optical system OS2 from the distance to the imaging position of the second object image when the object 100 is present at infinity is EP _2.

The first optical system OS1 and the second optical system OS2 are located between the front principal point (front principal plane) of the first optical system OS1 and the front principal point (front principal plane) of the second optical system OS2. Are arranged and arranged so that there is a difference (depth parallax) D in the depth direction (optical axis direction). That is, assuming that the distance (subject distance) from the front principal point of the first optical system OS1 to the subject 100 is a, the distance from the front principal point of the second optical system OS2 to the subject 100 is a + D.

By utilizing the reference to the described similar relationship to Figure 1, the magnification m ₁ of the first object image formed on the imaging surface of the first by the optical system OS1 first image sensor S1 is the following formula It can be represented by (8).

Here, EP _OD1 is the distance from the exit pupil of the first optical system OS1 to the image forming position of the first subject image when the subject 100 exists at the distance a, and EP _FD1 is the first The distance from the exit pupil of the optical system OS1 to the imaging surface of the first imaging element S1. The positional relationship between the distance EP _OD1 and the distance EP _{FD1 is determined by} the position of the first optical system OS1 such that when the first imaging system IS1 is configured, the subject 100 at an arbitrary distance a _FD1 is in the best focus. (Lens position) is determined. Δb _OD1 is the focal length f ₁ and the distance b _OD1 from the rear principal point of the first optical system OS1 to the image _forming position of the first subject image when the subject 100 exists at the distance a. Δb _FD1 is the difference between the focal length f ₁ and the distance b _FD1 from the rear principal point of the first optical system OS1 to the imaging surface of the first image sensor S1, and m _OD1 is , The magnification of the first subject image at the image forming position of the first subject image when the subject 100 exists at the distance a.

Since the above equations (1), (4) and (6) can be applied to the image formation by the first optical system OS1, the above equation (8) can be expressed by the following equation (9).

Here, a _FD1 is the distance from the front principal point of the first optical system OS1 to the subject 100 when the first subject image is in the best focus on the imaging surface of the first image sensor S1.

Similarly, the magnification m2 of the _second subject image formed on the imaging surface of the second imaging element S2 by the second optical system OS2 can be expressed by the following equation (10).

Here, EP _OD2 is the distance from the exit pupil of the second optical system OS2 to the imaging position of the second subject image when the subject 100 exists at a distance a + D, and EP _FD2 is the second This is the distance from the exit pupil of the optical system OS2 to the imaging surface of the second image sensor S2. The positional relationship between the distance EP _OD2 and the distance EP _{FD2 is determined by} the position of the second optical system OS2 such that the subject 100 at an arbitrary distance a _FD2 is in the best focus when the second imaging system IS2 is configured. (Lens position) is determined. Δb _OD2 is the focal length f ₂ and the distance b _OD2 from the rear principal point of the second optical system OS2 to the imaging position of the second subject image when the subject 100 exists at the distance a + D. Δb _FD2 is the difference between the focal length f ₂ and the distance b _FD2 from the rear principal point of the second optical system OS2 to the imaging surface of the second image sensor S2, and m _OD2 is Is the magnification of the second subject image at the image _forming position of the second subject image when the subject 100 exists at the distance a + D, and a _FD2 is the second subject image on the imaging surface of the second image sensor S2. Is the distance from the front principal point of the second optical system OS2 to the subject 100 when the best focus is obtained.

Thus, the imaging surface of the first magnification m ₁ of the first object image formed on the imaging surface of the first imaging device S1 by an optical system OS1, the second image pickup element S2 by the second optical system OS2 The image magnification ratio MR with respect to the magnification m2 of the _second object image formed above can be expressed by the following equation (11).

Here, K is a coefficient, which is determined from fixed values f ₁ , f ₂ , EP ₁ , EP ₂ , a _FD1 , and a _FD2 determined by the configurations of the first imaging system IS1 and the second imaging system IS2. It is represented by the following equation (12).

As apparent from the above equation (11), the first magnification m ₁ of a subject image formed on the imaging surface of the first by the optical system OS1 first image sensor S1, the second optical system OS2 The image magnification ratio MR of the _second subject image formed on the imaging surface of the second image sensor S2 to the magnification m2 of the _second subject image depends on the distance a from the subject 100 to the front principal point of the first optical system OS1. Change.

解 Also, when the above equation (11) is solved for the distance a, the general equation (13) for the distance a to the subject 100 can be obtained.

In the above equation (13), f ₁ , f ₂ , EP ₁ , EP ₂ , D, and K are fixed values determined by the configurations and arrangements of the first imaging system IS1 and the second imaging system IS2. If the image magnification ratio MR can be obtained, the distance a from the subject 100 to the front principal point of the first optical system OS1 can be calculated.

FIG 3, which is calculated based on the equation (13), and the magnification m ₁ of the first object image formed by the first optical system OS1 on the imaging surface of the first imaging device S1, the An example of the relationship between the image magnification ratio MR of the magnification m ₂ of the second subject image formed on the imaging surface of the second imaging element S2 by the second optical system OS2 and the distance a to the subject 100 is shown. Have been. As is apparent from FIG. 3, a one-to-one relationship is established between the value of the image magnification ratio MR and the distance a to the subject 100.

On the other hand, the image magnification ratio MR can be calculated by the following equation (14).

Here, sz is the actual size (height or width) of the subject 100, and Y _FD1 is the first subject image formed on the imaging surface of the first image sensor S1 by the first optical system OS1. The size (image height or image width), Y _FD2 is the size (image height or image width) of the second subject image formed on the imaging surface of the second image sensor S2 by the second optical system OS2. .

The size Y _FD1 of the first subject image can be actually measured from a first image obtained by capturing the first subject image by the first image sensor S1. Similarly, the size Y _FD2 of the second subject image can be actually measured from the second image acquired by the second image sensor S2 capturing the second subject image.

Specifically, the size Y _FD1 of the first subject image detects a plurality of feature points (for example, both ends in the height direction or the width direction) of the first subject image included in the first image. , And is obtained by measuring the distance between the detected feature points. On the other hand, the size Y _FD2 of the second subject image detects a plurality of feature points of the second subject image in the second image corresponding to the plurality of feature points of the detected first subject image, respectively. It is obtained by measuring the distance between the detected feature points. In the following description, a process for detecting a plurality of feature points of a second subject image in a second image corresponding to a plurality of feature points of a detected first subject image is referred to as a corresponding feature. This is called point detection processing. In the ranging camera of the present invention, the processing time required for the corresponding feature point detection processing is greatly reduced by using the epipolar line based on the epipolar geometry in the corresponding feature point detection processing.

FIGS. 4 and 5 show models for deriving epipolar lines used in the distance measuring camera of the present invention. FIG. 4 is an XZ plan view showing an arrangement of the first imaging system IS1 and the second imaging system IS2 in a model for deriving an epipolar line. FIG. 5 is a YZ plan view showing an arrangement of the first imaging system IS1 and the second imaging system IS2 in a model for deriving an epipolar line.

As shown in FIG. 4 and FIG. 5, the first imaging system IS1 and the second imaging system IS2 are composed of the optical axis of the first optical system OS1 of the first imaging system IS1 and the second imaging system IS2. The two optical systems OS2 are arranged such that the optical axes do not coincide. Therefore, a translation parallax occurs between the first subject image formed by the first optical system OS1 and the second subject image formed by the second optical system OS2. In the distance measuring camera of the present invention, utilized for the magnification m ₁ of the first object image which is the ratio of the magnification m ₂ of the second subject image Zobaihi MR calculates the distance a to the object 100 The translation parallax between the first subject image and the second subject image is not used for calculating the distance a to the subject 100. However, since a translational parallax exists between the first subject image and the second subject image, the principle of the epipolar line based on epipolar geometry as used in a stereo camera type ranging camera is described in the present invention. The present invention can also be applied to the first subject image and the second subject image obtained by the distance measuring camera of the present invention.

In general, as the model for deriving the epipolar line, a first arrangement of an imaging system IS1 and the second imaging system IS2 (parameter P _x about _parallax, P y, _D) only in consideration of the first imaging system IS1 and a second characteristic of the imaging system IS2 (above parameters _{_{_{_{f 1, f 2, EP 1}}}} , EP 2, a FD1, a FD2, PS 1, PS 2) pinhole model that does not consider is often used . However, the actual imaging systems IS1 and IS2 have many factors related to imaging of the optical systems OS1 and OS2 and the imaging devices S1 and S2. For this reason, a divergence occurs between the pinhole model ignoring such factors and reality, and it is not possible to accurately derive an epipolar line. On the other hand, in the distance measuring camera of the present invention, the epipolar line is derived by using a model in which the characteristics and arrangement of the first imaging system IS1 and the second imaging system IS2 shown in FIGS. It is possible to derive the epipolar line more accurately. The characteristics and arrangement of the first imaging system IS1 and the second imaging system IS2 in the models shown in FIGS. 4 and 5 are as shown in the following table as described with reference to FIG.

In the models shown in FIGS. 4 and 5, the coordinates of the front principal point of the first optical system OS1 of the first imaging system IS1 are the origin (0, 0, 0), and the coordinates of the second imaging system IS2 are The coordinates of the front principal point of the second optical system OS2 are (P _x , P _y , -D). Therefore, the distance between the optical axis of the first optical system OS1 and the optical axis of the second optical system OS2 in a direction perpendicular to the optical axis direction of the first optical system OS1 or the second optical system OS2. the distance P _is expressed by ^{_{^{P = (P x 2 + P}}} y 2) 1/2. Incidentally, the front principal point of the first optical system OS1 and the x-axis direction of the distance P _x of the front principal point of the second optical system OS2 called translation parallax in the x-axis direction, the first optical system OS1 the distance P _y in the y-axis direction between the front principal point and the front principal point of the second optical system OS2 of the y-axis direction translation parallax. As described above, the distance D in the z-axis direction between the front principal point of the first optical system OS1 and the front principal point of the second optical system OS2 is referred to as depth parallax.

In such a model, the feature point S of the subject 100 located at the coordinates (X, Y, a) is imaged using the first imaging system IS1 and the second imaging system IS2. In this case, first the coordinates of the feature point S in the image and (x _{1, y} _1), second in the image acquired by the second imaging system IS2 obtained by the first imaging system IS1 Let the coordinates of the feature point S be (x ₂ , y ₂ ).

In the following description, coordinates having an arbitrary reference point as the origin are referred to as world coordinates, and coordinates having the origin at the front principal point of the first optical system OS1 of the first imaging system IS1 are referred to as the first imaging system. The coordinates of the second imaging system IS2, which are referred to as the camera coordinates of the second imaging system IS2, are referred to as the camera coordinates of the second imaging system IS2, and the coordinates in the first image ( For example, x ₁ , y ₁ ) is referred to as image coordinates of the first image, and coordinates (eg, x ₂ , y ₂ ) in the second image are referred to as image coordinates of the second image. In the models shown in FIGS. 4 and 5, the origin of the world coordinates is the front principal point of the first optical system OS1 of the first imaging system IS1. Accordingly, in the models shown in FIGS. 4 and 5, the origin of the world coordinates coincides with the origin of the camera coordinates of the first imaging system IS1.

The world coordinates are converted into camera coordinates by an external matrix of the imaging system. Further, the camera coordinates are converted into image coordinates by an internal matrix of the imaging system. Therefore, the world coordinates (X, Y, a) of the feature point S are converted into the image coordinates (x ₁ , y ₁ ) of the first image by the external matrix and the internal matrix of the first imaging system IS1. Similarly, the world coordinates (X, Y, a) of the feature point S are converted into the image coordinates (x ₂ , y ₂ ) of the second image by the external matrix and the internal matrix of the second imaging system IS2. .

First, the image coordinates (x ₁ , y ₁ ) of the _first image acquired by the first imaging system IS1 will be considered. When the feature point S is imaged by the first imaging system IS1, the world coordinates (X, Y, a) of the feature point S are calculated using the external matrix of the first imaging system IS1 and the camera coordinates of the first imaging system IS1. (X ′ ₁ , y ′ ₁ , a ′). However, as described above, the world coordinates in the models shown in FIGS. 4 and 5 have the origin (reference point) at the front principal point of the first optical system OS1 of the first imaging system IS1. There is no rotation or position shift between the world coordinates in the model shown in FIG. 5 and the camera coordinates of the first imaging system IS1. This state can be represented by the following equation (15). The matrix of 4 rows and 4 columns in the following equation (15) is the external matrix of the first imaging system IS1. Since there is no rotation or position shift between the world coordinates in the models shown in FIGS. 4 and 5 and the camera coordinates of the first imaging system IS1, the external matrix of the first imaging system IS1 is a unit matrix.

Next, the camera coordinates (x ′ ₁ , y ′ ₁ , a ′) of the first imaging system IS1 of the feature point S are determined by the internal matrix of the first imaging system IS1 using the image coordinates (x ₁ , y ₁ ). Internal matrix of the first imaging system IS1 is described above with reference to FIG. 2, similar to the relationship between the size Y _FD1 size sz a first object image of an object 100 represented by the above formula (7) Can be derived. Therefore, the following equation (16) can be obtained. In the above equation (7), the size sz of the subject 100 and the size Y _FD1 of the first subject image are expressed in units of mm, but the following equation (16) expresses the image coordinates x1 of the _first image. It is in pixel units.

Similarly, when the image coordinates y1 of the _first image is obtained, the following equation (17) can be obtained.

Here, K ₁ and L ₁ in the above equations (16) and (17) are determined by fixed values f ₁ , EP ₁ , a _FD1 , and PS ₁ determined by the configuration of the first imaging system IS1. . Therefore, K ₁ and L ₁ in the above equations (16) and (17) are fixed values uniquely determined by the configuration of the first imaging system IS1.

From the above equations (16) and (17), the following equation (18) representing the image coordinates (x ₁ , y ₁ ) of the first image of the feature point S can be obtained. The matrix of 3 rows and 4 columns in the following equation (18) is an internal matrix of the first imaging system IS1.

From the above equation (18), the coordinates (x ₁ , y ₁ ) of the feature point S of the subject 100 in the first image acquired by the first imaging system IS1 can be specified. Hereinafter, the feature point S of the subject 100 observed at the image coordinates (x ₁ , y ₁ ) of the first image is referred to as a feature point of the first subject image.

The external matrix of the first imaging system IS1 of 4 rows and 4 columns in the above equation (18) reflects the arrangement of the first imaging system IS1 (the arrangement of the first imaging system IS1 with respect to a reference point in world coordinates). In the equation (18), the internal matrix of the first imaging system IS1 of 3 rows and 4 columns represents the characteristics (fixed values f ₁ , EP ₁ , a _FD1 , and PS ₁ ) of the first imaging system IS1. Reflects.

Now consider the image coordinates of the second image acquired by the second imaging system _{_{IS2 (x 2, y 2)}} . The world coordinates (X, Y, a) of the feature point S are converted into camera coordinates (x ′ ₂ , y ′ ₂ , a ′) of the second imaging system IS2 by an external matrix of the second imaging system IS2. You. At this time, there may be a rotation or displacement of the second imaging system IS2 with respect to the front principal point of the first optical system OS1 of the first imaging system IS1, which is the origin of the world coordinates.

Rotation matrix R ^z of rotation of the rotation matrix R ^x, rotation matrix R ^y of the rotation about the ^y-axis, and z about axis of rotation around the x-axis is represented by the following formula (19).

Since all of the x-axis, y-axis, and z-axis of the second imaging system IS2 have a possibility of rotating with respect to the first imaging system IS1, the rotation matrix R ^x , the rotation matrix R ^y , and the rotation The product of the multiplication by the matrix ^Rz is the rotation matrix R of the second imaging system IS2. Therefore, the rotation matrix R of the second imaging system IS2 is represented by the following equation (20). In the following equation (20), the rotation matrix R is represented by R ^x , R ^y , R ^z , but a rotation matrix R ^x , a rotation matrix R ^y , and a rotation matrix R for obtaining the rotation matrix R are obtained. The order of multiplying ^z is not limited to this. For example, the rotation matrix R ^may be represented by R z ^· R y · ^{R x} and ^R y ^· R x · ^{R z,} and the like.

Further, as described above, the second imaging system IS2, relative to the first imaging system IS1, translation direction of the translation parallax P _x, and a P _y and the depth direction of the depth disparity D. These parallaxes can be represented by a parallel progression column t in the following equation (21).

The external matrix of the second imaging system IS2 is expressed by a combination of the rotation matrix R of Expression (20) and the translation sequence of Expression (21), and the camera coordinates (x _{_{'2, y' 2, a}} ') can be represented by the following formula (22). The matrix of 4 rows and 4 columns in the following equation (22) is the external matrix of the second imaging system IS2.

For the sake of simplicity, in the following description, it is assumed that there is no rotating element of the second imaging system IS2 with respect to the first imaging system IS1, as shown in FIGS. That is, in the above formula _{(22), θ x = 0} , θ y = 0, and θ _z = 0. Note that such an assumption is for the sake of simplicity of description, and is such that the second imaging system IS2 is rotating with respect to the first imaging system IS1 (that is, θ _x , θ _y , and Embodiments in which θ _z is not zero) are also within the scope of the present invention.

As described above, when θ _x = 0, θ _y = 0, and θ _z = 0, the above equation (22) is simplified, and the following equation (23) is obtained.

Next, the camera coordinates (x ′ ₂ , y ′ ₂ , a ′) of the second imaging system IS2 of the feature point S are calculated by the internal matrix of the second imaging system IS2 using the image coordinates (x ₂ , y ₂ ). For the same reason as the above equations (16) and (17), the image coordinates (x ₂ , y ₂ ) of the second image of the feature point S are expressed by the following equations (24) and (25).

Here, K ₂ and L ₂ in the above equations (24) and (25) are determined by fixed values f ₂ , EP ₂ , a _FD2 , and PS ₂ determined by the configuration of the second imaging system IS2. . Therefore, K ₂ and L ₂ in the above equations (24) and (25) are fixed values uniquely determined by the configuration of the second imaging system IS2.

From the above equations (24) and (25), the image coordinates (x ₂ , y ₂ ) of the second image of the feature point S can be expressed by the following equation (26). The matrix of 3 rows and 4 columns in the following equation (26) is an internal matrix of the second imaging system IS2.

From the above equation (26), the coordinates (x ₂ , y ₂ ) of the characteristic point S of the subject 100 in the second image acquired by the second imaging system IS2 can be specified. Hereinafter, the feature point S of the object 100 to be observed in the image coordinates of the second image (x _{2, y} _2), that feature point of the second object image.

The external matrix of the second imaging system IS2 of 4 rows and 4 columns in the above equation (26) reflects the arrangement of the second imaging system IS2 (the arrangement of the second imaging system IS2 with respect to a reference point in world coordinates). The internal matrix of the second imaging system IS2 of 3 rows and 4 columns in the above equation (26) indicates the characteristics (fixed values f ₂ , EP ₂ , a _FD2, PS ₁ ) of the second imaging system IS2. Reflects.

Furthermore, since X in the above equation (18) and X in the above equation (26) are the same, the following equation (27) for the distance a is obtained from the above equations (18) and (26).

Similarly, since Y in the above equation (18) and Y in the above equation (26) are the same, the following equation (28) for the distance a is obtained from the above equations (18) and (26).

Since the equation (27) and (28) are equivalent, summarized the coordinates x ₂ and y ₂ of the characteristic points of the second object image in the second image, the general formula for the epipolar lines shown below ( 29) can be obtained.

Α, β, and γ in the general formula (29) are fixed values f ₁ , f ₂ , EP ₁ , and EP ₂ determined by the configurations and arrangements of the first imaging system IS1 and the second imaging system IS2. _{_{_{_{, PS 1, PS 2, a}}}} FD1, a FD2, P x, P y, is determined by D. Therefore, α, β, and γ in the above equation (29) are fixed values uniquely determined by the configurations and arrangements of the first imaging system IS1 and the second imaging system IS2.

The formula (29) equations relating the coordinates _{x 2} and _{y 2} of the characteristic points of the second second object image in the image represented by the first coordinate in the image _(x _{1, y} 1) Represents the epipolar line in the second image corresponding to the feature point of the first subject image located at. That is, when an arbitrary feature point of the first subject image is detected at coordinates (x ₁ , y ₁ ) in the first image, an arbitrary feature point of the first subject image is detected in the second image. Is always present on the epipolar line represented by the above equation (29).

FIG. 6 shows an example of the epipolar line calculated as described above. When the subject 100 is imaged with the characteristics and arrangement of the first imaging system IS1 and the second imaging system IS2 shown in FIG. 6, the first image and the second image as shown in FIG. An image is obtained. In the example of FIG. 6, an upper vertex of a triangle included in the first image and the second image is set as an arbitrary feature point S of the subject 100. The coordinates having the center point as the origin (coordinates (0, 0)) in each image are the image coordinates of each image.

A feature point of the first subject image (an upper vertex of the triangle in the first image) is detected at a position (x ₁ , y ₁ ) = (972.0, −549.0) in the first image. In this case, the feature points of the second subject corresponding to the feature points of the first subject image always exist on the epipolar line in the second image represented by the above equation (29). In the illustrated example, the feature points of the second subject corresponding to the feature points of the first subject image are coordinates (x ₂ , y ₂ ) in the second image = (568.7, −229.5). Exists.

As described above, by deriving the epipolar line in the second image by using the above equation (29), the search on the epipolar line can be performed without searching the entire region of the second image. A feature point of the second subject image corresponding to an arbitrary feature point of one subject image can be detected. In the corresponding feature point detection processing for detecting the feature points of the second subject image in the second image respectively corresponding to the plurality of feature points of the first subject image, the epipolar line based on the epipolar geometry as described above By performing a search for a feature point using, the processing time required for the corresponding feature point detection process can be significantly reduced. For this reason, the ranging camera of the present invention achieves a significant reduction in processing time for calculating the distance a to the subject 100 based on the image magnification ratio MR between the subject images.

Further, as compared with the pin pole model which is often used to derive the epipolar line as described above, the models shown in FIGS. 4 and 5 are different from those of the first imaging system IS1 and the second imaging system IS2. The feature is that both characteristics and arrangement are considered. In particular, the characteristics (fixed values f ₁ , EP ₁ , a _FD1 , PS ₁ ) of the first imaging system IS1 are reflected in the internal matrix of the first imaging system IS1 of 3 rows and 4 columns in the above equation (18). The characteristics (fixed values f ₂ , EP ₂ , a _FD2 , PS ₂ ) of the second imaging system IS2 are reflected in the internal matrix of the second imaging system IS2 of 3 rows and 4 columns in the above equation (26). Have been. For this reason, a plurality of feature points of the second subject image in the second image can be detected more accurately than in the case of using the conventional pinhole model.

The distance measuring camera according to the present invention uses the epipolar line based on the epipolar geometry as described above in the corresponding feature point detection processing, and detects the first object detected to measure the size _YFD1 of the first object image. A plurality of feature points of the second subject image in the second image respectively corresponding to the plurality of feature points of the image are detected. The distance between a plurality of feature points of the detected second subject image is measured, and the size _YFD2 of the second subject image is obtained. The obtained size Y _FD1 of the first subject image and the size Y _FD2 of the second subject image are obtained by calculating the image magnification ratio MR of the magnification m ₁ of the first subject image and the magnification m ₂ of the _second subject image. The distance a to the subject 100 is calculated based on the image magnification ratio MR.

As described above, the distance measuring camera according to the present invention includes the first object image including the first object image obtained by actually imaging the object 100 using the first imaging system IS1 and the second imaging system IS2. From the image and the second image including the second subject image, the size Y _FD1 of the first subject image and the size Y _FD2 of the second subject image are actually measured, and the above equation (14) MR = Y _FD2 / Y _FD1 it can be obtained by the Zobaihi MR with magnification m ₂ of the magnification m ₁ of the first subject image the second object image based on.

As is apparent from the above equation (11), the focal length _{f 1} of the first optical system OS1 is equal to the focal length _{f 2} of the first optical system OS1 _(f 1 _{= f} 2), first optical from the exit pupil of the system OS1, the distance EP ₁ to the imaging position of the first object image when the object 100 is at infinity is, from the exit pupil of the second optical system OS2, the far object 100 is infinite second equal to the distance EP ₂ to the imaging position of the object image _(EP 1 = _EP 2), and a front principal point of the first optical system OS1, the front principal point of the second optical system OS2 when When there is no depth parallax D in the depth direction (optical axis direction) (D = 0), the image magnification ratio MR does not hold as a function of the distance a, and the image magnification ratio MR is a constant. In this case, the change in the magnification m1 of the _first subject image according to the distance a to the subject 100 becomes the same as the change in the magnification m2 of the _second subject image according to the distance a to the subject 100. As a result, it becomes impossible to calculate the distance a from the first optical system OS1 to the subject based on the image magnification ratio MR.

As a special condition, even if f ₁ ≠ f ₂ , EP ₁ ≠ EP ₂ and D = 0, if f ₁ = EP ₁ and f ₂ = EP ₂ , the image magnification ratio MR is This does not hold as a function of a, and the image magnification ratio MR is a constant. Even in such a special case, it is impossible to calculate the distance a from the first optical system OS1 to the subject based on the image magnification ratio MR.

Therefore, in the distance measuring camera of the present invention, the first optical system OS1 and the second optical system OS2 are configured and arranged so that at least one of the following three conditions is satisfied. change in the magnification m ₁ of the first object image in accordance with a is adapted to differ from the change in the magnification m ₂ of the second object image in accordance with the distance a to the object 100.

The focal length _{f 1} of the (first condition) first optical system OS1, the focal length _{f 2} of the second optical system OS2 are different from each other _{(f 1} ≠ _f 2)
(Second condition) from the exit pupil of the first optical system OS1, the distance EP ₁ to the imaging position of the first object image when the object 100 is at infinity, the injection of the second optical system OS2 from the pupil, and the distance EP ₂ to the imaging position of the second object image when the object 100 is at infinity is different from each other _(EP ₁ ≠ EP 2)
(Third condition) A difference D in the depth direction (optical axis direction) exists between the front principal point of the first optical system OS1 and the front principal point of the second optical system OS2 (D ≠ 0).

In addition, even if at least one of the first to third conditions is satisfied, the special case described above (f ₁ ≠ f ₂ , EP ₁ ≠ EP ₂ , D = 0, f ₁ = EP ₁ and f ₂ = EP ₂ ), the image magnification ratio MR does not hold as a function of the distance a, and the distance a from the first optical system OS1 to the subject 100 is calculated based on the image magnification ratio MR. Becomes impossible. Therefore, in order to calculate the distance a from the first optical system OS1 to the subject 100 based on the image magnification ratio MR, in the distance measuring camera of the present invention, the image magnification ratio MR is established as a function of the distance a. The fourth condition is further satisfied.

Therefore, the image magnification ratio is calculated from the size Y _FD1 of the first subject image and the size Y _FD2 of the second subject image actually measured from the first image and the second image acquired by using the distance measuring camera of the present invention. By calculating the MR, the distance a from the front principal point of the first optical system OS1 to the subject 100 can be calculated.

Hereinafter, a ranging camera of the present invention for calculating the distance a to the object 100 based on Zobaihi MR with magnification m ₁ of the first object image and the magnification m ₂ of the second object image, in the accompanying drawings A detailed description will be given based on the preferred embodiment shown.

<First embodiment>
First, a first embodiment of the distance measuring camera of the present invention will be described with reference to FIG. FIG. 7 is a block diagram schematically showing the distance measuring camera according to the first embodiment of the present invention.

The distance measuring camera 1 shown in FIG. 7 includes a control unit 2 for controlling the distance measuring camera 1, a first optical system OS1 for collecting light from the subject 100, and forming a first subject image. A first imaging system IS1 having a first imaging device S1 for capturing a first subject image and acquiring a first image including the first subject image, and a first optical system OS1. On the other hand, the first optical system OS1 is arranged so as to be shifted by a distance P in a direction perpendicular to the optical axis direction of the first optical system OS1, and collects light from the subject 100 to form a second subject image. A second imaging system IS2 having a second optical system OS2, a second imaging device S2 for capturing a second subject image, and obtaining a second image including the second subject image; size acquisition for acquiring the size _{Y FD2} size _{Y FD1} and a second object image of the first object image 3, the magnification m ₁ of the first object image and Zobaihi MR with magnification m ₂ of the second object image, association information which stores association information associating the distance a to the object 100 a storage unit 4, the first size Y _FD1 and a second object image of the first object image obtained as the ratio of the size Y _FD2 of the subject image magnification m ₁ and a second acquired by the size acquiring section 3 based on Zobaihi MR with magnification m ₂ of the subject image, a distance calculation unit 5 for calculating the distance a to the object 100, the first image acquired by the first image sensor S1 or the second A three-dimensional image generation unit 6 for generating a three-dimensional image of the subject 100 based on the second image acquired by the image sensor S2 and the distance a to the subject 100 calculated by the distance calculation unit 5 And any information such as the liquid crystal panel , An operation unit 8 for inputting an operation by a user, a communication unit 9 for executing communication with an external device, and data communication between components of the distance measuring camera 1. And a data bus 10 for performing transmission and reception.

The distance measuring camera 1 of the present embodiment has the focal length f _{1 of} the first optical system OS1 among the above three conditions required to calculate the distance a to the subject 100 based on the image magnification ratio MR. When the focal length _{f 2} of the second optical system OS2 is different _{(f 1} ≠ _{f 2)} first so the condition is met that the first optical system OS1 and a second optical system OS2 is configured It is characterized by having been done. On the other hand, in the present embodiment, the first optical system OS1 and the second optical system OS2 satisfy the other two conditions (EP ₁ ≠ EP ₂ and D ≠ 0) among the above three conditions. Not configured and deployed. Further, the distance measuring camera 1 of the present embodiment is configured to satisfy a fourth condition that the image magnification ratio MR is satisfied as a function of the distance a.

Therefore, the above general formula (13) for calculating the distance a to the subject 100 using the image magnification ratio MR is simplified by the conditions of EP ₁ = EP ₂ = EP and D = 0, and the following formula (30) ).

Here, the coefficient K is represented by the following equation (31).

Ranging camera 1 of this embodiment, the magnification m ₂ magnification m ₁ and a second object image of the first object image by imaging a subject 100 by the first imaging system IS1 and the second imaging system IS2 Is calculated, and the distance a to the subject 100 is calculated using the above equation (30).

Further, in the distance measuring camera 1 of the present embodiment, the size acquisition unit 3 includes a plurality of feature points (for example, in the height direction) of the first subject image in the first image acquired by the first image sensor S1. _{Alternatively} , the size _YFD1 of the first subject image is acquired by detecting the distance between the plurality of feature points. Further, the size obtaining unit 3 detects a plurality of feature points of the second subject image in the second image respectively corresponding to the plurality of feature points of the detected first subject image, and detects the plurality of feature points. The size Y _FD2 of the second subject image is obtained by measuring the distance between them.

Further, in the distance measuring camera 1 of the present embodiment, a corresponding feature check for detecting a plurality of feature points of the second subject image in the second image respectively corresponding to the plurality of feature points of the first subject image. In the output processing, an epipolar line based on epipolar geometry is used. The general formula (29) representing the epipolar line is simplified by the condition of EP ₁ = EP ₂ = EP and D = 0, and can be expressed by the following formula (32).

In the distance measuring camera 1 of the present embodiment, by searching on the epipolar line in the second image represented by the above equation (32), the second camera corresponding to each of the plurality of feature points of the first subject image is searched. A plurality of feature points of the second subject image in the image can be detected. Thus, a plurality of feature points of the second subject image can be detected without searching the entire area of the second image, and the processing time required for the corresponding feature point detection process can be significantly reduced. . As a result, a processing time for calculating the distance a to the subject 100 based on the image magnification ratio MR between the subject images can be significantly reduced.

Hereinafter, each component of the distance measuring camera 1 will be described in detail. The control unit 2 transmits and receives various data and various instructions to and from each component via the data bus 10 and controls the distance measuring camera 1. The control unit 2 includes a processor for executing arithmetic processing, and a memory for storing data, programs, modules, and the like necessary for controlling the distance measuring camera 1. Executes the control of the distance measuring camera 1 by using data, programs, modules, and the like stored in the memory. The processor of the control unit 2 can provide a desired function by using each component of the distance measuring camera 1. For example, the processor control unit 2, by using the distance calculation unit 5, based on Zobaihi MR with magnification m ₁ of the first object image and the magnification m ₂ of the second object image to the object 100 Can be executed for calculating the distance a.

The processor of the control unit 2 includes, for example, one or more microprocessors, a microcomputer, a microcontroller, a digital signal processor (DSP), a central processing unit (CPU), a memory control unit (MCU), and an image processing unit. (GPU), a state machine, a logic circuit, an application specific integrated circuit (ASIC), or an arithmetic unit that performs arithmetic operations such as signal operations based on computer readable instructions such as a combination thereof. In particular, the processor of the controller 2 is configured to fetch computer readable instructions (eg, data, programs, modules, etc.) stored in the memory of the controller 2 and perform operations, signal operations and controls. I have.

The memory of the control unit 2 includes a volatile storage medium (eg, RAM, SRAM, DRAM), a non-volatile storage medium (eg, ROM, EPROM, EEPROM, flash memory, hard disk, optical disk, CD-ROM, digital versatile disk ( DVD), a magnetic cassette, a magnetic tape, a magnetic disk), or a removable or non-removable computer-readable medium including a combination thereof.

In the memory of the control unit 2, fixed values f ₁ , f ₂ , EP ₁ , EP ₂ , a _FD1 , and a fixed values f ₁ , f ₂ determined by the configuration and arrangement of the first imaging system IS1 and the second imaging system IS2 are provided. _{_{_{_{FD2, PS 1, PS 2,}}}} P x, P y, and D, as well, are derived from these fixed values, the formula for calculating the distance a to the object 100 (13) (or, simplified The fixed values L ₁ , L ₂ , K, used in equation (30) above and the general equation (29) (or the simplified equation (32)) above for the epipolar line in the second image. K ₁ , K ₂ , α, β, and γ are stored in advance.

The first imaging system IS1 has a first optical system OS1 and a first imaging element S1. The first optical system OS1 has a function of condensing light from the subject 100 and forming a first subject image on the imaging surface of the first image sensor S1. The first imaging element S1 has a function of capturing a first subject image formed on an imaging surface and acquiring a first image including the first subject image. The second imaging system IS2 has a second optical system OS2 and a second imaging element S2. The second optical system OS2 has a function of condensing light from the subject 100 and forming a second subject image on the imaging surface of the second image sensor S2. The second imaging element S2 has a function of capturing a second subject image formed on the imaging surface and acquiring a second image including the second subject image.

In the illustrated embodiment, the first imaging element S1 and the first optical system OS1 that constitute the first imaging system IS1 are provided in the same housing, and constitute the second imaging system IS2. Although the second image sensor S2 and the second optical system OS2 are provided in another identical housing, the present invention is not limited to this. An embodiment in which the first optical system OS1, the second optical system OS2, the first image sensor S1, and the second image sensor S2 are all provided in the same housing is also within the scope of the present invention. is there.

The first optical system OS1 and the second optical system OS2 include one or more lenses and optical elements such as a diaphragm. As described above, the first optical system OS1 and a second optical system OS2 is a focal length _{f 1} of the first optical system OS1 and the focal length _{f 2} of the second optical system OS2 are different from each other ( f ₁ ≠ f ₂ ). Thus, changes in accordance with the distance a to the first object 100 magnification m ₁ of a subject image formed by the first optical system OS1 is, the second object image formed by the second optical system OS2 It has become the change for the distance to the subject 100 of the magnification m ₂ and different. Zobaihi MR is the ratio of the magnification m ₂ of such first magnification m ₁ of the first subject image obtained by the configuration of the optical system OS1 and a second optical system OS2 second object image Is used to calculate the distance a to the subject 100.

As shown in the figure, the optical axis of the first optical system OS1 and the optical axis of the second optical system OS2 are parallel but not coincident. Further, the second optical system OS2 is arranged shifted by a distance P in a direction perpendicular to the optical axis direction of the first optical system OS1.

Each of the first image sensor S1 and the second image sensor S2 has a CMOS image sensor having a color filter such as an RGB primary color filter or a CMY complementary color filter arranged in an arbitrary pattern such as a Bayer array. It may be a color image sensor such as a CCD image sensor or a black and white image sensor without such a color filter. In this case, the first image obtained by the first image sensor S1 and the second image obtained by the second image sensor S2 are color or monochrome luminance information of the subject 100.

Further, each of the first image sensor S1 and the second image sensor S2 may be a phase sensor that acquires phase information of the subject 100. In this case, the first image obtained by the first image sensor S1 and the second image obtained by the second image sensor S2 are phase information of the subject 100.

The first optical system OS1 forms a first subject image on the imaging surface of the first imaging device S1, and the first imaging device S1 acquires a first image including the first subject image. . The acquired first image is sent to the control unit 2 and the size acquisition unit 3 via the data bus 10. Similarly, a second subject image is formed on the imaging surface of the second image sensor S2 by the second optical system OS2, and a second image including the second subject image is formed by the second image sensor S2. Is obtained. The acquired second image is sent to the control unit 2 and the size acquisition unit 3 via the data bus 10.

The first image and the second image sent to the size obtaining unit 3 are used for obtaining the size Y _FD1 of the first subject and the size Y _FD2 of the second subject. On the other hand, the first image and the second image sent to the control unit 2 are used for image display by the display unit 7 and communication of image signals by the communication unit 9.

The size acquisition unit 3 acquires a size Y _FD1 of the first subject and a size Y _FD2 of the second subject from the first image including the first subject image and the second image including the second subject image. It has the function to do. Specifically, the size obtaining unit 3 detects a plurality of feature points of the first subject image in the first image, and measures a distance between the detected plurality of feature points of the first subject image. Thereby, the size _YFD1 of the first subject image is obtained. Further, the size obtaining unit 3 detects a plurality of feature points of the second subject image in the second image respectively corresponding to the plurality of feature points of the detected first subject image, and detects the detected second feature point. The size Y _FD2 of the second subject image is obtained by measuring the distance between a plurality of feature points of the subject image.

Specifically, the size obtaining unit 3 receives a first image from the first image sensor S1, and further receives a second image from the second image sensor S2. After that, the size obtaining unit 3 detects arbitrary plural feature points of the first subject image in the first image. The method by which the size obtaining unit 3 detects arbitrary plural feature points of the first subject image in the first image is not particularly limited, and the size obtaining unit 3 uses various methods known in the art. , An arbitrary plurality of feature points of the first subject image in the first image can be detected. The coordinates (x ₁ , y ₁ ) of each of the plurality of feature points detected by the size obtaining unit 3 are temporarily stored in the memory of the control unit 2.

In one example, the size acquisition unit 3 performs a filtering process such as Canny on the first image, and extracts an edge portion of the first subject image in the first image. Thereafter, the size acquisition unit 3 detects any of the extracted edge portions of the first subject image as a plurality of feature points of the first subject image, and measures a separation distance between the plurality of feature points. Thus, the size Y _FD1 of the first subject image is obtained. In this case, the size acquisition unit 3 detects edge portions corresponding to both ends in the height direction of the first subject image as a plurality of feature points of the first subject image, and determines a separation distance between the feature points as the second feature point. The size (image height) _YFD1 of the first subject image may be used, or edge portions corresponding to both ends in the width direction of the first subject image may be detected as a plurality of feature points of the first subject image. The distance between the points may be set as the size (image width) _YFD1 of the first subject image.

After obtaining the size Y _FD1 of the first subject image, the size obtaining unit 3 sets the plurality of second subject images in the second image corresponding to the plurality of feature points of the detected first subject image, respectively. The corresponding feature point detection process for detecting the feature point is performed.

Specifically, the size obtaining unit 3 first detects and detects the coordinates (x ₁ , y ₁ ) of the plurality of feature points of the first subject image stored in the memory of the control unit 2. One of the plurality of feature points of the first subject image is selected. Thereafter, the size acquisition unit 3 determines an area of a predetermined size centered on the selected feature point in the first image (for example, an area of 5 × 5 pixels centered on the selected feature point, 7 × 7 (A pixel area, etc.), and a search block for the selected feature point is obtained. The search block is used to search for a feature point of the second subject image in the second image corresponding to the selected feature point of the first subject. The obtained search block is temporarily stored in the memory of the control unit 2.

Thereafter, the size obtaining unit 3 uses the fixed value stored in the memory of the control unit 2 to convert the epipolar line corresponding to the feature point of the selected first subject image into the above equation (32) (or , General formula (29)). Thereafter, the size acquiring unit 3 detects a feature point of the second subject image in the second image corresponding to the feature point of the selected first subject image by searching on the derived epipolar line. .

Specifically, the size obtaining unit 3 searches for the search block for the feature point of the selected first subject image stored in the memory of the control unit 2 and the pixel on the epipolar line in the second image. A convolution operation (convolution integration) between the search block and the epipolar line peripheral region having the same size as the search block is executed to calculate a correlation value between the search block and the epipolar line peripheral region. The calculation of the correlation value is performed along the derived epipolar line in the second image. The size acquisition unit 3 determines the center pixel (that is, the pixel on the epipolar line) of the epipolar line peripheral region having the highest correlation value as the second pixel in the second image corresponding to the selected feature point of the first subject image. 2 are detected as characteristic points of the subject image. The calculated coordinates (x ₂ , y ₂ ) of the feature point of the second subject image are temporarily stored in the memory of the control unit 2.

When performing a convolution operation between the search block and the epipolar line peripheral region, interpolation of pixels with respect to the search block or the second image may be performed. Any method known in the art for accurately obtaining such a correlation value between two regions may be used in the corresponding feature point detection processing.

Such processing is performed until the feature points of the second subject image in the second image corresponding to all of the detected feature points of the first subject image are detected. Are repeatedly executed by changing the characteristic points of Therefore, the size acquiring unit 3 derives a plurality of epipolar lines respectively corresponding to a plurality of feature points of the detected first subject image based on the above equation (32) (or general equation (29)), By searching on each of the plurality of epipolar lines as described above, a plurality of features of the second subject image in the second image corresponding to the plurality of feature points of the detected first subject image, respectively. Detect points. When the feature points of the second subject image in the second image corresponding to all of the detected feature points of the first subject image are detected, the corresponding feature point detection processing by the size acquisition unit 3 ends.

After executing the corresponding feature point detection process, the size acquisition unit 3 detects the size from the coordinates (x ₂ , y ₂ ) of the plurality of feature points of the second subject image temporarily stored in the memory of the control unit 2. The size _YFD2 of the second subject image is obtained by measuring the separation distance between the plurality of feature points of the second subject image thus obtained.

As described above, the epipolar line represented by the above equation (32) (or the general equation (29)) corresponds to the first imaging system IS1 and the second imaging system IS2 generally used in the related art. Instead of the pinhole model that does not consider the characteristics of the first imaging system IS1 and the second imaging system IS2 as shown in FIGS.

Therefore, the size obtaining unit 3 derives a plurality of epipolar lines in the second image using the conventional pinhole model, and detects a plurality of feature points of the second subject image in the second image. It is possible to more accurately detect a plurality of feature points of the second subject image in the second image as compared with. Thus, the distance a to the subject 100 can be measured more accurately.

Association information storage unit 4, and the magnification _{m 1} of the first object image and Zobaihi MR _(m 2 / _{m 1)} and magnification _{m 2} of the second object image, the front of the first optical system OS1 An arbitrary non-volatile recording medium (for example, a hard disk or a flash memory) for storing association information for associating a distance (subject distance) a from the principal point to the subject 100. Association information stored in the association information storage unit 4, from Zobaihi MR (m 2 / _m ₁₎ between the magnification m ₁ of the first object image and the magnification m ₂ of the second object image, This is information for calculating the distance a to the subject 100.

Typically, the associating information stored in the associating information storage unit 4 includes the above equation (30) (or the general equation (30)) for calculating the distance a to the subject 100 based on the image magnification ratio MR. 13)). Alternatively, the association information stored in the association information storage unit 4 may be a look-up table in which the image magnification ratio MR and the distance a to the subject 100 are uniquely associated. By referring to such association information stored in the association information storage unit 4, the distance a to the subject 100 can be calculated based on the image magnification ratio MR. When the association information is the above-described expression for calculating the distance a to the subject 100, the fixed value stored in the memory of the control unit 2 is also referred to, in addition to the association information, A distance a to 100 is calculated.

The distance calculator 5 calculates the magnification m ₁ of the _first subject image obtained as a ratio between the size Y _FD1 of the first subject image acquired by the size acquiring unit 3 and the size Y _FD2 of the second subject image, and It has a function of calculating the distance a to the subject 100 based on the image magnification ratio MR with the magnification m 2 of the subject image of No. ₂ . Specifically, the distance calculation unit 5 calculates the above equation (14) MR = Y _FD2 based on the size Y _FD1 of the first subject image and the size Y _FD2 of the second subject image acquired by the size acquisition unit 3. / by _{Y FD1,} calculates the magnification _{m 1} of the first object image and Zobaihi MR with magnification _{m 2} of the second object image. Thereafter, the distance calculation unit 5 refers to the association information stored in the association information storage unit 4 (if the association information is the above-described expression for calculating the distance a to the subject 100, The distance a to the subject 100 is calculated (specified) based on the image magnification ratio MR (see also the fixed value stored in the memory of the control unit 2).

The three-dimensional image generation unit 6 calculates the distance a to the subject 100 calculated by the distance calculation unit 5 and the color or black-and-white luminance information of the subject 100 acquired by the first imaging system IS1 or the second imaging system IS2. A function of generating a three-dimensional image of the subject 100 based on the first image or the second image). Here, the “three-dimensional image of the subject 100” refers to the calculated distance a to the subject 100 associated with each pixel of the two-dimensional image representing the color or monochrome luminance information of the normal subject 100. Means the data that exists. When the first imaging device S1 of the first imaging system IS1 and the second imaging device S2 of the second imaging system IS2 are phase sensors that acquire phase information of the subject 100, a three-dimensional image is obtained. The generation unit 6 is omitted.

The display unit 7 is a panel-type display unit such as a liquid crystal display unit, and according to a signal from a processor of the control unit 2, the color of the subject 100 acquired by the first imaging system IS1 or the second imaging system IS2. Alternatively, black-and-white luminance information or phase information (first image or second image) of the subject 100, the distance a to the subject 100 calculated by the distance calculating unit 5, the subject 100 generated by the three-dimensional image generating unit 6, 3D image, information for operating the distance measuring camera 1, and the like are displayed on the display unit 7 in the form of characters or images.

The operation unit 8 is used by a user of the distance measuring camera 1 to execute an operation. The operation unit 8 is not particularly limited as long as the user of the distance measuring camera 1 can perform an operation. For example, a mouse, a keyboard, a numeric keypad, a button, a dial, a lever, a touch panel, and the like can be used as the operation unit 8. . The operation unit 8 transmits a signal corresponding to an operation by the user of the distance measuring camera 1 to the processor of the control unit 2.

The communication unit 9 has a function of inputting data to the ranging camera 1 or outputting data from the ranging camera 1 to an external device. The communication unit 9 may be configured to be connectable to a network such as the Internet. In this case, by using the communication unit 9, the distance measuring camera 1 can communicate with an external device such as a web server or a data server provided outside.

As described above, in the distance measuring camera 1 of the present embodiment, the first optical system OS1 and the second optical system OS2 are formed by the focal length f1 of the _first optical system OS1 and the focal length of the second optical system OS2. f ₂ are different from each other (f ₁ ≠ f ₂ ), whereby the change in the magnification m ₁ of the first subject image with respect to the distance a to the subject 100 and the distance a to the subject 100 And the change in the magnification m2 of the _second subject image with respect to Therefore, the distance measurement camera 1 of the present invention, based on the Zobaihi MR _(m 2 / _{m 1)} between the magnification _{m 1} of the first object image and the magnification _{m 2} of the second object image to the object 100 Can be uniquely calculated.

In the distance measuring camera 1 of the present embodiment, an epipolar line based on epipolar geometry is used in the corresponding feature point detection processing executed by the size acquisition unit 3. Therefore, the processing time required for the corresponding feature point detection processing can be significantly reduced, and the processing time required for calculating the distance a to the subject 100 can be significantly reduced.

Further, the epipolar line represented by the above equation (32) (or the general equation (29)) does not consider the characteristics of the first imaging system IS1 and the second imaging system IS2 generally used in the related art. Instead of the pinhole model, it is derived using a model that considers both the characteristics and arrangement of the first imaging system IS1 and the second imaging system IS2 shown in FIGS. Therefore, a plurality of epipolar lines in the second image are derived using the conventional pinhole model, and compared with a case where a plurality of feature points of the second subject image in the second image are detected. A plurality of feature points of the second subject image in the second image can be accurately detected. Thereby, the accuracy of the measurement of the distance a to the subject 100 by the distance measuring camera 1 can be improved.

<Second embodiment>
Next, a distance measuring camera 1 according to a second embodiment of the present invention will be described in detail with reference to FIG. FIG. 8 is a block diagram schematically showing a distance measuring camera according to the second embodiment of the present invention.

Hereinafter, the distance measuring camera 1 of the second embodiment will be described focusing on the differences from the distance measuring camera 1 of the first embodiment, and the description of the same items will be omitted. The ranging camera 1 of the present embodiment is the same as the ranging camera 1 of the first embodiment, except that the configurations of the first optical system OS1 and the second optical system OS2 are changed.

The distance measuring camera 1 according to the present embodiment uses the exit pupil of the first optical system OS1 among the above three conditions required to calculate the distance a to the subject 100 based on the image magnification ratio MR. the distance EP ₁ to the imaging position of the first object image when the object 100 is at infinity, the exit pupil of the second optical system OS2, second object image when the object 100 is at infinity of the distance EP ₂ to the imaging position, different (EP ₁ ≠ EP ₂₎ a second so the condition is met that, the first optical system OS1 and a second optical system OS2 is configured It is characterized by. On the other hand, in the present embodiment, the first optical system OS1 and the second optical system OS2 satisfy the other two conditions (f ₁ ≠ f ₂ and D ≠ 0) among the above three conditions. Not configured and deployed. Further, the distance measuring camera 1 of the present embodiment is configured to satisfy a fourth condition that the image magnification ratio MR is satisfied as a function of the distance a.

The general formula (13) for calculating the distance a to the subject 100 based on the image magnification ratio MR is simplified by the conditions of f ₁ = f ₂ = f and D = 0, and is expressed by the following formula (33). Can be represented.

Here, the coefficient K is represented by the following equation (34).

The above general formula (29) representing the epipolar line is simplified by the conditions of f ₁ = f ₂ = f and D = 0, and can be expressed by the following formula (35).

Thus, in the distance measuring camera 1 of this embodiment, the exit pupil of the first optical system OS1, the distance EP ₁ to the imaging position of the first object image when the object 100 is at infinity, from the exit pupil of the second optical system OS2, and the distance EP ₂ to the imaging position of the second object image when the object 100 is at infinity is different from each other (EP ₁ ≠ _EP 2), first optical system OS1 and a second optical system OS2 is constituted by this, the first subject image with respect to the distance a to the object 100 and the change in the magnification m _1, the second for the distance a to the object 100 change the magnification m ₂ of the object image, which is different from each other. Therefore, the distance measurement camera 1 of this embodiment, based on Zobaihi MR _(m 2 / _{m 1)} between the magnification _{m 1} of the first object image and the magnification _{m 2} of the second subject image, the subject 100 Can be uniquely calculated.

Further, in the distance measuring camera 1 of the present embodiment, by searching on the epipolar line in the second image represented by the above equation (35), the second camera corresponding to each of the plurality of feature points of the first subject image is searched. A plurality of feature points of the second subject image in the second image can be detected. Thus, a plurality of feature points of the second subject image can be detected without searching the entire area of the second image, and the processing time required for the corresponding feature point detection process can be significantly reduced. . As a result, a processing time for calculating the distance a to the subject 100 based on the image magnification ratio MR between the subject images can be significantly reduced. As described above, according to the present embodiment, the same effects as those of the first embodiment can be exerted.

<Third embodiment>
Next, a distance measuring camera 1 according to a third embodiment of the present invention will be described in detail with reference to FIG. FIG. 9 is a block diagram schematically showing a distance measuring camera according to the third embodiment of the present invention.

Hereinafter, the distance measuring camera 1 of the third embodiment will be described focusing on the differences from the distance measuring camera 1 of the first embodiment, and the description of the same items will be omitted. The ranging camera 1 of the present embodiment is the same as the ranging camera 1 of the first embodiment, except that the configurations of the first optical system OS1 and the second optical system OS2 are changed.

The distance measuring camera 1 according to the present embodiment uses the front principal point of the first optical system OS1 among the above three conditions required to calculate the distance a to the subject 100 based on the image magnification ratio MR. And the first optical system OS1 and the second optical system OS2 so that the third condition that a difference D in the depth direction (optical axis direction) exists between the second optical system OS2 and the front principal point (D （0) is satisfied. The second optical system OS2 is configured and arranged. On the other hand, in the present embodiment, the first optical system OS1 and the second optical system OS2 satisfy the other two conditions (f ₁ ≠ f ₂ and EP ₁ ≠ EP ₂ ) among the above three conditions. Is not configured as such. Further, the distance measuring camera 1 of the present embodiment is configured to satisfy a fourth condition that the image magnification ratio MR is satisfied as a function of the distance a.

The general formula (13) for calculating the distance a to the subject 100 based on the image magnification ratio MR is simplified by the conditions of f ₁ = f ₂ = f and EP ₁ = EP ₂ = EP. It can be represented by (36).

Here, the coefficient K is represented by the following equation (37).

The above general formula (29) representing the epipolar line is simplified by the conditions of f ₁ = f ₂ = f and EP ₁ = EP ₂ = EP, and can be expressed by the following formula (38).

Thus, in the distance measuring camera 1 of the present embodiment, the difference D in the depth direction (optical axis direction) between the front principal point of the first optical system OS1 and the front principal point of the second optical system OS2. Is present (D ≠ 0), whereby the change in the magnification m ₁ of the first subject image with respect to the distance a to the subject 100 and the second change the magnification m ₂ of the object image, which is different from each other. Therefore, the distance measurement camera 1 of this embodiment, based on Zobaihi MR _(m 2 / _{m 1)} between the magnification _{m 1} of the first object image and the magnification _{m 2} of the second subject image, the subject 100 Can be uniquely calculated.

Further, in the distance measuring camera 1 of the present embodiment, by searching on the epipolar line in the second image represented by the above equation (38), the second camera corresponding to each of the plurality of feature points of the first subject image is searched. A plurality of feature points of the second subject image in the second image can be detected. Thus, a plurality of feature points of the second subject image can be detected without searching the entire area of the second image, and the processing time required for the corresponding feature point detection process can be significantly reduced. . As a result, a processing time for calculating the distance a to the subject 100 based on the image magnification ratio MR between the subject images can be significantly reduced. As described above, according to the present embodiment, the same effects as those of the first embodiment can be exerted.

As described above in detail with reference to each embodiment, the distance measuring camera 1 of the present invention uses the first image acquired using the first imaging system IS1 and the second imaging system IS2. By calculating the image magnification ratio MR from the size Y _FD1 of the first subject image actually measured from the acquired second image and the size Y _FD2 of the second subject image, the front main unit of the first optical system OS1 is calculated. The distance a from the point to the subject 100 can be calculated.

In the corresponding feature point detection processing for measuring the size Y _FD2 of the second subject image, an epipolar line based on epipolar geometry is used. Therefore, a plurality of feature points of the second subject image can be detected without searching the entire area of the second image, and the processing time required for the corresponding feature point detection processing can be greatly reduced. As a result, a processing time for calculating the distance a to the subject 100 based on the image magnification ratio MR between the subject images can be significantly reduced.

Further, in each of the above embodiments, two optical systems of the first optical system OS1 and the second optical system OS2 are used, but the number of optical systems used is not limited to this. For example, an embodiment further including an additional optical system in addition to the first optical system OS1 and the second optical system OS2 is also within the scope of the present invention. In this case, the additional optical system changes the magnification of the subject image formed by the additional optical system with respect to the distance a to the subject 100 with respect to the distance a to the subject with the magnification m1 of the _first subject image. The configuration and arrangement are different from the change and the change with respect to the distance a to the subject at the magnification m2 of the _second subject image.

In the first to third embodiments, the first to third embodiments satisfy any one of the above three conditions required to calculate the distance a to the subject 100 based on the image magnification ratio MR. The first optical system OS1 and the second optical system OS2 are configured and arranged, but the first optical system OS1 and the second optical system OS2 are configured so that at least one of the above three conditions is satisfied. The present invention is not limited to this as long as it is configured and arranged. For example, an aspect in which the first optical system OS1 and the second optical system OS2 are configured and arranged such that all or any combination of the above three conditions is satisfied is also within the scope of the present invention. .

<Ranging method>
Next, a distance measuring method executed by the distance measuring camera 1 of the present invention will be described with reference to FIGS. FIG. 10 is a flowchart for explaining a distance measuring method executed by the distance measuring camera of the present invention. FIG. 11 is a flowchart showing details of the corresponding feature point detection process executed in the distance measuring method shown in FIG.

Note that the ranging method described in detail below is performed using the ranging camera 1 according to the above-described first to third embodiments of the present invention and an arbitrary device having the same function as the ranging camera 1. However, the description will be made assuming that the processing is executed using the distance measuring camera 1 according to the first embodiment.

距 The distance measuring method S100 shown in FIG. 10 is started when the user of the distance measuring camera 1 executes an operation for measuring the distance a to the subject 100 using the operation unit 8. In step S110, a first image of the subject formed by the first optical system OS1 is captured by the first imaging element S1 of the first imaging system IS1, and a first image including the first subject image is formed. Is obtained. The first image is sent to the control unit 2 and the size acquisition unit 3 via the data bus 10. Similarly, in step S120, the second imaging device S2 of the second imaging system IS2 captures the second subject image formed by the second optical system OS2, and the second imaging device includes the second subject image. Are acquired. The second image is sent to the control unit 2 and the size acquisition unit 3 via the data bus 10. The acquisition of the first image in S110 and the acquisition of the second image in S120 may be performed simultaneously or separately.

After the acquisition of the first image in step S110 and the acquisition of the second image in step S120, the distance measuring method S100 proceeds to step S130. In step S130, the size obtaining unit 3 detects arbitrary plural feature points of the first subject image in the first image. Arbitrary plural feature points of the first subject image detected by the size acquiring unit 3 in step S130 are, for example, both ends in the height direction of the first subject image or both ends in the width direction of the first subject image. Department. The coordinates (x ₁ , y ₁ ) of each of the plurality of feature points of the first subject image detected by the size obtaining unit 3 are temporarily stored in the memory of the control unit 2.

In step S140, the size acquisition unit 3 refers to the coordinates (x ₁ , y ₁ ) of the plurality of feature points of the first subject image temporarily stored in the memory of the control unit 2 and detects the coordinates. The size _YFD1 of the first subject image is obtained by measuring the distance between a plurality of feature points of the first subject image. The size Y _FD1 of the first subject image acquired in step S140 is temporarily stored in the memory of the control unit 2.

Thereafter, in step S150, the size acquisition unit 3 detects a plurality of feature points of the second subject image in the second image corresponding to the plurality of feature points of the first subject image detected in step S130, respectively. Corresponding feature point detection processing for performing the FIG. 11 shows a flowchart illustrating details of the corresponding feature point detection processing executed in step S150.

In step S151, the size acquisition unit 3, with reference to the respective coordinates of a plurality of feature points of the first subject image stored in the control unit 2 memory (x _{1, y} _1), were detected One of a plurality of feature points of one subject image is selected. Next, in step S152, the size obtaining unit 3 determines a region of a predetermined size centered on a feature point of the selected first subject image in the first image (for example, a 5 × area centered on the feature point). A region of 5 pixels, a region of 7 × 7 pixels, etc.) is cut out, and a search block for the selected feature point is obtained. The obtained search block is temporarily stored in the memory of the control unit 2.

Next, in step S153, the size acquisition unit 3 uses the fixed value stored in the memory of the control unit 2 to select a second point corresponding to the feature point of the first subject image selected in step S151. The epipolar line in the image is derived based on the general formula (29) described above (or the expression representing the simplified epipolar line in each embodiment). Thereafter, in step S154, the size acquisition unit 3 searches the feature block of the selected first subject image stored in the memory of the control unit 2 for the feature point of the selected first subject image and the derived epipolar image in the second image. A convolution operation (convolution integration) is performed between the search block and the epipolar line peripheral region having the same size as the search block, and a correlation value between the search block and the epipolar line peripheral region is calculated. The calculated correlation value is temporarily stored in the memory of the control unit 2. The calculation of the correlation value is also referred to as block matching, and is performed along the derived epipolar line in the second image.

When the calculation of the correlation value along the epipolar line in the second image ends, the process of step S150 shifts to step S155. In step S155, the size acquisition unit 3 determines the center pixel (that is, the pixel on the epipolar line) of the epipolar line peripheral region having the highest correlation value as the second pixel corresponding to the selected feature point of the first subject image. It is detected as a feature point of the second subject image in the image. The calculated coordinates (x ₂ , y ₂ ) of the feature point of the second subject image are temporarily stored in the memory of the control unit 2.

After that, in step S156, it is determined whether all of the plurality of feature points of the first subject image detected in step S130 have been selected in step S151. If all of the plurality of feature points of the first subject image detected in step S130 have not been selected in step S151 (step S156 = No), the process of step S150 returns to step S151. In step S151, an unselected one of a plurality of feature points of the first subject image is newly selected, and the feature point of the selected first subject image is changed. The processing of steps S151 to S155 is performed until the feature points of the second subject image in the second image corresponding to all the feature points of the detected first subject image are detected. Are repeatedly executed by changing the feature points of the subject image.

場合 If all of the plurality of feature points of the first subject image detected in step S130 have been selected in step S151 (step S156 = Yes), the processing in step S150 ends. When the process in step S150 ends, the distance measuring method S100 proceeds to step S160.

Returning to FIG. 10, in step S160, the size obtaining unit 3 obtains the size Y _FD2 of the second subject image by measuring the distance between a plurality of feature points of the detected second subject image. The size Y _FD2 of the second subject image acquired in step S160 is temporarily stored in the memory of the control unit 2.

When the size _{Y FD2} size _{Y FD1} and a second object image of the first object image is acquired by the size acquiring unit 3, the distance measuring method S100, the process proceeds to step S170. In step S170, the distance calculation unit 5 calculates the above equation (14) from the size Y _FD1 of the first subject image and the size Y _FD2 of the second subject image temporarily stored in the memory of the control unit 2. based on the _Y _FD2 / Y _FD1, it calculates the Zobaihi MR with magnification _{m 2} of the magnification _{m 1} of the first subject image the second object image. Next, in step S180, the distance calculation unit 5 refers to the association information stored in the association information storage unit 4, and calculates the distance a to the subject 100 based on the calculated image magnification ratio MR. . If the association information is the above-described expression for calculating the distance a to the subject 100, the distance calculation unit 5 adds the fixed information stored in the memory of the control unit 2 in addition to the association information. The distance a to the subject 100 is calculated with reference to the value.

In step S180, when the distance calculation unit 5 calculates the distance a to the subject 100, the distance measuring method S100 proceeds to step S190. In step S190, the three-dimensional image generation unit 6 determines whether the distance a to the subject 100 calculated by the distance calculation unit 5 and the color or monochrome of the subject 100 acquired by the first imaging system IS1 or the second imaging system IS2. A three-dimensional image of the subject 100 is generated based on the luminance information (the first image or the second image). If the first imaging device S1 of the first imaging system IS1 and the second imaging device S2 of the second imaging system IS2 are phase sensors that acquire phase information of the subject 100, step S190 is performed. Omitted.

Thereafter, the color or black-and-white luminance information or phase information (first image and / or second image) of the subject 100 acquired in the steps up to here, the distance a to the subject 100, and / or 3 The two-dimensional image is displayed on the display unit 7 or transmitted to an external device by the communication unit 9, and the distance measuring method S100 ends.

Although the distance measuring camera of the present invention has been described based on the illustrated embodiment, the present invention is not limited to this. Each component of the present invention can be replaced with any component that can exhibit the same function, or any component can be added to each component of the present invention.

Those skilled in the art and art to which the invention pertains will be able to make modifications to the described ranging camera arrangements without significantly departing from the principles, concepts and scope of the invention. A ranging camera having a modified configuration is also within the scope of the present invention. For example, a mode in which the distance measuring cameras of the first to fourth embodiments are arbitrarily combined is also within the scope of the present invention.

The numbers and types of components of the distance measuring camera shown in FIGS. 7 to 9 are merely examples for explanation, and the present invention is not necessarily limited to this. Embodiments in which any component is added or combined or any component is deleted without departing from the principle and intent of the present invention are also within the scope of the present invention. Further, each component of the distance measuring camera may be realized by hardware, may be realized by software, or may be realized by a combination thereof.

The number and types of steps of the distance measuring method S100 shown in FIGS. 10 and 11 are merely examples for explanation, and the present invention is not necessarily limited to this. Embodiments in which any step is added or combined for any purpose or any step is deleted without departing from the principle and intention of the present invention are also within the scope of the present invention.

In the ranging camera of the present invention, in a corresponding feature point detection process for detecting a plurality of feature points of the other subject image corresponding to a plurality of feature points of the one subject image, an epipolar line based on epipolar geometry is used. A search for the used feature point is executed. Therefore, the processing time for calculating the distance to the subject based on the image magnification ratio between the subject images can be reduced. Therefore, the present invention has industrial applicability.

Claims

A first optical system for condensing light from a subject to form a first subject image; and a first image including the first subject image by capturing the first subject image A first imaging system having a first imaging element for acquiring
The first optical system is arranged so as to be shifted in a direction perpendicular to the optical axis direction of the first optical system, condenses the light from the subject, and forms a second subject image. A second optical system for forming, and a second image sensor for capturing the second subject image to obtain a second image including the second subject image. Imaging system,
By detecting a plurality of feature points of the first subject image in the first image and measuring a distance between the plurality of feature points of the first subject image, Acquiring a size, detecting a plurality of feature points of the second subject image in the second image respectively corresponding to the plurality of feature points of the first subject image, A size obtaining unit for obtaining a size of the second subject image by measuring a distance between the plurality of feature points of the image;
The magnification of the first subject image and the magnification of the second subject image obtained as a ratio of the size of the first subject image acquired by the size acquiring unit to the size of the second subject image. A distance calculator for calculating the distance to the subject based on the image magnification ratio with
The size acquiring unit searches the plurality of epipolar lines in the second image corresponding to the plurality of feature points of the first subject image, respectively, to thereby obtain the second image in the second image. A distance measuring camera for detecting the plurality of feature points of a subject image.
The size acquisition unit is configured to: correspond to the plurality of feature points of the first subject image based on a model in which characteristics and arrangements of the first imaging system and the second imaging system are considered. The distance measuring camera according to claim 1, wherein the plurality of epipolar lines in the image are derived.
The distance measuring camera according to claim 2, wherein the plurality of epipolar lines in the second image respectively corresponding to the plurality of feature points of the first subject image are represented by the following equation (1).

Here, x 1 and y 1 are respectively x and y coordinates in any one of the first images of the plurality of feature points of the first subject image, and x 2 and y 2 are respectively , X and y coordinates of feature points of the second subject image in the second image corresponding to the arbitrary one of the plurality of feature points of the first subject image, P x and P y Are the values in the x-axis direction and the y-axis direction of the translational parallax between the front principal point of the first optical system and the front principal point of the second optical system, respectively, and D is the A depth parallax between the first optical system and the second optical system in the optical axis direction of the first optical system or the second optical system, and PS 1 is a pixel size of the first image sensor. in and, PS 2 is a pixel size of the second image sensor, f 1 is the focal of said first optical system Distance, f 2 is the focal length of the second optical system, the exit pupil of EP 1 is the first optical system, to the imaging position of the first object image in the case where the subject is present at infinity distance, EP 2 from the exit pupil of the second optical system, the distance to the imaging position of the second object image when the object exists at infinity, a FD1 is the first image sensor The distance from the front principal point of the first optical system to the subject when the first subject image is in the best focus on the imaging surface of a, and a FD2 is the imaging surface of the second imaging device. The distance from the front principal point of the second optical system to the subject when the second subject image is the best focus.
The first optical system and the second optical system may be configured such that a change in the magnification of the first subject image according to the distance to the subject is equal to or smaller than the second magnification according to the distance from the subject. The ranging camera according to claim 1, wherein the ranging camera is configured to be different from the change in the magnification of the subject image.
The first optical system and the second optical system are configured such that a focal length of the first optical system and a focal length of the second optical system are different from each other, whereby The change in the magnification of the first subject image according to the distance to the subject is different from the change in the magnification of the second subject image according to the distance to the subject. The distance measuring camera according to claim 4.
The first optical system and the second optical system are formed by the first optical system formed from the exit pupil of the first optical system when the subject is at infinity. From the distance to the imaging position of the subject image and the exit pupil of the second optical system, the focusing of the second subject image formed by the second optical system when the subject is at infinity. The distance to the image position is configured to be different, thereby, the change in the magnification of the first subject image according to the distance to the subject, according to the distance to the subject The distance measuring camera according to claim 4, wherein the magnification of the second subject image is different from the change of the magnification.
A depth parallax in the optical axis direction of the first optical system or the second optical system exists between a front principal point of the first optical system and a front principal point of the second optical system, Thus, the change in the magnification of the first subject image according to the distance to the subject is different from the change in the magnification of the second subject image according to the distance to the subject. The distance measuring camera according to any one of claims 4 to 6, wherein