JP3620710B2 - Imaging device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an imaging apparatus that measures a three-dimensional shape of a subject and a subject distance using a captured image obtained by imaging the subject.
[0002]
[Prior art]
In a conventional imaging apparatus (hereinafter also referred to as a camera) that captures a subject such as a person or a moving object, there are various configurations in which the distance to the subject is calculated or data on the three-dimensional shape of the subject is obtained. Or has been proposed.
[0003]
Among the above configurations, various types of distance measuring means for a camera with an AF (Auto Focusing) function have been proposed and put into practical use, particularly as a mechanism used for focusing an imaging optical system by obtaining a distance to a subject (subject distance). Has been made.
As for the above-mentioned ranging means for focusing (hereinafter referred to as “AF ranging means”), for example, Kokura: Modern camera and lens technology (Photo Kogyo Publishing Co., Ltd., 1982). ).
[0004]
As described in the above-mentioned known documents, the AF distance measuring means can be roughly divided into an active type and a passive type, which will be described later, depending on the distance measuring method. Prior art including each will be described below.
[0005]
(1) Active AF ranging means (hereinafter referred to as “first prior art”).
This type of prior art configuration projects a physical wave such as light or ultrasonic waves from a camera to a subject, and performs distance measurement using a wave that is reflected and returned.
In the configuration using ultrasonic waves, the camera includes sound generation means for projecting ultrasonic waves and reception means for receiving ultrasonic waves reflected back from the subject, and the ultrasonic waves emitted from the sound generation means are received. The configuration is such that the time until the object is reflected by the subject and received by the receiving means is measured to obtain the subject distance.
In the configuration for projecting infrared rays, the camera includes light emitting means for projecting infrared rays and light receiving means for receiving infrared rays reflected by the subject, and projects the infrared rays to the subject. In the operation to receive reflected infrared rays, know the projection angle with respect to the reference line of the camera, the light reception angle, and the separation distance (baseline length) between the light emitting means and the light receiving means, and obtain the subject distance based on the triangulation method It was the composition.
In particular, the above-described configuration for projecting infrared rays is advantageous in terms of reliability of distance measurement, simple configuration and low cost, and focusing distance measuring means such as a lens shutter type film camera and a digital camera. It is often used as.
[0006]
(2) Passive AF distance measuring means (hereinafter referred to as “second prior art”).
In this configuration, a physical wave is not projected onto the subject, and the subject distance is calculated mainly using a light beam that passes through the imaging lens.
First, in a configuration called a “phase difference detection method” (also referred to as TCL: Through Camera Lens), a part of a light beam for imaging on an image sensor such as a photographic film or a CCD (Charge Coupled Device) is extracted and imaged. An optical system is provided that forms a double image in which a shift occurs in proportion to a shift from the focal plane of the image (position of the above-described photographic film or image sensor). Then, the focusing operation of the imaging optical system is performed in the direction in which the double images match to obtain a focused imaging screen, and distance measurement is also possible by knowing the moving position of the optical system involved in the focusing at that time.
The above-described phase difference detection method has high focusing accuracy and has sufficient performance even in a long-focus imaging optical system, and is therefore frequently used in single-lens reflex cameras and early video cameras.
[0007]
Similarly, the “video AF method” can be cited as a method included in the second prior art. In this configuration, a high-frequency component is extracted from the image signal of the subject output using the image sensor, and the focusing optical system performs a focusing operation in a direction in which the high-frequency component increases, thereby obtaining a focused video image. You can also know the subject distance from the position of the optical system involved in focusing. When implemented in a video camera, the configuration related to imaging and the configuration for ranging can be greatly shared, and the configuration is simple and high accuracy is obtained. Therefore, it is frequently used in video cameras and digital cameras.
[0008]
The first prior art and the second prior art described above are mainly used as focusing means of an autofocus camera, and are configured to be able to calculate a subject distance as a secondary.
On the other hand, apart from these, there is a conventional technique for directly calculating the subject distance and the three-dimensional shape of the subject, which will be described below.
[0009]
(3) A configuration in which a subject is imaged from a plurality of viewpoints and three-dimensional shape recognition is performed using parallax (hereinafter referred to as “third conventional technology”).
In this configuration, a camera is arranged at each of a plurality of viewpoints separated by the distance of the baseline length, the same subject is photographed from different angles, and the subject distance and the subject distance based on the triangulation method are used by using the image signal output from each camera. It is intended to obtain the three-dimensional shape data of the subject.
It is not just a means of focusing, it can know the distance of any point in the camera's imaging screen, and it can automatically generate three-dimensional shape data of the subject, so security cameras such as parking lots, crime prevention It is being put to practical use in a wide range of applications such as surveillance cameras, construction site applications that store three-dimensional image data, police cameras, and CG (Computer Graphics) data capture cameras.
[0010]
(4) A configuration in which a subject is photographed from a plurality of viewpoints using a single camera (hereinafter referred to as “fourth prior art”).
In the third prior art described above, at least two cameras are required, but this configuration is intended to exhibit the same effect using only one camera, and Japanese Patent Laid-Open No. 7-181024. It is disclosed in the publication. The fourth prior art will be described below.
[0011]
The configuration of the fourth prior art is shown in FIG.
As shown in the figure, in this configuration, an operator shoots a subject by holding one video camera in his / her hand and swinging it left and right, etc., and using an acceleration sensor or the like provided in the video camera, Output a signal.
Furthermore, an optical flow for detecting an optical flow (a vector distribution indicating how each part in one image has moved to become the other image) for a plurality of images input to the image input unit 120. A detection unit 121, an image region extraction unit 123 that detects an edge region in the image, and a triangulation calculation unit 122 that calculates distance information based on the principle of triangulation using an optical flow are provided.
[0012]
Further, a direct method calculation unit 124 that directly calculates distance information from the constraint formula of the character space gradient method and self-motion information is provided.
That is, the depth map synthesis is performed on the outline of the subject calculated by the triangulation calculation unit 122 based on the principle of triangulation and the distance information obtained by the direct method calculation unit 124 by the direct method near the edge where the distance changes rapidly. The depth map output unit 126 outputs a measurement result synthesized by the unit 125 and improved in shape measurement accuracy as a whole.
[0013]
[Problems to be solved by the invention]
In the above-described conventional technique for measuring the object distance and measuring the three-dimensional shape of the object, the following problems to be solved exist.
[0014]
First, the first prior art and the second prior art described above have been put into practical use as a distance measuring means for a camera having an AF function, and at a specific point of a subject photographed by the camera, for example, at the center of the screen. Since it only has the function of measuring the distance of only the point of the subject that is located, it is not possible to measure the distance of the point of the subject that has a three-dimensional extent in the space at the same time, or recognize the three-dimensional shape of the subject I couldn't.
[0015]
In addition, according to the third prior art described above, it is necessary to shoot a subject by arranging at least two cameras at viewpoints at different positions that are separated from each other, so that the configuration is large, complicated, and expensive. There was a problem.
[0016]
Further, according to the fourth prior art described above, one camera is used, although it has the advantage that it can measure the distance by imaging a subject from a plurality of different viewpoints using only one video camera. Since the images are sent out from the respective viewpoints while moving, the images picked up from different viewpoints are separated on the time axis.For example, even when a difference occurs between the left and right picked-up images, It is difficult to distinguish whether the difference is due to the three-dimensional shape of the subject caused by the difference in viewpoint (parallax) or the difference caused by the subject moving in the space within the camera movement time. It was.
Therefore, the fourth conventional technique has a problem that it cannot accurately measure a subject that changes in time by moving in space.
[0017]
Therefore, the present invention has been made in view of the above-described problems, and in particular, the first image memory or the second image memory is captured from the same viewpoint as the captured image recorded, and immediately before or after. The same as the third image memory output, the third image memory for recording the frame image, the motion vector calculation means for calculating the motion vector that is the difference between the first image memory output and the second image memory output. A time difference vector calculating unit that calculates a time difference vector that is a difference from the captured image of the viewpoint, and a disparity vector that calculates a disparity vector that is a disparity between the first viewpoint and the second viewpoint by subtracting the element of the time difference vector from the motion vector. By comprising a calculation means and a subject distance calculation means for calculating a separation distance between the subject and the image sensor based on a triangulation method using a parallax vector. Realized by a simple configuration performed accurately measure even for temporally with the change subject, and an object thereof is to provide an imaging apparatus for measuring the three-dimensional shape and the subject distance of a subject.
[0018]
[Means for Solving the Problems]
In order to solve the above-described problems, the present invention provides an imaging apparatus having the following configuration.
An imaging optical system (imaging means, lens, CCD) 2 including an imaging element is connected to a first viewpoint (a viewpoint located on the left side of the subject) and a second viewpoint (toward the subject) across the subject (person, moving object). Moving means (optical system driving circuit) 1 for moving between the viewpoints located on the right side),
A first image memory (n-th reference frame memory) 6 for recording a captured image captured from the first viewpoint in units of frames;
A second image memory (n-th reference frame memory) 7 for recording a captured image captured at the second viewpoint in units of frames;
A third image memory (the (n + 1) th reference frame) that is captured from the same viewpoint as the viewpoint of the captured image recorded by the first image memory 6 or the second image memory 7 and that records the immediately preceding or immediately following frame image. Memory) 5,
Motion vector calculation means (motion vector memory) for calculating a motion vector (total movement vector of the subject (image) in the screen) Vto which is a difference between the output of the first image memory 6 and the output of the second image memory 7 (B)) 9 and
Time difference vector calculation means for calculating a time difference vector (movement vector in the screen generated by the movement of the subject in the space) Vm that is a difference between the output of the third image memory 5 and the captured image of the same viewpoint. Motion vector memory (A)) 8;
Disparity vector calculation for calculating a disparity vector (movement vector in the screen generated by the difference in viewpoint) Vpara which is a disparity between the first viewpoint and the second viewpoint by subtracting the element of the time difference vector Vm from the motion vector Vto. Means (motion vector calculation circuit) 10;
An imaging device comprising: an object distance calculating means (distance information data adding circuit) 11 for calculating a separation distance between the object and the imaging device (CCD) based on a triangulation method using the parallax vector Vpara. Device (camera).
[0019]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, preferred examples of embodiments of the present invention will be described with reference to FIGS. 1 to 5.
FIG. 1 is a configuration diagram of a camera according to a first embodiment of the present invention, FIG. 2 is a schematic diagram of a shooting screen including a subject image in the embodiment of the present invention, and FIG. 3 is a second embodiment according to the present invention. FIG. 4 is a block diagram of the camera of the third embodiment according to the present invention, and FIG. 5 is a block diagram of the camera of the fourth embodiment according to the present invention.
[0020]
Prior to the description of the embodiment according to the present invention, the technical focus used by the present embodiment to solve the above-described problems will be described.
FIG. 2 schematically shows the position of the subject in the screen when the subject moving in the space is imaged from different viewpoints at different times. The left and right screens viewed from different viewpoints are shown superimposed on one sheet.
[0021]
In FIG. 2, (1) is an image of a subject taken from a first viewpoint, for example, a viewpoint located on the left side of the subject. (1) 'is an image of a subject taken from a second viewpoint, for example, a viewpoint located on the right side of the subject, and there is a delay on the time axis by one frame with respect to (1). {Circle around (2)} is an image obtained by photographing the subject again from the first viewpoint, and is further delayed by one frame with respect to {circle around (1)}.
[0022]
Here, if the subject moving in the space is photographed from the first viewpoint and the second viewpoint, and the movement of the subject image in the screen is expressed by a vector, the following expression (1) is obtained. .
Vto = Vpara + Vm (1)
Vto: total movement vector of the subject (image) in the screen Vpara: movement vector in the screen caused by the difference in viewpoint Vm: movement vector in the screen caused by movement of the subject in the space
As described above, the imaging apparatus according to the present embodiment extracts only the above Vpara to obtain the three-dimensional data of the subject, and the following formula (2) can be easily obtained from the formula (1).
Vpara = Vto−Vm (2)
[0024]
When the above equation (2) is specifically applied to the movement of one frame time described with reference to FIG. 2, first, the total movement vector in the screen is a change from (1) to (1). From each of them as a vector,
Vto = (1) '-(1) ... Formula (3)
[0025]
Since the subject moved from (1) to (2) in two frame times by moving the subject, assuming that the subject is moving at a constant speed,
Vm = (1/2). ((2)-(1 ▼)) Formula (4)
[0026]
The movement vector based on the difference of viewpoints to be obtained after all is
Vpara = Vto-Vm = (1) '-(1)-(1/2) ((2)-(1))
= (1) '-(1/2) ((1) + (2)) (5)
[0027]
In this embodiment, when calculating the movement vector due to the difference in viewpoints based on the above formula (2) or formula (5), the captured image from the first viewpoint and the second in the time to be calculated are calculated. The movement vector (time difference vector) resulting from the movement of the subject in the space by using the shooting screen from the first viewpoint of the next frame instead of calculating using only the captured image from the viewpoint of The imaging apparatus is configured to obtain the three-dimensional shape data of the subject with higher accuracy than in the past by calculating only the movement vector caused by the difference in viewpoint (parallax) with higher accuracy.
[0028]
FIG. 1 shows the configuration of the image pickup apparatus of the first example according to the embodiment of the present invention.
In FIG. 1, an optical system driving circuit 1 is an actuator that moves the position of an imaging unit 2 at a time interval corresponding to one frame alternately between a first viewpoint and a second viewpoint. The image pickup means 2 includes a photographic lens, a CCD, and peripheral circuits, shoots a subject, and outputs a photographic image in units of frames. The synchronization signal means 3 issues a synchronization signal to the optical system drive circuit 1 and the switch 4 to synchronize the movement of the imaging means 2 and the image output in units of frames. The switch 4 selectively outputs a frame image output from the imaging unit 2 to each frame memory described later in accordance with the synchronization signal.
[0029]
The (n + 1) th reference frame memory 5 records a photographed image photographed from the first viewpoint that is delayed by two frames from the timing to be measured, and outputs it in response to a command.
The nth reference frame memory 6 records a photographed image taken from the first viewpoint at the timing to be measured and outputs it in response to a command.
The nth reference frame memory 7 records a photographed image taken from the second viewpoint at the timing to be measured and outputs it in response to a command. As described above, the frame image recorded by the nth reference frame memory 7 is delayed by one frame on the time axis with respect to the frame image recorded by the nth reference frame memory 6.
[0030]
The motion vector memory (A) 8 uses the frame images recorded by the (n + 1) th reference frame memory 5 and the nth reference frame memory 6 and uses a motion vector (above-mentioned) associated with the movement of the subject in space as viewed from the first viewpoint. Corresponding to Vm) calculated.
In calculating the motion vector, as described above, assuming that the subject moves at a constant speed, one half of the movement vector of the two frame images may be used as the motion vector, or Other calculation methods may be used.
[0031]
The motion vector memory (B) 9 uses the respective frame images recorded in the nth reference frame memory 6 and the nth reference frame memory 7 in the images photographed from the first viewpoint and the second viewpoint. A total movement vector of the subject (corresponding to Vto described above) is calculated.
The motion vector calculation circuit 10 calculates the above equation (5) from the movement vector Vm accompanying the spatial movement of the subject calculated by the motion vector memory (A) 8 and the total movement vector Vto calculated by the motion vector memory (B) 9. A movement vector Vpara accompanying the movement of the viewpoint obtained in 2) is calculated.
[0032]
The calculated movement vector Vpara is a movement vector (disparity vector) generated only by the movement of the viewpoint where the component generated by the movement of the subject in the space is canceled as described above.
[0033]
The distance information data adding circuit 11 calculates distance information of each point of the subject using the movement vector Vpara calculated and output by the motion vector calculation circuit 10. For the calculation, a method based on the triangulation method used in the above-described prior art may be used. Then, the distance information of each point of the subject is added to the image taken from the first viewpoint recorded and output by the nth reference frame memory 6 and output.
[0034]
Next, a second example according to the embodiment of the present invention will be described.
FIG. 3 is a configuration diagram of the imaging apparatus (camera) of the present embodiment.
[0035]
In FIG. 3, the optical system drive circuit 1, the image pickup means 2, the synchronization circuit 3, the switch 4, the (n + 1) th reference frame memory 5, the nth reference frame memory 6, and the nth reference frame memory 7 are the same as those in the first embodiment. Since it is the same structure, description is abbreviate | omitted.
[0036]
The motion vector memory 12 in FIG. 3 is recorded and output by the (n + 1) th reference frame memory 5, and is recorded and output by the nth reference frame memory 7 and a captured image with a delay of 2 frames captured from the first viewpoint. A subject movement vector Vpara generated by a difference in viewpoint is calculated from a captured image with a delay of one frame taken from two viewpoints.
For the calculation, for example, the following calculation method is used.
[0037]
As described above, when the subject image moves on the shooting screen as shown in FIG. 2, the movement vector Vpara generated by the difference in viewpoint can be obtained by the following equation.
Vpara = (1) '-(1/2). ((1) + (2)) (5)
However, (1) ': subject vector in the image photographed from the second viewpoint, (1): subject vector in the image photographed from the first viewpoint, (2): first subject vector after two frames is there.
[0038]
Here, if there is a premise that the movement of the subject in the space is a constant speed, (1) and (2) are not independent, and one can be obtained from the other. For example, if the movement vector (velocity vector) of the subject per unit time is Vk,
(2) = (1) + Vk · t (6)
Holds. However, t is a time corresponding to 2 frames.
[0039]
From Equation (5) and Equation (6),
Vpara = (1) '-(1/2). ((2) -Vk.t + (2)) = (1)'-(2)-(1/2) .Vk.t (7)
The velocity vector Vk can be easily obtained by, for example, obtaining an average velocity of the subject image moving within the screen using a continuous imaging screen from the first viewpoint.
[0040]
The distance information data adding circuit 13 obtains distance information of each point of the subject using Vpraa output from the motion vector memory 12, and adds the distance information to the captured image output from the nth reference frame memory 6.
Since the present embodiment has the above-described configuration, the configuration can be simplified as compared with the first embodiment.
Note that a method other than the above can be considered as the method by which the motion vector memory 12 calculates Vpara.
[0041]
Next, a third example according to the embodiment of the present invention will be described.
FIG. 4 is a configuration diagram of the camera according to the present embodiment. Since the configuration not shown is the same as that of the second embodiment described above with reference to FIG. 3, the illustration is omitted.
[0042]
In this embodiment, the determination circuit 14 and the motion vector setting means 15 are newly provided to cope with the case where the motion of the subject is fast and the motion vector cannot be calculated.
The determination circuit 14 determines whether or not Vpara output from the motion vector memory 12 is greater than or equal to a predetermined limit value. If the Vpara is lower than the limit value, the distance information is determined using Vpara as in the second embodiment. When the data addition circuit 13 calculates and adds distance information, and if it is equal to or greater than the limit value, the motion vector setting means 15 sets Vpara to a predetermined maximum value, and Vpara is erroneously calculated Even when the subject is hidden behind a shadow and is not reflected on the screen, erroneous distance information is prevented from being calculated from the meaningless Vpara value.
[0043]
Next, a fourth example according to the embodiment of the present invention will be described.
FIG. 5 is a diagram showing the configuration of the image pickup means in the camera of this embodiment, and the other configurations are the same as those of the above embodiment.
[0044]
In the first to third embodiments, the image pickup means is provided with one set of an optical system and an image pickup element, and the position is alternately moved by an actuator.
On the other hand, in this embodiment, two optical systems and one image sensor are used, and an optical path switching shutter that selectively switches light beams emitted from the two optical systems is used.
Compared to a configuration in which a set of optical systems is moved, the moving weight is small, and the configuration is simplified.
[0045]
【The invention's effect】
As described above in detail, the present invention is a third image memory that is captured at the same viewpoint as the viewpoint related to the captured image recorded in the first image memory or the second image memory and records the immediately preceding or immediately following frame image. A motion vector calculating means for calculating a motion vector that is a difference between the first image memory output and the second image memory output, and a time difference vector that is a difference between the third image memory output and the captured image of the same viewpoint Using the disparity vector, a disparity vector calculating means for calculating a disparity vector that is a disparity between the first viewpoint and the second viewpoint by subtracting the element of the time difference vector from the motion vector, and a disparity vector. And subject distance calculation means for calculating the distance between the image sensor and the image sensor based on the triangulation method. Well realized by a simple configuration performs measurement, those that result the effect is to provide an imaging apparatus for measuring the three-dimensional shape and the subject distance of a subject.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of a camera according to a first embodiment of the present invention.
FIG. 2 is a schematic diagram of a photographing screen including a subject image in the embodiment of the present invention.
FIG. 3 is a configuration diagram of a camera according to a second embodiment of the present invention.
FIG. 4 is a configuration diagram of a camera according to a third embodiment of the present invention.
FIG. 5 is a configuration diagram of a camera according to a fourth embodiment of the present invention.
FIG. 6 is a configuration diagram of a conventional imaging apparatus.
[Explanation of symbols]
1 Optical system drive circuit (moving means)
2 Imaging means, lens, CCD (imaging optical system with imaging element)
5 n + 1 reference frame memory (third image memory)
6 nth reference frame memory (first image memory)
7 nth reference frame memory (second image memory)
8 motion vector memory (A) (time difference vector calculation means)
9 Motion vector memory (B) (motion vector calculation means)
10 Motion vector calculation circuit (disparity vector calculation means)
11 Distance information data addition circuit (subject distance calculation means)
Vm Movement vector in the screen (time difference vector) caused by the movement of the subject in the space
Vpara In-screen movement vector (disparity vector) caused by differences in viewpoint
Vto Total movement vector (motion vector) of the subject (image) in the screen

Claims (1)

Moving means for moving an imaging optical system including an imaging element between a first viewpoint and a second viewpoint sandwiching the subject;
A first image memory that records a captured image captured from the first viewpoint in units of frames;
A second image memory for recording a captured image captured from the second viewpoint in units of frames;
A third image memory that is imaged at the same viewpoint as the viewpoint of the captured image recorded by the first image memory or the second image memory, and that records the immediately preceding or immediately following frame image;
Motion vector calculating means for calculating a motion vector that is a difference between the first image memory output and the second image memory output;
Time difference vector calculation means for calculating a time difference vector which is a difference between the third image memory output and the captured image of the same viewpoint;
A disparity vector calculating means for calculating a disparity vector that is a disparity between the first viewpoint and the second viewpoint by subtracting an element of the time difference vector from the motion vector;
An imaging apparatus comprising: a subject distance calculation unit that calculates a separation distance between a subject and the imaging element based on a triangulation method using the parallax vector.

Images