JP2611173B2 - Positioning method and device using fisheye lens - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a positioning method and apparatus for obtaining three-dimensional positioning information of a plurality of moving targets using a fisheye lens.

[0002]

2. Description of the Related Art In a positioning device based on current image information, a target is usually tracked by a camera having two telephoto lenses. In this method, the target is obtained at the center of the field of view of each camera. The direction of the camera (the direction of the optical axis of the lens) is adjusted so that the position of the target is calculated by triangulation from the installation positions of the two cameras and the angle information of the optical axes. Therefore, an adjusting device including a servo system for accurately pointing the two cameras to the target and an angle measuring device for accurately measuring the optical axis direction of the cameras are required.

[0003]

As described above, when tracking a target with an ordinary camera, high response is required for both the servo system and the angle measurement system. However, when the moving speed of the target is high, it is very difficult to perform positioning while simultaneously tracking with the current technology. Moreover,
In such a system, one device can track only one target, and tracking two or more targets requires the same number of devices as the system as a system. . Thus, a single device cannot track a plurality of targets at the same time.

[0004]

Means for Solving the Problems In the first invention, a system coordinate system is arbitrarily provided, and optical axes are individually determined for a plurality of fisheye lenses which capture a moving target existing in the system coordinate system. From each fish-eye image provided by a plurality of imaging devices using a fish-eye lens, a moving target existing on the fish-eye image is detected, and the moving target and the visual interface center of the imaging device in the system coordinate system are detected. The number of straight line equations that pass is determined by the number corresponding to the number of imaging devices, and a group of straight line equations, which is a set of the plurality of linear equations, is determined for each detected moving target, and each linear target corresponds to each moving target. The intersection of the linear equation group is determined for each moving target, and the three-dimensional position coordinates of each moving target in the system coordinate system are determined.
Further, in the second invention, the position data of each moving target object with respect to the time is stored in a memory together with the measurement time, and the position data stored in the memory is subjected to tracking processing for each moving target object. The trajectory of a moving target is obtained. Further, the third invention is:
A plurality of fish-eye lenses for capturing a moving target, an imaging device for capturing a fish-eye image obtained by the fish-eye lens, and an image processing unit for performing image processing on the fish-eye image captured by the imaging device to detect the moving target. A set of straight-line equations passing through the moving target and the center of the visual interface of the imaging device corresponding to the number of imaging devices based on the optical axis of each fish-eye lens and the system coordinate system; Calculating for each moving target, calculating the intersection of the linear equation group corresponding to each moving target for each moving target, and calculating the three-dimensional position coordinates of the moving target. A memory for storing the three-dimensional position coordinates of each moving target calculated by the arithmetic unit, and data for displaying the three-dimensional position coordinates of each moving target stored in this memory in three dimensions on a monitor screen Processing equipment , It is made of a display device for displaying a three-dimensional position coordinates of the moving target. Further, in the fourth invention, each of the moving targets stored in the memory is stored in a memory.
A tracking processing unit that performs tracking processing of the three-dimensional position coordinates to determine the trajectory of each moving target; a data processing device that displays the trajectory of each moving target three-dimensionally on a monitor screen; And a display device for displaying the wake.

[0005]

With respect to a moving target object (hereinafter, simply referred to as a target object) captured in a fisheye image, a set of linear equations corresponding to the number of imaging devices using a fisheye lens is defined as a set of linear equations. In addition to obtaining the number corresponding to the object, the intersection of this group of linear equations is determined for each target, and
The position information of the target is obtained by obtaining the dimensional position coordinates.

[0006]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to FIGS. FIG. 1 is a configuration diagram of a main part of the present invention
FIG. 2 is a fisheye image with a fisheye lens having a viewing angle of 180 °, FIG. 3 is a fisheye image with three fisheye lenses and positioning information, FIG.
Is an explanatory diagram for positioning, and FIGS. 5 to 7 are explanatory diagrams showing examples of the arrangement of fisheye lenses. 1 to 3, reference numeral 1 denotes a fish-eye lens, which is combined with a photoelectric conversion element 2 such as a CCD to constitute an imaging device 3. In this embodiment, three imaging devices 3a, 3b, and 3c are used. ing. By installing the imaging device 3 using the fisheye lens 1, FIG.
As shown in FIG. 3, the optical axis L (L _a , L _b , L _c ) is determined. Also, the system coordinate system (X, Y, Z) of the system
Is arbitrarily determined when the fisheye lens 1 is installed. Therefore, three image pickup device 3 _a, 3 _b, 3 respectively captured by the _c fisheye image 4 _a, 4 _b, 4 system coordinates _c, as shown in FIG. 3, respectively (X _a, Y _a , Z _a ), (X
_{b, Y b, Z b)} , is given by _{(X c, Y c, Z} c).

Reference numeral 5 denotes an interface for digitally converting the fisheye image 4 picked up by the image pickup device 3 for image processing. Reference numeral 6 denotes an image processing unit, which detects moving targets T ₁ , T ₂ , T ₃ ... In the fish-eye image 4 by image processing. Reference numeral 7 denotes a calculation unit, which is a system coordinate system (X, Y,
Z) Three linear equations are obtained from the optical axis L of the fisheye lens 1 above. For example, the target _{T 1} and the imaging devices ₃ a,
3 _b, 3 3 straight lines equation passing through the field of view plane centers of the _c are obtained, respectively. That is, a set of linear equations in which one set of linear equations corresponding to the number of fish-eye lenses 1 (in this embodiment, one set of three linear equations) is used for each target T ₁ ,
Are obtained for T ₂ , T _3, ..., Respectively, and three-dimensional position coordinates are calculated from the intersections of the linear equation groups. 8
Is a memory, and 9 is a tracking processing unit, which sequentially performs tracking processing on the fisheye image 4 to obtain target objects T ₁ , T ₂ , T ₃ ,.
Wake is determined. Reference numeral 10 denotes a data processing device for outputting an image, which converts a track of the target object T ₁ , T ₂ , T ₃ ,. Data processing is performed. A recording unit 12 temporarily stores the fisheye image 4 captured by the imaging device 2.

Next, the principle of positioning will be described. As shown in FIG. 2, the viewing angle of the fisheye lens 1 is generally 180 °.
And the optical axis L is a perpendicular line passing through the center of the image plane, and the linear equation in this system coordinate system is determined by the installation location of the fisheye lens 1 (showing the imaging device 3 using the fisheye lens 1). The fish-eye image 4 formed by the fish-eye lens 1 having a viewing angle of 180 ° is circular, and the center O of the fish-eye image 4 when viewing the targets T ₁ , T ₂ , T ₃ ,. Is directly in front of the fisheye lens 1 (optical axis L
Direction, and points A and B in the left-right direction indicate points at 90 ° in the left-right direction of the fisheye lens 1, and points C and D in the up-down direction indicate points directly above and below the fisheye lens 1, respectively. The system coordinate system (X, Y, Z) is arbitrarily determined when the fisheye lens 1 is installed. In addition, the target objects T ₁ , T ₂ ,
T ₃ when viewed ..., the shape of the target _{T 1, T 2, T 3} ··· are distorted along the circumference of the fish-eye image 4 with distance from the center O, the target T _1, T ₂ , T ₃ ... Are difficult to recognize by their shapes. The present inventors, as a target T _1, T _2, T ₃ recognizes ... means such as an aircraft, the target T ₁ to point (strobe) to flashing tail of the aircraft, T _2, T ₃ ... The target T ₁ ,
If T ₂ , T ₃ ... Are point-like, the target T
Is not distorted even in the circumferential direction of the fisheye image 4.

[0009] Therefore, a target _{T 1, T 2, T 3} ··· shape and determines a point (spot), in FIG. 3, the target T _1, T ₂ as a point-shaped target 2 Is set to exist. Thus, a plurality of imaging devices 3 using the fisheye lens 1 respectively (3a, 3b, 3c ···) gives fisheye image 4 (4a, 4b, 4c) in the two target T _1, T ₂ is The positioning information that exists and is common to the target objects T ₁ and T ₂ includes the installation coordinates (X) of the three imaging devices 3a, 3b, and 3c, respectively, depending on the installation location of the imaging device 3.
_{a, Y a, Z a)} , (X b, Y b, Z b), (X c, Y
_c, Z _c) and three optical axis _{L a (θ a, ψ a} ),
L _b (θ _b , ψ _b ) and L _c (θ _c , ψ _c ) are respectively determined.

[0010] Then, the positioning information regarding target T _1, the target T ₁ of the optical axis _{L a (θ a, ψ a} ) when viewed from the first image pickup device 3a displacement angle from (theta _{a1, ψ a1)} and the second target T ₁ of the when viewed from the imaging device 3b in the optical axis direction _{L b (θ b, ψ b} ) displacement angle from (θ _b1, ψ _b1) and third imaging The displacement angle (θ _c1 , ψ _c1 ) from the optical axis direction L _c (θ _c , ψ _c ) of the target T ₁ as viewed from the device 3c is obtained. Similarly, the positioning information regarding target T _2,
Target T ₂ of the optical axis _{L a (θ a, ψ a} ) when viewed from the first image pickup device 3a displacement angle from (θ _a2, ψ _a2) and second
The optical axis direction L _b of the target T ₂ as viewed from the imaging device 3b of FIG.
(Θ _b, ψ _b) displacement angle from (θ _b2, ψ _b2) and third
The optical axis direction L _c of the target T ₂ as viewed from the imaging device 3c of FIG.
(Θ _c, ψ _c) displacement angle from (θ _c2, ψ _c2) is obtained.

As described above, in the system coordinate system, the linear equation of the optical axis L of the fisheye lens 1 (depending on the installation position and the optical axis direction L) and the target T identified on the fisheye image 4
_1, the position data of T ₂ - from data for fisheye image 4 obtained by one of the fisheye lens 1, all of the target T _1, T ₂ · · · and the imaging device 3 on the fisheye image 4 Is given by a linear equation in a three-dimensional space connecting. Therefore, FIG.
As shown in FIG.
When two targets T ₁ , T ₂ are present on three fish-eye images 4 (4 _a , 4 _b , 4 _c ) respectively obtained by the three imaging devices 3 (3a, 3b, 3c), The target T (T ₁ ,
If the position of T ₂ ) is read from the image information, three linear equations f _1a , f _1b , connecting one target T ₁ and the visual interface centers of the three imaging devices 3a, 3b, 3c respectively.
f _1c is obtained. Similarly, the target T _2, target T ₂ and three image pickup device 3 _a, 3 _b, 3 _c visibility surface center and the three straight lines equation f _2a connecting respectively, f _2b, f _2c is obtained Can be As described above, a set of linear equations having a set of linear equations corresponding to the number of the imaging devices 3 corresponds to each target T ₁ ,
Every T _2, that is, the two sets are obtained. Since the target T is located at the intersection where the number of straight lines corresponding to the number of the imaging devices 3 given by the set of straight line equations obtained in this way intersects one place, if this intersection is calculated , This point gives the position coordinates of the target T in the three-dimensional space.

Next, the operation of the positioning device for actually obtaining the positioning information of the target T based on the positioning principle will be described with reference to FIGS. First, in the case where there are a plurality of targets T, each target T must be captured in the field of view of at least three imaging devices 3 in order to completely eliminate false images. Therefore, each of the three imaging devices 3 (3a, 3b, 3c) has
When positioning the target T in the 0 ° field of view, FIG.
As shown in FIG. 2, in this embodiment, three imaging devices 3 (3
a, 3b, 3c) are respectively on the same plane with the optical axis L (L
_a , _Lb , _Lc ) are set in parallel. However, the angle resolution of the fisheye lens 1 is maximum in the optical axis direction L, and decreases as the distance from the direction increases. In addition, when the target T is on a straight line connecting the two imaging devices 3 on the installation plane of the imaging device 3, counterfeiting may occur.

Therefore, when positioning the target T, it is necessary to optimize the arrangement of the image pickup device 3 in accordance with the purpose such as which area is to be positioned with high accuracy. For example,
When a viewing angle of 180 ° is not required, as shown in FIG. 6, the three imaging devices 3 are installed such that the optical axes L _a , L _b , and L _c intersect at the front, so that the second imaging device 3 can be used. It is possible to reduce the positioning accuracy in front of the imaging device 3b, and to improve the positioning accuracy in the left-right direction of a wider front area than the case shown in FIG. Also, as shown in FIG. 7, if a total of five imaging devices 3b, 3c, 3d, 3e are arranged three-dimensionally in the vertical and horizontal directions with the imaging device 3a as the center, the vertical direction is also wider. The positioning accuracy for the area can be improved. Further, when a viewing angle of 180 ° or more is required, the imaging device 3 may be replaced with a regular hexahedron or regular 12
It is possible to improve the positioning accuracy by arranging it three-dimensionally like a planar body, and also by using a plurality of image pickup devices combined in a similar manner. That is, the positioning area is determined based on the positional relationship and the number of the fisheye lenses 1 (imaging device 3).

If the field-of-view image produced by the fisheye lens 1 is sufficiently large, the photoelectric conversion element 2 (hereinafter referred to as CCD 2)
It is possible to use 400,000 elements or more than one million elements as the number of elements of the CCD 2. The number of elements of the CCD 2 is increased to improve the resolution, and the number of imaging devices used is reduced to improve the positioning accuracy. It is also possible. The requirements as described above are examined, and the number and arrangement of the imaging devices 3 using the fisheye lens 1 are determined from the positioning viewing angle, the positioning area, the positioning accuracy, and the like. When the arrangement of the imaging devices 3 is determined, the optical axis L of each fisheye lens 1 is determined. That is, the normal direction of the fisheye lens 1 is the optical axis L. At the same time, a description is made as a three-dimensional linear equation in the system coordinate system.

Next, three imaging devices 3 (3a, 3b, 3
From c), three fisheye images 4 ( _4a , _4b , _4c ) are obtained as shown in FIG. This fisheye image 4
Are photoelectrically converted by the CCD 2 into video signals. When it is not necessary to obtain the positioning information of the target T in real time, for example, when tracking a wake or the like, this video signal is temporarily stored in the recording unit 12 and is later processed by the image processing unit 6. . The video signal of each fish-eye image 4 in the CCD 2 is converted into a digital signal via the interface 5 and input to the image processing unit 6. The image processing unit 6 performs image processing for detecting the target T, detects the target T captured in each fish-eye image 4, and shifts the target T with respect to the optical axis L. The number of vertical angles and the number of horizontal angles are detected, and the image information is
(X _a1 , Y _a1 ), (X _a2 , Y _a2 ) ... (X _b1 , Y
_b1 ), ( _Xb2 , _Yb2 ) ..., ( _Xc1 , _Yc1 ), (X
_c2 , Y _c2 ).

The arithmetic unit 7 determines the equation of the optical axis L based on the system coordinate system, and passes through the optical axis L to indicate the center of the visual interface of the imaging device 3 of the fisheye lens 1. A straight line that passes through the point O (X ₀ , Y ₀ , Z ₀ ) and intersects with the target T is obtained. Equation of this straight _{line, (x-x 0a) /} α a = (y-y 0a) / β a = (z-
z _0a ) / γ _a (xx _0b ) / α _b = (y−y _0b ) / β _b = (z−
z _0b ) / γ _b (xx _0c ) / α _c = (y−y _0c ) / β _c = (z−
As many linear equation groups as one set represented by z _0c ) / γ _c and three linear equations are obtained for the number of targets T. In this way, the linear equation group of a set of linear equations of the number corresponding to the number of fisheye 1 is, are respectively calculated two sets of the target _{T (T 1, T 2)} . Next, the intersection of the two sets of straight-line equations is calculated by the calculation unit 7 to obtain a list of intersections. Only three points having the same output (points having the same number of outputs corresponding to the number of fisheye lenses 1) are extracted from the list of the intersections, and these points are defined as three linear equations, that is, a group of linear equations. The intersection represents the position data of the target T. When the number of fisheye lenses 1 is three, a false image is generated as an intersection of two linear equations,
It does not occur at the intersection of the three linear equations. Therefore, as the number of fisheye lenses 1 increases, the probability of occurrence of a false image decreases.

In this way, the position of the intersection of the linear equations of the number corresponding to the number of fisheye lenses 1 constituting the group of linear equations at one point also corresponds to the system coordinate system. Are calculated as the three-dimensional spatial coordinates (X _n , Y _n , Z _n , t _m ) of the target at
This value indicates three-dimensional position coordinates. Here, t indicates time. That is, when the number of the targets T is N,
N sets of linear equation groups each including three linear equations obtained from the three imaging devices 3 are obtained, and the N sets of linear equation groups intersect each other at a maximum of (3N−1). )!
The intersections are connected at points, and among these, the intersections at which three straight lines intersect are only N points corresponding to the number of the targets T. For example, if the number of targets T to be located is ten,
The linear equation obtained from the three imaging devices 3 is 29! The intersections of the points are connected. Of these, the intersections where the three linear equations intersect (the target T is the intersection) are only 10 points corresponding to the number of the targets T. Are three-dimensional position coordinates at times t ₁ , t ₂ , t _3.
, And the three-dimensional position coordinates are stored in the memory 8.

The data stored in the memory 8 is stored in a tracking processor 9 at times t ₁ , t ₂ , t _3.
Are tracked, that is, the trajectory of each coordinate is obtained, and the wake of the target T is obtained in the image viewed from above. Since this wake is coordinate data at consecutive times of three-dimensional positioning, it can be visualized as a three-dimensional trajectory with respect to an arbitrary spatial coordinate system. Processing for three-dimensional display on the monitor screen is performed, converted into a video signal, displayed on the display device 11 as a continuous image, and subjected to knowledge processing by a controller or the like.

[0019]

According to the present invention, a system coordinate system is arbitrarily provided, and optical axes are individually determined for a plurality of fish-eye lenses that capture a moving target existing in the system coordinate system, and a plurality of imaging using each fish-eye lens is performed. From each fish-eye image provided by the device, a moving target existing on the fish-eye image is detected, and a linear equation passing through the moving target in the system coordinate system and the center of the visual interface of the imaging device is calculated. Are obtained for each of the detected moving targets, and the intersections of the linear equations corresponding to each of the moving targets are determined by: Since three-dimensional position coordinates of each moving target in the system coordinate system are determined for each moving target, a plurality of moving targets can be simultaneously obtained by one system. Kuraishi, it is possible to determine the coordinates can be tracked. Further, the position data of each moving target with respect to the time is stored in a memory together with the measured time, and the position data stored in the memory is subjected to tracking processing for each moving target, and the wake of each moving target is tracked. Since it is determined, the wake of the moving target can be analyzed.

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing an embodiment of the present invention.

FIG. 2 is a fisheye image provided by a fisheye lens 1;

3 shows an embodiment of the present invention and is a fisheye image provided by three fisheye lenses 1. FIG.

FIG. 4 is an explanatory diagram showing an embodiment of the present invention.

FIG. 5, showing an embodiment of the present invention, is a diagram illustrating an example of arrangement of imaging devices.

FIG. 6, showing an embodiment of the present invention, is a diagram illustrating an example of arrangement of imaging devices.

FIG. 7 illustrates an embodiment of the present invention and is a diagram illustrating an example of arrangement of imaging devices.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Fish-eye lens 3 ... Imaging device 4 ... Fish-eye image 5 ... Interface 6 ... Image processing part 7 ... Operation part 8 ... Memory 9 ... Tracking processing part 10. · Data processing device 11 · · · Display device 12 · · · Recording unit T · · · moving target

Claims (5)

(57) [Claims] 1. A system coordinate system is arbitrarily provided, and optical axes are individually determined for a plurality of fisheye lenses that capture a moving target existing in the system coordinate system, and provided by a plurality of imaging devices using the plurality of fisheye lenses. The moving target object present on the fisheye image is detected from each of the fisheye images obtained, and a linear equation passing through the moving target object in the system coordinate system and the center of the visual interface of the imaging device is determined by the imaging device. Are obtained for each of the detected moving targets, and intersections of the linear equation groups respectively corresponding to the moving targets are obtained. Is determined for each moving target, and the three-dimensional position coordinates of each moving target in the system coordinate system are determined. Positioning methods. 2. The position data of each of the targets with respect to the time is stored in a memory together with the measured time, and the position data stored in the memory is subjected to tracking processing for each of the moving targets, and the movement of each of the targets is performed. The positioning method using a fisheye lens according to claim 1, wherein a wake of the target is obtained. 3. A plurality of fish-eye lenses for capturing a target, an imaging device for capturing a fish-eye image obtained by the fish-eye lens, and a moving target detected by processing the fish-eye image captured by the imaging device. An image processing unit that calculates the number of linear equations that pass through the moving target and the center of the visual interface of the imaging device by the number corresponding to the number of imaging devices, based on the optical axis of each fish-eye lens and the system coordinate system. Is calculated for each of the moving targets, and the intersection of the linear equation group corresponding to each of the moving targets is calculated for each of the moving targets. An operation unit for calculating the three-dimensional position coordinates of the object; a memory for storing the three-dimensional position coordinates of each of the moving objects calculated by the operation unit; Dimension Positioning device using a fisheye lens, characterized in that it consists of the coordinates, and a data processing device for displaying three-dimensional on the monitor screen, and a display device for displaying a three-dimensional position coordinates of the moving target. 4. A tracking processing unit that performs tracking processing on the three-dimensional position coordinates of each of the moving targets stored in the memory to determine a wake of each of the moving targets, and a wake of each of the moving targets. The positioning device using a fisheye lens according to claim 3, comprising: a data processing device for displaying three-dimensionally on a monitor screen; and a display device for displaying a track of the moving target. 5. The positioning device using a fish-eye lens according to claim 3, wherein at least three fish-eye lenses are used.