JPH07114642A - Measuring instrument for mobile object - Google Patents

Abstract

PURPOSE:To find a coordinate conversion coefficient, etc., while actually a photograph by calculating three-dimensional spatial coordinates of an object on the basis of the distance of a vector to a restraint plane where a mobile object is present and the angle of rotation of a camera. CONSTITUTION:A controller 14 is equipped with a position arithmetic unit 16 which calculates the three-dimensional spatial coordinates of the object, and a driving control unit 17 which obtains information on the angle of rotation, zooming, focusing, a stop, etc., from a zoom lens 12 and a motor-driven universal head 13 and supplies the information to the position arithmetic unit 16. In this case the position of the object on a screen is obtained from an image position measuring instrument and the position is converted into a position on a projection surface (VT surface) as a plane in the space of the image position measuring instrument. Then the vector from the center of the projection to the center of rotation of the universal head 13 is regarded as an aimed vector. The vector is rotated around the center of rotation of the universal head 13 by the tilt of the universal head 13. The point where the straight line obtained by extending the rotated aimed vector and the restraint plane cross each other is the position of the object in the three-dimensional space.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is used in an apparatus for picking up an image of a moving object and measuring the picked up image. INDUSTRIAL APPLICABILITY The present invention can be used when a plurality of moving image videos are used in combination, such as a case where a video is used for measurement or a video is edited for use in broadcasting.

The present invention is particularly suitable for use in a device for tracking and imaging a moving athlete such as a sports competition and extracting the three-dimensional spatial coordinate information thereof.

[0003]

2. Description of the Related Art As a method of measuring a moving object, a three-dimensional spatial coordinate of the moving object is obtained by imaging the moving object and processing a video signal thereof.

The Applicant, as a device for measuring such a moving object, superimposes the information obtained from the camera platform or the image pickup device as position information on the video signal obtained by the image pickup device mounted on the camera platform. The technology is Japanese Patent Application No. 5-082178,
Japanese Patent Application No. 05-082209 describes a technique for calculating the three-dimensional spatial coordinates of a moving object from a signal obtained by capturing an image of the moving object, and Japanese Patent Application No. 5-20209 describes a technique for predicting or measuring the movement of the moving object and controlling the camera device. 138805, Japanese Patent Application No. 5-139629 for a technique for superimposing the obtained three-dimensional spatial coordinate information of a moving object on video data, and Japanese Patent Application No. 5-137857 for a technique for identifying a moving object from a video signal. Sometimes unpublished).

[0005]

The techniques proposed by the applicant of the present invention have various points to be improved.

First, in order to calculate the three-dimensional space coordinates of the target object from the picked-up image, the information such as the rotation angle and the position of the image pickup surface obtained from the camera device or the platform is converted into the three-dimensional real space coordinates. Various parameters such as the coordinate conversion coefficient are necessary for this. However, said application does not mention how to determine these parameters. Further, it is not preferable to use design values for these parameters because there is a manufacturing error for each camera device or platform. Furthermore, these parameters change due to aging and temperature changes. Therefore, it is necessary to specifically obtain these parameters without disassembling the device or the like in the actual imaging field.

Accurately obtaining these parameters is to increase the measurement accuracy of a moving object, and is necessary to increase the accuracy of automatic tracking control when a moving object is automatically tracked and imaged. Therefore, it is necessary to obtain different parameters for each device at the actual shooting location.

An object of the present invention is to improve the above-mentioned prior application, and to provide a measuring device capable of obtaining highly accurate three-dimensional spatial coordinate information.

Another object of the present invention is to provide an apparatus capable of obtaining a coordinate conversion coefficient or the like used for calculation of a three-dimensional spatial coordinate of a moving object while actually photographing.

Another object of the present invention is to enable tracking control of a moving object with high accuracy.

[0011]

The present invention includes an image pickup means for picking up an image of a moving object, a drive means for setting an image pickup direction and an image pickup range of the image pickup means, and a video signal obtained by the image pickup means. The image processing means for calculating the position of the moving object in the image, and the position of the moving object in the image and the information about the image capturing direction and the image capturing range of the image capturing means when the video signal is obtained are moved. In a measuring device for a moving object, which comprises a position calculating means for calculating coordinates of an object in real space, the position calculating means sets a projection center of a lens of a camera of the image pickup means, and the projection center is point-symmetrical. Means for converting the object of the moving object imaged as a position on the projection surface at a position point-symmetrical to the image sensor surface, and up to the object of the image converted by this converting means. And a means for calculating a three-dimensional spatial coordinate of the target object based on the distance from the projection center to the constraining plane where the moving object exists and the rotation angle of the camera. .

Further, the position of the projection center of the image which is the center position of the lens of the camera of the image pickup means and the moving object 3
A means for calculating a transformation variable necessary for computing the three-dimensional space coordinates as a temporary value, capturing a reference point fixed on the constraining plane and having a known spatial position coordinate, and computing the spatial position coordinate; Means for determining the projection center and the conversion variable by imaging the reference point again while gradually changing the projection center of the camera and the conversion variable based on the result.

Further, there is provided means for operating the drive means of the image pickup means for a plurality of operating points and obtaining the drive amount of the drive means as an electric signal, and the position computing means is provided at the plurality of obtained operating points. 3D space of the target moving object by obtaining the relational expression between the projection center position and the drive amount based on the relations at the plurality of operating points. Means for computing the coordinates may be included.

Further, it comprises a movement predicting means for predicting the moving direction of the moving object based on the obtained coordinates in the real space of the moving object, and a drive control means for controlling the driving means by the predicting means. The movement predicting means includes means for predicting the position of the moving object a plurality of times at a time interval shorter than the position calculation interval of the position calculating means, and the drive control means is shorter than the calculation interval of the position calculating means. And a means for providing the drive means with a control target value based on the predicted position value.

Further, the image pickup means may include a color camera, and the image processing means may include color extracting means for separating the moving object to be measured from the background by color in the picked-up image.

Further, it comprises a movement predicting means for predicting a moving direction of the moving object based on the obtained coordinates in the real space of the moving object, and a drive control means for controlling the driving means by the predicting means. The drive control means obtains in advance illuminance distribution or white balance distribution information in the image pickup range of the image pickup means, and controls the drive means based on this information to adjust the aperture or white balance of the camera device of the image pickup means. Means may be included.

[0017]

In the present invention, not only the image but also the real space 3
The dimensional space position coordinates can be measured together with the image of the target object, and can be recorded or output in synchronization with the image.
The measurement of the three-dimensional spatial position coordinates is performed on the three-dimensional coordinates of the target object itself, the three-dimensional coordinates of the imaged range, or the representative point such as the center of the imaged range.

The calculation of the three-dimensional spatial position coordinates of the target object is
Consider the focal plane of the lens of the video camera device as the projection center O, and the image pickup surface of the CCD and the projection plane VT that is symmetrical about this projection center O. On the projection surface VT, an imaged image of the subject, which is the reverse of the image obtained on the imaging surface of the CCD, is obtained. A belt reel from the projection center O for the subject on the projection surface VT is set. Then, based on the belt from the projection center O, the three-dimensional space coordinates of the subject are calculated from the distance and the angle to the constraining plane where the subject exists.

Further, when calculating the three-dimensional space coordinates of the subject, various parameters such as necessary coordinate conversion coefficients are actually on the constraining plane, and reference points whose three-dimensional space coordinates are known in advance are respectively set. Parameters such as camera device and zoom are determined by imaging while zooming or moving the platform of the device.

Further, in controlling the pan / tilt head and the zoom lens, the number of times of imaging by the camera device depends on the imaging method. Therefore, based on the position information of the target object obtained from the picked-up image, the pan / tilt head or When the zoom lens is controlled, it may not be possible to follow the movement of the target object.Therefore, the moving position of the target object is predicted multiple times during the imaging time interval, and the control target value is driven based on the predicted value. The automatic tracking control of the image pickup means can be smoothly performed by providing the image pickup means.

Further, the target object is extracted from the background by performing color identification by the color camera device. Also,
A map of the illuminance distribution or white balance within the movable range of the moving object is created in advance, and the aperture of the camera device is adjusted.

[0022]

Embodiments of the present invention will be described below with reference to the drawings.

The following embodiment will be described as an embodiment in which the movement of the player to be measured has less movement in the height direction than in the horizontal direction and can be regarded as a plane.

(First Embodiment) The following is an embodiment in the case where the movement of the target object (player) is less in the height direction than in the horizontal direction and can be regarded as a plane as in the swimming competition.

FIG. 1 is a block diagram of the whole moving object measuring apparatus and editing apparatus. The whole includes a camera head 1, a data recording section 2, a moving image editing section 3, and a time base unit 4 and a stopwatch 5 for time management.

In photographing, as shown in FIG. 2, the three-dimensional position of the target object 7 is measured by the camera head 1 simultaneously with the image. Further, the camera head 1 automatically moves according to the movement of the target object 7 and tracks the target object.

Since the movement of the target object 7 may be limited to the plane 6, if the position of the camera head 1 with respect to the plane 6 and the direction when the target object 7 is viewed from the camera head 1 are known, the plane will be a plane. The position of the target object in three dimensions can be found by finding the intersection of the straight lines. At the same time, the photographed range of the plane 6 can be calculated from the angle of view of the lens.

FIG. 3 is a conceptual diagram of the entire structure of the camera head 1, which includes a video camera 11, a zoom lens 12, the video camera 11 and the zoom lens 12.
It is equipped with an electric pan head 13 for mounting. This camera head 1
Corresponds to the camera head / arithmetic unit shown in Japanese Patent Application No. 5-138805. A head controller 14 that controls the electric platform 13 and the zoom lens 12 includes a position calculation device 16 that calculates three-dimensional spatial coordinates of a target object,
The drive control is performed for the zoom lens 12 and the electric pan head 13, and the drive information is obtained from the zoom lens 12 and the electric pan head 13 such as the rotation angle, zoom, focus, and diaphragm amount, and the position calculation device 16 is provided with the information. The controller 17 is provided. Further, the image captured by the video camera 11 is output to the data recording unit as a moving image, and an image processing device 15 that performs image processing for extracting a target object to be measured is provided. A plurality of camera heads 1 are provided as shown in FIG.

FIG. 4 shows the configuration of the camera head 1 shown in FIG. 3 in more detail. Video camera 11
Is a color video camera, the output of which is output as a moving image to the data recording unit 2 or the moving image editing unit 3. Further, as a device corresponding to the image processing device 15, a learning type color extraction device 21 for extracting from the background of the object of the subject.
And an image position measuring device 22 for calculating the position of the subject on the screen. The learning type color extraction device 21 is connected to a confirmation TV monitor 36, and the image position measurement device 22 is connected to an operation TV monitor 37. The output of the image position measuring device 22 is input to the spatial coordinate calculating circuit 26 which calculates the spatial coordinates of the object in the position calculating device 16.

The zoom lens 12 includes an iris driving motor 121 and an iris position detecting potentiometer 12.
2, focus drive motor 123, focus position detection potentiometer 124, zoom drive motor 12
5. The potentiometer 126 for zoom position detection is provided. Further, the electric platform 13 is a motor 13 for driving the platform horizontal axis.
1, a pan head horizontal axis position detecting potentiometer 132, a pan head vertical axis driving motor 133, and a pan head vertical axis position detecting potentiometer 134.

In this embodiment, the position control servo circuit 201
The servo control circuit 20 is provided with the servo circuits 20 to 205, and the control output of the head controller 14 is guided to these position control servo circuits 201 to 205. The position control servo circuits 201 to 205 control the positions of the iris driving motor 121, the focus driving motor 123, the zoom driving motor 125, the pan head horizontal axis driving motor 131, and the pan head vertical axis driving motor 133, respectively. Used.

The head controller 14 includes an iris position detecting potentiometer 122, a focus position detecting potentiometer 124, a zoom position detecting potentiometer 126, a pan head horizontal axis position detecting potentiometer 132, and a pan head vertical axis position detecting potentiometer 134.
A / D that converts the output from each to a digital signal
A converter 23 is provided. The control signal for each motor is D
/ A converter 24 to each position control servo circuit 20
It is output to 1-205.

The head controller 14 calculates the direction of the video camera 11 based on the output of the A / D converter 23. The camera direction calculation circuit 28 outputs the direction calculation circuit 28 of the camera and the image position measuring device 22. The spatial coordinate calculation circuit 26 that calculates the three-dimensional spatial position coordinates of the target object based on the output, the movement prediction circuit 27 that calculates the predicted movement position of the target object based on this calculation output, and the movement prediction calculation result Based on this, a camera direction determining circuit 30 for determining the direction of the video camera, a zoom amount determining circuit 31 for determining the zoom amount of the zoom lens 12, and a focus determining circuit 32 for determining the focus amount are provided. These circuits 3
The outputs of 0 to 32 are input to the motor control circuit 33 that determines the control amount of each motor. The motor control circuit 33
The control amount of each motor is determined, and its control signal is sent via the D / A converter 24 to each position control servo circuit 201-.
Output to 205.

The calculated three-dimensional spatial position coordinates of the moving object are output from the interface 34 to the data recording unit 2 or the moving image editing unit 3 as three-dimensional data output. The interface 34 is also connected to a control terminal device 38 including a display unit for displaying the position of an object, a terminal device, and an operation unit, and a control signal given from the interface is provided to the direction determination circuit 30 of each camera and the zoom. It is given to the amount determination circuit 31 and the focus determination circuit 32. In this way, this embodiment can be manually operated.

FIG. 5 is a diagram for explaining the configuration of the video camera 11, the zoom lens 12, and the electric pan head 13 in the camera head 1 for picking up an image of the target object 7. The video camera 11 is normally photographed in units of 1/60 seconds per field.

The camera lens may have a fixed focus, but a zoom lens is convenient when photographing a wide range. In this embodiment, the electric zoom lens is provided with a potentiometer and the rotation angle of the zoom ring is read as a voltage value so that the current angle of view can be always known. Further, the servo is constructed by using the data, and the zoom can be moved to an arbitrary position by the command of the head controller 14. A potentiometer was used as the rotation angle sensor in this example, but other devices such as a rotary encoder may be used. In this example, the angle of view of the lens is mechanically detected by the angle of rotation of the zoom ring. It is also possible to measure in real time optically using the ranking period.

Since the actual angle of view of the lens and the output of the sensor are mechanically coupled, variations due to individuals and
Changes over time occur. Variations due to individuals can be dealt with by having inspection data at the factory and having correction data. Regarding the change with time, at the time of calibration of the camera head for the three-dimensional coordinate calculation, it is possible to correct the correction data by photographing a known size while changing the zoom.

The lens also has a motor and potentiometer attached to the focus and iris,
Servo is assembled. Similar to zooming, the head controller 14 can be moved by a command, and the head controller 14 can know the current value. Focus control can also use so-called autofocus algorithms such as examining the high-frequency components of the image obtained from the camera, but this system can calculate the distance between the camera and the target object because the three-dimensional position of the target object is known. , Control according to the distance. There is also a method of changing the iris control according to the image from the camera as well, but the so-called auto iris algorithm is not optimal when direct reflection of illumination light occurs such as in swimming.
Here, the movable range of the target object, that is, the brightness of the entire pool or some points is measured, an illuminance map of the pool is created in advance, and the iris is controlled according to the position of the target object. This can also do white balance as well. If the illumination is uniform, the white balance does not change, but if there are multiple illuminations with different color temperatures, it is necessary to change the white balance depending on the location. The algorithm of auto white balance using captured images is not good for images where most of the screen like pool is occupied by a single color other than white. This example creates a white balance map for the entire pool, similar to the iris control, and adjusts the white balance of the camera accordingly.

The electric pan head 13 can change the orientation of the camera between a pan direction for horizontal rotation and a tilt direction for vertical rotation. A potentiometer is attached to each axis so that you can know the current orientation of the camera. Also, like the lens, it has a servo system and can be moved in any direction by a command from the head controller. Unlike the lens system, the platform has a large moment of inertia, so it takes time to reach the position indicated by the servo circuit. Therefore, instead of using the position indicated to the servo system, the direction of the camera synchronized with the image uses the reading of the potentiometer. Although the potentiometer is used as the rotation angle sensor here, any other device such as an optical rotary encoder may be used as long as the angle information can be obtained.

As shown in FIG. 4, the head controller 1
Reference numeral 4 has five A / D converters 23 for reading potentiometers and five D / A converters 24 for generating a position command voltage for the servo system. In this example, the analog voltage is used as an interface with the servo meter, but parallel or serial digital signals may be used.

Next, the automatic tracking control will be described.

In the case of automatic tracking, the target object and the background must be automatically separated in the image obtained from the camera. In this example, a color camera is used as the video camera 11, and the learning type color extraction device 21 separates the colors. The target object has a color that is not in the background, for example, a swimming cap. Ordinary swimming caps are suitable for separating from the background because they use colors that stand out in the pool. Further, in this example, the target object is separated from the background by using a color, but if the separation is possible, anything such as brightness or correlation with a specific pattern may be used. The position of the target object separated from the background in the image is calculated by the image position measuring device 22. In this example, the position of the centroid of the separated graphic is calculated on the screen. Automatic tracking controls the platform and lens so that the target object is always displayed at a specific position such as the center of the image.

In the method of controlling the pan head so that the difference between the position of the target object and the target position in the image is always small by using only the data of the image position measuring device 22, the distance between the target object and the camera is If it changes or the zoom of the lens changes, the ratio of the difference on the screen and the rotation amount of the platform changes, so the gain of the control loop changes and it is difficult to control optimally. . Therefore, in this example, the three-dimensional position measurement of the target object and the pan head control are separately performed as follows.

First, the actual three-dimensional position of the target object is obtained from the position of the target object in the image, the orientation of the camera and the parameters of the lens. Next, the direction and tilt of the camera at which the obtained three-dimensional position becomes a target position such as the center of the screen is calculated. Then, the camera platform is controlled so that the camera faces in that direction. Further, in this example, since the movement of the target object is limited to the plane, the position of the target object within this constraining plane is calculated. If the position of the target object is known in this way, the distance between the target object and the camera can be known. Therefore, the focus of the lens is controlled so as to match the distance. In addition, the zoom is controlled according to the distance so that the target object appears to be approximately the same size.

When controlling, the target value is indicated to the servo system by voltage. This instruction cannot be controlled smoothly because the interval is too wide at 1/60 second intervals in one field. Since the measurement of the three-dimensional position of the target object can be performed only in one field interval, the movement of the target object in the three-dimensional space is predicted. The target value can be set in the servo system by using this predicted value, and the interval can be narrowed regardless of the field cycle. In this example, a smooth movement is obtained by giving a target instruction four times per field at intervals of about 4 milliseconds. For the prediction of the motion of the target object, linear interpolation that extends the difference from the previous field as it is is used, but for example, the gravitational acceleration is always added to the moving object in the gravitational field. If you know the property of, you may take the prediction method according to the physical law. It can also be predicted with a Kalman filter or the like.

Next, a method of calculating the three-dimensional coordinates of the target object will be described.

As shown in FIG. 6, the coordinate system is a right-handed coordinate system, and the origin is the center of rotation of the platform. The Y axis is taken perpendicular to the constraining plane in which the target object moves. The two axes of the platform are 0 degrees in the Y-axis direction. (If both axes are 0 degrees,
The optical axis of the camera is parallel to the Y axis. ) Also, as shown in FIG. 7, when the two axes of the platform are set to 0 degrees, the image pickup surface (CCD surface) of the camera is offset from the center of rotation in the XYZ directions.
They are X, OffsetY, OffsetZ apart. As shown in FIG. 9, in the zoom lens, two projection centers N1 and N2 must be considered. However, in order to facilitate the calculation, in this example, assuming that the distance to the subject is sufficiently larger than the distances N1 and N2, the projection center is approximated by a thin lens and rays closer to the axis. Use a pinhole camera model like 8. In an actual image pickup system, the lens is deviated from such an ideal state and has a non-linear element. In order to consider the non-linear element, it is necessary to process with a large numerical table instead of linear numerical calculation. However, if the automatic tracking control works well enough and the target object moves to the center of the image, the three-dimensional position measurement target will come near the center of the lens, and the best part of the lens characteristics can be used. , It is not difficult to choose a lens that is sufficient even with linear approximation to obtain the strength of the platform installation location and the optical accuracy that exceeds the accuracy of the angle sensor.

In order to obtain the three-dimensional spatial coordinates of the target object, the position of the target object on the screen is obtained from the image position measuring device and the position is determined by the plane in the space of the image position measuring device, as shown in FIG. It is converted to a position on a certain projection surface (VT surface). Since the VT plane is located in a point-symmetrical position with respect to the CCD plane with respect to the projection center, the size of the image on the VT plane is the same as the size of the image formed by the lens on the image sensor. Further, since the center of the image does not always coincide with the point (principal point) where the optical axis intersects the projection surface, it is necessary to correct the deviation by translating the position of the image position measuring device. As shown in FIG. 11, a vector from the projection center to the position of the target object on the VT plane is the attention vector. Next, the vector of interest is rotated about the center of rotation of the platform by the same amount as the inclination of the platform. 12
The point where the straight line obtained by extending the vector of interest rotated as described above and the constraint plane intersect is the position of the target object in the three-dimensional space.
Further, by using the vectors up to the four corners of the VT surface instead of the vector of interest, it is possible to calculate the field of view on the constrained plane, which is the imaged range. Next, using the obtained three-dimensional spatial position of the target object as a target point, an angle for controlling the platform so that the target point is imaged at a designated location on the screen, for example, the principal point, is determined. To move to the position of the principal point of the image,
The rotation angles of the two axes of the platform such that the center line of the camera passing through the principal point intersects the target point on the constraining plane as shown in 3 are obtained. In FIG. 13, the angle A is obtained by arc tangent using the distance to the constraining plane and the distance from the Y axis of the target point on the constraining plane. At the same time, the distance d from the center of rotation to the target point is also obtained. The angle B can be obtained by arc cosine using the distance d and the distance from the rotation center of the camera center line. Further, in the case of bringing to a position other than the principal point, the vector a indicating the designated point on the VT plane may be extended from the projection center as shown in FIG. 14, and the rotation angle that intersects with the target point on the constraining plane may be obtained.

The servo circuit 20 is controlled so that the rotation angle thus calculated is obtained.

Next, the actual parameters will be described. The calculation of the spatial coordinates described above and the calculation of the rotation angle of the platform are performed in order to convert the position of the projection center, the position of the imaging surface, and the coordinates obtained from the image position measuring device into the actual spatial coordinates. There are parameters such as coefficients. These parameters have a great influence on the accuracy, and it is necessary to accurately obtain the parameters in order to obtain the accuracy. However,
This kind of parameter always contains a manufacturing error and may be displaced during transportation. Further, when it is operating, various accelerations are applied, and it is possible to consider changes over time and changes in temperature. Therefore, it is not desirable to keep using the designed value. Therefore, in the present invention, these parameters are measured in an operable state without disassembling or the like. The measuring method is shown below.

The main point is that it does not move in the image even if the zoom is changed. The zoom is changed so that the fixed target object is imaged at the temporary principal point. When the position changes in the image due to the change in zoom, it is shifted from the correct principal point, so the temporary principal point is changed to find a point not depending on the zoom.

The projection center and the conversion coefficient of the coordinates of the image position measuring device are related to each other, and if either one is not accurately found, the other is not. Therefore, first, the temporary coordinate conversion coefficient and the position of the projection center are determined from design values and the like. A reference point fixed on the constraining plane is photographed as a target object. Next, the coordinates of the reference point are calculated while actually rotating the rotation axis of the platform. Rotate and calculate while gradually changing the projection center and the coordinate transformation coefficient. Since the point is fixed, the correct projection center and coordinate conversion coefficient can be obtained as a value that makes the calculation result constant regardless of the angle of the rotation axis. I haven't changed the zoom here yet. Since the position of the projection center moves when the zoom is changed, next, the corresponding projection center of each focal length is obtained while changing the zoom by using the coordinate conversion coefficient just obtained.

Here, examples of actual parameters will be shown. The zoom of the lens is changed by rotating the zoom ring with a motor, and the rotation angle is detected as a voltage by the potentiometer 126 for zoom position detection, and the A / D converter 2
It is read as a digital value at 3. Therefore, the controller can know only the value of the A / D converter 23. The position of the center of projection is a function of the zoom of the lens. Therefore, in order to obtain the actual position of the projection center, it must be obtained from the value of the A / D converter 23 of the zoom ring of the lens. As mentioned previously, in the present invention,
Since the parameters are based on actual measurement, it is not realistic to measure the relationship between the value of the A / D converter and the position of the projection center for all values because it takes a lot of time. Therefore, we measure only some points and find the relational expression from them. Once the coefficients of the relational expression have been determined, it is possible to use a table method in which the values of all points are calculated in advance. However, in this example, the calculation is performed every time because the speed of the computer is sufficient. FIG. 15 shows a graph of measured values of the A / D converter and the position of the projection center. In this lens, the reciprocal of the position of the projection center can be approximated by the cubic expression of the reading of the A / D converter.
FIG. 16 is a graph showing the relationship between the reciprocal of the position of the projection center and the reading of the A / D converter. The circle is the measured value, and the curve is the curve based on the cubic formula based on it. Similarly, for the focus, a relational expression is obtained from the reading of the A / D converter and the actual measured value of the focus of the lens. The relationship between the reciprocal of the distance and the A / D converter is shown in FIG. In this case, the relationship can be expressed by a linear expression. The same parameter determination method is used for the servo system. As described above, in tracking a target object, if the target object is close to the principal point, the influence of the error in the optical system is lessened and the measurement accuracy is improved. Therefore, it is important to properly configure the servo system. Although the servo system is more resistant to fluctuations than the optical system, it is still important to actually measure the parameters in the actual system. The situation is shown below. FIG.
Shows the step response of the horizontal rotation axis (thin solid line) and the characteristics of the model (thick solid line and broken line) in which the parameters were determined so as to be the closest to the measured value. An example of the transfer function obtained by this is shown in Expression 1. Even if you replace part or all of the lens or camera for maintenance,
Optimal control can be performed by using the method of measuring the characteristics of the actually operating system and determining the parameters from the characteristics.

[0054]

[Equation 1] Next, the automatic tracking control will be described.

As described above, since the field cycle of the television camera is about 60 Hz, that is, about 16.7 msec, the position of the target object can be obtained at 16.7 msec intervals, but in order to control the motor, 16.7 msec. Since the interval is too wide, in this example, it is divided into four equal intervals at about 4 msec intervals.
Specifically, the value of the potentiometer for detecting the rotation angle is read, and the command is sent to the motor by PID control based on the difference between the value and the target value at intervals of 4 msec. As described above, this target value is estimated from the data at 16.7 msec intervals based on the movement of the target object. In addition to this pan head control, head control includes lens zoom and focus. 19 and 20 show an example of the flowchart.

When the initial setting is completed and the automatic tracking operation is started,
First, the position of the target object is read from the image position measuring device.
The data of the image position measuring device is obtained in the field cycle of the TV camera, and this is used as the reference of the time timing (timer reset). On the other hand, in order to control the motor, the potentiometer for the rotation angle is read, the PID control is calculated, and a command is issued to the motor. As the target value at this time, the last value of the four target values obtained in the previous calculation is used. The position of the target object in the three-dimensional space is calculated from the information from the image position measuring device and the value of the potentiometer. Since the performance of the platform may not catch up with the movement of the target object,
Determine whether the target object is on the screen. When the target object is on the screen, four target values at 4 msec intervals are predicted from the law of motion of the target object. In this example, linear extrapolation is simply used. Four sets of rotation angles of the platform for adjusting the camera direction to this target value are calculated. This rotation angle becomes the target value for the motor. Knowing the angle of rotation of the platform allows us to determine if that angle is possible due to the dynamics of the system. If it is found that the target object moves faster than the dynamic characteristics of the system, and if it goes out of the screen as it is, move the zoom to the wide-angle side to widen the visible range to avoid losing sight. Otherwise, the zoom position is calculated from the distance between the target object and the camera so that the size of the target object is almost the same. Similarly, the focus position is calculated according to the distance. These values are the same as for the platform, and are the target values, and the control is performed to match this target. This completes the calculation with the data from one image and only controls the motor every 4 ms.

The above is the case where the object is in the image, but if the object is not found, the following processing is performed. Two types of processing are possible depending on the time during which the object is invisible. This is when you really lose sight,
It happens for some reason, for example, when an obstacle crosses the front of the camera for a moment and it disappears for a moment. 2
Correspond to the street. The time during which the target object cannot be seen is easy to measure in units of the field frequency of 16.7 ms. If the hidden time is short, the position of the target object is predicted from the movement of the target object up to now, and the target object is controlled to be there, and at the same time, the zoom is slightly widened to widen the prediction range. If the hidden time is long, it is determined that the user has lost sight of it, and the process according to the preset algorithm is performed. For example, a process of scanning a set range to automatically search for a target object, stopping at the time of losing sight, or returning to the origin can be considered. The actual calculation speed is 2 to 3 milliseconds for calculation in a system using a floating point coprocessor, even if a microcomputer is used. Therefore, in the processing in the unit of 4 msec in one field, the three do not perform any processing relating to this and just wait for the passage of time. If it is determined that the processing is within 4 ms, the processing can be performed in this. In this example, communication processing for transmitting the obtained three-dimensional information to the outside is performed.

(Second Embodiment) In the first embodiment, the image data is captured by using the camera head for automatic tracking, but it may be difficult to extract it from the background depending on the target object. When such an object is automatically tracked, the influence of noise or the like may increase and the accuracy may deteriorate. Therefore, there is a method in which a person uses the same head to chase the target object. To move the camera
There is also a method in which a person follows a target object by using the camera head of the first embodiment as it is and operating the remote control while looking at the monitor. However, when a human chases an object, he or she also uses a monitor screen, but in most cases, he or she makes a comprehensive judgment by using other information, such as the movement seen directly or the surrounding situation. Therefore, the tracking ability becomes higher when a person is actually at the camera and directly operates the camera instead of operating the remote controller. Therefore,
It is better to use a camera suitable for human operation and attach sensors such as angle to the rotation axis and lens of the camera. In this case, since the position of the target object in the image cannot be detected, the accurate three-dimensional position of the target object cannot be calculated. However, depending on the orientation of the camera and the angle of view,
The field of view of the camera in dimensional space can be calculated. Further, since it is considered that the target object is usually photographed so as to be in the vicinity of the center of the visual field, it is also possible to calculate the coordinates of the center of the visual field and automate the control such as focus according to the calculation.

[0059]

As described above, according to the present invention, in the moving object measuring apparatus, (1) the measuring accuracy of the moving object can be increased, and (2) the three-dimensional moving object which is different for each camera device. The parameters for obtaining the spatial position coordinates can be obtained on the spot, (3) the automatic tracking control of the camera head can be accurately and smoothly performed, and (4) the moving object to be measured can be accurately obtained from the background. There is an effect that can be taken out.

[Brief description of drawings]

FIG. 1 is a block diagram showing an example of the overall configuration of an image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is a diagram showing a manner of photographing a moving object according to the first embodiment of the present invention.

FIG. 3 is a diagram showing a conceptual diagram of the overall configuration of a camera head of a first embodiment of the present invention.

FIG. 4 is a diagram showing a detailed configuration of a camera head.

FIG. 5 is a diagram showing a configuration of a video camera, a zoom lens, and an electric pan head in the camera head.

FIG. 6 is a diagram illustrating a coordinate system of a camera head.

FIG. 7 is a diagram illustrating a relationship between an image pickup surface of a camera and a coordinate system.

FIG. 8 is a diagram illustrating a projection relationship of a subject by a pinhole camera model.

FIG. 9 is a diagram illustrating a projection relationship of a subject by a zoom lens.

FIG. 10 is a diagram for explaining conversion of a subject into a VT plane by the image position measuring device.

FIG. 11 is a diagram illustrating an attention vector of a subject.

FIG. 12 is a diagram for explaining a relationship between a vector of interest and a subject on a constraint plane.

FIG. 13 is a diagram illustrating calculation of three-dimensional spatial coordinates of a subject.

FIG. 14 is a diagram for explaining calculation of three-dimensional spatial coordinates of a subject.

FIG. 15 is a diagram showing an example of actually measured values of the relationship between the value of the A / D converter and the position of the projection center.

FIG. 16 is a diagram showing an example of the relationship between the value of the A / D converter and the reciprocal of the position of the projection center.

FIG. 17 is a diagram showing an example of the relationship between the reciprocal of the distance and the value of the A / D converter.

FIG. 18 is a diagram showing the characteristics of the model based on the measured values of the step response of the horizontal rotation axis and the parameters determined so as to be close to the measured values.

FIG. 19 is a diagram showing an example of a control flowchart of the camera head.

FIG. 20 is a diagram showing an example of a control flowchart of a camera head.

[Explanation of symbols]

 1 camera head 2 data recording unit 3 moving image editing unit 4 time base unit 5 stopwatch 11 video camera 12 zoom lens 13 electric pan head 14 head controller 15 image processing device 16 position calculation device 17 drive control device 20 servo circuit 21 learning type Color extraction device 22 Image position measurement device 23 A / D converter 24 D / A converter 26 Spatial coordinate calculation circuit 27 Movement prediction circuit 28 Camera direction calculation circuit 30 Camera direction determination circuit 31 Zoom amount determination circuit 32 Focus determination circuit 33 Motor Control Circuit 34 Interface 36 TV Monitor for Confirmation 37 TV Monitor for Operation 38 Control Terminal Device 121 Iris Drive Motor 122 Iris Position Detection Potentiometer 123 Focus Drive Motor 124 Focus Position Detection Tensioner 125 Zoom drive motor 126 Zoom position detection potentiometer 131 Pan head horizontal axis drive motor 132 Pan head horizontal axis position detection potentiometer 133 Pan head vertical axis drive motor 134 Pan head vertical axis position detection potentiometer 201-205 Position control servo circuit

Claims (6)

[Claims] 1. An image pickup unit for picking up an image of a moving object, a drive unit for setting an image pickup direction and an image pickup range of the image pickup unit, and a position in an image of the moving object included in a video signal obtained by the image pickup unit. The coordinates in the real space of the moving object are calculated from the image processing means obtained by calculation, the position in the image obtained by the image processing means, and the information on the image pickup direction and the image pickup range of the image pickup means when the video signal is obtained. In the apparatus for measuring a moving object, the position calculating means sets a projection center of a lens of the camera of the imaging means, and the object of the moving object imaged with the projection center as point symmetry. To a position on the projection surface that is point-symmetrical to the image sensor surface, and a vector from the projection center up to the subject of the image converted by this conversion means. A moving object measuring apparatus, which includes means for setting a torque and calculating a three-dimensional spatial coordinate of the target object based on the distance from the vector to the constraining plane where the moving object exists and the rotation angle of the camera. . 2. An image pickup means for picking up an image of a moving object, a driving means for setting an image pickup direction and an image pickup range of the image pickup means, and a position in an image of the moving object included in a video signal obtained by the image pickup means. The coordinates in the real space of the moving object are calculated from the image processing means obtained by calculation, the position in the image obtained by the image processing means, and the information on the image pickup direction and the image pickup range of the image pickup means when the video signal is obtained. In the measuring device for a moving object, the position of the projection center of the image, which is the central position of the lens of the camera of the image pickup means, and the three-dimensional spatial coordinates of the moving object are calculated. A means for calculating the spatial position coordinates by imaging a reference point that is fixed on the constraining plane and whose spatial position coordinates are known by setting a variable as a temporary value. Measuring device of the moving object, characterized in that it comprises means for the camera projection center and said conversion variables little by little by imaging again reference point while changing seeking projection center and conversion variables Zui. 3. An image pickup means for picking up an image of a moving object, a driving means for setting an image pickup direction and an image pickup range of the image pickup means, and a position in an image of the moving object included in a video signal obtained by the image pickup means. The coordinates in the real space of the moving object are calculated from the image processing means obtained by calculation, the position in the image obtained by the image processing means, and the information on the image pickup direction and the image pickup range of the image pickup means when the video signal is obtained. In the apparatus for measuring a moving object, the position calculating means is provided with means for operating the driving means of the imaging means for a plurality of operating points and obtaining the drive amount of the driving means as an electric signal. Calculates the relationship between the obtained electrical signals at the multiple operating points and the projection center position of the camera lens, and performs projection based on the relationship at the multiple operating points. Position and measuring device of the moving object, characterized in that it comprises a means for calculating the three-dimensional coordinates of the target moving object obtained relation expression between the drive quantity. 4. An image pickup means for picking up an image of a moving object, a driving means for setting an image pickup direction and an image pickup range of the image pickup means, and a position in an image of the moving object included in a video signal obtained by the image pickup means. The coordinates in the real space of the moving object are calculated from the image processing means obtained by calculation, the position in the image obtained by the image processing means, and the information on the image pickup direction and the image pickup range of the image pickup means when the video signal is obtained. A position calculation means for calculating the moving direction of the moving object based on the obtained coordinates in the real space of the moving object, and a drive control means for controlling the driving means by the predicting means. In the moving object measuring device provided with, the movement predicting means includes means for predicting a position of the moving object a plurality of times at a time interval shorter than an interval of position calculation of the position calculating means. In addition, the drive control means includes means for giving a control target value based on the predicted position value to the drive means at a time interval shorter than the calculation interval of the position calculation means. apparatus. 5. An image pickup means for picking up an image of a moving object, a driving means for setting an image pickup direction and an image pickup range of the image pickup means, and a position in an image of the moving object included in a video signal obtained by the image pickup means. The coordinates in the real space of the moving object are calculated from the image processing means obtained by calculation, the position in the image obtained by the image processing means, and the information on the image pickup direction and the image pickup range of the image pickup means when the video signal is obtained. In the measuring device for moving objects, the image pickup means includes a color camera, and the image processing means extracts color from the background of the moving object to be measured from the captured image. A moving object measuring apparatus comprising means. 6. An image pickup means for picking up an image of a moving object, a driving means for setting an image pickup direction and an image pickup range of the image pickup means, and a position in an image of the moving object included in a video signal obtained by the image pickup means. The coordinates in the real space of the moving object are calculated from the image processing means obtained by calculation, the position in the image obtained by the image processing means, and the information on the image pickup direction and the image pickup range of the image pickup means when the video signal is obtained. A position calculation means for calculating the moving direction of the moving object based on the obtained coordinates in the real space of the moving object, and a drive control means for controlling the driving means by the predicting means. In the moving object measuring device, the drive control means may obtain in advance illuminance distribution or white balance distribution information in the imaging range of the imaging means. A measuring device for a moving object, comprising means for controlling the drive means based on this information to adjust the aperture or white balance of the camera device of the image pickup means.