JP3631266B2 - Measuring device for moving objects - Google Patents

Description

[0001]
[Industrial application fields]
The present invention is used in an apparatus that images a moving object and performs measurement using the captured image. The present invention is particularly suitable for use in an apparatus that images a moving athlete such as a sports competition, extracts the athlete's three-dimensional spatial coordinate information, and performs various image processing based on the three-dimensional spatial position information.
[0002]
[Prior art]
As a method for measuring a moving object, three-dimensional spatial position information of the moving object is obtained by imaging the moving object and processing the video signal.
[0003]
As an apparatus for measuring such a moving object, the applicant specializes in a technique of superimposing information obtained from the camera platform or the imaging device as position information on a video signal obtained by the imaging device placed on the camera platform. Japanese Patent Application No. 5-082178 discloses a technique for calculating a three-dimensional spatial coordinate of a moving object from a signal obtained by imaging the moving object. Japanese Patent Application No. Hei 5-082209 predicts or measures the movement of the moving object and controls the camera device. Japanese Patent Application No. 5-138805, Japanese Patent Application No. 5-139629, which superimposes the obtained three-dimensional spatial coordinate information of a moving object on video data, and Japanese Patent Application No. 5-5 Japanese Patent Application No. 5-261223 filed a technology relating to measurement of various parameters of the imaging means and a control device for the imaging means. In addition, an image captured by an imaging unit that measures the three-dimensional spatial coordinates of such a moving object is converted into an image captured by a camera on an arbitrary spatial position or time axis based on the measured three-dimensional spatial coordinate information. The application was filed as Japanese Patent Application No. 5-221363 (both unpublished at the time of filing this application).
[0004]
[Problems to be solved by the invention]
The technology applied by the above-mentioned applicant is based on a method of tracking by a camera head with a tracking function such as a camera mounted on at least one pan head for each player, and a plurality of competitions as it is. There is a problem that is not suitable for imaging a competition in which a person runs on the same field or moves in a three-dimensional space.
[0005]
One of them is that an error or error associated with imaging occurs.
[0006]
For example, in a competition where there are many athletes to be measured, such as soccer, if two or more camera heads are used per player, the measurement error is reduced to the performance of the camera head, but many expensive camera heads are used. Need to be placed. Further, in the case of using the constraint plane technique in which the player moves on a fixed plane, this is an error if the player who is the target object to be imaged moves away from the plane. In particular, when taking images at a shallow angle with respect to the constrained plane, the error becomes large. In imaging such as soccer, the camera position is low relative to the width of the competition field, so it can only be observed at a shallow angle and extracted. Further, there is a problem that an error becomes large in the three-dimensional space coordinate information. Furthermore, since the posture of the player changes greatly, there is a problem that an error in the position of the player becomes large.
[0007]
Further, as the player moves, the angle formed by the player and the camera head also changes. For example, even when the players overlap each other, the three-dimensional image calculated from the image captured by the camera head tracking the player. An error may occur in the spatial information.
[0008]
In addition, it is important for the camera head to produce a stable image with high accuracy, but it is a mechanically configured mechanism and is often installed in a relatively poor environment such as outdoors. Inspection is required. In this case, since what is used for industrial measurement needs continuous measurement, it was necessary to suspend the measurement for inspection.
[0009]
An object of the present invention is to reduce the number of use of expensive camera heads with a pan head, and the imaging means images a plurality of players by combining a plurality of imaging means for imaging a plurality of target objects, and a plurality of imaging means Another object of the present invention is to provide a measuring apparatus that can measure the three-dimensional spatial information of each player based on the image information.
[0010]
Another object of the present invention is to provide a measuring apparatus capable of increasing redundancy as information by imaging a player while dynamically switching a plurality of imaging means depending on the position of the player, and reducing errors and errors due to imaging of moving objects. It is to provide.
[0011]
Another object of the present invention is to perform accurate measurement over a long period of time without being influenced by environmental conditions such as uneven illumination and weather conditions by automatically or periodically calibrating the color camera. An object of the present invention is to provide a measuring device that can be used.
[0012]
[Means for Solving the Problems]
The first aspect of the present invention is a technique for measuring a plurality of moving objects by imaging a plurality of moving objects with a small number of imaging means, and imaging the plurality of moving objects. In the moving object measuring apparatus for extracting the three-dimensional spatial information from the image, three or more imaging means for imaging the moving object are provided, and each of the imaging means is a plurality of imaging targets in the field of view of the imaging means. And a means for calculating the three-dimensional information of each target object based on the information of the target object in the image from each imaging means. With this configuration, it is possible to obtain the three-dimensional spatial position information of many moving objects with less imaging means.
[0013]
Further, with respect to the images of the plurality of target objects in the common visual field, the target object is extracted for each target object based on the property of the image of the target object, and the position in the image of each of the plurality of target objects is obtained. An internal position measuring means and a 3D position calculating means for calculating 3D spatial information of a plurality of target objects based on the in-image position information measured by the means can be provided. With this configuration, it is possible to capture a plurality of target objects within the field of view of the imaging unit and obtain three-dimensional spatial information of these target objects.
[0014]
Further, the imaging means can include a fixed camera whose camera parameters are fixed, and a camera with a tracking function whose camera parameters are variable. Thereby, the number of expensive cameras with a tracking function can be reduced.
[0015]
The in-image position calculation means generates a window surrounding the target object image for each of the plurality of captured target objects, and moves the window according to the movement of the target object, and a plurality of targets visible from the imaging means. A priority table in which priorities between a plurality of windows are set for each image unit based on the positional relationship of objects for each image unit, and a target that is hidden for a plurality of target objects with overlapping windows by referring to this table Means for inhibiting the output of the image information of the object. With this configuration, it is possible to perform image processing when a plurality of target objects are overlapped, and to suppress the occurrence of measurement errors.
[0016]
Also included is a means for comparing at least three or more three-dimensional spatial information of the moving object obtained based on the image of the moving object imaged by three or more imaging means and determining whether or not there is an error in measurement. be able to. With this configuration, occurrence of measurement errors can be suppressed and measurement accuracy can be improved.
[0017]
According to a second aspect of the present invention, there is provided a technique relating to dynamic switching of imaging means, in a moving object measuring apparatus that images a moving object and extracts three-dimensional spatial information from the captured moving object image. The three or more imaging means for imaging the moving object are combined to image at least one moving object at the same time, so that the imaging means is at an appropriate position for measurement. It is characterized by comprising means for switching one imaging means of the imaging means to an imaging means at another appropriate position. With this configuration, data redundancy can be increased and measurement errors can be reduced.
[0018]
Moreover, it is preferable that the means for switching the imaging means to the imaging means at another appropriate position includes means for providing hysteresis in the boundary region of the switching condition. This prevents frequent switching.
[0019]
It is also preferable to include means for performing switching after a predetermined time has elapsed since the previous switching. Thereby, it is possible to prevent frequent switching when the moving speed of the target object is low, and to prevent unnatural switching of images.
[0020]
Further, the switching of the imaging means is performed so that the angle between the three or more imaging means becomes wide. With this configuration, it is possible to perform measurement with a small measurement error as a whole.
[0021]
In addition, an imaging unit that newly participates by switching obtains an angle and a field angle at which a target object to be captured can be captured based on an image from another imaging unit, and an area in which the target object to be captured exists in the image Means for setting a window and learning a detection condition for extracting a target object in the window. With this configuration, it is possible to automatically set the imaging unit to newly participate when switching the imaging unit.
[0022]
Further, it may include means for comparing measurement data and determining whether there is an error measurement based on image data before and after the imaging means is switched. As a result, errors before and after switching can be detected and a failure can be found.
[0023]
In addition, when there are a plurality of moving objects, it is preferable to include means for assigning priority to moving objects to be imaged and preferentially assigning imaging means to objects with high priority. When there is a limit on the number of imaging means, priority can be given to multiple moving objects depending on the target object or scene, so that the moving object or scene that you want to see most preferentially can be imaged under the best conditions. it can.
[0024]
A third aspect of the present invention is a technique relating to camera calibration, in a moving object measuring apparatus that images a plurality of moving objects and extracts three-dimensional spatial information from the captured moving object image. There are provided three or more imaging means for imaging the moving object, and the imaging means includes a color camera, has a calibration point position teaching and recording means, and automatically images a white calibration point at the teaching position. And white balance means. As a result, even if the imaging conditions fluctuate due to uneven illumination or weather conditions, the white balance of the camera can be readjusted and measurement with high accuracy can be performed in response to the fluctuations in the conditions.
[0025]
It is also possible to provide a color sample as a calibration point, automatically image the calibration point of the color sample at the teaching position, sample this sample color, and record it as the target color of the target object. Accordingly, it is possible to perform highly accurate measurement with respect to fluctuations in imaging conditions.
[0026]
Furthermore, it is preferable to install a plurality of calibration points. As a result, calibration can be performed according to the position of the imaging camera, the position of the target object, etc., the calibration time can be shortened, and calibration corresponding to the state of the illumination light beam can be performed.
[0027]
[Action]
When the three-dimensional spatial position of the target object is measured on the assumption that the target object moves on a fixed plane that is a fixed plane, one target head may be imaged with one camera head. However, in order to measure the three-dimensional spatial position of the target object without providing a constraint on the constraint plane, one target object needs to be imaged by a plurality of camera heads. In this case, when measuring a plurality of target objects at the same time, if the distance between the target objects is short, it is possible to image a plurality of target objects in the field of view of one camera head. The position in the image of a plurality of target objects in the screen imaged by this one camera head is obtained, and the position information in the image of the same target object imaged by a plurality of camera heads (at least two camera heads) is obtained. Based on this, it is possible to calculate three-dimensional spatial information (hereinafter referred to as three-dimensional information) of a plurality of target objects.
[0028]
The position in the image of the plurality of target objects in the same image is obtained by providing a window in the image for each target object, extracting the target object from the background by feature extraction such as color, and obtaining its centroid, It is obtained by calculating the position in the image for each target object. Then, based on the in-image position information, the in-image position information for each target object obtained from images from a plurality of cameras and the position information of the camera (in the case of a tracking camera head, a camera that fluctuates due to the tracking) 3D information for each target object can be calculated. As this camera, a fixed camera can also be used. In this case, the measurement accuracy is inferior to that of a camera head with a tracking function, but only the camera parameters that fluctuate cannot be obtained. There is basically no difference.
[0029]
In this case, since the positional relationship between the plurality of target objects and the camera constantly changes as the target object moves, a plurality of targets that can be seen from the camera based on the calculated three-dimensional information of the target object. The object positional relationship is set as a priority table in units of images (for each field, for example), and the image information output from the camera masks the image information of the target object hidden by the target object. Try to reduce.
[0030]
Since the relationship between the target object and the camera head does not need to be fixed, the assignment of which target image is captured by which camera head is dynamically changed. In this case, the assignment of a plurality of camera heads is performed by, for example, imaging a target object with three camera heads, leaving one of them consciously left, and imaging with a camera head at another appropriate imaging position. By switching to, it dynamically changes according to the movement of the target object. The change of the camera head is continuously switched to imaging by a more appropriate camera head depending on the angle formed by the target object and the three camera heads, or the position of the camera head and the target object. As a result, the information captured by the camera head has a high degree of redundancy, making it easier to detect errors during imaging and reducing errors.
[0031]
Furthermore, since the redundancy of information is increased even when the camera head fails, measurement can be performed without using the failed camera head.
[0032]
When this camera head is switched, the newly added camera head sets the angle and angle of view so that the target object is captured within the field of view based on information from the camera head that is currently imaging. The imaging range is set. Then, the detection condition for extracting the feature of the target object is learned, and the target object is extracted.
[0033]
In addition, a white or color calibration point is provided for imaging, and white or color calibration is performed periodically or manually during imaging. As a result, even if there are fluctuations in shooting conditions such as uneven illumination and fluctuations in sunlight, the camera parameters and the like can be readjusted in response to these fluctuations, so that more accurate measurement can be performed over a long period of time.
[0034]
【Example】
Embodiments of the present invention will be described below with reference to the drawings.
[0035]
(First embodiment)
FIG. 1 shows an example of the arrangement of image pickup means according to the first embodiment of the present invention. In this case, a plurality of image pickup target objects are photographed and their three-dimensional space positions are calculated. In this example, an example is shown in which three-dimensional information of a player is measured by photographing with six tracking heads MH1 to MH6 and six fixed cameras FC1 to FC6 in which a soccer game is arranged.
In such a configuration, the present embodiment is a feature of the present invention in a moving object measuring apparatus that images a player, which is a plurality of moving objects, and extracts three-dimensional information of the player from the captured image of the player. Three or more tracking heads MH1 to MH6 and fixed cameras FC1 to FC6 as imaging means for imaging the moving object are provided, and the tracking heads MH1 to MH6 and fixed cameras FC1 to FC6, which are the respective imaging means, Means for calculating a three-dimensional information of each target object based on information of each player in the image from each camera, including means for commonly imaging a plurality of players to be imaged within the field of view of the camera I have.
[0036]
The operation of this embodiment will be described below.
[0037]
When there are a large number of target objects such as soccer and the movement is wide and complex, if a single target object has two heads, a very large number of heads are required. For a wide shooting range, it is often possible to shoot only at a shallow angle, and a player far from the head may become invisible in the shadow of the player in front. In that case, it is necessary to observe from another direction.
[0038]
In the three-dimensional position measurement, the target objects photographed by the two heads need to be the same. Therefore, when there are a plurality of target objects, it is necessary to correspond with the images photographed by the two heads. If the teams are different, they can be easily identified by the color of the uniform, but if the players of the same team are messed up, people will not understand. Therefore, there is a possibility that the correspondence between the two images is wrong. If there is only two heads, there is no detection method even if the correspondence is wrong. However, if there are three or more heads, it is possible to detect that the correspondence is wrong. In this case, it is possible to detect that the images of at least one of the heads are incorrectly associated unless all the heads are associated with each other in the same manner, that is, unless two players are completely replaced. Since the viewing direction is different if the three heads are observing in different directions, the possibility of exchanging the two is quite low.
[0039]
For this reason, it is always necessary to photograph one target object with three heads from different directions. In many cases, what is desired as an image or data is close to the goal. Therefore, it is desirable to arrange the head so that the accuracy near the goal can be obtained.
[0040]
FIG. 1 shows an example of the arrangement of heads and the arrangement of calibration points. In this example, six tracking heads MH1 to MH6 and six fixed cameras FC1 to FC6 are arranged. The tracking head can change the direction of the camera, the angle of view of the lens, and the like. Usually, three units are used as a set to measure the target object. In the example of FIG. 1, three units MH1, MH2, and MH3 form a set. What can be measured at this time is a hatched area in the figure at a shared portion of the field of view of the three heads. A wide range of measurements is possible if the angle of view of the lens of the tracking head is widely controlled, but the tracking head is characterized by capturing the target object as much as possible and reducing the influence of the background to achieve accurate and stable measurement. Therefore, the target object is photographed without widening the angle of view. For this reason, a fixed camera is used in combination to avoid the formation of a target object that cannot be captured. The fixed camera is photographed with a wide-angle lens, and is adjusted so that at least two, usually three, fields of view are included in the measurement target range, for example, the court. In this example, the number and arrangement of the tracking head and the fixed camera are the same, but this is not necessary.
[0041]
FIG. 1 shows an example in which four calibration points CP1, CP2, CP3, and CP4 are arranged. Any one of the calibration points can be imaged from all the heads. This calibration point can be used for calibration of the head position, lens parameters, and the like. Here, the calibration point is used for setting the white balance of the color camera and the color sample of the target moving object. Each calibration point is as shown in FIG. It is sufficient to prepare as many sample colors as necessary, but here, only the sample colors of the team players' uniforms are prepared on one side. COL1 is a white hemisphere, COL2 is a sample color hemisphere of one team player's shirt, and COL3 is a sample color hemisphere of the other team player's shirt.
[0042]
With respect to the accuracy of the position measurement of the target object, the tracking head has a small observation range and a high precision of the pan head, so that the target object has a considerably high three-dimensional position accuracy. On the other hand, in the case of a fixed camera, even if a high-pixel image sensor is used, the accuracy of the lens is not so good because a wide angle is observed with a lens.
[0043]
When expressing the position of each player in a plane, the accuracy of the position can also be expressed as shown in FIG. The players in the shared portion 21 of the field of view of three or more tracking cameras are expressed by small points as indicated by reference numeral 22 because their positions are obtained with high precision, while the other players are indicated by reference numeral 23. The position is expressed by a large circle. The radius of this circle expresses the accuracy that can be achieved by the number of pixels of the fixed camera. In the case of a fixed camera, the accuracy can be calculated according to the location if the camera is located. If the combination to be used when obtaining the three-dimensional coordinates is determined, the accuracy can be calculated accordingly. Therefore, it is possible to dynamically change the size of the circle.
[0044]
Next, the operation of measuring a plurality of target objects with one tracking head or camera will be described. The measurement system may be the same in basic image processing except that the tracking head and the fixed camera differ in whether or not the camera parameter changes.
[0045]
FIG. 3 shows a block diagram of three-dimensional position measurement. In this example, three tracking heads or fixed cameras are combined into one set, but more can be combined. Further, three tracking heads may be used, or three tracking heads and a fixed camera may be mixed. When fixed cameras are mixed, care must be taken when calculating the three-dimensional position because the accuracy can be fixed.
[0046]
For the images taken by the three cameras 31, the position of the target object in the image is measured by the in-image position measurement unit 32 corresponding to each camera. Since there are a plurality of target objects in one image, a plurality of position data in the image can be obtained simultaneously. In order to obtain the three-dimensional coordinates, positions in the image obtained by at least two cameras are required. In this example, three cameras are used, and three-dimensional coordinate calculation uses two of them. Since there are three types of combinations, there are three sets of three-dimensional position calculation units 34. From these, three-dimensional coordinate information 35 corresponding to the number of target objects is output. In FIG. 3, the output of the in-image position measurement unit 32 and the three-dimensional position calculation unit 34 are connected 33 corresponding to the combination, but in reality, the three-dimensional position calculation is performed by software in the computer. Instead of the wiring as shown in this figure, a communication line is used.
[0047]
If the correspondence is not wrong when three-dimensionally measuring a plurality of target objects, the outputs of the three sets of three-dimensional position calculation units 34 should be the same, but the calculation of the three-dimensional position is caused by some reason such as incorrect correspondence. If you can't do it properly, the three sets of data will have different values. Using this fact, the collation unit 36 checks whether or not the output three-dimensional position is correct, and outputs the information simultaneously with the position. Since an error is always included in the calculation of the three-dimensional coordinates, the three sets of data are unlikely to be completely matched. The collation unit 36 determines whether there is an error in consideration of this fact. The output of the verification unit 36 is three-dimensional information 37 and error signals 38 corresponding to the number of target objects. The processing when the error signal 38 is output is basically manual processing by the operator such as correcting the target object if the correspondence of the target object is wrong, but the number of simultaneous photographing cameras may be increased. Thus, it is possible to automate to some extent by increasing the error information such as in which camera image the error has occurred.
[0048]
Next, the in-image position measurement unit 32 will be described with reference to FIG. The in-image position measurement unit includes a detection circuit (DET0 to DET3) 41 that extracts features of a target object such as a color from an input image signal and converts them into a binarized signal, and the converted binarization. Based on the signal, an object table (OBJ TBL) 42 in which a combination of the extracted binarized signals is set, a window generator (WIND0 to WIND4) 43 that generates a window for each target object, and a set window A priority table (PR TBL) 44 that outputs information on the target object extracted corresponding to the priority of the position, and a position counter that counts the information for each output target object and outputs the position information in the image ( 0-4) 45 and a controller 46.
[0049]
An image photographed by the camera is binarized using a difference in color, a difference in luminance, pattern correlation, or the like (detection circuits DET0 to DET3). For example, when different colors are used, when an image as shown in FIG. 5 is taken, the shirt colors of the players at both ends are extracted as shown in FIG. Fig. 7 shows the trunk color of the player. The middle player is the same color as the shirt and trunks. If the centroid of this extracted figure is used as the position in the player's image, the middle player and the players at both ends will not be the same. Therefore, the extracted signals are not used as they are, but are used in combination. The object table 42 of FIG. 4 performs this function. Since the middle player can remain extracted, the table passes as it is. For the players at both ends, the object table 42 is set so that the OR of the signal from which the shirt is extracted and the signal from which the trunks are extracted is taken. The OR signal is as shown in FIG. In this example, three colors are extracted from the image (D0, D1, D2), and two signals are obtained from them. OD2 in FIG. 4 is not used.
[0050]
When the extracted signal is one target object in the screen like the middle player, the centroid position can be set as the target object position. In some cases, the extracted signal alone is indistinguishable. Therefore, use a window. In FIG. 8 and FIG. 9, the dotted line is a window, and the respective centroid positions are calculated within the window. The window generator 43 in FIG. 4 creates the position and size of this window. In this example, since there are five (WIND0 to WIND4), up to five target objects can be photographed simultaneously with the same image. The centroid calculation of the target object is performed in the window, but the windows are given priority, and the processing is changed for each window with respect to the overlapping of the windows.
[0051]
The priority will be briefly described. When windows having different priorities are overlapped on two target objects as shown in FIG. 10 and overlapped with each other, a portion where there is another window as shown in FIG. As shown in FIG. 12, processing such as a window in which the target object can be seen regardless of other windows can be performed. In FIG. 11, the original target object is hidden by another window and the centroid moves slightly. In FIG. 12, another target object also enters the window, and the centroid position of the figure including both is measured as shown in FIG. If the window is moved to the centroid position in accordance with the movement of the target object, the window shown in FIG. 13 when the two target objects are separated from each other has a different target object. There is a possibility of chasing. For this reason, the correspondence between the two cameras is lost, and an error occurs in the three-dimensional calculation. However, if the same target object is mistaken for all cameras, the error cannot be detected. Therefore, the window priority is changed for each camera. As a result, even if the target object is mixed up, it becomes different for each camera and can be detected as an error. The priority table 44 in FIG. 4 creates this priority. The priority can also be set when the number of windows is three or more. Since the priority table 44 can be rewritten in the field cycle, the priority of each camera can be changed in the field cycle based on the three-dimensional position of the player. The output of the priority table 44 is integrated by the position counter 45 and output as a position in the screen. The controller 46 uses this position to update the window size and position for the next field for each field. By performing the above processing, it is possible to measure a plurality of target objects with one camera image while adding information about whether the value is correct.
[0052]
The window technique for measuring the positions of a plurality of target objects in the above-mentioned screen and the technique for counting the positions of the target objects are shown in the motion measurement device (JP-A-3-26281) previously proposed by the applicant. It can be realized with technology. That is, by using a technique for generating a plurality of windows and moving the windows according to the movement of the target object and a technique for counting the coordinate positions of the target object, the coordinate positions in the images of the target objects are determined as the target object. Can be determined for each.
[0053]
The three-dimensional position calculation unit 34 may use, for example, the three-dimensional spatial coordinate calculation technique of the target object described in Japanese Patent Application No. 5-261223 related to the prior application. This technology uses a constrained plane. Based on the image position measurement result, the target object is considered to be a vector from the projection center of the camera to the position of the target object in the screen (for example, the centroid). By using parameters measured in advance, the three-dimensional space coordinates of the target object on the constraint plane can be calculated. Even when the constraint plane is not used, the three-dimensional space coordinates can be calculated by setting a projection plane corresponding to the constraint plane of the target object for each image. Further, a region of the three-dimensional real space coordinate system in which the target object moves described in Japanese Patent Application No. 5-82209 described in the prior application is set, and this is mapped to a two-dimensional region to calculate the three-dimensional space coordinates of the target object. Technology can also be used.
[0054]
In order to calculate the three-dimensional information of the target object, a target whose value is known in advance is imaged with a tracking head or a fixed camera in the field, for example, and the three-dimensional information is calculated. It is preferable to obtain in advance parameters such as the tracking head parameters necessary for the calculation of the three-dimensional information and the relationship between the fixed camera and the target.
[0055]
When measuring an object that moves over a wide range, such as soccer, for a long time, there are differences in lighting conditions depending on the location, and there are variations in time due to sunny or cloudy conditions. In such a case, if the position of the target object is measured using the color information of the target object, the target object may not be measured well. In such a case, it is effective to readjust the white balance of the camera. It is also effective to re-register the color to be measured (the color of the uniform in the case of soccer). It is better to do both. In order to do this, the colors of white and the target uniform are set in fixed positions as sample colors in advance, and by monitoring the sample colors, it is possible to keep up with changes in illumination and the like.
[0056]
Here, adjustment of the white balance of the camera and registration and re-registration of the target color are performed using calibration points. The procedure and processing are as follows.
[0057]
1. At the time of camera head installation or system power-on, put the calibration point in the field of view, specifically move the pan head manually to bring the calibration point into the field of view, then specify the sample color hemisphere with the cursor, Register the position of the sample color. If necessary, white balance adjustment and sample color registration are executed simultaneously.
[0058]
2. The white balance is automatically readjusted at the registered position when not measuring. Specifically, this readjustment is as follows. It is sufficient to perform the same operation as that performed manually at the position of the registered sample hemisphere.
[0059]
3. The sample color is automatically re-registered with the color registered during non-measurement.
[0060]
4). 1. Immediately before entering a new measurement 2. Or 3. Re-register.
[0061]
5. 1. Regularly set the camera head to the non-measurement state. Or 3. Execute. In the meantime, the other camera head substitutes for measurement or enters a measurement state without redundancy.
[0062]
6). When there are a plurality of calibration points, the calibration point is closest to the target object to be measured.
[0063]
Since calibration can be performed automatically or simply as necessary, measurement over a long period of time can be performed stably. For this reason, it is possible to perform measurement that is resistant to uneven illumination and changes. In addition, the system can be made less susceptible to changes in weather conditions.
[0064]
By arranging a plurality of calibration points, it is desirable to monitor under conditions as close as possible to the state of the moving object. The calibration point to be calibrated may be selected adaptively depending not only on the distance to the target object but also on the position of the target object and the direction of the sunlight. Furthermore, if the calibration point and the moving object are close in position, the operation time of the camera head for calibration can be shortened. It is also possible to easily select an unreasonable position from the movable range of the camera head. It is also possible to estimate the variation of the lighting condition in advance by combining the data of a plurality of calibration points and reflect this in grasping the measurement accuracy. These are particularly effective for a wide range of measurements.
[0065]
(Second embodiment)
Next, a change in tracking head allocation will be described.
[0066]
In this example, three heads are used as one set for measurement. In the arrangement of FIG. 1, there are six heads, and it is decided which three of them are used. As shown in FIG. 1, the simplest method is to switch to three on this side when the coat on this side is the area of interest as shown in FIG. 1, and to switch to three on the other side when on the other side. However, in this case, when the center line is crossed, three units are switched simultaneously. Since there is always some error in 3D measurement, the data is not continuous at the moment of switching. In particular, when a uniform or the like is extracted by color for position measurement in the image and the centroid is calculated, the position of the centroid changes because the appearance of the player changes when the camera viewing direction changes. Since these three units are switched at the same time, a large deviation occurs when the three-dimensional coordinates are calculated. In addition, the condition of illumination may change depending on the variation of the camera and the lighting condition depending on the viewing direction. Furthermore, in automatic tracking, it is necessary to ensure the continuity of the three-dimensional coordinates when switching is performed in order to apply the servo using the three-dimensional position of the target object. For this reason, in order to minimize the influence of switching, in this example, only one of the three heads is switched at a time. Even if the switching fails for some reason, at least two heads continuously measure, so that the three-dimensional position data is not interrupted and can be recovered.
[0067]
If the image taken with the head is important, for example, when shooting a video of a popular player, the head can be switched in the sense of selecting the shooting direction according to the posture of the player, but for that purpose, May control another shooting camera based on the measurement data.
[0068]
In addition, when the performance of the head is not sufficient due to the installation state of the head or the like, when the accuracy is lowered if the head is far from the head, the head may be switched to a nearby head.
[0069]
Measuring the target object with the three heads is most important in terms of confirming the reliability of the measurement data. For this purpose, it is preferable to capture the target object from a different angle as much as possible. In this example, the head is switched so as to capture from a different angle as much as possible.
[0070]
FIG. 14 shows an example of switching. In this example, a state of switching when the attention area captured by three cameras moves from A in the upper left to F in the lower right is shown. When the attention area is A, images are taken by the head 1, the head 2, and the head 6. When the region of interest moves and comes to the position B, the head 1, head 3, and head 6 are now photographed. During this time, head 2 switched to head 3. Similarly, the head 6 is switched to the head 5 at the position C. Further, at the position D, the head 5 is switched to the head 4. At the position E, the head 1 is switched to the head 6, and at the position F, the head 6 is switched to the head 5. As described above, only one of the three switching modes is switched to the head arranged next to it.
[0071]
This is because, for example, when the target object is in the position from A to B, the angle formed by the head 1 and the head 2 becomes small. In this case, in order to increase the reliability of data, the head 1 and the head 6 are used. The head 3 that increases the angle is selected and switched. In this case, when switching to the head 3, there is another player in front of the target player viewed from the head 3, and if the player is behind the player, the head 1 can be left and not switched. . This can be easily determined by using the priority table set for each camera of the first embodiment described above. Therefore, even if the switching area is entered in this way, it is possible not to switch the head depending on the condition.
[0072]
Since the switching of these heads can be calculated in advance depending on the position of the attention area, it is also possible to simply switch at the position of the attention area. Even in such a case, the switching is provided with hysteresis so that it does not flutter at the switching point. In this example, since there is symmetry in the arrangement of the heads, it is not determined which three to use only by the position of the region of interest. Therefore, in consideration of the moving direction of the region of interest, only one of the three heads is switched to the next head. Further, even when the switching position is exceeded, if the moving speed of the region of interest is smaller than the switching direction, the region of interest is not switched immediately but waits for a certain time. When the speed in the switching direction is very fast, the switching operation is performed before the switching position is reached, and a certain time is taken for switching.
[0073]
When tracking a plurality of target objects with a combination of a plurality of heads, it is often the case that all the target objects cannot simultaneously satisfy the same condition, for example, the number of heads and the angle formed by the heads. This is particularly difficult when the number of heads is minimized.
[0074]
In such a case, in the present invention, priorities are assigned to the target objects, and the heads are assigned so as to satisfy the conditions from the target objects having higher priorities. For example, the target object with the highest priority is always assigned three heads with the best angle. Use the other heads to track the target object of the next priority. At this time, even if the number of heads is sufficient, the condition for the angle becomes worse. If the priority is further lowered, the number of heads to be allocated is reduced or a fixed camera is used.
[0075]
Although the priority order can be fixedly assigned to the target object, it can be dynamically changed as necessary so that the priority order of the target object that is most desired to be seen is highest according to the scene. In this case, a priority order can be given to the target object by an operator or the like. If there is a possibility that data continuity may be lost if you change the assignment of all the heads immediately after changing the priority order, at the stage where the assignment of some heads changes due to changes in the position of the target object, etc. It is preferable that the assignment is performed according to the new priority order so that the assignment of all the heads is not changed immediately.
[0076]
In addition, at the switching point, four vehicles are temporarily observed. In the meantime, three different sets of data are obtained simultaneously. If there is a discrepancy between the two sets of data, the continuity of the data is maintained by gradually correcting it so that it becomes continuous during the switching time. For this reason, switching takes a certain amount of time.
[0077]
In the case of switching, one new unit will be entered in addition to the three conventional heads. The setting of the new unit is performed as follows. FIG. 15 shows the case where there is one target object for simplicity. As shown in FIG. 15, the newly entered head H2 has a different viewing angle from the conventional head H1. Therefore, the angle and the angle of view must be determined first. For example, paying attention to the position of one target object that is being tracked, the angle and angle of view at which the object can be photographed are determined. For a single target object, the size can be predicted using the area in the image that has been taken so far, and therefore the angle of view is determined so that it has the same screen ratio as the previous image. In addition, when a window is used with a plurality of target objects, the windows are set to have the same size. At this time, the size of the window needs to be converted into a position where the target object is located, not the size in the screen. FIG. 16 shows a case where a window is used. In this figure, the shooting range of the other party's head and the position of the window are displayed in the image, but in this way, the control of the switching point can be debugged. After the position of the window is set, next, a detection condition for the target object in the image is set. Detection conditions may be fixed for each camera, but detection using color may change from sunlight to artificial light over time, or shade conditions may change over time. In this example, a new setting is made at the time of switching. Newly-entered heads can calculate where and how large the target object will appear in the image using the measurement data of the existing heads. The window is set using this value, and at that time, several representative points of the target object are obtained. For example, select a three-dimensional position and its surrounding area. The new entry head learns the color of the position in this area and uses it as the extraction condition as the color of the target object. Further, in the case of binarization of luminance, a binarization level is newly set using a luminance histogram in a newly set window. In these cases, the data may be changed only within a certain range from the previous data. Alternatively, the newly detected detection condition is compared with the previous detection condition, and if the difference is large, an error may be generated. As a result, the detection condition can be updated at the same time as the head is switched.
[0078]
In addition, if it is used together with a means for recording an image, it is possible to know whether or not the switching is correct later by recording only the switching state.
[0079]
The above-described learning of detection conditions for a new entry head is performed by, for example, the technology of a color identification device (Japanese Patent Application No. 5-145084) for identifying and extracting a certain color from a color image previously filed by the applicant and the above-described video What is necessary is just to use the technique (Japanese Patent Application No. 5-137857) which isolate | separates a target object from a signal.
[0080]
As described above, when the conditions for a new entry head are met, measurement can be performed with a new set of three units including the new entry head. If the difference between this data and the three data so far seems to be large, it can be reset, or the operator can be alerted as an error as a switching failure. If there are consecutive errors, it is possible that a new entry head has failed. Therefore, even if an error occurs, if the measurement is not stopped, the switching itself is stopped, or the switching to another adjacent head is performed, the influence of the head failure can be minimized and continuous data can be obtained. In order to perform such processing, it is preferable to take a certain time for switching in consideration of the speed of the region of interest.
[0081]
【The invention's effect】
The present invention has the following effects by the configuration described above.
[0082]
(1) Without using many expensive camera heads having a tracking function, it is possible to capture a competition in which a large number of athletes move by using a small number of imaging cameras and measure the three-dimensional information of a plurality of moving objects. .
[0083]
(2) Since a target object can be imaged by combining a fixed camera with a wide field of view and a tracking camera head with high measurement accuracy, the number of necessary imaging cameras can be reduced.
[0084]
(3) Since the assignment of the imaging camera can be dynamically changed according to the movement of the moving object, three-dimensional information can be obtained with a small error.
[0085]
(4) Since the target object is imaged simultaneously by at least three imaging cameras, it is possible to detect the occurrence of an error even if an error occurs in one imaging camera.
[0086]
(5) Since imaging is performed by three or more imaging cameras, information with a high degree of redundancy can be obtained for the target object, so that an error can be dealt with.
[0087]
(6) Newly entered imaging cameras are automatically controlled so that their imaging target is within the field of view using information from other cameras, and the camera also automatically extracts the target object through learning. You can switch to
[0088]
(7) By checking the data before and after entering the market, it is possible to detect errors caused by switching cameras.
[0089]
(8) The state where the image is switched at the switching boundary because the image is not switched suddenly by switching the imaging camera with hysteresis or after a certain time has elapsed since the previous switching. Does not occur, and the switching of images can be performed smoothly and smoothly without giving an unnatural feeling.
[0090]
(9) Cameras are assigned with priorities assigned to the target objects, so that necessary and accurate imaging can be performed and sufficient information can be obtained with respect to target objects that require the highest priority and highly accurate information. be able to.
[0091]
(10) Since calibration points are provided and the white balance of the camera is readjusted regularly or automatically using color samples, there are fluctuations in imaging environment conditions such as uneven illumination and changes in sunlight. However, accurate measurement can be performed even when the target object is measured for a long time.
[0092]
(11) Since a plurality of calibration points are arranged, the operation time for camera calibration can be shortened, and the calibration can be performed using the calibration point closest to the state of the target object. Can be calibrated most closely.
[Brief description of the drawings]
FIG. 1 is a diagram for explaining an arrangement example of camera heads and calibration points according to a first embodiment of the present invention.
FIG. 2 is a diagram showing an example in which the accuracy of the photographed player position is expressed.
FIG. 3 is a block diagram showing the configuration of an apparatus for measuring a three-dimensional position of a moving object.
FIG. 4 is a block diagram illustrating a configuration of an in-image position measurement unit.
FIG. 5 shows an example of an image obtained by photographing a target object.
FIG. 6 is an example showing an image of a shirt extracted from an image by a color difference.
FIG. 7 shows an example of an image of trunks extracted from an image by a color difference.
FIG. 8 is a diagram showing an example of a window.
FIG. 9 is a diagram showing an example of a window.
FIG. 10 is a diagram showing an example in which two windows overlap each other.
FIG. 11 shows an example in which priority processing is performed on a window.
FIG. 12 shows an example in which priority processing is performed on a window.
FIG. 13 shows an example of a window that causes an error.
FIG. 14 is a diagram illustrating a combination in which camera heads are switched according to movement of a target object.
FIG. 15 is a diagram showing the relationship between the field of view of a newly entered camera head and the camera head so far.
FIG. 16 is a diagram illustrating an image of a camera head up to that point and an image of a camera head of a new entry.
FIG. 17 is a diagram illustrating an example of calibration points.
[Explanation of symbols]
21 Shared field of view
22, 23 Player position
31 Camera
32 In-image position measurement unit
33 connections
34 3D position calculation unit
35 3D coordinate information
36 Verification unit
37 3D position information
38 Error signal
41 Detection circuit
42 Object Table
43 Window generator
44 Priority table
45 Position counter

Claims (7)

In a moving object measuring apparatus that images a moving object and extracts three-dimensional spatial information from the image of the picked moving object, three or more imaging means for picking up the moving object are combined to obtain at least one moving object. One imaging means of the three or more imaging means is switched to an imaging means at another appropriate position according to the movement of the moving object so that an imaging means at an appropriate position for simultaneous imaging and measurement is assigned. An apparatus for measuring a moving object, comprising: means. 2. The moving object measuring apparatus according to claim 1, wherein the means for switching to the imaging means at another appropriate position includes means for providing hysteresis in a boundary region of the switching condition . 2. The moving object measuring apparatus according to claim 1 , wherein the means for switching to the imaging means at another appropriate position includes means for switching after a predetermined time has elapsed since the previous switching . The apparatus for measuring a moving object according to any one of claims 1 to 3, wherein the imaging means is switched so that an angle between the three or more imaging means is widened . The imaging means that newly participates by switching obtains an angle and a field angle at which a target object to be imaged can be captured based on an image from another imaging means, and opens a window in an area in the image where the target object to be imaged exists. The moving object measuring apparatus according to claim 1 , further comprising means for learning a detection condition for extracting a target object in the window . 6. The moving object measuring apparatus according to claim 1, further comprising means for comparing measurement data and determining whether or not there is an error measurement based on image data before and after the imaging means is switched . The apparatus for measuring a moving object according to claim 1 , comprising a plurality of moving objects, and means for assigning priority to moving objects to be imaged and preferentially assigning imaging means to objects with high priority .

Images