JP2006190176A - Camera work parameter calculation method, device and program executing the method, and recording medium therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a camera work parameter calculation method necessary for a preprocess of sprite coding and moving object extraction of a video stored in a technological field of moving image processing, and to provide a device and a program executing the method, and a recording medium therefor. <P>SOLUTION: This camera work parameter calculation device has: a representative point calculation part 104 calculating a point that is a characteristic from a frame image inside the video as a representative point; a corresponding point calculation part 105 calculating a point corresponding to the representative point of a certain frame image from another frame image; a camera work parameter calculation part 106 performing exception value decision by redundant use of a set of point correspondence relation of the image, and calculating a camera work parameter except an exception value; a representative point calculation decision part 107 deciding whether the calculation of the representative point is required or not; and a frame image update part 103 updating the frame image and the representative point. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a camera work parameter calculation method, an apparatus for implementing the method, a program, and a storage medium for the method, and particularly in the field of moving image processing, necessary for pre-processing of moving object extraction of accumulated video, sprite encoding, and the like. The present invention relates to a simple camera work parameter calculation method, an apparatus for implementing this method, a program, and a storage medium thereof.

  Non-Patent Document 1 discloses a technique for classifying feature points belonging to a background using a camera model when a three-dimensional object is projected onto a camera. By using the fact that the trajectory vector of the feature point belonging to the background exists in the partial space representing the camera model, the trajectory of each point is classified into the background and other trajectories. This method calculates projective transformation parameters corresponding to camerawork using only points with trajectories belonging to the background, converts the background to create a background panorama using the median method, and uses this background panorama to extract moving objects. I was going. Also, feature point matching is used for motion vector calculation. Note that the feature point calculation method associated with the screen change is when the number of feature points due to feature point tracking failure has decreased.

The processing steps for background point trajectory classification are briefly described below.
(1) Extract three-point trajectories from L-frame trajectory vectors (2 L-dimensional vectors connecting L point trajectories).
(2) A moment matrix M2 (2L × 2L dimensions) around the center of gravity of the extracted vector is calculated.
(3) A unit eigenvector corresponding to two eigenvalues having a large M2 is calculated.
(4) A 2L × 2L projection matrix is calculated from the eigenvectors.
(5) Calculate the total number of trajectory vectors for which the discriminant value calculated from the projection matrix is less than or equal to a certain value.
(6) Repeat (1) to (5) at another three points to obtain a projection matrix that maximizes the number calculated in (5), and further use the matrix to obtain a discrimination value similar to (5). Using these, the trajectory vectors are classified into background points and other points.
(7) Projective transformation parameters are calculated from the correspondence of the calculated background points using the renormalization method and set as camera work parameters.
Jyoda, Kanaya et al. "Extracting moving objects from moving video camera images" (Information Processing Society of Japan CVIM March 2004).

  The method of Non-Patent Document 1 requires two steps of calculating a point belonging to the background based on the camera model and calculating a projective transformation parameter corresponding to the camera work parameter using the point belonging to the background. Met. For this reason, there are disadvantages that the number of calculation steps increases and the processing speed becomes slow. In addition, after calculating a feature point tracking history of L frame length for a certain feature point, it is necessary to determine whether or not the point belongs to the background, and a sudden image change such as a cut point occurs in the middle. If this occurs and the corresponding points are miscorresponding more than a certain number, normal parameter calculation cannot be performed, and it is also difficult to specify the point where the change has occurred.

  The present invention has been made in view of the above points, and has as its object to provide a camera work parameter calculation method capable of detecting the time at which a rapid change of the screen occurs at a relatively high calculation speed. Yes.

  In order to solve the above-described problem, the camera work parameter calculation apparatus according to claim 1 inputs a necessary frame image, and updates a processed frame image and a representative point input to and output from a representative point calculation unit. A frame image update unit that outputs a frame image necessary for processing, a representative point calculation unit that inputs a frame image in the video as necessary, calculates a characteristic point as a representative point, and outputs its coordinates, The frame image after a certain number of frames from the frame image is input as a reference frame from the frame image update unit, the representative point coordinates are input from the representative point calculation unit, and the representative point calculated by comparing the two images A corresponding point calculation unit that calculates a corresponding point from the reference frame and outputs a corresponding set of coordinates, and a representative point and the calculated corresponding point are input from the corresponding point calculation unit, A camera work parameter calculation unit that calculates and outputs camera work parameters by removing points with low reliability by calculating the meter redundantly, and inputs the corresponding points and camera work parameters calculated as representative points. A representative point calculation determination unit that determines whether or not the calculation of the representative point is necessary, and outputs the necessity to the representative point calculation unit; Here, in particular, by including the camera work parameter calculation unit and the representative point calculation determination unit, it is possible to stably process an edited video including a cut point at a relatively high processing speed. .

  The camera work parameter calculation apparatus according to claim 2 inputs a necessary frame image, updates a processed frame image and a representative point input / output from the representative point calculation unit, and a frame image necessary for processing. A frame image update unit that outputs a representative point, a frame image in the video as necessary, a feature point that calculates a characteristic point as a representative point, and outputs the coordinates, and a frame image A plurality of frame images at a later video time are input as reference frames from the frame image update unit, and representative point coordinates are input from the representative point calculation unit, and the frame image between the representative point calculated and the plurality of reference images is input. And a corresponding point calculation unit for calculating a point corresponding to the representative point from a plurality of reference frames and outputting a plurality of pairs of the corresponding coordinates, and a corresponding point output to the representative point and the plurality of corresponding points. To determine which reference frame should be adopted, and to input the reference frame number, representative point, and corresponding point set frame interval determination unit, and input the representative point and calculated corresponding point from the corresponding point calculation unit The camerawork parameter calculation unit that calculates and outputs the camerawork parameter by removing the pair corresponding to the point with low reliability by calculating the camerawork parameter redundantly, and the corresponding point calculated by the representative point and the camerawork A representative point calculation determination unit that inputs a parameter, determines whether or not the calculation of a representative point is necessary, and outputs the necessity to the representative point calculation unit; Here, in particular, in the camera work parameter calculation device according to claim 2, in addition to the configuration of claim 1, the frame interval determination unit and the configuration corresponding thereto are provided to further improve the processing speed stably. To be able to.

  The camera work parameter calculation unit according to claim 3, wherein the point combination redundancy selection unit selects a number of point pairs more than the minimum number of point sets necessary for parameter calculation from combinations of representative points and calculated corresponding points. A temporary parameter calculation unit that calculates a plurality of camera work parameters, a parameter verification unit that determines whether the value is reliable by performing threshold processing based on the degree of dispersion of the variance value, and the camera work using only the reliable value A camera work parameter determination unit for determining parameters. Here, the camera work parameter calculation apparatus according to claim 3 is provided with a point combination redundancy selection unit, a provisional parameter calculation unit, a parameter verification unit, a camera work parameter determination unit, and all of them. An effect similar to that produced by Document 1, that is, calculation of camera work parameters with suppressed influence of the movement of the moving object can be executed at a relatively high processing speed.

  In addition, the camera work parameter calculation unit according to claim 4 inputs a representative point and the calculated corresponding point, and selects a point combination redundancy selection unit that selects more points than the minimum number necessary for parameter calculation from among them A temporary parameter calculation unit that calculates a plurality of camera work parameters, a parameter verification unit that determines whether the values are reliable by performing threshold processing based on the degree of dispersion of the variance values, and the camera work parameters using only reliable values A camera work parameter determination unit that determines the camera work parameter, inputs the determined camera work parameter, determines whether it is normal, and stores or outputs the abnormal case as an uncalculated time. To do. Here, in claim 4, in addition to the camera work parameter calculation device of claim 3, a camera work parameter normality determination unit is provided to detect a sudden change in the image such as a cut point and perform stable parameter calculation. This makes it possible to detect a video section in which parameter calculation is unstable or unstable.

  In order to solve the above-mentioned problem, the camera work parameter calculation method according to claim 5 inputs a necessary frame image and inputs / outputs the processed frame image and the representative point input / output to / from the representative point calculation unit. Frame image update procedure for updating the frame image and outputting the frame image necessary for processing, and if necessary, the frame image in the video is input, the characteristic point is calculated as the representative point, and the coordinates are output The frame image after a certain number of frames from the frame image is input as a reference frame from the frame image update unit, the representative point coordinates are input from the representative point calculation unit, and the representative image is calculated by comparison between the two images. A corresponding point calculation procedure for calculating a point corresponding to a point from the reference frame and outputting the corresponding coordinate pair, and a pair of the representative point and the calculated corresponding point according to the corresponding point calculation procedure The camera work parameter calculation procedure for calculating and outputting the camera work parameter by removing the pair corresponding to the point with low reliability by calculating the camera work parameter redundantly, and the corresponding point and the camera work calculated as the representative point And a representative point calculation determination procedure for determining whether or not the calculation of a representative point is necessary by inputting a parameter and outputting whether or not the representative point calculation procedure is necessary. Here, in particular, by having the camera work parameter calculation procedure and the representative point calculation determination procedure, there is an effect that an edited video including a cut point can be stably processed at a relatively high processing speed.

  The camera work parameter calculation method according to claim 6 inputs a necessary frame image, updates a processed frame image and a representative point inputted / outputted from a representative point calculation procedure, and a frame image necessary for processing. A frame image update procedure for outputting a representative point and a frame point in the video as necessary, calculating a feature point as a representative point and outputting the coordinates as necessary, and a frame point update procedure from the frame image A plurality of frame images at a later video time are input as reference frames from the frame image update procedure, and representative point coordinates are input from the representative point calculation unit, and the frame image between the representative point calculated and the plurality of reference images is input. The corresponding point calculation procedure for calculating the points corresponding to the representative points from the plurality of reference frames and outputting the corresponding pairs of coordinates, and the representative points and the plurality of outputs. Input corresponding points to determine which reference frame should be adopted, frame interval determination procedure for outputting the reference frame number and representative point, pair of corresponding points, and representative points and calculated corresponding points as corresponding points Camerawork parameter calculation procedure that calculates and outputs camerawork parameters by removing the pair of low-reliability points by inputting the calculation procedure and calculating camerawork parameters redundantly, and the corresponding points calculated as representative points A representative point calculation determination procedure for inputting a point or a camera work parameter, determining whether or not a representative point calculation is necessary, and outputting whether or not the representative point calculation is necessary. Here, in particular, in the camera work parameter calculation method according to claim 6, in addition to the configuration of claim 5, the frame interval determination procedure and the corresponding procedure are provided, thereby further improving the processing speed stably. To be able to.

  8. The camera work parameter calculation method according to claim 7, wherein a point combination redundancy selection procedure for selecting a number of point pairs more than a minimum number of point pairs necessary for parameter calculation from combinations of representative points and calculated corresponding points. A provisional parameter calculation procedure for calculating a plurality of camera work parameters, a parameter verification procedure for determining whether the values are reliable by performing threshold processing based on the degree of dispersion of the variance values, and the camera work using only reliable values. And a camera work parameter determining procedure for determining the parameter. Here, the camera work parameter calculation method according to claim 7 has the above-described point combination redundancy selection procedure, provisional parameter calculation procedure, parameter verification procedure, and camera work parameter determination unit, all of which, The effect similar to the effect of No. 1, that is, the calculation of the camera work parameter while suppressing the influence of the movement of the moving object can be executed at a relatively high processing speed.

  A camera work parameter calculation method according to claim 8 is a point combination redundancy selection procedure in which a representative point and a calculated corresponding point are input, and more points than the minimum number necessary for parameter calculation are selected therefrom. , A provisional parameter calculation procedure for calculating a plurality of camera work parameters, a parameter verification procedure for determining whether the values are reliable by performing threshold processing according to the degree of dispersion of the variance values, and the camera work parameters using only reliable values A camera work parameter determination procedure for inputting the determined camera work parameter, and determining whether the camera work parameter is normal and storing or outputting the abnormal case as an uncalculated time. It is characterized by that. Here, in claim 8, in addition to the camera work parameter calculation method of claim 7, by having a camera work parameter normal determination procedure, it is possible to detect a sudden change in an image such as a cut point and to calculate a stable parameter. Alternatively, it is possible to detect a video section in which parameter calculation is unstable.

A camera work parameter calculation program for instructing a computer to execute a camera work parameter calculation method according to any one of claims 5, 6, 7, and 8 as claim 9. It is characterized by comprising.
According to a tenth aspect of the present invention, a storage medium storing the camera work parameter calculation program according to the ninth aspect is configured.

  According to the present invention, corresponding points between frames are calculated by performing representative point matching using feature points as representative points, and camera work parameters are calculated. Note that it is known that relatively stable tracking can be performed by selecting a corner point having a luminance gradient of a certain degree or more on the image, such as a square corner, as a point on the image calculated as a feature point. ing. Next, inter-frame camera work parameters such as projective transformation are calculated from the correspondence between the frames. In order to perform robust camera work parameter calculation, camera work parameters are calculated redundantly and outliers are removed. An outlier is a point trajectory that has a trajectory that cannot be considered as a background point, and is a trajectory on a moving object or a feature point on a moving object. That is, when calculating camera work parameters such as projective transformation, simultaneous linear equations are solved, but a plurality of temporary camera works are used by using more points than the representative points and corresponding points necessary for the calculation. The camera work parameters are calculated by reducing the outliers by calculating the parameters, and determining that the calculated camera work parameters have large variations as inappropriate camera work parameters. The number of camerawork parameters to be tentatively calculated may be any number that is less than the maximum number that can be calculated and greater than two. If the moving object moves in a way that does not apply to the camera work model, or if there are a sufficient number of representative points in the background, the camera work parameters can be set even if the majority of the pixels are occupied by pixels from the moving object. Can be calculated. Although this effect is also non-patent document 1, the present invention has invented a calculation method with relatively low calculation cost.

Regarding the processing speed, in addition to the above representative point matching, the frame interval in motion vector calculation is made variable every few frames instead of every frame, and in the case of frames with little motion or almost no camera work, skip multiple frames. Speed is improved by calculating camera work parameters. In addition, since the calculation of parameters is performed simultaneously with the removal of outliers using the correspondence between representative points between two frames, the calculation cost can be reduced compared to Non-Patent Document 1.
The calculation cost is compared with Non-Patent Document 1. In Non-Patent Document 1, there is a step of calculating eigenvalues and eigenvectors of a 2M × 2M dimensional matrix for background point classification used for outlier removal, and this calculation order is said to be O (M ² ). This order processing is required for a set of three feature points. On the other hand, the calculation order in the outlier removal at the time of camera work parameter calculation in FIG. 4 described later is O (M) in M frames for a set of three representative points. This is a calculation order not including camerawork parameter calculation described later. It is understood that both the processes for solving the simultaneous linear equations for calculating the camera work parameters are necessary in the same way, and that the process is fast in terms of outlier removal processing.

  As described above, it is possible to calculate camera work parameters that exclude outliers at a relatively high calculation speed. Then, a cut point where camerawork parameters cannot be calculated, or a time when a sudden image change occurs can be calculated at the same time.

Hereinafter, a more specific description will be given with reference to the illustrated embodiment.
FIG. 1 is a block diagram for explaining an embodiment of a camera work parameter calculation apparatus corresponding to claim 1.
The video storage unit 101 is a storage part that stores video to be input to the camera work parameter calculation device and to be processed.
The video input unit 102 is a device that inputs a video selected from the video storage unit 101 by a user's keyboard command or the like, or a video automatically selected from the video storage unit 101. If the input video is encoded, it is decoded using each existing video encoding technique and divided into frame images for input.

  The frame image update unit 103 is a part that selects and outputs a frame image to be input to the corresponding point calculation unit 105 in the next stage from the video input via the video input unit 102 or stores it in the memory 100. . In the initial reading by the frame image update unit 103, the processing target frame is the one with the smallest frame number (video time) in the memory 100, and the next frame is processed with the reference frame as the next frame or a subsequent frame image. I do. In the updating loop processing, the frame image before the current reference frame is cleared, the current reference frame is set as the next processing target frame, and the frame image to be the next reference frame image is input from the video input unit 102. Thus, the reference frame image is also input.

  The representative point calculation unit 104 is a device for inputting a processing target frame image, searching for a feature point from the image, and storing the coordinates as representative point coordinates. This process can be realized by an existing technique such as the technique described in Section 4 of Non-Patent Document 2. That is, for example, by selecting a desired number of points having a smaller eigenvalue of a 2 × 2 luminance gradient matrix of an image that is equal to or greater than a predetermined value from a larger value, a feature point corresponding to a corner is determined as a desired upper limit score. It is possible to calculate only. As a feature point calculation technique, it is of course possible to use other existing methods such as simply using a point having a large absolute value in both the x and y directions of the luminance gradient of the image as a criterion for determining a representative point. . The processing in this apparatus is performed when the representative point calculation flag is set in the read initial frame, the corresponding point calculation unit 105 in the subsequent stage, the representative point calculation determination unit 107, or the like.

The corresponding point calculation unit 105 is a part that searches the reference frame image for a point corresponding to the representative point on the input processing target frame image. This processing can be realized by using, for example, the technique described in Section 3 of Reference Document 1. Further, by performing threshold processing on the dissimilarity (“dissimilarity”) defined by the above method, it is possible to determine a point that has disappeared from the image. The coordinates of the feature points of the lost processing target frame may be deleted.
As another example of the calculation of corresponding points, a motion compensation vector in a macroblock to which the representative point coordinates belong and an image that has been encoded using block matching or motion compensation using the peripheral pixels of the representative point and its It is of course possible to use other methods for calculating corresponding points, such as using the sum of coordinates. Here, it can be determined that points whose evaluation values corresponding to points, such as the sum of residual absolute values of neighboring pixels, are below a certain threshold value have disappeared.

The camera work parameter calculation unit 106 is a part that calculates a camera work parameter from a set of corresponding points calculated between two frames of a processing target frame and a reference frame. This operation will be described in detail later with reference to the flowcharts of FIGS.
The representative point calculation determination unit 107 is a part that receives the result of the corresponding point calculation unit 105 and determines whether or not the feature point of the current frame should be calculated again. This operation will be described in detail later with reference to S204, S206, and S209 of FIG.
The processing result output unit 108 is a part that stores camerawork parameters, coordinates of extracted objects, feature points, frame images, and the like in each frame in a storage device or displays them so as to be visible to the user. In the memory 100, a processing target frame image, a reference frame image, a representative point calculation flag, a representative point coordinate set, a corresponding point coordinate set, and the like can be stored.

In addition, the data that flows through each connection is first given an ID, etc., so that it is possible to determine where the data should be processed. Each part inputs only its own data and ignores the rest. Data control shall be performed.
FIG. 2 is a flowchart describing the contents of a specific processing procedure of the camera work parameter calculation apparatus of FIG. This is also a specific description of the camera work parameter calculation method corresponding to the combination of the fifth embodiment and the seventh embodiment. In this flowchart, the retry flag is not used. The retry flag is used in FIG.

  (S201) The video is read from the video storage unit 101 to the video input unit 102, converted into an image for each frame by the video input unit 102 and output, and the initial processing target frame image (first frame) of the video and the initial The reference frame image (frame image of frame number F + 1, F is a natural number constant) is input to the memory. Also, the representative point calculation flag on the memory 100 is turned on. The processing target frame image and the reference frame image are output to the frame image update unit 103, and the process proceeds to S202.

   (S202) This is a specific example of the representative point calculation procedure of claims 5 and 6. The processing target frame image and the reference frame image stored in the memory 100 by S201 or S207 are input to the representative point calculation unit 104, and feature points on the processing target frame image are calculated when the representative point needs to be calculated. The coordinates are stored in the memory 100 as the representative point P = [p (i)]. This feature point calculation technique can be performed using Section 4 of Non-Patent Document 2 described above. It should be noted that the feature points can be calculated using other existing methods such as simply using the local maximum point of the absolute value in both the x and y directions of the luminance gradient of the image as a representative point. The number Num (P) of feature points obtained here is stored in the memory 100. Also, the representative point calculation flag is turned off. The processing target frame image, the coordinates of the representative point, and the reference frame image are output to the corresponding point calculation unit, and the process proceeds to S203. The contents of the update process in the frame image update unit 103 will be described in the description of S207. In addition, it is determined in step S209 whether the representative point needs to be calculated.

  (S203) This is a specific example of the corresponding point calculation procedure of claim 5. This process is a process executed by the corresponding point calculation unit 105. The processing target frame image and the reference frame image are input from the memory 100 and the representative point coordinates thereof are input from the representative point calculation unit, respectively, and a point corresponding to the representative point coordinates in the processing target frame image is searched from the reference frame image and calculated. The coordinates of the corresponding points are stored in the memory 100. For this procedure, the technique described in Section 3 of Non-Patent Document 2 can be used. In addition, the coordinates of the representative point of the lost processing target frame can be deleted. Another example of the corresponding point calculation is a motion compensation vector in a macroblock to which the representative point coordinates belong and an image that has been encoded using block matching or motion compensation using the peripheral pixels of the representative point, It is of course possible to use other methods for calculating corresponding points, such as using the sum of coordinates. Here, it can be determined that points whose evaluation values corresponding to points, such as the sum of residual absolute values of neighboring pixels, are below a certain threshold value have disappeared. In any example, the coordinates of the corresponding points calculated are stored in the memory 100, and the number Num (P ′) of the corresponding points is output to the representative point calculation determination unit 107. The representative point coordinates and the corresponding point coordinates are also output. Is output to the camera work parameter calculation unit 106, and the process proceeds to S204.

  (S204) This is a specific example of the representative point calculation determination procedure of claim 5. This process is a part of the process executed by the representative point calculation determination unit 107. The corresponding points and the number of corresponding points calculated in S203 are input from the memory 100, the process skip determination in this frame is performed, and whether or not the representative point recalculation is necessary is determined. First, a skip determination is made based on whether or not the number of corresponding points is extremely reduced. That is, if the number of corresponding points is smaller than a threshold different from the above, it is determined to skip, and the process proceeds to S206, and otherwise, the process proceeds to S205. As an example of the threshold value in the skip determination, the minimum score (numerical value M ′ in FIG. 3 to be described later) necessary for camera work parameter calculation in the next step S205, a value obtained by multiplying by a certain number of 1 or more, or a stored number It is possible to use various numerical values such as a value obtained by multiplying Num (P ′) by a certain number (0 to 1) or a certain number. Next, it is determined whether it is necessary to recalculate the representative points. What is used for this determination is, for example, the condition of the number of corresponding points and the positional relationship of the corresponding points. As a condition for the number of corresponding points, when the calculated number of corresponding points is larger than a certain numerical value Vpn, it is determined that it is not necessary to recalculate the representative points in the next processing, and when the number is less than Vpn, it is necessary to reacquire the representative points. judge. Here, as an example, a value obtained by multiplying the number of corresponding points Num (P ′) stored in S203 by a real number from 0 to 1 is adopted as Vpn. In addition, as a condition of the positional relationship of corresponding points, for example, it is necessary to calculate the distance to the adjacent corresponding point for each calculated corresponding point, and to re-represent the representative point in the next processing when the intermediate value falls below the threshold value It is determined that there is. As an example of the positional relationship condition, the minimum value of the variance value of each of the x and y components of the corresponding point coordinates may be used. As an example of the threshold value of the position condition, for a representative point calculated in S202, a certain ratio of the value of any of the positional relationship conditions can be used. As another example of the positional relationship between corresponding points, as described above, the average value of distances to adjacent points is equal to or less than a threshold value, or the sum of variance values for corresponding point coordinates x and y, or a small value. There is an example in which it is determined that the representative point needs to be taken again when the larger value or the larger value exceeds the threshold value. If it is determined that the representative point needs to be retaken according to either the number of corresponding points or the positional relationship, the representative point calculation flag is set (ON). Alternatively, the representative point recalculation determination may be performed using only one of the condition of the number of corresponding points and the condition of the positional relationship of the corresponding points.

The above conditions for the number of corresponding points are that many of the representative points are hidden behind the moving object in the video, and many corresponding points cannot be calculated. This is a determination process for re-taking representative points once the number of representative points has decreased too much. In addition, when the number of corresponding points is extremely reduced, it is considered that there is an abrupt discontinuity of the image such as a camera work or a cut point. In this case, since the camera work parameter cannot be calculated, the processing is skipped.
When the corresponding points are calculated using the technique described in Section 3 of Non-Patent Document 2, the corresponding points are concentrated along with the moving object when the moving object moves on the screen. Sometimes. At this time, most of the corresponding points may exist on the moving object at most, and in this case, it is impossible to calculate appropriate camerawork parameters, so the determination to re-represent the representative points It is intended to be used as

  (S205) The processing content corresponding to the camera work parameter calculation procedure of claims 5 and 6. This processing is the processing content of the camera work parameter calculation unit 106, and is a step of calculating camera work parameters from the representative points calculated in S203 and the corresponding points. The coordinates of the representative points and the coordinates of the corresponding points are input from the memory 100 to the camera work parameter calculation unit 106 to perform processing. Take more representative points than the minimum necessary for camerawork parameter calculation, apply them to the camerawork model, solve the linear equation multiple times, calculate multiple camerawork parameters, and calculate the variance of these multiple values. Such a value representing the degree of variation is subjected to threshold processing to calculate camera work parameters that exclude outliers such as moving objects and corresponding point calculation errors. Details of this processing will be described later with reference to FIG. The calculated camera work parameters are stored or output, and the process proceeds to S207. Further, the calculated camera work parameters, representative point coordinates, and corresponding point coordinates are output to the processing result output unit 108 and the representative point calculation determination unit 107, respectively.

(S206) In order to recalculate the representative point, the representative point calculation flag is turned on and the process proceeds to S207.
(S207) This is a specific example of the frame image update procedure according to claim 5. This is a processing procedure of the frame image update unit 103, and the frame number to be input next is determined at the timing when the signal is output by the operation of the representative point calculation determination unit 107. As for the processing contents, the memory 100 is updated with the current reference frame as the processing target frame in the next loop and the calculated corresponding point as the representative point in the next loop. Further, the frame image after the F frame next to the updated processing target frame is read into the memory 100, and the reference frame image is also updated. In this way, reading of frame images is skipped every F frames (F: natural number constant). That is, when the image read last time is an image of frame number f, it is assumed that the image read next is an image of frame number f + F. The frame number to be read next is output to the video input unit 102.

  (S208) It is determined whether there is a frame image to be read next, that is, whether the image has been read to the end of the video. This determination is performed as a function of the video input unit 102 when a frame number to be read from the frame image update unit 103 is input to the video input unit 102. If it is determined that the video has been read to the end, the process ends. In other cases, the video input unit 102 outputs a frame image having the requested frame number to the frame image update unit 103. Specifically, it is determined whether the value obtained by adding F to the frame number of the current reference frame does not exceed the total number of frames of the video. If not, the process proceeds to S209, and if it exceeds, the process ends.

(S209) This is a specific example of the representative point calculation determination procedure according to claim 5. This process is a process executed by the representative point calculation determination unit 107. In this step, it is determined by a representative point calculation flag whether or not it is necessary to recalculate the representative point. Here, whether or not the representative point calculation flag is set (ON) is determined with reference to the memory 100. If ON, the process proceeds to S202, and if OFF, the process proceeds to S203.
FIG. 9 is a diagram for explaining the details of the configuration of the camera work parameter calculation unit in FIG. Moreover, it is also a block diagram explaining the Example corresponding to Claim 3. Details of the operation will be described with reference to FIG.

In the point combination redundancy selection unit 901, a representative point and a set of corresponding points are input, and more points than the number necessary for camera work parameter calculation are selected from them, and whether parameter calculation is possible with the set of points. It is a device that determines and outputs the set if possible. The combination of points once selected is stored so as not to be selected twice. Details of the operation of this part will be described in S299, S300, S302, S309, and S310 of FIG.
The provisional parameter calculation unit 902 selects a set necessary for parameter calculation from the redundant point pairs selected by the point combination redundancy selection unit 901, calculates provisional camera work parameters, and outputs those values. It is a part to do. Further, a temporary parameter storage table 903 is provided as a table for storing the combinations of the representative points and the calculated temporary parameter values. This table is cleared when the final camera work parameter is output by the camera work parameter determination unit 905 and the processing target frame is changed. Details of the operation of this part will be described in S303 to S306 of FIG.

The parameter verification unit 904 verifies the variation of the plurality of temporary camera work parameters calculated by the temporary parameter calculation unit based on the degree of variation such as the variance value, and determines whether these temporary parameters are reliable values. If the values are reliable, those temporary parameters are output to the camera work parameter determination unit 905.
The camera work parameter determination unit 905 calculates one provisional camera work parameter from the camera work parameters determined to be reliable by the parameter verification unit, and the point combination redundancy selection unit 901 completes the processing for all combinations of points. The apparatus determines one camera work parameter from all the provisional camera work parameters calculated after the operation.

  FIG. 3 is a detailed flowchart for explaining the operation of FIG. 9 and the operation of S205 of FIG. 2, that is, the operation of the camera work parameter calculation apparatus according to claim 3. This is also a specific explanation of the camera work parameter calculation method described in claim 7. The input is a representative point set P = [p (i)] (i = 1, 2,..., Np) and its corresponding point set P ′ = [p ′ (i)] (where p (i) and p ′ (i) ) Is supported). The output is camerawork parameter C. The camerawork parameter is calculated using the representative point p = (x, y, 1) and its corresponding point p ′ = (x ′, y ′, 1) (both are in the same coordinate system). It is a linear transformation matrix such as a projective transformation expressed in Equation 1. In the case of the camera work model of Equation 1, the camera work parameter number M is 9, and the minimum score NR required for parameter calculation is 5. A specific example will be described later with reference to FIG.

(S299) The counter n is initialized with 1.

  (S300) This process is executed by the point combination redundancy selection unit 901, and is a specific example of the point combination redundancy selection procedure according to claims 7 and 8. A set of representative point coordinates P (referred to as a representative point set) and a corresponding set of corresponding point coordinates P ′ (referred to as a corresponding point set) are input first, and the processing is started by the input. Arbitrary NQ point sets Q = [q (j)] (where NQ> NR and j = 1, 2,..., NQ) from the representative point set P for which corresponding points have been calculated not to overlap. Take out. Similarly, a set Q ′ of corresponding points of Q extracted is also extracted. In the second and subsequent steps of S300, point sets Q having different combinations are extracted. The number NQ of points extracted in S300 is set to be larger than the number NR required for parameter calculation. The coordinates of Q and Q 'are stored in the memory, and the coordinates of Q and Q' are output to S302.

(S301) The counter m is initialized with 0, and the process proceeds to S302.
(S302) This process is executed by the point combination redundancy selection unit 901, and is a specific example of the point combination redundancy selection procedure according to claims 7 and 8. It is determined whether or not the parameter can be stably calculated using Equation 1 based on Q input from S301. That is, when any three points in Q are on the same straight line, the calculation is unstable.
For example, when arbitrary three points in Q are q ₁ = (x ₁ , y ₁ ), q ₂ = (x ₂ , y ₂ ), q ₃ = (x ₃ , y ₃ ), Th is given It is determined that the calculation is unstable when the condition of the following expression is not satisfied as the threshold value.
| (X ₃ −x ₁ ) (y ₂ −y ₁ ) − (y ₃ −y ₁ ) (x ₂ −x ₁ ) | / (| q ₃ −q ₁ | · | q ₂ −q ₁ |) / 2> Th (Th is a threshold value. Here, a constant is used as an example).
Incidentally, the meaning of the left side of the above equation is the sin value of the angle between the vector of q ₁ is the q _2, q ₃ of origin. Further, the condition of the distance between each point | q ₃ −q ₁ | <Th _d , | q ₂ −q ₁ | <Th _d (Th _d is a threshold value, a constant) can be further added as a condition. As another example of the distance condition, there is a method in which threshold processing is performed using a distance from the center of gravity of a triangle formed by three points. Further, it may be determined whether or not calculation is possible by an arbitrary logical operation of two conditions of distance and straight line determination.
If either of the above two conditions is not satisfied and the calculation is determined to be unstable, the process returns to S301 and the point set Q is selected again. Otherwise, Q and Q ′ are output and the process proceeds to S303. When determining that calculation is possible, the point combination redundancy selection unit 901 outputs Q and Q ′ to the provisional parameter calculation unit 902 in this processing step.

  (S303) Part of the processing executed by the provisional parameter calculation unit 902, and is a specific example of the provisional parameter calculation procedure according to claims 7 and 8. Q and Q ′ are input from the point combination redundancy selection unit 901. NR = [M / 2 if M is an even number, or (M + 1) / 2] if M is an odd number as the point set R, which is the number of points required for parameter calculation from the point sets Q and Q ′ taken out in S300 The corresponding point is extracted as R ′, and R and R ′ are output to S305. The temporary parameter calculation unit 902 stores R and R ′ in the memory.

Looking at Formula 1, two sets of formulas are established for a set of representative points and corresponding points, so the number of points required to calculate M parameters is the previous NR. In the second and subsequent steps of this step, point sets R with different combinations are taken out.
(S304) The counter m is incremented and the process proceeds to S305.

(S305) This is a part of the processing executed by the provisional parameter calculation unit 902, and is a specific example of the provisional parameter calculation unit according to claims 7 and 8. R and R ′ stored in S303 are used. Camera work parameters are calculated based on Equation 1 using R and R ′. That is, when Equation 1 is transformed to Equation 2, a linear equation from a _{11 to} a ₃₃ is obtained. Therefore, by substituting the values of R and R ′ for x, y, x ′, and y ′, respectively, the values of the respective parameters can be calculated when the required number of expressions is the same. The solution of the linear equation is performed using any existing method such as LU decomposition and regression.

x ′ · (xa ₁₃ + ya ₂₃ + a ₃₃ ) = (xa ₁₁ + ya ₂₁ + a ₃₁ )
y ′ · (xa ₁₃ + ya ₂₃ + a ₃₃ ) = (xa ₁₂ + ya ₂₂ + a ₃₂ ) (Formula 2)
The calculated value is stored in a location corresponding to the combination of the point number i of p (i) in the temporary parameter storage table 903, and the calculation of the same set of points is performed only once. Number of solving the above equation for this case is _NP C _NR times at maximum. The camera work parameter matrix calculated here is Cnm (one camera work parameter matrix Cnm per one combination of m and n). The calculated camera work parameter matrix nm is stored in the memory 100.

(S306) This is one of the processes executed by the provisional parameter calculation unit 902. It is determined whether R has taken all the combinations of a certain subset of Q. If all the certain subsets are taken, the camera work parameter matrix Cnm is output, and if not, the process proceeds to S307. Note that the total number of subsets R extracted from Q is _NQ C _NR combinations. As a method of taking a certain subset, for example, all of this total number may be taken, or a subset ( _NQ C _NR-1 ways) in which a certain point is necessarily combined may be taken.
Further, when all the certain subsets are taken, the temporary parameter calculation unit outputs the camera work parameter matrix Cnm to the parameter verification unit 904.

  (S307) This is a function of the parameter verification unit 904, and is a specific example of the parameter verification procedure according to claims 7 and 8. The camera work parameter Cnm is input from the temporary parameter calculation unit 902. In this processing step, the reliability of the calculated parameter is determined. When at least one point in the extracted point set R has a different motion from the camera work model represented by Equation 1, the calculated parameter value varies depending on m. In this case, since the calculated parameter value is also a low reliability value, it is not used for the final parameter calculation (S311).

For example, the reliability of each component of the parameter matrix Cnm is determined by the magnitude of the variance value for m. If it exceeds a certain threshold, the value is discarded as a parameter with low reliability and the process proceeds to S310. If it is equal to or less than a certain threshold, the parameter is stored as a parameter with high reliability and the process proceeds to S308.
As another example of the value for determining the reliability, there is a method in which a difference between parameters is taken for two m having different Cnm, and the m _0th value from the smallest difference value is used for each parameter. There is also a method of randomly selecting the difference value and using the value for the determination.

(S308) This is a part of processing executed by the camera work parameter determination unit 905, and is a specific example of the camera work parameter determination procedure according to claims 7 and 8. An average value is calculated for each component of the parameter matrix Cmn calculated from the representative point set Q, and stored as a matrix Cn.
If it is determined in S307 that the reliability is high, the parameter verification unit 904 outputs the matrix Cn to the camera work parameter determination unit 905.
(S309) The counter n is incremented and the process proceeds to S310.

(S310) This is a part of the processing executed by the point combination redundancy selection unit 901, and is a specific example of the point combination redundancy selection unit 901 according to claims 7 and 8. It is determined whether or not the partial point set Q has taken all possible combinations of the point set P. If the partial set Q has been taken, the process proceeds to S311; otherwise, the process proceeds to S300. Note that the total number of subsets Q extracted from P is only _{NP CNQ} combinations.

  (S311) This is a part of processing executed by the camera work parameter determination unit 905, and is a specific example of the camera work parameter determination procedure according to claims 7 and 8. When a signal indicating that the parameters for all the points have been calculated from the point combination redundancy selection unit 901, processing is performed using Cn in the memory 100. The camera work parameter C is calculated from Cn. In the calculation method, for example, an intermediate value of each component of Cn is C. As another example of determining C, there is a method of setting a mode value in a histogram after quantizing each parameter of Cn components, or an average value.

  In addition, the determination in S307 causes all the calculated camera work parameters to be discarded because they are low in reliability. If Cn is empty, it is determined that the calculation is impossible, and the camera work parameter value indicating abnormality is output. The value of the camera work parameter indicating an abnormality may be any value that cannot be normally calculated as a camera work parameter, such as a parameter matrix in which all components are zero. This value is used when abnormality determination is performed in S502 of the embodiment of FIG. In the embodiment of FIG. 2, if the calculation is impossible, the process may be forcibly terminated, or the calculation may be terminated by outputting a predetermined parameter value.

The camera work parameter determination unit 905 outputs the camera work parameter value C calculated as the output of the camera work parameter calculation unit 106 as described above.
FIG. 4 is a flowchart showing a specific example when the camera work model of FIG. 3 is represented by Equation 3 (M = 4, NR = 2) and NQ = 3. Here, a ₁ is a zoom parameter, a ₂ is a rotation parameter, a ₃ is an x-axis movement parameter, and a ₄ is a y-axis movement parameter.
(S400) The counter n is initialized.

  (S401) Part of the processing executed by the point combination redundancy selection unit 901. The representative point set P and its corresponding point set P ′ are input. Arbitrary three point sets Q = [q (1), q (2), q (3)] from the representative point set P and their corresponding points Q ′ = [q ′ (1), q ′ ( 2), q ′ (3)] are extracted and P, P ′, Q, and Q ′ are output. In the second and subsequent steps of this step, point sets Q and Q 'having different combinations are taken out.

  (S402) This is part of the processing executed by the point combination redundancy selection unit 901. It is the same processing as S302, and it is determined whether or not the parameter can be calculated using Equation 3 by Q. That is, Q and Q ′ are input, and the calculation instability determination is performed using the three points q (1), q (2), and q (3) on the same basis as S302. If the calculation is unstable, the process returns to S401. Reselect the point set Q. Otherwise, Q and Q 'are output and the process proceeds to S403.

  (S403) This is a part of the processing executed by the provisional parameter calculation unit 902. Q and Q ′ are input, and all combinations of two points, which are points necessary for parameter calculation, from Q and corresponding points Q ′ are obtained. Output along with the corresponding extraction point. That is, [q (1), q (2)], [q (2), q (3)], [q (3), q (1)]. Note that the set of these two points corresponds to R in FIG.

(S404) This is part of the processing executed by the provisional parameter calculation unit 902. Three kinds of two-point combinations, which are all the extracted combinations, are input, and camera work parameters Cn _{1 to} Cn ₃ (Cnm, m = 1, 2, 3 in FIG. 3) are calculated based on Equation 3. In other words, when Equation 3 is transformed into Equation 4, a linear equation from a _{1 to} a ₄ is obtained, and the values of the representative point (x, y) and its corresponding point (x ′, y ′) are substituted into Equation 4. Then, since it becomes two linear equations, four equations can be established using the three two-point combinations extracted in S403, and the value of one parameter matrix can be calculated. Existing methods are used to solve simultaneous linear equations. The parameter matrix calculated from [q (1), q (2)] is Cn ₁ , the parameter matrix calculated from [q (2), q (3)] is Cn ₂ , [q (3), q The parameter matrix calculated from (1)] is expressed as Cn ₃ . Each calculated parameter matrix is stored in a table, and thereafter, the parameter matrix of the calculated set of points is referred to the table. As a result, the total number of times to solve this equation is Np (Np-1) / 2.
x′a ₁ + ya ₂ −a ₃ −x = 0
y′a ₁ −xa ₂ −a ₄ −y = 0 (Formula 4)

(S405) This is a process executed by the parameter verification unit 904. The calculated Cn ₁ to Cn ₃ are input, and | Cn ₁ -Cn ₂ |, | Cn ₂ -Cn ₃ |, | Cn ₃ -Cn ₁ | The minimum value is taken out for each parameter in, and the value is measured. If there are two or more of the three differences whose values are greater than a certain threshold value, the reliability is low and the process proceeds to S407. If the number is less than 1, Cn ₁ to Cn ₃ are output because the reliability is high, and the process proceeds to S406. The parameter threshold value may be set for each parameter component. Note that a variance value may be calculated for each component of the parameter to measure the level of reliability, or an arbitrary difference value such as | Cn ₁ −Cn ₂ | may be used alone to determine the magnitude of the difference value of each parameter. You can measure the level of reliability.

(S406) This is a part of the function of the camera work parameter determination unit 905. The calculated parameter matrices Cn _{1 to} Cn ₃ are input, the average value is calculated for each of the components, and these are stored in the memory as the matrix Cn. Further, the counter n is incremented and the process proceeds to S407.
(S407) This is part of the processing executed by the point combination redundancy selection unit 901. It is determined whether or not the partial point set Q has taken the entire combination of the point set P. If the partial set Q is taken, Cn is output and the process proceeds to S408, and if not, the process proceeds to S401. Note that the total number of selection methods from P to Q is the maximum number of Cn, and the number is as follows: _{NPC 3} = NP (NP-1) (NP-2). Here, it is possible to determine whether or not a predetermined subset has been taken, instead of determining with Q being the total combination of P.

(S408) This is part of the processing executed by the camera work parameter determination unit 905. Cn is input, and camera work parameter C is calculated and output. Here, as an example, an intermediate value of each component of Cn is adopted as C. As another example of the calculation method of Cn to C, the average value of Cn may be C, or C may be determined by calculating the mode value after quantization for each parameter of Cn.
The camera work parameter determination unit 905 outputs the camera work parameter value C calculated as the output of the camera work parameter calculation unit 106 as described above.

As another method for calculating camera work parameters in S404, parameters a- _{1 to a- 4} are calculated from two points [q (1), q (2)], and the remaining one point q (3) is S405. There is a method used for error determination. At this time, the following changes are made at the same time in S405 and S406. In S405, an error when q (3) is applied to Equation 3 is calculated as an error, and error determination is performed by thresholding it. That is, the magnitude of the error is determined by thresholding the value of Equation 5 as an error. In S406, there are a method of using the calculated value as it is as the parameter Cn, a method of recalculating Cn ₁ to Cn ₃ and taking the average value in the same manner as the method described in FIG.

_{Max (| x'a~ 1 + ya~ 2} -a~ 3 -x |, | y'a~ 1 -xa~ 2 -a~ 4 -y |)
(Formula 5)
FIG. 10 is a flowchart in which a camera work parameter normality determination unit is added to FIG. Only the camera work parameter normality determination unit 1006 added to FIG. 9 will be described.
The camera work parameter normality determination unit 1006 is a part for determining whether or not the calculated camerawork parameter is normal. The operation of this part will be described in detail in S502.

  FIG. 5 shows an embodiment corresponding to the combination of claim 1 and claim 4, and is a flowchart in which the camerawork parameter normality determination is added to the flowchart of FIG. Similarly, the flowchart is a specific description of the camera work parameter calculation method corresponding to the combination of claim 5 and claim 8. That is, the calculation apparatus itself that executes the flowchart of FIG. 5 is a camera work parameter calculation apparatus that is substantially the same as that of FIG. 1, but the apparatus of FIG. 5 is illustrated as a camera work parameter calculation unit 106 in FIG. 10 instead of FIG. Use the camerawork parameter calculator described. The retry flag of the contents of the memory 100 is used. The processing content in the flowchart of FIG. 5 is added with S502 which is an additional step in the processing of FIG. In the following, only processes that are changed or added from FIG. 2 will be described with reference to FIGS. 1 and 10 as well.

  (S501) A video is input from the video storage unit 101 to the video input unit 102, converted into an image for each frame by the video input unit 102, and output. The initial processing target frame (first frame), and then F An initial reference frame (frame image of frame number F + 1) that is a frame image after the frame is input to the memory. Further, the representative point calculation flag and the retry flag are turned off. The processing target frame and the reference frame are output, and the process proceeds to S202.

  (S502) This process is executed by the camera work parameter normality determination unit 1006 in FIG. 10, and is a specific example of the camera work parameter normality determination procedure according to claim 8. The calculated camera work parameter is input to determine whether the value is normal. As an example of the judgment method, if the camera work parameter calculated in the previous loop is normal, the sum of squares of the difference between that value and the current calculated value is taken, and the case where the value does not exceed the threshold is normal . The case where the camera work parameter calculated in the previous loop is determined to be normal and the value exceeds the threshold is regarded as abnormal. If the camera work parameter calculated in the previous loop is determined to be abnormal, whether the parameter value is within a certain range is determined as normal or abnormal. Although the previous loop does not exist in the initial frame, the camera work parameter calculation of the previous loop is regarded as an abnormality determination, and is determined in the same manner. When the camera work parameter calculated in the previous loop becomes abnormal, the same processing as in the case of the initial frame is performed. As another example of the determination method, there is also a method of determining whether the calculated parameter value is normal or abnormal only by whether or not each value of the calculated parameter is within a certain range. In any example, if it is determined to be normal, the process proceeds to S207, and if it is determined to be abnormal, the process proceeds to S512.

  When many representative points are hidden behind sudden camerawork, cut points in edited video, etc., or behind the moving object in the video, etc., or the cause is several frames When the number of representative points is reduced, the corresponding points decrease due to the accumulation of the cause between frames, or the video time of this frame is a cut point. In order to determine whether the frame is possible, the determination process for calculating the representative point again and performing the corresponding point search is S204, and the flag necessary for not performing the same recalculation is the retry flag. The skip determination of S204 is a determination when the corresponding point search has failed and the number of points has become extremely small. In this determination of S507, many miscorresponding points occur in the calculation of the corresponding points. This is a determination condition when discontinuity cannot be determined. This result is output in S514.

(S512) This step corresponds to when the calculated corresponding point or the camera work parameter takes an abnormal value. It is determined whether the retry flag is set (ON). If it is set, the process proceeds to S514. If it is not set, the process proceeds to S513. The retry flag indicates that the corresponding point or the camera work parameter is determined not to be calculated in S204 and S502 and indicates that the representative point has been recalculated and retraced, and is a flag necessary for not performing the same process twice.
(S513) When the calculated camera work parameter is abnormal, in order to test whether or not the cause is a decrease in the representative point and the corresponding point in the background portion, the representative point is taken again and the same between the same frames. This is the step that comes when processing. Here, both the representative point calculation flag and the retry flag are set, and the process proceeds to S209.

(S514) In the step that comes when the calculated corresponding point or camera work parameter has an abnormal value and the retry flag is ON, the representative point calculation flag is turned ON, the retry flag is turned OFF, and the process proceeds to S207. At this time, as an output of the camera work parameter calculation unit 106, for example, a predetermined parameter value is output. Further, this video time may be recorded as the time when the discontinuity occurs. The case where this step is reached is a case where the camera work parameter cannot be calculated due to the discontinuity of the video between frames.
FIG. 6 shows an embodiment having both claims 2 and 4. In other words, this corresponds to the addition of the frame interval determination unit 609 for determining the frame interval to the embodiment of FIG. The other functions are basically the same as those in FIG. 4, but the processing content is changed so that the corresponding points are calculated using a plurality of reference frame images. Specific changes will be described later with reference to FIG.

The advantage of widening the frame interval is that the camera work detection process is reduced in a video section with relatively little movement, leading to an improvement in processing speed. However, if it is too wide, it may not be possible to take corresponding points when large camera work occurs, and camera work changes such as camera reciprocation may not be taken. Therefore, an upper limit F _Max is provided for the frame interval for processing. Also, by calculating the corresponding points using a plurality of reference frames and comparing the number of corresponding points, if the movement is large, that is, if the corresponding points are greatly reduced, the frame interval is shortened, and the corresponding points decrease little. If not, set the frame interval longer.

A frame interval determination unit 609 determines which reference frame is used to calculate the camera work parameter from the number of corresponding points calculated in a plurality of reference frames. Specific processing contents will be described with reference to FIG.
FIG. 7 is a flowchart showing the specific processing contents of FIG. 6 and corresponds to an embodiment in which claims 6 and 8 are combined. In FIG. 5, S501, S203, S204, and S207 are changed to S701, S703, S704, and S707, respectively. Only the changed part will be described with reference to FIG.

(S701) The video is input from the video storage unit 101 to the video input unit 102, converted into an image for each frame by the video input unit 102 and output, and the first frame which is the initial processing target frame is input. Further, any f frames (f ≦ F _Max +1) are selected from the next second frame to the F _Max +1 frame and are input. Further, the representative point calculation flag and the retry flag are turned off. The processing target frame and the reference frame are stored and output in the memory, and the process proceeds to S202.
(S703) This process is executed by the corresponding point calculation unit 105. A corresponding point calculation procedure according to claim 6 is a specific example. The processing target frame image and the reference frame image are input from the memory, the representative point coordinates thereof are input from the representative point calculation unit, respectively, and the points corresponding to the representative point coordinates in the processing target frame image are respectively searched from the f reference frame images, The coordinates of the corresponding points calculated are stored in the memory. That is, the processing in the corresponding point calculation unit 105, the same processing as S203, is performed f times. The coordinates of the corresponding points of the calculated reference frame images f and the number of points are output to the representative point calculation determination unit 107, and the representative point coordinates and the corresponding point coordinates are output to the camera work parameter calculation unit 106. The process proceeds to S704.

  (S704) This process is executed by the frame interval determination unit 609, and is a specific example of the frame interval determination procedure according to claim 6. The corresponding points for f frames calculated in S703, which is the process of the corresponding point calculation unit 105, and the number of the corresponding points are input from the memory, and the same skip determination as in S204 is performed on the corresponding points on all reference frames. And determine which reference frame is appropriate to use. If it is determined in step S204 that it is necessary to skip all reference frames, the process proceeds to step S512. If it is determined that it is not necessary to skip only one reference frame, the reference frame number is stored, and the corresponding point is output to S205. When it is determined that there is no need to skip a plurality of reference frames, the calculated number of corresponding points is compared. Comparison is made in order from the number of corresponding points of the reference frame with the smallest frame number, and it is checked whether the number is decreasing. In the same case, the larger frame number is adopted in order. When the number is decreased, the reference frame number is stored, and the corresponding point set is output to S205. Note that the condition for determining that the number of corresponding points is determined to have decreased by a certain value or more and the condition for determining that the number of corresponding points is the same may be processed to be less than a certain value. Further, the representative point calculation flag set determination similar to S204 is performed on the selected reference frame, and the state of the representative point calculation flag is changed according to the determination.

(S707) The frame image updating unit 103 updates the reference frame currently determined to the processing target frame in the next loop, and the corresponding point calculated for the reference frame is updated to the representative point in the next loop. . Also, the frame interval fp of the currently determined reference frame with respect to the processing target frame determined in the current loop is input, and the frame interval determining unit 609 sets the reference frame number in the next loop to be read in f frames. decide. That is, the reference frame image is also updated by storing in the memory 100 f frames determined from the F _Max frame images from the next to the updated processing target frame to after the F _Max frame. The processing executed by the frame interval determination unit 609 is also a specific example of the frame interval determination procedure according to claim 6.

As a specific example of the reference frame selection method, there is a method of selecting an image with a fixed frame interval before and after the frame interval fp and with F _Max as an upper limit.
For example, when selecting f = 2, F _Max = 10, fp = 9, and a constant interval of 2, the reference frame is selected as the reference image with 9-2 = 7th frame of the read frame and 9 + 2 = 11> 10. . For example, when selecting f = 5, F _Max = 10, fp = 9, and a fixed interval 2, counting from the read frame 9-2 ² = 5th, 9-2 ¹ = 7th, 9 + 0 = 9th, 9 + 2 = 11> 10 The 10th is selected as the reference image. In this example, the tenth image is selected twice, but if it is selected once, the number of images selected is four.

As another example of the reference frame selection method, the next f frames are selected at random from the processing target frame after the update, or F _Max frame images are changed to f frames at regular intervals. You can choose to. In addition, when the camerawork parameter between frames cannot be calculated (retry flag OFF, representative point calculation flag ON), it is considered that there has been a large image change during this period, and in order to investigate in detail where the image change has occurred It is also possible to take out f frames for each frame from the next frame to be processed and re-execute. Alternatively, it is possible to calculate using other numerical calculations and conditions other than those described above.

All the reference frame numbers determined last are output to the frame image update unit 103, and f reference frames are read into the memory 100.
An example in which the embodiment of FIG. 7 of the present invention is performed using the embodiment of camera work calculation of FIG. 4 is shown below. FIG. 8A shows an image obtained by superimposing a difference to see the background coincidence. FIG. 8B shows a processing target frame image, a reference frame image and a representative point before the difference is obtained. Each is shown in c). The frame interval between FIGS. 8B and 8C is 7 frames. The representative points shown in the figure are converted using the calculated camera work parameters to calculate the difference between the representative point of the frame to be processed and the corresponding point of the reference frame. The point is determined to be a point of an object with movement other than the background, and each representative point is classified by changing the value of the point to a point that belongs to the background with a small value, and displayed by changing the shade of the point Is. Here, a circle shown in gray indicates a background, and a white circle indicates an outlier. This is a display method similar to Non-Patent Document 1. Further, the pixel values remaining in a line shape at the right end and the upper and lower ends in FIG. 8A are pixels in which the difference between the camera work occurs in the right and zoom-out directions.

Between these frames, the camera moves slightly to the right and zooms out, and the background object moves to the left and the center of the image on the image, but the telop (not explicitly shown in FIG. 8) has stopped, The moving objects are moving differently. That is, the telop and the moving object do not conform to the camera work parameters, but it can be seen that the background can be matched regardless of the representative points on the telop and the moving object.
Some of the representative points on the moving object are classified as points belonging to the background, but this is because the movement of that point happens to be the same as the camera work parameter, or the corresponding point can be calculated accurately. This is because it is determined that the camera work parameter is accidentally moved. On the other hand, some of them are classified as not belonging to the background even though they are representative points on the background, but this is because there is a slight shift in the corresponding point calculation due to the effects of video noise and the like. This is probably because a corresponding point slightly different from the motion indicated by the calculated camera work parameter is calculated.

Further, it has been determined that the camerawork parameter cannot be calculated between frames in which a significant change in the image such as a cut point has occurred.
As an optimum embodiment of the present invention, an example in which the embodiment illustrated and described with reference to FIG. 6 is executed by the processing method of the flowchart of FIG. The camera work parameter calculation method uses the method described with reference to FIG.
By the way, the camera work parameter calculation device of the present invention can be configured by a DSP (Digital Signal Processor). Further, it may function by executing a program from a computer. In this case, the program is recorded on a C-ROM, a floppy (registered trademark) disk, a magnetic disk, or the like by being loaded into a program memory in the computer. The program memory may be downloaded by communication.

  [Reference 1] Jianbo Shi, Carlo Tomasi. "Good Features to Track", IEEE Conference CVPR 94, June 1994.

The block diagram explaining the Example corresponding to Claim 1. The operation | movement flowchart of the Example of FIG. The detailed flowchart explaining the Example corresponding to Claim 5. 3 is a detailed flowchart of S205 of FIG. The detailed flowchart explaining the Example corresponding to Claim 7. The operation | movement flowchart which showed the specific example of the procedure of the camera work parameter calculation of FIG. The detailed flowchart explaining the Example corresponding to Claim 8. The operation | movement flowchart of the Example which added the camera work parameter normal determination part to the Example of FIG. The flowchart which shows the Example which employ | adopted the procedure of Claim 8 to the camera work calculation procedure in Claim 5. The block diagram explaining the Example which added the frame space | interval determination part to the Example of FIG. The operation | movement flowchart of the Example of FIG. The flowchart which shows the Example which employ | adopted the procedure of Claim 8 to the camera work calculation procedure in Claim 6. The figure which shows the example of application with respect to the image | video using a calculation parameter. The block diagram explaining the Example corresponding to Claim 3. The block diagram which shows the detailed structure of the camera work parameter calculation part of the Example of FIG. The block diagram explaining the Example corresponding to Claim 4. The figure which shows the structure which added the camera work parameter normal determination part to the camera work parameter calculation part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Memory 101 Image | video accumulation | storage part 102 Image | video input part 103 Frame image update part 104 Representative point calculation part 105 Corresponding point calculation part 106 Camerawork parameter calculation part 107 Representative point calculation determination part 108 Processing result output part 609 Frame space | interval determination part 901 Point combination Redundancy selection unit 902 Provisional parameter calculation unit 903 Provisional parameter storage table 904 Parameter verification unit 905 Camera work parameter determination unit 1006 Camera work parameter normality determination unit

Claims (10)

  A representative point calculating unit that calculates a characteristic point from a frame image in the video as a representative point, a corresponding point calculating unit that calculates a point corresponding to a representative point of a certain frame image from another frame image, and an image point Performs reliability determination using redundant pairs of correspondence relationships, and determines whether it is necessary to calculate representative points, and a camera work parameter calculation unit that calculates camera work parameters by excluding pairs with low reliability points A camera work parameter calculation apparatus comprising: a representative point calculation determination unit; and a frame image and a frame image update unit that updates a representative point.   A representative point calculation unit that calculates a characteristic point from a frame image in the video as a representative point; a corresponding point calculation unit that calculates a point corresponding to a representative point of a certain frame image from another frame image; A frame interval determination unit that determines the reference frame to be adopted by comparing the number of corresponding points on a plurality of reference frames and a pair of image point correspondences are used redundantly to make a reliability determination. A camera work parameter calculation unit that calculates camera work parameters excluding the set, a representative point calculation determination unit that determines whether the calculation of the representative point is necessary, and a frame image update unit that updates the frame image and the representative point. A camera work parameter calculation device provided. In the camera work parameter calculation device according to any one of claims 1 and 2,
The camera work parameter calculation unit includes a point combination redundancy selection unit that selects more points than the minimum number necessary for parameter calculation from the representative points and the calculated corresponding points, and a plurality of combinations of the selected redundant points. A temporary parameter calculation unit that calculates a camera work parameter, a parameter verification unit that determines whether the value is reliable by performing threshold processing based on the degree of dispersion of the variance value, and the like, and determines the camera work parameter using only the reliable value A camera work parameter calculation device comprising a camera work parameter determination unit. In the camera work parameter calculation device according to any one of claims 1 and 2,
The camera work parameter calculation unit includes a point combination redundancy selection unit that selects more points than the minimum number necessary for parameter calculation from the representative points and the calculated corresponding points, and a plurality of combinations of the selected redundant points. A temporary parameter calculation unit that calculates a camera work parameter, a parameter verification unit that determines whether the value is reliable by performing threshold processing based on the degree of dispersion of the variance value, and the like, and determines the camera work parameter using only the reliable value A camera work parameter calculation apparatus comprising: a camera work parameter determination unit; and a camera work parameter normality determination unit that determines whether the calculated camera work parameter is normal and stores an abnormal case as an uncalculatable time.   A representative point calculation procedure for calculating a characteristic point from a frame image in a video as a representative point, a corresponding point calculation procedure for calculating a point corresponding to a representative point of a certain frame image from another frame image, and an image point Performs reliability determination using redundant pairs of correspondence relationships, determines the camera work parameter calculation procedure for calculating camera work parameters excluding point correspondence pairs with low reliability, and determines whether representative points need to be calculated A camera work parameter calculation method comprising: a representative point calculation determination procedure; and a frame image update procedure for updating a frame image and a representative point.   A representative point calculation procedure for calculating a characteristic point from a frame image in the video as a representative point; a corresponding point calculation procedure for calculating a point corresponding to a representative point of a certain frame image from another frame image; A frame interval determination procedure for determining the reference frame to be adopted by comparing the number of corresponding points on a plurality of reference frames and a pair of image point correspondences are used redundantly to make a reliability determination, and a point corresponding to a low reliability point A camera work parameter calculation procedure for calculating camera work parameters excluding the set, a representative point calculation determination procedure for determining whether or not a representative point needs to be calculated, and a frame image update procedure for updating the frame image and the representative point. A camera work parameter calculation method. In the camera work parameter calculation method according to any one of claims 5 and 6,
As a camera work parameter calculation procedure, a point combination redundancy selection procedure for selecting more points than the minimum number required for parameter calculation from representative points and calculated corresponding points, and a plurality of combinations of the selected redundant points are selected. A provisional parameter calculation procedure for calculating camera work parameters, a parameter verification procedure for determining whether or not a reliable value is obtained by performing threshold processing based on the degree of dispersion of the variance value, and the camera work parameter is determined using only the reliable value. A camera work parameter determination procedure. In the camera work parameter calculation method according to any one of claims 5 and 6,
As a camera work parameter calculation procedure, a point combination redundancy selection procedure for selecting more points than the minimum number required for parameter calculation from representative points and calculated corresponding points, and a plurality of combinations of the selected redundant points are selected. A provisional parameter calculation procedure for calculating camera work parameters, a parameter verification procedure for determining whether or not a reliable value is obtained by performing threshold processing based on the degree of dispersion of the variance value, and the camera work parameter is determined using only the reliable value. A camera work parameter calculation method comprising: a camera work parameter determination procedure; and a camera work parameter normal determination procedure for determining whether or not the calculated camera work parameter is normal and storing an abnormal case as an uncalculatable time.   A camera work parameter calculation program for instructing a computer to execute the camera work parameter calculation method according to any one of claims 5 to 8.   A storage medium storing the camera work parameter calculation program according to claim 9.