JP2008225591A - Camera work detection method, equipment, program, and its recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To precisely detect a camera work generated in a moving image frame with a relatively small amount of calculation. <P>SOLUTION: A camera work detection method includes steps of comparing sizes of prediction errors of in-frame prediction and inter-frame prediction with respect to the right end, left end, upper end and lower end external edge blocks in a frame, thus deciding whether or not the correlation of a time direction in the region of the external edge in the frame is smaller than the correlation of the direction of a space, and discriminating whether or not the frame includes a pan in the right direction, includes a pan in the left direction, includes a tilt in the upward direction and includes a tilt in the downward direction according to the decision result. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a camera work detection method for use in a high-efficiency image signal encoding method and the like, and more particularly, to a method for detecting a change in a time axis direction in each frame in a moving image frame in which camera work has occurred.

  In a frame in which a large change in the time axis direction (high-speed camera pan / tilt, scene change, etc.) has occurred, the sensitivity to the image quality of that frame extremely decreases. This is a visual characteristic known as the time masking effect. Therefore, an approach that actively reduces the amount of code is effective for frames in which a large change in the time axis direction has occurred, since even if the amount of code is reduced, a decrease in subjective image quality is difficult to detect. In this case, it is important to detect a large change in the time axis direction in each frame.

  Conventionally, global motion estimation in global motion compensation or sprite coding has been studied for global motion detection in a frame caused by camera work [see Patent Document 1].

Global motion refers to the amount of displacement common to the entire frame caused by camera work. In the conventional method, global motion is estimated based on the displacement (local motion) estimated in blocks. In global motion compensation and sprite coding, since the direction and size of global motion are both important, an approach to estimate global motion by integrating all local motions in a frame is taken. For this reason, it is necessary to consider all local motions in the frame.
Japanese Patent Laid-Open No. 9-182081

  When the purpose is to detect a frame in which a large change in the time axis direction occurs, only the magnitude of the shift amount is a problem, and the direction is not necessarily required. For this reason, a method that considers all local motions in a frame, as in the conventional method described above, divides the calculation amount into obtaining unnecessary information. Since the amount of computation and the coding efficiency are in a trade-off relationship, the waste of the amount of computation leads to a decrease in the coding efficiency in the limited computer resource environment.

  Further, in the conventional method, the difference between the motion vector estimated in block units and the true motion amount is not taken into consideration. For this reason, there is a possibility that the desired global motion will deviate from true camera work. If the estimated global motion is estimated to be larger than the true motion amount, the code amount reduction approach based on the global motion described above causes fatal image quality degradation.

  Accordingly, an object of the present invention is to establish a global motion magnitude detection method based on local motion at a specific position in a frame. Furthermore, for a block at a specific position, the reliability of the local motion is indirectly evaluated through a comparison of the correlation in the spatial direction and the correlation in the temporal direction. The purpose is to avoid being disturbed.

  In order to solve the above-described problem, the present invention provides an area that does not exist in the next frame and cannot be associated with a pixel in the next frame (referred to as a non-connected area). Since the correlation with the previous frame is impossible, the correlation in the temporal direction is significantly reduced, while the correlation in the spatial direction is irrelevant to whether the correlation with the previous frame is possible, Camerawork is detected as follows.

  For example, if the temporal correlation is smaller than the spatial correlation in the rightmost region in the frame, the frame is determined to be a frame including a right pan. In the following description, the determination of the frame including the right pan is described as an example. However, the determination of the left pan, the upward tilt, and the downward tilt can be performed in the same manner. It should be noted that these four directions can be determined at the same time, or only a predetermined specific direction can be determined.

  Further, the accuracy of discrimination can be improved by comparing the temporal correlation and the spatial correlation not only for the immediately preceding frame but also for the immediately following frame as follows. That is, in another method according to the present invention, in a frame including a pan in the right direction, the correlation in the time direction with the immediately preceding frame is smaller than the correlation in the spatial direction in the rightmost region in the frame, and the reference / When the relationship of the referenced frame is reversed on the time axis, when the correlation in the time direction becomes larger than the correlation in the spatial direction, it is determined that the frame includes a pan in the right direction.

  In addition, when the correlation in the time direction is larger than the correlation in the spatial direction in the upper and lower end regions, it is possible to improve the accuracy of the determination by adding a frame including a pan in the right direction as a determination condition. it can.

  In the above invention, when comparing the correlation in the spatial direction and the correlation in the temporal direction, the frame is divided into blocks of a predetermined size, and inter-frame prediction by motion compensation is performed for each block included in the outer edge region. When the error is compared with the size of the intra-frame prediction error based on the intra prediction in the current frame, and the number or ratio of blocks in which the intra-frame prediction error is smaller than the inter-frame prediction error is larger than a predetermined threshold, the temporal correlation is spatial It is determined that the correlation is smaller than the direction correlation.

  As described above, the outer edge blocks of the frame are classified according to the magnitude relationship between the prediction errors of the intra-frame prediction and the inter-frame prediction, and used for camera work determination.

  In the above invention, the camera work detection process is repeated while gradually increasing the width of the outer edge region from the frame end in a stepwise manner, and the magnitude relationship between the temporal direction correlation and the spatial direction correlation is repeated. It is also possible to detect the size of the camera work from the width of the outer edge area when is reversed.

  According to the present invention, it is possible to detect a large change in the time axis direction (high-speed camera pan / tilt, scene change, etc.) in each frame. This makes it possible to assign an adaptive code amount for each frame in moving picture coding that uses the time masking effect to reduce the code amount in a frame in which a large change in the time axis direction has occurred. , Which leads to efficient code amount reduction. In addition, according to the present invention, it is possible to avoid waste of calculation amount necessary for detecting a frame including a large change in the time axis direction. Since the amount of computation and coding efficiency are in a trade-off relationship, the amount of computation saved by the detection process can be used for other processing, so that the coding efficiency can be improved even in a limited computer resource environment. It is done.

  The basic concept and embodiment of the present invention will be described below with reference to the drawings.

[Unconnected area]
In the case of an image including camera work such as camera zoom, pan, and tilt, some regions close to the frame boundary may not be present in the frame in the next frame. For example, in the case of an image including a right camera pan, a part of the outer edge region at the right end of the frame does not exist in the immediately preceding frame. As described above, an area that does not exist in the immediately preceding frame and cannot be associated with a pixel in the immediately preceding frame and a pixel in the frame is referred to as a non-connected area.

  FIG. 1 shows an example of a non-connected region by camera work of right panning. A shaded portion in the frame of FIG. 1B indicates a non-connected region. In such a non-connected region, it is impossible to associate with the immediately preceding frame in FIG.

  On the other hand, the correlation in the spatial direction is irrelevant to whether or not it can be associated with the immediately preceding frame, and does not decrease because it is a non-connected region. In the present invention, focusing on this property, global movement is detected.

[Intraframe / interframe prediction]
The frame is divided into rectangular areas (hereinafter referred to as blocks) of a certain size. The block size N × N is given from the outside. Pay attention to the block located at the outer edge of the frame (hereinafter referred to as the outer edge block).

  For example, when the inside of a frame is divided into blocks as shown in FIG. 2, the outer edge block hits a shaded portion. Here, a method for determining camera work when the outer edge block is arranged as shown in FIG. 2 will be described first. Note that it is also possible to adaptively change the arrangement of the outer edge block according to the amount of motion to be extracted, and to detect the size of the camera work as will be described in detail later.

  The size of a prediction error in intra prediction (described later) and a prediction error in forward inter prediction (described later) is determined for the outer edge block in the frame. As a result of this determination, blocks in which the prediction error in intra prediction is smaller than the prediction error in forward inter prediction are classified into categories called non-forward inter blocks, and otherwise, categories called forward inter blocks. Classify into:

  Next, the size of the prediction error in intra prediction (described later) and the prediction error in backward inter prediction (described later) are determined. As a result of this determination, blocks in which the prediction error in intra prediction is smaller than the prediction error in backward inter prediction are classified into categories called non-backward interblocks, and otherwise, categories called backward interblocks. Classify into:

  Intra prediction is a prediction method that refers to pixel values in the same frame. There is a prediction method defined in H.264. H. In H.264, nine types of prediction methods are prepared for luminance components of 4 × 4 pixels, and four types of prediction methods for luminance components of 16 × 16 pixels. A method for minimizing a prediction error is selected from these 13 types of prediction methods, and the prediction error at that time is set as a prediction error in intra prediction. H. According to the H.264 standard, a decoded pixel value is used as a reference pixel value used for prediction, but here, an original pixel value at a corresponding position is used. Note that other prediction methods can be used.

On the other hand, the forward inter prediction method uses block-based motion compensation prediction. In this method, blocks in the encoding target frame f _t (p) (pε {(x, y) | 0 ≦ x ≦ X−1, 0 ≦ y ≦ Y−1}) (screen size X × Y). Each time, the following inter-frame prediction is performed.

f _t ′ (p) = f _t−1 (p−v _i ), pεB _i
Here, B _i is the i-th block, and v _i is a motion vector that satisfies the following equation.

Here, arg (subscript v) min (subscript vεR _i ) returns v that minimizes the next function. That is, in the search range R _i, v that minimizes the prediction error in the block B _i is selected as the motion vector v _i. At this time, a prediction error corresponding to the motion vector v _i is set as a prediction error in inter prediction. Thus, the forward inter prediction uses a past frame located immediately before on the time axis as a reference frame. On the other hand, backward inter prediction is a prediction method using a future frame located immediately after on the time axis as a reference frame.

  The outline of the present invention will be briefly described with reference to FIGS. FIG. 3 is a diagram for explaining detection of a right pan, and FIG. 4 is a diagram for explaining detection of an upward tilt.

  For example, as shown in FIG. 3, in a frame in which a camera work of panning in the right direction has occurred, the area at the right end of the frame is a non-connected area in relation to the previous frame. Here, the number of non-front interblocks Will increase. On the other hand, in the leftmost region of the frame, the number of non-forward inter blocks is relatively small. Further, in the relationship with the immediately following frame, the leftmost area of the frame becomes a non-connected area, and the number of non-back interblocks increases here. On the other hand, in the right end region of the frame, the number of non-backward inter blocks is relatively small. Further, in the right pan, the number of front inter blocks increases in the outer edge areas at the upper and lower ends. Therefore, if such outer edge blocks are classified and used for camera work detection, the problem that global motion obtained by the conventional method may deviate from true camera work can be solved. .

  Panning in the left direction can also be detected by making a similar determination with the left and right in FIG. 3 reversed. Further, as shown in FIG. 4, the upward tilt can also be detected using the classification of the outer edge block, and similarly the downward tilt can be detected. Hereinafter, these determination conditions will be described in more detail.

[Determination of camera work focusing on outer edge block (1)]
Since the outer edge block that is a non-connected region has a low correlation in the time direction, when referring to the immediately preceding frame, there is a high possibility of being classified as a non-forward inter-block. It is likely to be classified as an interblock. Therefore, based on the number (or ratio) of non-front interblocks and non-backward interblocks in the outer edge block, it is determined whether the correlation in the time direction or the correlation in the spatial direction is strong. The conditions described below are combined to make the judgment conditions for each camera work. Here, it is assumed that the frame size is X × Y, the block size is N × N, and that N is a divisor of X and Y.

The following threshold parameters (α _r , α _l , β _t , β _b , γ _t , γ _b , γ _r , γ _l ) are assumed to be positive values of 100 or less, and specific values are given from the outside. Shall be. These can be determined, for example, through a preliminary experiment on a test image having statistical properties close to those of the encoding target image. If the parameter value is set to a large value, the criteria for using the parameter becomes stricter and the number of corresponding blocks decreases.

Condition (1-f): Of the outer edge blocks (Y / N) at the right end, α _r % or more is a non-forward inter block Condition (1-b): Of the outer edge blocks (Y / N) at the right end , Α _r % or more is a non-rear interblock Condition (2-f): Of the leftmost outer edge blocks (Y / N), α _l % or more is a non-forward interblock Condition (2-b): Of the leftmost outer edge blocks (Y / N), α _l % or more is a non-rear interblock. Condition (3-f): Of the uppermost outer edge blocks (X / N), β _t % or more is non Condition (3-b) to be a front inter block: β _t % or more of non-back inter blocks among outer edge blocks (X / N) at the upper end Condition (4-f): outer edge block (X / N at the lower end) among N), conditions or beta _b% are non-forward inter block (4 b): Of the lower end of the outer blocks (X / N pieces), beta _{below b%} or more in a non-backward inter block, the right direction of the pan, leftward pan upward tilt, the determination of downward tilt Indicates conditions.

Criteria for determining right pan:
-Condition (1-f) is satisfied-Condition (1-b) is not satisfied-Condition (2-f) is not satisfied-Condition (2-b) is satisfied-The outer edge block at the upper end is γ _t % or more and forward Inter block ・ The outer edge block at the lower end is γ _b % or more, and it is a front inter block. This is due to the following reasons. If the rightmost outer edge block is a non-connected region with respect to the previous frame, the condition (1-f) is highly likely to be satisfied. There is a high possibility that 1-b) is not satisfied.

  That is, when the condition (1-f) is satisfied and the condition (1-b) is not satisfied, the rightmost image of the frame is not included in the past frame but included in the future frame. It can be assumed that there is a high possibility that If the leftmost outer edge block is a connected area with respect to the immediately preceding frame, it is highly likely that the condition (2-f) will not be satisfied. There is a high possibility of satisfying. That is, when the condition (2-b) is satisfied and the condition (2-f) is not satisfied, the image at the left end of the frame is included in the past frame and is not included in the future frame. It can be assumed that there is a high possibility that Furthermore, satisfying the remaining two conditions can be presumed that there is a high possibility that a certain amount of movement has occurred in the outer edge blocks other than the right and left ends. In such a situation, since the image that did not exist in the previous frame at the right end of the frame coincides with the right pan camera work, the above condition is set as the right pan camera work generation condition.

  In the same way, the judgment conditions for each camera work are set as follows.

Judgment conditions for left panning:
-Condition (2-f) is satisfied-Condition (2-b) is not satisfied-Condition (1-f) is not satisfied-Condition (1-b) is satisfied-The outer edge block at the upper end is γ _t % or more and forward Inter block ・ The outer edge block at the bottom is γ _b % or more and the front inter block is judged.
-Condition (3-f) is satisfied-Condition (3-b) is not satisfied-Condition (4-f) is not satisfied-Condition (4-b) is satisfied-The outer edge block on the right end is γ _r % or more and forward Inter block ・ Left edge outer block is γ _l % or more, and front inter block is down Tilt judgment condition:
-Condition (4-f) is satisfied-Condition (4-b) is not satisfied-Condition (3-f) is not satisfied-Condition (3-b) is satisfied-The right edge block is γ _r % or more and forward Interblock ・ The outer edge block at the left end is γ _l % or more, and the judgment condition for each camera work that is a front interblock is as above.

[Determination of camera work focusing on outer edge block (Part 2)]
A method for further reducing the amount of calculation for the camera work determination (part 1) focusing on the outer edge block will be described below. In order to reduce the amount of computation, it is determined whether the correlation in the time direction or the correlation in the spatial direction is strong based on the number of non-forward inter blocks in the outer edge block. The combination of the following conditions is used as a judgment condition for each camera work.

Condition (1-f): Of the outer edge blocks (Y / N) at the right end, α _r % or more is a non-forward inter-block Condition (2-f): Of the outer edge blocks (Y / N) at the left end , Α _l % or more is a non-forward inter block Condition (3-f): Out of the outer edge blocks (X / N) at the upper end, β _t % or more is a non-forward inter block Condition (4-f): Out of the outer edge blocks (X / N) at the lower end, β _b % or more is a non-front interblock. Below are the judgment conditions for right pan, left pan, upward tilt, and downward tilt. .

Criteria for determining right pan:
• Condition (1-f) is satisfied • Condition (2-f) is not satisfied • Upper edge outer block is γ _t % or more, front inter block • Lower edge outer block is γ _b % or more, front inter block The left panning criteria:
• Condition (2-f) is satisfied • Condition (1-f) is not satisfied • The outer edge block at the upper end is γ _t % or more and a front inter block • The outer edge block at the lower end is γ _b % or more and a front inter block Yes Upward tilt judgment condition:
• Condition (3-f) is satisfied • Condition (4-f) is not satisfied • Right outer edge block is γ _r % or more, front inter block • Left end outer edge block is γ _l % or more, front inter block Yes Down tilt judgment condition:
• Condition (4-f) is satisfied • Condition (3-f) is not satisfied • Right outer edge block is γ _r % or more, front inter block • Left end outer edge block is γ _l % or more, front inter block Yes [Camera work size detection method]
A method for detecting the size of the camera work will be described below. The input of this processing is a block side length N [pixel] and a camerawork size P [pixel] to be extracted (a norm of a motion vector representing the camerawork). The size P [pixel] of the camera work to be extracted is P = (k + a) N (k is a positive integer). Here, a is a parameter set in advance in the range of 0 <a ≦ 1.

  In accordance with the determination conditions described in [Determination of Camerawork Focusing on Outer Edge Block (Part 1)] or [Determination of Camerawork Focusing on Outer Edge Block (Part 2)], a determination process for camera work in four directions is performed. . At this time, if camerawork in either direction is detected, a process for identifying the size is added. For example, when detecting a frame including a left camera pan having a size P = (2 + a) N [pixels] or more, a determination process is added to the dark shaded block in FIG.

  In the following, specific processing is divided into the case of using [determination of camera work focusing on outer edge block (part 1)] and the case of using [determination of camera work focusing on outer edge block (part 2)]. Indicates.

<When using camera work determination (part 1) focusing on the outer edge block>
Note that the determination conditions for the right pan, the left pan, the upward tilt, and the downward tilt shown below all refer to the conditions described above.

(i) Determination condition for determining that panning of P = (k + a) N [pixels] or more has occurred in the right direction:
[Y / N] -2 in the vertical direction adjacent to the left side of the outer edge block at the right end of the frame and k-1 in the horizontal direction ([Y / N ] -2) For the (k−1) blocks, the following is satisfied. Here, [·] is a Gaussian symbol, and [x] represents the maximum integer not exceeding the real number x.

-Condition (1-f) is satisfied-Condition (1-b) is not satisfied-Condition (2-f) is not satisfied-Condition (2-b) is satisfied
(ii) Determination condition for determining that panning of P = (k + a) N [pixels] or more has occurred in the left direction:
[Y / N] -2 in the vertical direction adjacent to the right side of the outer edge block at the left end of the frame and k-1 in the horizontal direction ([Y / N ] -2) For the (k−1) blocks, the following is satisfied.

-Condition (2-f) is satisfied-Condition (2-b) is not satisfied-Condition (1-f) is not satisfied-Condition (1-b) is satisfied
(iii) Determination condition for determining that a tilt of P = (k + a) N [pixels] or more has occurred in the upward direction:
It satisfies the above-described upward tilt determination condition, and is composed of k−1 pieces in the vertical direction adjacent to the lower side of the outer edge block at the upper end of the frame and [X / N] −2 pieces in the horizontal direction (k−1). ) × ([X / N] −2) blocks satisfy the following.

-Condition (3-f) is satisfied-Condition (3-b) is not satisfied-Condition (4-f) is not satisfied-Condition (4-b) is satisfied
(iv) Determination condition for determining that a tilt of P = (k + a) N [pixels] or more has occurred in the downward direction:
The above-described downward tilt determination condition is satisfied, and k−1 pieces in the vertical direction and [X / N] −2 pieces in the horizontal direction are adjacent to the upper side of the outer edge block at the lower end of the frame (k−1). For x ([X / N] -2) blocks, the following is satisfied:

-Condition (4-f) is satisfied-Condition (4-b) is not satisfied-Condition (3-f) is not satisfied-Condition (3-b) is satisfied <Determination of camera work focusing on outer edge block (No. When using 2)>
The determination conditions for right pan, left pan, upward tilt, and downward tilt shown below are all shown in the camera work determination (part 2) focusing on the outer edge block described above. Refers to a condition.

(i) Determination condition for determining that panning of P = (k + a) N [pixels] or more has occurred in the right direction:
[Y / N] × ((Y / N) × (Y / N) in the vertical direction and k−1 in the horizontal direction, which satisfy the above-described right pan determination condition and are adjacent to the left side of the outer edge block at the right end of the frame. For k−1) blocks: Here, [·] is a Gaussian symbol, and [x] represents the maximum integer not exceeding the real number x.

-Condition (1-f) is satisfied-Condition (2-f) is not satisfied
(ii) Determination condition for determining that panning of P = (k + a) N [pixels] or more has occurred in the left direction:
[Y / N] × ((Y / N) in the vertical direction and k−1 in the horizontal direction satisfying the aforementioned left pan determination condition and adjacent to the right side of the outer edge block at the left end of the frame. For k−1) blocks:

-Condition (2-f) is satisfied-Condition (1-f) is not satisfied
(iii) Determination condition for determining that a tilt of P = (k + a) N [pixels] or more has occurred in the upward direction:
The above-described upward tilt determination condition is satisfied, and k−1 pieces in the vertical direction and [X / N] pieces in the horizontal direction are adjacent to the lower side of the outer edge block at the upper end of the frame (k−1) × For [X / N] blocks, the following is satisfied.

-Condition (3-f) is satisfied-Condition (4-f) is not satisfied
(iv) Determination condition for determining that a tilt of P = (k + a) N [pixels] or more has occurred in the downward direction:
(K−1) × [[k / 1] in the vertical direction and [X / N] in the horizontal direction, which satisfy the above-described downward tilt determination condition, and is adjacent to the upper side of the outer edge block at the lower end of the frame. For X / N] blocks:

The condition (4-f) is satisfied. The condition (3-f) is not satisfied. Note that the above-described front interblock and non-front interblock are determined based on a block column (or block row) located outside the frame. ). When the condition is not satisfied in a column (or block row) of a certain block, the determination in the subsequent block column (or block row) is omitted.

If the vertical tilt and the horizontal pan are satisfied at the same time, the camera work is determined to be oblique. For example, if it is determined that panning of P = kN [pixels] or more has occurred in the left direction and panning of P = mN [pixels] or more has occurred in the upward direction, It is determined that camera work of (k ² + m ² ) ^1/2 × N [pixels] or more has occurred in the upward direction. These are the same when either of the above-mentioned [determination of camera work focusing on outer edge block (part 1)] or [determination of camera work focusing on outer edge block (part 2)] is used.

[flowchart]
An embodiment of the processing of the present invention will be described with reference to FIG.
Step S1: A mode determination process is performed on the outer edge block. Details of this processing will be described later with reference to FIG.
Step S2: The correlation strength in the time direction is measured for each of the outer edge blocks at the right end, left end, upper end, and lower end. Details of this processing will be described later with reference to FIGS.
Step S3: It is determined whether or not rightward panning camera work is included in the frame. Details of this processing will be described later with reference to FIG.
Step S4: When the determination result in step S3 is input and the determination result is a true value, the process proceeds to step S5, and a flag indicating the pan direction is set to 2. On the other hand, if the determination result is a false value, the process proceeds to step S6.
Step S6: It is determined whether or not the left pan camera work is included in the frame. Details of this processing will be described later with reference to FIG.
Step S7: When the determination result in step S6 is input and the determination result is a true value, the process proceeds to step S9, and a flag indicating the pan direction is set to 1. On the other hand, if the determination result is a false value, the process proceeds to step S8, and a flag indicating the pan direction is set to 0.
Step S10: It is determined whether or not camera work with an upward tilt is included in the frame. Details of this processing will be described later with reference to FIG.
Step S11: When the determination result in step S10 is input and the determination result is a true value, the process proceeds to step S12, and a flag indicating the tilt direction is set to 2. On the other hand, if the determination result is a false value, the process proceeds to step S13.
Step S13: It is determined whether or not camera work of a downward tilt is included in the frame. Details of this processing will be described later with reference to FIG.
Step S14: The determination result in step S13 is input, and if the determination result is a true value, the process proceeds to step S16, and a flag indicating the tilt direction is set to 1. On the other hand, if the determination result is a false value, the process proceeds to step S15, and a flag indicating the tilt direction is set to 0.
Step S17: When the pan direction flag and the tilt direction flag are input, and the pan direction flag is not 0 or the tilt direction flag is not 0, it is determined that camera work is included. Otherwise, it is determined that camera work is not included.
Step S18: It is determined whether or not processing has been completed for all frames. If the determination result is a true value, the processing ends. If it is a false value, the process proceeds to step S19.
Step S19: Set the processing target as the next frame, return to step S1, and repeat the process in the same manner.

<Flow of Step S1 (Part 1)>
FIG. 7 shows the flow of detailed processing in step S1 of FIG. The details of the mode determination process performed on the outer edge block in step S1 will be described below with reference to FIG.
Step S101: The frame is read and stored in the frame buffer.
Step S102: A reference frame (forward prediction reference frame) when performing inter-frame prediction is read from the frame and stored in the frame buffer.
Step S103: The frame is read, intra-frame prediction is performed on the block in the frame, and the prediction error in the block is written to the register. As a method of intra-frame prediction, for example, H.264. There is a method defined in H.264.
Step S104: The frame and the forward prediction reference frame are read as inputs, inter-frame prediction is performed on the block in the frame by motion compensation, and the prediction error in the block is written to the register. As a method of inter-frame prediction by motion compensation, for example, H.264. There is a method defined in H.264.
Step S105: When the intra-frame prediction error and the inter-frame prediction error calculated in steps S103 and S104 are input, a comparison process is performed. If the intra-frame prediction error is smaller than the inter-frame prediction error, step S105 is performed. The process proceeds to S106. Otherwise, the process proceeds to step S107.
Step S106: The mode of the block is set as a non-front inter-block, the mode information is written together with the position information of the block, and the process proceeds to step S113.
Step S107: The mode of the block is set as the front inter block, the mode information is written together with the position information of the block, and the process proceeds to Step S113.
Step S108: A reference frame (backward prediction reference frame) when performing backward inter-frame prediction is read for the frame and stored in the frame buffer.
Step S109: Read the frame and backward prediction reference frame, perform inter-frame prediction by motion compensation for the block in the frame, and write the prediction residual sum in the block. As a method of inter-frame prediction by motion compensation, for example, H.264. There are 264 methods.
Step S110: When the intra-frame prediction error and the inter-frame prediction error calculated in steps S103 and S109 are input, a process for comparing both is performed. If the intra-frame prediction error is smaller than the inter-frame prediction error, step S110 is performed. The process proceeds to S111. Otherwise, the process proceeds to step S112.
Step S111: The mode of the block is set as a non-rear inter-block, and the mode information is written together with the position information of the block, and the process proceeds to step S113.
Step S112: The mode of the block is set as a backward inter block, the mode information is written together with the position information of the block, and the process proceeds to Step S113.
Step S113: It is determined whether or not processing has been completed for all outer edge blocks. If the determination result is a true value, the processing is terminated and the process returns to the caller. In the case of a false value, the process proceeds to step S114.
Step S114: The process target is set as the next outer edge block, and the processes after steps S103, S104, and S109 are similarly repeated.

<Process Flow of Step S1 (Part 2)>
As the processing in step S1 in FIG. 6, in addition to the processing method shown in FIG. 7, it is also possible to use a method in which the amount of calculation is further reduced as shown in FIG. The process shown in FIG. 21 has a configuration in which some processes are omitted from FIG. 7. Since each process is the same as the process described with reference to FIG. 7, further detailed description thereof is omitted.

<Flow of Step S2 (Part 1)>
8 and 9 show the detailed processing flow in step S2 of FIG. Hereinafter, the details of the process of measuring the strength of the correlation in the time direction with respect to the outer edge block in step S2 will be described using FIG. Note that FIG. 9 is the same as FIG. 8, and thus detailed description regarding each processing step is omitted.
Step S201: The result of the mode identification process for the rightmost outer edge block output in FIG. 7 is input, the process of counting the number of blocks classified as non-forward inter blocks is performed, and the counted frequency value is registered. Export to Further, the result of the mode identification process for the rightmost outer edge block output in FIG. 7 is used as an input, the number of blocks classified as non-rear interblocks is counted, and the counted frequency value is written to the register. .
Step S202: Using the predetermined threshold value and the frequency value of the non-front inter-block output in step S201 as input, calculate the ratio of the non-front inter-block to the rightmost outer edge block, and calculate the ratio and the threshold value Make a comparison. If the ratio is equal to or greater than the threshold, the forward weak time correlation flag for the rightmost outer edge block is set to 1. Otherwise, the forward weak time correlation flag is set to 0.
Step S203: Using the predetermined threshold value and the frequency value of the non-rear interblock output in step S201 as input, calculate the ratio of the non-rear interblock to the rightmost outer edge block, and calculate the same ratio and the magnitude of the threshold value. Make a comparison. If the ratio is equal to or greater than the threshold, the backward weak time correlation flag for the rightmost outer edge block is set to 1. Otherwise, the backward weak time correlation flag is set to 0.
Step S204: Using the result of the mode identification process for the outer edge block output in FIG. 7 as an input, the process of counting the number of blocks classified as the front inter block is performed, and this counted frequency value is stored in the register. Write out. Similarly, a process of counting the number of blocks classified as backward inter blocks is performed, and the counted frequency value is written to a register.
Step S205: Using the result of the mode identification process for the lower edge block output in FIG. 7 as an input, perform the process of counting the number of blocks classified as the front inter block, and store the counted frequency value in the register Write out. Similarly, a process of counting the number of blocks classified as backward inter blocks is performed, and the counted frequency value is written to a register.
Step S206: Using the predetermined threshold value and the frequency value output in Steps S204 and S205 as inputs, calculate the ratio of the front inter block and the ratio of the rear inter block to the outer edge block at the upper end. Each ratio is compared with the threshold value. If either of these two ratios is equal to or greater than the threshold, the moving region flag for the outer edge block at the upper end is set to 1. Otherwise, the moving area flag is set to 0.
Step S207: Using the predetermined threshold value and the frequency value output in steps S204 and S205 as inputs, calculate the ratio of the front inter block and the ratio of the rear inter block to the outer edge block at the lower end. Each ratio is compared with the threshold value. If either of these two ratios is equal to or greater than the threshold, the moving region flag for the outer edge block at the lower end is set to 1. Otherwise, the moving area flag is set to 0.
Steps S211 to S217: The same as steps S201 to S207.

<Processing flow of step S2 (2)>
As the processing in step S2 in FIG. 6, in addition to the processing methods shown in FIGS. 8 and 9, it is also possible to use a method in which the amount of calculation is further reduced as shown in FIGS. The processing shown in FIG. 22 and FIG. 23 has a configuration in which some processing is omitted from FIG. 8 and FIG. 9, and each processing is the same as the processing explained in FIG. Description is omitted.

<Processing flow of step S3 (part 1)>
FIG. 10 shows the detailed processing flow in step S3 of FIG. The details of the right pan determination process in step S3 will be described below with reference to FIG.
Step S301: The forward weak time correlation flag for the rightmost outer edge block output in FIG. 8 is read as input and written to the register.
Step S302: Using the forward weak time correlation flag for the rightmost outer edge block read in Step S301 as an input, whether or not it is 1 is determined. If it is 1, the process proceeds to Step S303; The process proceeds to step S314.
Step S303: The backward weak time correlation flag for the rightmost outer edge block output in FIG. 8 is read as input and written to the register.
Step S304: Using the backward weak time correlation flag for the rightmost outer edge block read in step S303 as an input, a determination process is performed as to whether or not the flag is zero. If the flag is zero, the process proceeds to step S305. Otherwise, the process proceeds to step S314.
Step S305: The forward weak time correlation flag for the leftmost outer edge block output in FIG. 8 is read as input and written to the register.
Step S306: The forward weak time correlation flag for the leftmost outer edge block read in Step S305 is input, and it is determined whether or not the flag is zero. If the flag is zero, the process proceeds to Step S307. Otherwise, the process proceeds to step S314.
Step S307: The backward weak time correlation flag for the leftmost outer edge block output in FIG. 8 is read as input and written to the register.
Step S308: The backward weak time correlation flag for the leftmost outer edge block read in Step S307 is input, and it is determined whether or not the flag is 1. If the flag is 1, the process proceeds to Step S309. Otherwise, the process proceeds to step S314.
Step S309: The moving area flag for the outer edge block at the top output in FIG. 8 is read as input and written to the register.
Step S310: The moving region flag for the uppermost outer edge block read in step S309 is input, and it is determined whether the flag is 1. If the flag is 1, the process proceeds to step S311. Otherwise, the process proceeds to step S314.
Step S311: The moving area flag for the outer edge block at the lower end outputted in FIG. 8 is read as input and written to the register.
Step S312: The moving region flag for the lower edge block read in step S311 is input and a determination process is performed to determine whether the flag is 1. If the flag is 1, the process proceeds to step S313. Otherwise, the process proceeds to step S314.
Step S313: It is determined that a right pan is included in the frame.
Step S314: It is determined that right panning is not included in the frame.

  11, 12, and 13 are the same as FIG. 10, and show left pan determination processing, upward tilt determination processing, and downward tilt determination processing, respectively.

<Processing flow of step S3 (part 2)>
As the processing in step S3 in FIG. 6, in addition to the processing method shown in FIG. 10, it is also possible to use a method in which the amount of calculation is further reduced as shown in FIG. The process shown in FIG. 24 has a configuration in which some of the processes are omitted from FIG. 10. Since each process is the same as the process described with reference to FIG.

  25, 26, and 27 are the same as those in FIG.

[Device diagram]
An apparatus diagram of an embodiment of the present invention is shown in FIG. Hereinafter, each part of the camera work detection device will be described with reference to FIG.
Outer edge block determination unit 101: Performs mode determination processing on the outer edge block and writes the determination result to the block type storage unit 102. Details of this processing unit will be described later with reference to FIG.
Right end outer edge block selection unit 103: The block type read from the block type storage unit 102 is input, and the block type of the right end outer edge block is extracted.
Right end outer edge block flag setting unit 104: Measures the strength of correlation in the time direction for the right end outer edge block and writes a flag representing the same strength to the right end outer edge block flag storage unit 105. Details of this processing unit will be described later with reference to FIG.
Left edge outer edge block selection unit 106: The block type read from the block type storage unit 102 is input, and the block type of the left edge outer edge block is extracted.
Left end outer edge block flag setting unit 107: Measures the strength of correlation in the time direction with respect to the left end outer edge block, and writes a flag representing the same strength in the left end outer edge block flag storage unit 108. Details of this processing unit will be described later with reference to FIG.
Upper edge block selection unit 109: The block type read from the block type storage unit 102 is input, and the block type of the upper edge block is extracted.
Upper edge block flag setting unit 110: Measures the strength of the correlation in the time direction for the upper edge block, and writes a flag representing the intensity in the upper edge block flag storage unit 111. Details of this processing unit will be described later with reference to FIG.
Lower edge outer edge block selection unit 112: The block type read from the block type storage unit 102 is input, and the block type of the lower edge outer edge block is extracted.
Lower end outer edge block flag setting unit 113: Measures the strength of correlation in the time direction for the outer edge block at the lower end, and writes a flag representing the same strength to the lower end outer edge block flag storage unit 114. Details of this processing unit will be described later with reference to FIG.
Camera work determination unit 120: In order to determine camera work, the following right direction pan determination unit 121, left direction pan determination unit 122, upward tilt determination unit 123, and downward tilt determination unit 124 are provided.
Right pan determination unit 121: It is determined whether or not right pan camera work is included in the frame. Details of the determination processing performed here will be described later with reference to FIG.
Left panning determination unit 122: Determines whether left panning camerawork is included in the frame. Details of the determination processing performed here will be described later with reference to FIG.
Upward tilt determination unit 123: It is determined whether or not camera work with an upward tilt is included in the frame. Details of the determination processing performed here will be described later with reference to FIG.
Downward tilt determination unit 124: It is determined whether or not a downward tilt camera work is included in the frame. Details of the determination processing performed here will be described later with reference to FIG.

<Configuration of outer edge block determination unit 101>
A configuration diagram of the outer edge block determination unit 101 is shown in FIG.
Forward prediction reference frame storage unit 201: Stores a reference frame (forward prediction reference frame) when performing forward inter-frame prediction for the frame.
Frame storage unit 202: Stores the frame.
Backward prediction reference frame storage unit 203: Reads a reference frame (backward prediction reference frame) when performing backward interframe prediction for the frame, and stores it in the frame buffer.
Intra-frame prediction error calculation unit 204: Using the frame read from the frame storage unit 202 as an input, performs intra-frame prediction on the block in the frame, and calculates the prediction residual sum in the block as intra-frame prediction Write to the error storage unit 205. As a method of intra-frame prediction, for example, H.264. There is a method defined in H.264.
Inter-frame prediction error calculation unit 206: Using the frame read from the frame storage unit 202 and the forward prediction reference frame read from the forward prediction reference frame storage unit 201 as input, motion compensation is performed on the block in the frame. And the prediction residual sum in the same block is written in the inter-frame prediction error storage unit 207. As a method of inter-frame prediction by motion compensation, for example, H.264. There is a method defined in H.264.
Inter-frame prediction error calculation unit 208: Using the frame read from the frame storage unit 202 and the backward prediction reference frame read from the backward prediction reference frame storage unit 203 as input, motion compensation is performed on the block in the frame. And the prediction residual sum in the same block is written in the inter-frame prediction error storage unit 209. As a method of inter-frame prediction by motion compensation, for example, H.264. There is a method defined in H.264.
Prediction error comparison unit 210: An intra-frame prediction error and an inter-frame prediction error read from the intra-frame prediction error storage unit 205 and the inter-frame prediction error storage unit 207 are input, a process for comparing both is performed, and the comparison result is blocked It is passed to the type output unit 211.
Block type output unit 211: Reads the comparison result between the intra-frame prediction error and the inter-frame prediction error output from the prediction error comparison unit 210 as input, and when the intra-frame prediction error is smaller than the inter-frame prediction error, As a non-front inter block, this mode information is written in the block type storage unit 214 together with the position information of the block. Otherwise, the mode information is written in the block type storage unit 214 together with the position information of the block as a front inter block.
Prediction error comparison unit 212: Inputs the intra-frame prediction error and the inter-frame prediction error read from the intra-frame prediction error storage unit 205 and the inter-frame prediction error storage unit 209, performs a process of comparing both, and blocks the comparison result It is passed to the type output unit 213.
Block type output unit 213: Reads the comparison result of the intra-frame prediction error and the inter-frame prediction error output from the prediction error comparison unit 212 as input, and when the intra-frame prediction error is smaller than the inter-frame prediction error, As a non-rear interblock, this mode information is written in the block type storage unit 214 together with the position information of the block. Otherwise, this mode information is written in the block type storage unit 214 together with the position information of the block as a backward inter block.
Final block determination unit 215: The above process is performed for all outer edge blocks.

  As the device configuration of the outer edge block determination unit 101, as shown in FIG. 28, a configuration in which the calculation amount of the device configuration is further reduced is possible. The apparatus configuration shown in FIG. 28 is a configuration in which a part of the apparatus configuration shown in FIG. 15 is omitted, and the processing units are the same as those in FIG.

<Configuration diagram of right end outer edge block flag setting unit 104, left end outer edge block flag setting unit 107, upper end outer edge block flag setting unit 110, and lower end outer edge block flag setting unit 113>
Detailed configurations of the right end outer edge block flag setting unit 104, the left end outer edge block flag setting unit 107, the upper end outer edge block flag setting unit 110, and the lower end outer edge block flag setting unit 113 will be described with reference to FIG.
Outer edge block position specifying unit 300: The specified position of the outer edge block to be detected by the camera parameter is input, and the specified position is written in the outer edge block position specifying storage units 304 and 310. Furthermore, the target position for inter block determination is written in 313 for the designated position that has been input. Here, the position of the outer edge block is one of the right end, the left end, the upper end, and the lower end.
Block type storage unit 301: Stores the block type of the outer edge block.
Non-front inter-block counting unit 302: The block type for the outer edge block read from the block type storage unit 301 is input, and the number of blocks classified as non-front inter-blocks is counted for each position of the outer edge block. The counted frequency value is written in the counter storage unit 303 for each position of the outer edge block. Here, the position of the outer edge block is one of the right end, the left end, the upper end, and the lower end.
Non-front inter-block ratio determination unit 305: classify into non-front inter-blocks for the threshold value read from the threshold value storage unit 307 and the outer edge block (outer edge block at the specified position) at the position specified by the outer edge block position specification storage unit 304 The frequency value of the read block is read from the counter storage unit 303 and used as an input, the ratio of the non-front inter-block to the outer edge block at the specified position is calculated, and the ratio is compared with the threshold value. If the ratio is equal to or greater than the threshold, the forward weak time correlation flag for the outer edge block is set to 1. Otherwise, the flag is set to 0, and the value of the flag is stored in the forward weak time correlation flag storage unit 306. Write out.
Non-rear inter-block counting unit 308: The block type for the outer edge block read from the block type storage unit 301 is input, and the number of blocks classified as non-rear inter-blocks is counted for each position of the outer edge block. The counted frequency value is written in the counter storage unit 309 for each position of the outer edge block. Here, the position of the outer edge block is one of the right end, the left end, the upper end, and the lower end.
Non-rear inter-block ratio determining unit 311: Classification into non-rear inter-blocks for the threshold value read from the threshold value storage unit 313 and the outer edge block at the position specified by the outer edge block position specification storage unit 310 (outer edge block at the specified position) The frequency value of the obtained block is read from the counter storage unit 309 and used as an input, and the ratio of the non-rear inter-block to the outer edge block at the designated position is calculated, and the ratio is compared with the threshold value. If the ratio is equal to or greater than the threshold, the backward weak time correlation flag for the outer edge block is set to 1. Otherwise, the flag is set to 0, and the value of the flag is stored in the backward weak time correlation flag storage unit 312. Write out.
Front inter-block ratio determining unit 315: the front inter-block or the rear inter-block with respect to the threshold value read from the threshold value storage unit 317 and the outer edge block (outer edge block at the specified position) at the position designated by the outer edge block position designation storage unit 314 The frequency values of the blocks classified into the blocks are read from the counter storage units 303 and 309 and used as inputs, and the proportion of the front inter block and the proportion of the rear inter block are calculated with respect to the outer edge block at the designated position. The ratio of each ratio is compared with the threshold value. If the front interblock occupancy is greater than or equal to the threshold or the rear interblock occupancy is greater than or equal to the threshold, the dynamic region flag for the outer edge block is set to 1, otherwise the flag is set to 0, The value of the flag is written in the moving area flag storage unit 316.

  The apparatus configuration of the right edge block flag setting unit 104, the left edge block flag setting unit 107, the upper edge block flag setting unit 110, and the lower edge block flag setting unit 113 described above is a configuration in which the calculation amount shown in FIG. 29 is further reduced. Is also possible. The apparatus configuration shown in FIG. 29 is a configuration in which a part of the apparatus configuration shown in FIG. 16 is omitted, and the processing units are the same as those in FIG.

<Configuration of right pan determination unit 121>
FIG. 17 shows a configuration diagram of the right pan determination unit 121. Hereinafter, the detailed configuration of the right direction pan determination unit 121 will be described with reference to FIG.
Right side outer edge block forward weak time correlation flag storage unit 401: Stores the forward weak time correlation flag for the rightmost outer edge block read from the right end outer edge block flag storage unit 105 in FIG.
Right outer edge block forward weak time correlation flag non-zero value determination unit 402: The forward weak time correlation flag for the rightmost outer edge block read from the right outer edge block forward weak time correlation flag storage unit 401 is input, and the flag is 1 or not. If the flag is 1, the process proceeds to the process of the right outer edge block backward weak time correlation flag storage unit 403. Otherwise, the process proceeds to the process of the right pan flag setting unit 413.
Right outer edge block backward weak time correlation flag storage unit 403: Stores a backward weak time correlation flag for the rightmost outer edge block read from the right end outer edge block flag storage unit 105 in FIG.
Right outer edge block backward weak time correlation flag zero value determination unit 404: The backward weak time correlation flag for the rightmost outer edge block read from the right outer edge block backward weak time correlation flag storage unit 403 is input, and the flag is zero or not. If the flag is zero, the process proceeds to the process of the left outer edge block forward weak time correlation flag storage unit 405. Otherwise, the process proceeds to the process of the right direction pan flag setting unit 413.
Left outer edge block forward weak time correlation flag storage unit 405: Stores a front weak time correlation flag for the leftmost outer edge block read from the left outer edge block flag storage unit 108 of FIG.
Left side outer edge block forward weak time correlation flag zero value determination unit 406: The front weak time correlation flag for the leftmost outer edge block read from the left outer edge block forward weak time correlation flag storage unit 405 is input, and the flag is zero or not. If the flag is zero, the process proceeds to the process of the left outer edge block backward weak time correlation flag storage unit 407. Otherwise, the process proceeds to the process of the right pan flag setting unit 413.
Left-side outer edge block backward weak time correlation flag storage unit 407: Stores the backward weak time correlation flag for the left-most outer edge block read from the left-end outer edge block flag storage unit 108 in FIG.
Left side outer edge block backward weak time correlation flag non-zero value determination unit 408: The backward weak time correlation flag for the leftmost outer edge block read from the left outer edge block backward weak time correlation flag storage unit 407 is input, and the flag is 1 or not. If the flag is 1, the process proceeds to the process of the upper outer edge block motion area flag storage unit 409. Otherwise, the process proceeds to the process of the right pan flag setting unit 413.
Upper edge block motion area flag storage unit 409: Stores a motion area flag for the upper edge block read from the upper edge block block storage unit 111 in FIG.
Upper edge block motion region flag non-zero value determination unit 410: The motion region flag for the uppermost outer edge block read from the upper edge block motion region flag storage unit 409 is input to determine whether the flag is 1 or not. If the flag is 1, the process proceeds to the process of the lower outer edge block motion area flag storage unit 411. Otherwise, the process proceeds to the process of the right pan flag setting unit 413.
Lower outer edge block motion area flag storage unit 411: Stores a motion area flag for the lower edge outer edge block read from the lower edge outer edge block flag storage unit 114 of FIG.
Lower outer edge block motion region flag non-zero value determination unit 412: Processing for determining whether or not the flag is 1 by using the motion region flag for the lower outer block read from the lower outer block motion region flag storage unit 411 as an input If the flag is 1, the process proceeds to the right pan flag setting unit 414. Otherwise, the process proceeds to the right pan flag setting unit 413.
Right pan flag setting unit 414: It is determined that right pan is included in the frame, and the result is set in the right pan flag storage unit 415.
Right pan flag setting unit 413: Determines that the frame does not include right pan, and sets the result in the right pan flag storage unit 415.

<Configuration of Left Direction Pan Determination Unit 122>
A configuration diagram of the left pan determination unit 122 is shown in FIG. The processing content by the left direction pan determination unit 122 in FIG. 18 is substantially the same as the processing content by the right direction pan determination unit 121 in FIG.

<Configuration of Upward Tilt Determination Unit 123>
FIG. 19 shows a configuration diagram of the upward tilt determination unit 123. The processing content by the upward tilt determination unit 123 in FIG. 19 is substantially the same as the processing content by the right pan determination unit 121 in FIG.

<Configuration of Downward Tilt Determination Unit 124>
A configuration diagram of the downward tilt determination unit 124 is shown in FIG. The processing content by the downward tilt determination unit 124 in FIG. 20 is substantially the same as the processing content by the right pan determination unit 121 in FIG.

  As an apparatus configuration of the right direction pan determination unit 121, a configuration in which the calculation amount shown in FIG. 30 is further reduced is possible. The apparatus configuration shown in FIG. 30 is a configuration obtained by omitting a part of the apparatus configuration shown in FIG. 17, and the processing units are the same as those in FIG.

  Further, the apparatus configuration shown in FIGS. 31, 32, and 33 is substantially the same as that shown in FIG. 30, and thus the description thereof will not be repeated.

  The camerawork detection process described above can also be realized by a computer and a software program. The program can be provided by being recorded on a computer-readable recording medium or via a network. is there.

It is a figure explaining an unconnected field. It is a figure explaining the block division | segmentation in an flame | frame, and an outer edge block. It is a figure explaining the outline | summary (detection of the panning of a right direction) of this invention. It is a figure explaining the outline | summary (detection of the upward tilt) of this invention. It is a figure explaining the detection of the magnitude | size of the camera work by the dynamic setting of an outer edge block. It is a process flowchart of an Example. It is a detailed process flowchart of step S1 shown in FIG. It is a detailed process flowchart of step S2 shown in FIG. It is a detailed process flowchart of step S2 shown in FIG. It is a detailed process flowchart of step S3 shown in FIG. It is a detailed process flowchart of step S6 shown in FIG. It is a detailed process flowchart of step S10 shown in FIG. It is a detailed process flowchart of step S13 shown in FIG. It is an apparatus block diagram of an Example. It is a detailed block diagram of the outer edge block determination part shown in FIG. It is a detailed block diagram of the right end outer edge block flag setting part shown in FIG. It is a detailed block diagram of the right direction pan determination part shown in FIG. It is a detailed block diagram of the left direction pan determination part shown in FIG. It is a detailed block diagram of the upward direction tilt determination part shown in FIG. It is a detailed block diagram of the downward direction tilt determination part shown in FIG. 7 is another detailed processing flowchart of step S1 shown in FIG. 6. 7 is another detailed process flowchart of step S2 shown in FIG. 6. 7 is another detailed process flowchart of step S2 shown in FIG. 6. 7 is another detailed processing flowchart of step S3 shown in FIG. 6. It is another detailed process flowchart of step S6 shown in FIG. It is another detailed process flowchart of step S10 shown in FIG. It is another detailed process flowchart of step S13 shown in FIG. FIG. 15 is another detailed configuration diagram of the outer edge block determination unit shown in FIG. 14. FIG. 15 is another detailed configuration diagram of the right end outer edge block flag setting unit shown in FIG. 14. It is another detailed block diagram of the right direction pan determination part shown in FIG. It is another detailed block diagram of the left direction pan determination part shown in FIG. FIG. 15 is another detailed configuration diagram of the upward tilt determination unit shown in FIG. 14. FIG. 15 is another detailed configuration diagram of the downward tilt determination unit shown in FIG. 14.

Explanation of symbols

101 Outer edge block determination unit 102 Block type storage unit 103 Right end outer edge block selection unit 104 Right end outer edge block flag setting unit 105 Right end outer edge block flag storage unit 106 Left end outer edge block selection unit 107 Left end outer edge block flag setting unit 108 Left end outer edge block flag storage unit 109 Upper edge block selection unit 110 Upper edge block flag setting unit 111 Upper edge block flag storage unit 112 Lower edge block selection unit 113 Lower edge block flag setting unit 114 Lower edge block flag storage unit 120 Camera work determination unit 121 Right direction pan determination Unit 122 Left direction pan determination unit 123 Up direction tilt determination unit 124 Down direction tilt determination unit

Claims (9)

In a camera work detection method for detecting camera work generated in a moving image frame,
A correlation determination process for determining whether the temporal correlation with a frame immediately before a certain range of region at the right edge, left edge, upper edge or lower edge of the frame is smaller than the spatial correlation in the frame;
When the correlation in the time direction in the rightmost region in the frame is smaller than the correlation in the spatial direction, the frame is determined to be a frame including a pan in the right direction;
Or, when the correlation in the time direction in the leftmost region in the frame is smaller than the correlation in the spatial direction, the frame is determined to be a frame including a left pan,
Or, when the correlation in the time direction in the uppermost region in the frame is smaller than the correlation in the spatial direction, the frame is determined to be a frame including an upward tilt,
Alternatively, when the temporal correlation in the lower end region of the frame is smaller than the spatial correlation, the frame has a camera work determination process for determining that the frame includes a downward tilt. A camera work detection method characterized by the above. In a camera work detection method for detecting camera work generated in a moving image frame,
The magnitude of the temporal correlation with the immediately preceding frame and the temporal correlation with the immediately following frame and the spatial correlation with the corresponding frame in the range of the right edge, left edge, top edge, or bottom edge of the frame. Correlation judgment process to compare each,
In the rightmost region in the frame, the temporal correlation with the immediately preceding frame is smaller than the spatial correlation, and the temporal correlation with the immediately following frame is greater than the spatial correlation. In addition, it is determined that the frame includes a right pan,
Or, in the leftmost region in the frame, the temporal correlation with the immediately preceding frame is smaller than the spatial correlation, and the temporal correlation with the immediately following frame is larger than the spatial correlation. If this is the case, it is determined that the frame includes a left pan.
Alternatively, in the uppermost region in the frame, the temporal correlation with the immediately preceding frame is smaller than the spatial correlation, and the temporal correlation with the immediately following frame is larger than the spatial correlation. The frame is determined to be a frame including an upward tilt,
Alternatively, in the lower end region in the frame, the temporal correlation with the immediately preceding frame is smaller than the spatial correlation, and the temporal correlation with the immediately following frame is larger than the spatial correlation. In this case, the camera work detection method includes a camera work determination process for determining that the frame is a frame including a downward tilt. In the camerawork detection method according to claim 1 or 2,
In the camera work determination process, when it is determined that the frame includes the right pan or the left pan, the time direction correlation is higher than the spatial direction correlation in the upper and lower end regions. It is further determined that the frame is larger, and is determined to be a frame including pan only when the condition is satisfied.
When it is determined that the frame includes the upward tilt or the downward tilt, it is further provided that the temporal correlation is greater than the spatial correlation in the left and right end regions. And a camerawork detection method characterized by determining that the frame includes a tilt only when the condition is satisfied. In the camera work detection method according to claim 1, claim 2 or claim 3,
In the correlation determination process, the frame is divided into blocks of a predetermined size, and for each block included in the outer edge region, an inter-frame prediction error based on motion compensation and an intra-frame prediction error based on intra prediction in the frame. When the number or ratio of blocks in which the intra-frame prediction error is smaller than the inter-frame prediction error is larger than a predetermined threshold, it is determined that the temporal correlation is smaller than the spatial correlation A camera work detection method characterized by the above. In the camera work detection method according to any one of claims 1 to 4,
When camerawork is detected in the camerawork determination process, the correlation determination process and the camerawork determination process are repeated while gradually expanding the outer edge region in the frame along the direction of the camerawork. A camera work detection method, comprising: detecting a size of a camera work according to a determination result of the camera work determination process. In a camera work detection device that detects camera work generated in a moving image frame,
Correlation determining means for determining whether a temporal correlation with a frame immediately before a certain range of region at the right edge, left edge, upper edge or lower edge of the frame is smaller than a spatial correlation in the frame;
When the correlation in the time direction in the rightmost region in the frame is smaller than the correlation in the spatial direction, the frame is determined to be a frame including a pan in the right direction;
Or, when the correlation in the time direction in the leftmost region in the frame is smaller than the correlation in the spatial direction, the frame is determined to be a frame including a left pan,
Or, when the correlation in the time direction in the uppermost region in the frame is smaller than the correlation in the spatial direction, the frame is determined to be a frame including an upward tilt,
Or, when the temporal correlation in the lower end region in the frame is smaller than the spatial correlation, the frame includes camera work determination means for determining that the frame includes a downward tilt. A camera work detection device. In a camera work detection device that detects camera work generated in a moving image frame,
The magnitude of the temporal correlation with the immediately preceding frame and the temporal correlation with the immediately following frame and the spatial correlation with the corresponding frame in the range of the right edge, left edge, top edge, or bottom edge of the frame. A correlation determination means for comparing each;
In the rightmost region in the frame, the temporal correlation with the immediately preceding frame is smaller than the spatial correlation, and the temporal correlation with the immediately following frame is greater than the spatial correlation. In addition, it is determined that the frame includes a right pan,
Or, in the leftmost region in the frame, the temporal correlation with the immediately preceding frame is smaller than the spatial correlation, and the temporal correlation with the immediately following frame is larger than the spatial correlation. If this is the case, it is determined that the frame includes a left pan.
Alternatively, in the uppermost region in the frame, the temporal correlation with the immediately preceding frame is smaller than the spatial correlation, and the temporal correlation with the immediately following frame is larger than the spatial correlation. The frame is determined to be a frame including an upward tilt,
Alternatively, in the lower end region in the frame, the temporal correlation with the immediately preceding frame is smaller than the spatial correlation, and the temporal correlation with the immediately following frame is larger than the spatial correlation. In this case, the camera work detection device includes camera work determination means for determining that the frame is a frame including a downward tilt.   A camera work detection program for causing a computer to execute the camera work detection method according to any one of claims 1 to 5.   A computer-readable recording medium recording a camera work detection program for causing a computer to execute the camera work detection method according to any one of claims 1 to 5.

Images