JP4552553B2 - Image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing program, and more particularly to an image processing apparatus and an image processing program for performing image processing by estimating an optical flow.

As a conventional technique for performing image processing by estimating an optical flow, there is a proposal of an invention for automatically detecting a target moving object from a moving image obtained from a camera that performs moving shooting. As an embodiment of the present invention, as shown in FIGS. 2A, 2B, 2C, and 2D, image processing for detecting two falling rocks is described. FIG. 2A shows a frame image of a single color falling rock A and a patterned falling rock B. FIG. FIG. 2B shows an actual monitoring area in which a motion vector indicating the movement of a falling rock on the image obtained from the input frame image of FIG. 2A and the frame image one frame after (or before) is obtained. A velocity vector obtained by mapping to a plane is calculated by an optical flow estimation process, and a velocity field vector obtained by coordinate arrangement of the velocity vector is shown. In this velocity field vector, the process of dividing the rock fall block according to the directionality of the vector is a state called velocity field labeling shown in FIG. Next, based on the velocity field labeling result of FIG. 2 (c), as shown in FIG. 2 (d), by estimating the direction of the movement of the rock fall, the shape and area of the rock fall are estimated. It is configured to detect the presence or absence.
Furthermore, as shown in FIG. 3, when calculating a motion vector, a parameter indicating the degree of spatial smoothness of the motion vector is used as a regularization parameter (see Patent Document 1). The optical flow estimation process using the regularization parameter is obtained by adding an improvement using a normalization parameter to a known normalization method introduced in another document (Patent Document 2).

On the other hand, there is a proposal of an invention for detecting the movement of a camera work, that is, a video camera to be imaged by estimating an optical flow. For example, according to a proposed moving image processing apparatus, a difference image diff1 (t, x, y) is calculated from a frame image f (t-1, x, y) adjacent to a processing target frame image f (t, x, y). y) is obtained, and the magnitude D (t) of change between two frame images (horizontal NX and vertical NY regions) involved in the difference is obtained. Next, with respect to the magnitude of change (D (t) / (NX × NY)) obtained by normalization, it is determined whether or not the scene is cut by threshold processing. Next, the movement of the camera is determined by two methods (method (a) and method (b)). In the method (a), the head and the last frame of the corresponding scene are compared, and if the region where the change is large is large, it is determined that the camera has moved. In the method (b), for each pixel, the variance of the luminance value throughout the entire scene is obtained. If there are many pixels having a large variance, it is determined that there is camera movement. For example, as shown in FIG. 4, the movement of the camera can be obtained by the movement of four small areas set for the image. That is, a period in which the four small areas are moving in the same direction is detected as a vertical and horizontal movement of the camera (see Patent Document 3).
JP 2002-74370 A Japanese Patent Laid-Open No. 9-297851 JP-A-6-333048

However, as in Patent Document 1, the display method that represents the motion vector with an arrow has a problem that the actual motion of the image cannot be visually grasped. In particular, when analyzing sports watching images, it is difficult to analyze the fine movements of the players with arrows. For example, in a table tennis game as shown in FIG. 25, it is difficult to visually grasp the actual image motion even if the arm, racket, and ball motion vectors obtained by the optical flow estimation process are represented by arrows. It is.
Further, as in the above-mentioned Patent Document 3, only by detecting the period in which the four small regions are moving in the same direction as the vertical and horizontal movements of the camera, the video camera moves vertically, horizontally, or diagonally, that is, the camera moves in parallel. Can be detected, but there is a problem that zoom camera work for enlargement or reduction and camera work for rotation cannot be detected. For this reason, it is difficult to analyze the movement of a player in a sports game such as soccer or the battle of the game based on the captured moving image. Alternatively, it is difficult to study imaging technology (video grammar) by analyzing camera work in a movie based on the reproduced moving image.
An object of the present invention is to provide an image processing apparatus and an image processing program capable of visually grasping a motion vector of an image by an optical flow estimation process.
In addition, the present invention detects various camera works such as translation, zoom, and rotation by optical flow estimation processing, enables image analysis, and extracts an image to be analyzed, An object of the present invention is to provide an image processing apparatus and an image processing program capable of displaying the movement of the image.

The image processing apparatus according to claim 1, a motion estimation means for estimating the optical flow is a motion vector of a plurality of frame images corresponding to the moving screen continuous in time series, estimated by the motion estimation unit Optical Vector calculation means for calculating a vector distribution based on the flow, display control means for moving and displaying the display position of the image according to the vector distribution calculated by the vector calculation means on the display screen of the predetermined display means, When the optical flow estimated by the motion estimation unit is a vector distribution of a plurality of frame images corresponding to continuous imaging screens of moving images captured by the predetermined imaging unit, the vector distribution resulting from the movement of the imaging unit A vector determination unit for determining the captured image. If the dispersion of the optical flow vector distribution in a plurality of unit regions having a predetermined number of pixels is within a predetermined threshold, the vector distribution resulting from the movement of the imaging means is determined to be in a parallel movement or stationary state, When the variance of the optical flow vector distribution in the plurality of unit regions exceeds a predetermined threshold, the unit region having the minimum vector amount is detected, and the unit region of the minimum vector amount and any other unit When the vector direction of the arbitrary unit region is substantially the same direction or the substantially opposite direction with respect to the direction between the regions, the vector distribution resulting from the movement of the imaging unit is determined to be enlarged or reduced, When the vector direction of an arbitrary unit region is a substantially right angle direction, the vector distribution resulting from the movement of the imaging means is determined as rotation, and the display control means And displaying by moving the display position of the image based on a relationship between the vector distribution image of vector distribution and the frame image due to the movement of the imaging means is determined by the vector determining means. Further, in this case, as described in claim 2, the display control means displays only the imaging screen when the vector distribution due to the movement of the imaging means is determined to be parallel movement by the vector determination means. It may be configured.

The image processing program according to claim 3, wherein a first step of estimating an optical flow, which is a motion vector of a plurality of frame images corresponding to a time-sequential moving image screen, is estimated by the first step. A second step of calculating a vector distribution based on the optical flow, and a display position of the image is moved and displayed in accordance with the vector distribution calculated by the second step on a display screen of a predetermined display means. When the optical flow estimated by the third step and the first step is a vector distribution of a plurality of frame images corresponding to continuous imaging screens of moving images captured by the predetermined imaging unit, the imaging unit A fourth step of determining a vector distribution resulting from the movement of the image processing apparatus; The fourth step is a program for executing the imaging when the variance of the optical flow vector distribution in a plurality of unit areas having a predetermined number of pixels in the captured image is within a predetermined threshold. If the vector distribution resulting from the movement of the means is determined to be parallel or stationary, and the variance of the optical flow vector distribution in the plurality of unit areas exceeds a predetermined threshold, the unit area having the minimum vector amount is determined. And when the vector direction of the arbitrary unit region is substantially the same direction or substantially opposite to the direction between the unit region of the minimum vector amount and another arbitrary unit region, If the vector distribution resulting from the movement of the imaging means is determined to be enlarged or reduced, and the vector direction of the arbitrary unit area is substantially perpendicular, the imaging The vector distribution resulting from the movement of the means is determined to be rotation, and the third step includes a vector distribution resulting from the movement of the imaging means determined by the fourth step and a vector distribution of the image in the frame image. The display position of the image is moved and displayed based on the relationship. Further, in this case, as described in claim 4, the third step includes only an imaging screen when the vector distribution resulting from the movement of the imaging unit is determined to be parallel movement by the fourth step. May be displayed.

According to the image processing apparatus and the image processing program of the present invention, it is possible to obtain an effect that the motion vector of the image can be visually grasped by the optical flow estimation process.
According to the image processing apparatus and the image processing program of the present invention, the optical flow estimation process detects various camera works such as translation, zoom, and rotation, and enables image analysis. It is possible to extract an image to be analyzed and display the movement of the image.

Hereinafter, a first embodiment and a second embodiment of an image processing apparatus according to the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating the configuration of the image processing apparatus according to the first embodiment. In FIG. 1, a CPU 1 is connected to a ROM 2, a RAM 3, an image input interface (I / F) 4, a switch unit 5, an arithmetic unit 6, a frame memory 7, and an image output interface 8 via a system bus. The entire apparatus is controlled by sending and receiving commands and data.

  The ROM 2 stores a control program including an image processing program and initial data at initialization at power-on. The RAM 3 is a work area of the CPU 1 and is provided with various register, flag, and variable areas necessary for execution of the CPU 1 program. The image input interface 4 inputs image data from a video camera or other video resources. In addition to the power switch, the switch unit 5 includes a screen division switch for instructing screen division, a scene division switch for instructing scene division, a frame interval switch for specifying a frame interval between two frame images to be subjected to optical flow estimation processing, And other switches. The calculation unit 6 performs calculations related to the optical flow estimation process and other calculations described later in response to a calculation command from the CPU 1. The frame memory 7 stores the image data input from the image input interface 4 in units of frames. The image output interface 8 outputs the image data of the frame processed by the CPU 1 and the calculation unit 6 to the display 9 for display.

FIG. 2 is a flowchart of the main routine of the CPU 1 in the first embodiment. First, after predetermined initialization (step SA1), switch processing is performed (step SA2). FIG. 3 is a flowchart of the switch process. It is determined whether or not the screen division switch is turned on (step SB1). When this switch is turned on, "0" is set in the register div (step SB2). Further, it is determined whether or not the constant movement amount display is set by the display form setting switch (step SB3). If the constant movement amount display is set, “0” is set in the register mode (step SB4). If the constant movement amount display is not set, it is determined whether or not the constant frequency display is set (step SB5). When the constant frequency display is set, “1” is set in the mode (step SB6).
If the screen division switch is not on in step SB1, it is determined whether or not the scene division switch is on (step SB7). If this switch is on, div is set to “1” (step SB8).

  After setting mode to “0” in step SB4, or after setting mode to “1” in step SB6, or after setting “1” to div in step SB8, or in step SB7 If the division switch is not on, it is determined whether or not the frame interval switch is on (step SB9). If the switch is on and the frame interval is designated, the frame interval designated in the register f is determined. Is set (step SB10). When this value is “1”, the optical flow estimation process is performed on the adjacent frame images, and when this value is “10”, the optical flow estimation process is performed on the frame images at intervals of 10 frames. If the frame interval switch is not on, it is determined whether or not another switch is on (step SB11). If the other switch is on, processing corresponding to that switch is performed (step SB12). ). If the constant frequency display is not set in step SB5, other processing is performed (step SB13).

  After setting the numerical value of the frame interval in step SB10, or after the processing of step SB12 or step SB13, the process returns to the flowchart of FIG. 2 to perform image input processing (step SA3). In this process, image data input from an imaging means such as a video camera via the image input interface 4 is stored in the frame memory 7. Note that this image input processing may be performed according to an interrupt instead of a routine work. For example, in the case of NTSC, image input processing is performed according to an interrupt that occurs approximately every 1/30 seconds.

  Next, it is determined whether or not the value of the register div is “0” (step SA4). If this value is “0”, screen division processing is performed (step SA5). If the value of div is not “0”, it is determined whether or not the value of div is “1” (step SA6). If this value is “1”, scene division processing is performed (step S6). Step SA7). After the screen division processing in step SA5 or after the scene division processing in step SA7, image output processing is performed (step SA8). In this process, a frame image subjected to screen division processing or scene division processing is output to the display 9 via the image output interface 8. After this image output process, or when div is neither “0” nor “1”, other processes are performed (step SA9), and the process proceeds to step SA2 to perform the switch process.

  FIG. 4 is a flowchart of the screen division process in step SA5 of FIG. First, the frame F (n) to be processed is selected from the frame memory 7 (step SC1). For example, the latest frame is selected. Next, the previous frame F (n−f) by the numerical value of the frame interval stored in the register f is selected (step SC2). The numerical value of the frame interval is set according to the moving speed in the moving image. Then, the optical flow between the two selected frames F (n) and F (n−f) is calculated (step SC3).

  Next, the two frames F (n) and F (n−f) are divided into M × N (= J) blocks B (1) to B (J) (step SC4). For example, a frame having 640 pixels in the horizontal direction and 480 pixels in the vertical direction is divided into 10 blocks in the horizontal direction and 8 blocks B (1) to B (80) in the vertical direction. Accordingly, the number of pixels in each block is 3840 (horizontal pixel number 64 × vertical pixel number 60).

  Next, the variable j is set to “1” (step SC5), and the following loop processing is performed while incrementing the value of j. The average velocity vectors Vx (j) and Vy (j) of the block B (j) designated by the value of j are calculated (step SC6), and the calculated average velocity vectors Vx (j) and Vy (j) are stored in the RAM 3. Store (step SC7). Thereafter, the value of j is incremented (step SC8), and it is determined whether or not the value of j exceeds the maximum value J (step SC9).

一方、ｍｏｄｅの値が「１」である場合には、オプティカルフローの計算によるベクトル分布に従って、ブロックの画像を振動数一定でディスプレイ９に表示する（ステップＳＣ１２）。あるブロックについて振動数一定で表示する場合には、そのブロックにおいて指定した任意の点（ｘ０，ｙ０）は、その点（ｘ０，ｙ０）とベクトル分布に応じた点（ｘ０＋Ｖｘ，ｙ０＋Ｖｙ）とを結ぶ直線上を、速度ベクトルに比例した移動量で移動する。
ステップＳＣ１１又はステップＳＣ１２の表示の後は、図２のメインルーチンに戻る。 If the value of j is within the value of J, the process proceeds to step SC6 to repeat the loop process. If the value of j exceeds the value of J, the loop process is terminated and the value of the register mode is reached. Is “0” or “1” (step SC10). When the value of mode is “0”, the image of the block is displayed on the display 9 with a constant movement amount according to the vector distribution obtained by the optical flow calculation (step SC11). When displaying a certain block with a constant movement amount, an arbitrary point (x0, y0) designated in the block is connected to the point (x0, y0) and the following equation 1 corresponding to the vector distribution. It moves on a straight line at a frequency proportional to the velocity vector.

On the other hand, if the value of mode is “1”, the image of the block is displayed on the display 9 at a constant frequency according to the vector distribution calculated by the optical flow (step SC12). When a certain block is displayed with a constant frequency, an arbitrary point (x0, y0) designated in the block includes the point (x0, y0) and a point (x0 + Vx, y0 + Vy) corresponding to the vector distribution. It moves on the connecting line with a movement amount proportional to the velocity vector.
After the display of step SC11 or step SC12, the process returns to the main routine of FIG.

  FIG. 5 is a diagram showing an image displayed on the display 9 at a constant frequency. This figure shows an example in which a frame is divided into 80 blocks of 10 in the horizontal direction and 8 in the vertical direction. Each block is moved and displayed based on the average velocity vector. Compared with the conventional image shown in FIG. 25, it is possible to visually grasp the motion vector of the image by the optical flow estimation process despite the illustration of the still image. Actually, when this is viewed on a screen displayed as a moving image, the motion vector of the image can be recognized remarkably.

  FIG. 6 is a flowchart of the scene dividing process in step SA7 in the main routine of FIG. In a plurality of processing target frames stored in the frame memory 7, an optical flow of each frame is calculated (step SD1). In the optical flow calculation, a frame is divided into unit area areas each including a predetermined number of pixels (for example, 16 pixels × 16 pixels), and a motion vector for each area is calculated. Next, one frame is taken out (step SD2). For example, the latest frame is taken out. Then, the vector variance Dv is calculated based on the optical flow (step SD3).

  7, 8, and 9 are diagrams showing various camera work states by motion vectors for each area. In FIG. 7A, the vectors of the respective areas are uniformly directed to the right. In FIG. 7 (2), the vectors of the respective areas are uniformly in the upper right direction. In FIG. 8A, the vector of each area has a clockwise spiral shape around the hatched area. In FIG. 8B, the vector of each area has a counterclockwise spiral shape centering on the hatched area. In FIG. 9 (1), the vector of each area is in the direction toward the area indicated by hatching. In FIG. 9 (2), the vectors of the respective areas are directed outward from the areas indicated by hatching. In FIG. 7 to FIG. 9, the number of unit areas is reduced. However, in actuality, when the horizontal direction is a frame of 640 pixels and the vertical direction is 480 pixels, the area is 16 pixels × 16. If it is a pixel, it will be 1200 areas consisting of 40 horizontal x 30 vertical.

  Next, it is determined whether or not the motion vector variance Dv calculated in step SD3 is larger than a threshold value Tv (step SD4). Since the vectors of the areas shown in FIG. 7 are uniform in the same direction, the variance Dv is smaller than the threshold value Tv. Therefore, the camera work in FIG. 7 is classified as a parallel movement (step SD5). Note that images that are very close to still images with very little movement are also classified as parallel movements.

  On the other hand, the vectors of the areas shown in FIGS. 8 and 9 are in a dispersed state with no uniformity. In this case, in step SD4, it is determined that the variance Dv is larger than the threshold value Tv. In this case, the area of the minimum vector amount is detected (step SD6). Specifically, the area of the minimum vector amount is detected based on the optical flow calculated in step SD1. In FIGS. 8 and 9, the area indicated by hatching is the area of the minimum vector amount.

  Next, camera work determination calculation is performed (step SD7). In this calculation, the relationship between the direction of the straight line connecting the area of the minimum vector amount and the other area and the direction of the motion vector in the other area is analyzed. When the center position of the area of the minimum vector amount is the starting point coordinate (x, y), and the angle between the straight line connecting the starting point coordinate and the center position of the other area and the horizontal direction (X-axis direction) is θxy, As shown in FIG. 8 (2), in a state where the vector of each area is in a counterclockwise spiral shape, the vector V of that area is as shown in FIG. 10 (1).

Therefore, the horizontal vector Vx and the vertical vector Vy in the vector V are expressed by the following equations.
Vx = -V · sinθxy
Vy = V · cosθxy
Next, a straight line connecting the starting point coordinate and the center position of another area, a normal direction component dr, and a tangential direction component dx are obtained by the following equations.
dr = Vy · cos θxy−Vx · sin θxy
= V.cos.theta.xy.cos .theta.xy-V (-sin .theta.xy) .sin .theta.xy = V
dz = Vy · sin θxy + Vx · cos θxy
= V.cos.theta.xy.sin.theta.xy. + V. (-Sin .theta.xy) .cos .theta.xy = 0

  That is, in the state where the vector of each area is in a counterclockwise spiral shape, the component in the normal direction after normalization with the vector V is substantially “1” even if the error of the optical flow calculation is taken into consideration. The component in the tangential direction is almost “0”.

On the other hand, as shown in FIG. 9 (2), when the vector of each area is directed outward from the hatched area, the vector V of the area is as shown in FIG. 10 (2). .
Therefore, the horizontal vector Vx and the vertical vector Vy in the vector V are expressed by the following equations.
Vx = V · cosθxy
Vy = V · sinθxy
Next, a straight line connecting the starting point coordinate and the center position of another area, a normal direction component dr, and a tangential direction component dx are obtained by the following equations.
dr = Vy · cos θxy−Vx ·
= V · sinθxy · cosθxy−V · cosθxy · sinθxy = 0
dz = Vy · sin θxy + Vx · cos θxy
= V · sin θxy · sin θxy · + V · cos θxy · cos θxy = V

  That is, when the vectors in each area are directed outward from the hatched areas, the normal direction component after normalization with the vector V is almost the same even if the optical flow calculation error is taken into account. “0”, the tangential component is almost “1”.

This is equivalent to calculating the inner product of two vectors. As shown in FIG. 11 (1), Vu (| Vu | = 1) is a linear unit vector connecting the starting point coordinates and the center position of another area, and Vn, Vn and Vu are vectors of other arbitrary areas. If the vector angle of α is α, the inner product of the vectors is expressed by the following equation.
| Vn | ・ | Vu | ・ cosα = | Vn | ・ cosα
Alternatively, since the inner product of two vectors is generally represented by the sum of the product of horizontal components and the product of vertical components, if the angle between the vector Vn and the horizontal direction is β, the inner product of the vectors is It is also expressed by the following formula.
| Vn | cosβ · | Vu | cosθxy + | Vn | · sinβ · | Vu | sinθxy
= | Vn | cosβ · cosθxy + | Vn | · sinβ · sinθxy
= | Vn | .cos (β-θxy)
Since θxy is a horizontal angle with the vector Vu, β−θxy = α,
| Vn | · cos (β−θxy) = | Vn | · cosα

  Therefore, as shown in FIG. 11 (2), when the unit vector Vu and the vector Vn1 are substantially perpendicular, that is, as shown in FIG. 8 (1) or (2), when the camera work is rotated, α Is about 90 °, the inner product normalized by | Vn | is about “1”. On the other hand, when the unit vector Vu and the vector Vn2 are substantially parallel, that is, as shown in FIG. 9 (1) or (2), when the camera work is enlarged or reduced, α is about 180 ° or 0 °. Therefore, the inner product normalized by | Vn | is about “0”.

Therefore, it is determined whether or not the value of dr is larger than the value of dz (step SD8). If the value of dr is larger than the value of dz, the camera work is classified as rotation (step SD9). On the other hand, when the value of dr is not larger than the value of dz, the camera work is classified as enlarged or reduced (step SD10). After the camera work is classified in step SD5, step SD9, and step SD10, it is determined whether or not the frame to be processed remains in the frame memory 7 (step SD11). Then, the loop process up to step SD11 is repeated.
If no frame to be processed remains in the frame memory 7, the scene-divided image is displayed on the display 9 via the image output interface 8 (step SD 12). Then, the process returns to the main routine of FIG.

After classifying the camera work, exclude the influence of the camera work from the captured screen, image processing to analyze the movement of the target image, and exclude the influence of the moving image from the captured screen, Image processing for analyzing the movement of the camera work itself becomes possible.
FIG. 12 is a diagram illustrating an example of image processing for analyzing the motion of the target image. FIG. 12 (1) is a moving image in which a soccer game is imaged while moving the video camera in parallel, and motion vectors of v1, v2, and v3, which are vector distributions calculated by an optical flow for three players, respectively. Has moved. Assuming that the camera work vector distribution is vc toward the right direction, the vector distributions v1, v2, v3 of each player are as shown in FIG. 12 (2). , V2 ′, v3 ′. Therefore, if the camera work vector distribution is offset from the vector distribution of each player, the player image can be extracted from the moving image based on the player's own vector distributions v1 ′, v2 ′, and v3 ′.
Then, as shown in FIG. 12 (3), the image of each player extracted from the frame can be moved and displayed according to the respective vector distribution. As a result, by analyzing the movement of each player during the game, it is possible to study the policy of future practice and the measures for the next game.

FIG. 13 is a diagram illustrating an example of image processing for analyzing the movement of the camera work itself. FIG. 13 (1) shows consecutive frames when a soccer game is imaged while moving the video camera. On the screen of this frame, since the ball flies and the players are running, the movement of the camera work itself is difficult to understand. Therefore, an image having a motion vector faster than a predetermined threshold is excluded.
FIG. 13 (2) shows a continuous frame when a ball having a motion vector faster than the threshold and an image of a player are excluded. The movement of the camera work itself can be extracted from the frame of FIG. 13 (2) and easily analyzed. The camera work in this case is a simple one that is parallel from left to right, but in camera work that has been decoded by rotation, enlargement / reduction, etc., the camera vector is excluded by excluding images with motion vectors that are faster than a predetermined threshold. A remarkable effect can be obtained in analyzing the movement of the workpiece itself. For example, in the case of movie shooting, it is possible to analyze how camera work differs depending on the director, movie genre, etc., that is, movie grammar analysis.

  In this case, only when the movement of the camera work itself is a parallel movement may be displayed. In this case, by easily recognizing the attacking team by the moving direction of the video camera, it is possible to clearly grasp the flow of the entire game.

  FIG. 14 is a diagram for specifying the imaging position of the video camera. In a soccer field, background images without movement such as center circles, lines, goals, etc. are stored in the frame memory 7 in advance, and in the time-series frames (t), frames (t + 1),. The camera work is analyzed by matching with the background image.

As described above, according to the first embodiment, the CPU 1 estimates an optical flow that is a motion vector of a plurality of frame images corresponding to a time-sequential moving image screen, and determines a predetermined number of frame images. A vector distribution based on the optical flow estimated for each of the partial images is calculated, and the display position of the partial image is moved and displayed according to the calculated vector distribution. For example, the moving amount of the partial image is displayed with a constant amount according to the calculated vector distribution. Alternatively, the frequency of the partial image is displayed with a constant frequency according to the calculated vector distribution.
Therefore, the motion vector of the image can be visually grasped by the optical flow estimation process.

Further, according to the first embodiment, the CPU 1 determines the vector of the video camera when the estimated optical flow is a vector distribution of a plurality of frame images corresponding to continuous imaging screens of moving images captured by the video camera. The distribution is determined, and the display position of the partial image is moved and displayed based on the relationship between the vector distribution of the video camera and the vector distribution of the image in the frame image.
For example, the display position of the partial image is moved and displayed based on the vector distribution obtained by canceling the vector distribution of the video camera from the vector distribution of the frame image captured by the video camera. In this case, a specific image (player image in FIG. 12) corresponding to the vector distribution obtained by canceling the vector distribution of the video camera is extracted, and the display position of the specific image is moved based on the canceled vector distribution. To display.
Therefore, an image to be analyzed can be extracted by the optical flow estimation process, and the movement of the image itself can be displayed regardless of the movement of the camera work.
Alternatively, based on the vector distribution of the video camera, the display position of the background image without movement captured by the video camera is moved and displayed.
Therefore, the optical flow estimation process detects various camera works such as translation, zoom, and rotation to enable image analysis.

Further, the CPU 1 determines the vector distribution of the video camera when the variance of the optical flow vector distribution in a plurality of unit regions having a predetermined number of pixels (16 pixels × 16 pixels) in the captured image is within a predetermined threshold. If the dispersion of the optical flow vector distribution in a plurality of unit regions exceeds a predetermined threshold value, the unit region having the minimum vector amount is detected, and the minimum vector amount is determined. When the vector direction of the arbitrary unit area is substantially the same or substantially opposite to the direction between the unit area and another arbitrary unit area, the vector distribution of the video camera is enlarged or reduced. If the vector direction of the arbitrary unit area is substantially perpendicular, the vector distribution of the video camera is determined to be rotation.
Therefore, various camera work such as translation, zooming, and rotation can be accurately detected.
In this case, the CPU 1 displays only the imaging screen when the vector distribution of the video camera is determined to be parallel movement.
Therefore, in a sports game, the team on the attacking side can be easily recognized based on the moving direction of the video camera, so that the overall flow of the game can be clearly grasped.

Next, a second embodiment of the image processing apparatus according to the present invention will be described.
The configuration of the image processing apparatus in the second embodiment is the same as the block diagram of the first embodiment shown in FIG. Further, the image processing operation in the second embodiment is the same as the operation in the first embodiment except for a part of the scene division processing. Therefore, the description of the same processing as that of the first embodiment will be omitted, and the content of the first embodiment will be described while contents different from those of the first embodiment will be described.

  FIG. 15 is a flowchart of the scene division process in the second embodiment. As is clear from a comparison between FIG. 15 and FIG. 6 in the first embodiment, in FIG. 15, after step SD5 in which camera work is classified as translation or stationary, after step SD9 in which rotation is classified, or After step SD10 classified as enlargement or reduction, a speed parameter corresponding to the classification is calculated (step SD13) except for the case of stillness (when the speed is zero).

When the camera work is translated, if the horizontal velocity parameter at the coordinate (x, y) of an arbitrary point is Vx and the vertical velocity parameter is Vy, the horizontal vector at the coordinate (x, y) The distribution vx and the vertical vector distribution vy are expressed by the following equations. Therefore, the velocity parameter Vx in the horizontal direction and the velocity parameter Vy in the vertical direction can be obtained by the following equations.
vx = x + Vx
vy = y + Vy

When the camerawork is rotating, assuming that the rotation speed parameter when the coordinates (x, y) of an arbitrary point is rotating around the coordinates (xc, yc) is Vθ, the coordinates (x, y) The vector distribution vx in the horizontal direction and the vector distribution vy in the vertical direction are expressed by the following equations. Therefore, the rotation center positions xc and yc and the rotation speed parameter Vθ can be obtained by the following equations.
vx = (x−xc) cosVθ− (y−yc) sinVθ
vy = (x−xc) sinVθ− (y−yc) cosVθ

  In addition, when the camerawork and the subject are both rotating, when analyzing the subject's movement, the subject's movement is based on the vector distribution obtained by offsetting the vector distribution by the camera's rotational speed from the vector distribution by the subject's rotational speed. Analyze. On the other hand, when analyzing the motion of the camera work, that is, the video grammar, the camera work is analyzed based on the vector distribution obtained by canceling the vector distribution based on the rotation speed of the subject from the vector distribution based on the camera rotation speed.

  FIG. 16 is a diagram showing a screen of a videotape or DVD on which a movie by a well-known director is recorded. Three persons p0, p1, and p2 and other persons or sets of buildings not shown are photographed. When the camera moves from the scene of FIG. 16A to a scene of FIG. 16B after a unit time (for example, 1 second), the video grammar that is the motion of the camera work is analyzed by the motion vector optical flow. To do. As shown in the images of these two scenes, the image of the person p0 does not move, the image of the person p1 moves to the left by the distance d1, and the image of the person p2 moves to the right by the distance d2. ing. Therefore, in this case, it can be analyzed that the camera is rotating around the person p0. That is, the movement distances d1 and d2 per unit time represent the motion vectors of the persons p1 and p2.

FIG. 17 is an overhead view of camera work based on the analysis result. In the scene of FIG. 16A, the camera 11 is at the position of the dotted line in the circle 12 indicating the rotational movement trajectory of FIG. 17, and in the scene of FIG. 16B, the camera 11 has the motion vector θ centered on the person p0. It is in the position of the solid line rotated and moved by. In FIG. 17, a screen G1 shown by a two-dot chain line represents the movement of the image of the person p1, and a screen G2 shown by a two-dot chain line represents the movement of the image of the person p2. Assuming that the distance between the person p0 and the person p1 is r1, and the distance between the person p0 and the person p2 is r2, the following approximate expression is established.
r1 · θ = d1, r2 · θ = d2
That is, the following equation can be derived from this equation.
d2 / d1 = r2 / r1
Therefore, by analyzing the motion vectors of the images of the persons p1 and p2 in the video of FIG. 16, the positional relationship between the three persons p0, p1, and p2 can be analyzed.
FIG. 18 is a diagram in which an actual shooting site is predicted based on such video grammar analysis. As shown in this figure, the camera 11 is rotationally moved in the direction of the arrow along a circular orbit 12 centered on the person p0.

  FIG. 19 is a diagram showing the location scene screen of this movie. In FIGS. 19A, 19B, and 19C, which are continuous imaging screens, the human image 13 is displayed in the same size. . However, the image of the townscape in the background is getting smaller. According to the optical flow of these three scene motion vectors, it is analyzed that the camera work is to shoot at a low position, that is, at a low angle, by rotating the camera in the right direction and zooming to the left and rear in accordance with the progress of the person. .

  FIG. 20 shows a scene of this movie walk. When the scene in FIG. 20A changes to the scene in FIG. 20B, the person moves to the right by a distance of d1, and the background moves to the left by a distance of d2. Therefore, according to the optical flow of the two scene motion vectors, it is analyzed that the camera work is panning that moves the camera to the right. It can also be seen that the person moves a distance of (d1 + d2) during the time from the scene of FIG. 20A to the scene of FIG. 20B.

  FIG. 21 is a diagram showing a close-up scene of a person in this movie. Since the scene changes to the scene of FIG. 21A, the scene of FIG. 21B, and the scene of FIG. 21C, the camera moves from the bottom to the top according to the optical flow of these three scene motion vectors. It is analyzed that it is camera work to shoot while. Although not shown in the drawing, camera work is also performed in which the camera is moved from the top to the bottom, contrary to the scene of FIG. In other words, it is camera work of vertical video grammar.

  FIG. 22 is a diagram showing a scene in the cockpit of a fighter in the battle scene of this movie. Since the scene changes to the scene of FIG. 22A, the scene of FIG. 22B, and the scene of FIG. 22C, and the image of the horizontal line rotates to the right, according to the optical flow of these three scene motion vectors. As the camera in the cockpit rotates to the left of the fighter, it is analyzed that the camera work is taken while rotating. The rotating camera work shown in FIG. 16 is a camera work that rotates about a predetermined point on the shooting site as a vertical axis, whereas the rotating camera work shown in FIG. 22 is centered on the optical axis of the camera. It is a camera work that rotates.

FIG. 23 is a series of diagrams showing different types of camera work according to the passage of time in this movie, and changes in the screen according to the types of camera work are indicated by arrows. As shown in this figure, rotation (rotation with a predetermined point as a vertical axis) camera work, zoom camera work, pan camera work, pitch camera work, panning camera work, although not shown in the embodiment, The screen changes as shown by an arrow in accordance with the camera work of the diagonal shake that is a combination of the vertical shake and the camera work of the rotation (rotation around the optical axis of the camera).
Each type of camera work may be displayed with each scene number, or each type of camera work may be displayed with the start time and end time of each scene.

  FIG. 24 is a diagram showing a screen when the video grammar is analyzed by a series of camera work of the movie. As shown in this figure, a series of moving images is divided into sections corresponding to camera work as a vector distribution classification, that is, scenes (1, 2, 3,...) Composed of a plurality of screens, one for each divided section The image (one screen of the display 9) is moved and displayed according to the vector distribution of the section. In this case, moving display in each section, that is, rotation of the image around the vertical axis or horizontal axis, zoom that is enlargement or reduction of the screen, panning to move the image horizontally or diagonally, image up or down As shown in FIG. 24, the images in the divided sections are displayed with a space between them so that the pitching movement, the arbitrary display position on the screen, that is, the rotation around the optical axis of the camera, and the like are possible.

  Furthermore, in this case, it is also possible to select and display an arbitrary scene, and to extract a plurality of divided sections only for a specific camera work, for example, a camera work of rotating an image around the vertical axis, It is also possible to analyze the video grammar in more detail by comparing the rotation of each image. Alternatively, it is possible to analyze video grammar for each work by extracting and comparing specific camera work from other works of the same director or works of other prominent directors.

In the analysis of video grammar, when the camera work is zoom as shown in FIG. 19, the coordinates (x, y) of an arbitrary point are expanded outward with the coordinates (xc, yc) as the center. Or, when the image is reduced inward with the coordinate (xc, yc) as the center, if the enlargement or reduction speed parameter is Vz, the horizontal vector distribution vx and the vertical vector at the coordinate (x, y) The distribution vy is expressed by the following formula. Accordingly, the enlargement / reduction center position xc, yc and the enlargement / reduction speed parameter Vz can be obtained by the following equations.
vx = (x−xc) Vz
vy = (y−yc) Vz

As described above, according to the second embodiment, the CPU 1 calculates the speed parameter at an arbitrary position of the frame image that is the target of optical flow estimation, and the partial image at a speed according to the calculated speed parameter. Move the display position of.
Thus, the optical flow estimation process detects various camera work velocities such as translation, zoom, and rotation, allowing quantitative analysis of the camera work.

  In each embodiment, the interval between two frames for which the optical flow is estimated is not particularly limited, but the optical flow is normally estimated between adjacent frames. In this case, the value of the register f is set to “1” in step SB10 of the switch process shown in FIG. However, if the motion of the moving image to be analyzed is considerably slower than the frame interval of 33.3 ms, the amount of motion vector between adjacent frames becomes very small, and analysis is possible even if optical flow is estimated. Have difficulty. In such a case, the value set in f is increased, and the interval between two frames for performing optical flow estimation is lengthened.

Moreover, in each embodiment, the image processing by the optical flow has been described taking a soccer game as an example, but it is obvious that the present invention can be applied to image processing for other sports games or images other than sports. is there.
In each embodiment, the specific image is extracted based on the shape of the image. However, the specific image may be extracted based on the color information of the image.
In the first embodiment, in order to specify the imaging position of the video camera shown in FIG. 14, the camera work is analyzed by matching with the background image. However, the camera work is analyzed directly by hand. You may make it perform.

In the first embodiment and the second embodiment, the invention of the image processing apparatus including the ROM 2 in which the image processing program is stored in advance has been described. However, the flexible disk (FD), CD, DVD, etc. A configuration in which an image processing program read from an external storage medium is installed and executed in a writable non-volatile memory such as a fresh ROM or a hard disk in a general-purpose information processing apparatus such as a personal computer, a minicomputer, an office computer, or a workstation. Is possible. Alternatively, a configuration in which an image processing program downloaded via a network such as the Internet is installed in such a nonvolatile memory and executed is also possible.
That is, by installing a program in a general-purpose information processing apparatus and executing image processing, it is possible to realize not only an image processing apparatus invention as a so-called product invention but also an image processing program invention. Further, since a flexible disk, CD, or DVD itself storing such an image processing program can be sold independently, it goes without saying that the invention of the storage medium can also be realized.

The image processing program has a configuration shown in the flowchart of the CPU in each of the above embodiments. That is,
A first step of estimating an optical flow that is a motion vector of a plurality of frame images corresponding to a time-sequential moving image screen; a second step of dividing the frame image into a predetermined number of partial images; A third step of calculating a vector distribution based on the optical flow estimated in the first step for each of the partial images divided in step 2, and the third step on the display screen of a predetermined display means. And a fourth step in which the display position of the image is moved and displayed according to the vector distribution calculated in this step.

  In the third step, an image is displayed with a movement amount corresponding to the vector distribution calculated in the second step.

  In the third step, an image is displayed at a frequency corresponding to the vector distribution calculated in the second step.

  The method further includes a fourth step of dividing each frame image corresponding to a moving image screen continuous in time series into a predetermined number of partial images, wherein the second step is a partial image divided by the fourth step. A vector distribution based on the optical flow estimated by the first step is calculated.

  A vector resulting from movement of the imaging means when the optical flow estimated by the first step is a vector distribution of a plurality of frame images corresponding to continuous imaging screens of moving images captured by the predetermined imaging means. A fifth step of determining a distribution, wherein the third step includes a vector distribution resulting from the movement of the imaging means determined by the fifth step and a vector distribution of an image in the frame image; The display position of the partial image is moved and displayed based on the relationship.

  The third step is a partial image based on a vector distribution in which the vector distribution resulting from the movement of the imaging unit determined by the fifth step is offset from the vector distribution of the frame image captured by the imaging unit. The display position of is moved and displayed.

  A sixth step of extracting a specific image corresponding to the vector distribution in which the vector distribution resulting from the movement of the imaging unit is canceled based on the partial image divided by the image dividing unit; In this step, the display position of the specific image is moved and displayed based on the canceled vector distribution.

  In the third step, the display position of the background image having no motion imaged by the imaging unit is moved and displayed based on the vector distribution resulting from the movement of the imaging unit determined in the fifth step. .

  In the fifth step, when the dispersion of the optical flow vector distribution in a plurality of unit regions having a predetermined number of pixels in the captured image is within a predetermined threshold, the vector distribution resulting from the movement of the imaging unit is calculated. When the variance of the optical flow vector distribution in the plurality of unit regions exceeds a predetermined threshold value, the unit region having the minimum vector amount is detected and the minimum vector amount is determined. Vector distribution resulting from the movement of the imaging means when the vector direction of the arbitrary unit region is substantially the same direction or the substantially opposite direction with respect to the direction between the unit region and any other unit region If the vector direction of the arbitrary unit region is a substantially perpendicular direction, the vector distribution resulting from the movement of the imaging means is calculated. It determines that.

  In the third step, only the imaging screen when the vector distribution resulting from the movement of the imaging means is determined to be parallel movement in the fifth step is displayed.

  The third step calculates a speed parameter at an arbitrary position of the frame image that is the target of the optical flow estimation in the first step, and sets the display position of the partial image at a speed according to the calculated speed parameter. Moving.

  The third step divides a series of moving images into sections corresponding to the classification of the vector distribution calculated in the second step, and one image for each divided section according to the vector distribution of the section. Moving.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment. The flowchart of the main routine in 1st Embodiment. The flowchart of the switch process of the main routine in 1st Embodiment. The flowchart of the screen division | segmentation process of the main routine in 1st Embodiment. The figure which shows the example of a display of the screen division | segmentation in 1st Embodiment. The flowchart of the scene division | segmentation process of the main routine in 1st Embodiment. The figure which shows the vector distribution for every unit area | region when camerawork is a parallel movement in 1st Embodiment. The figure which shows the vector distribution for every unit area | region when camerawork is rotation in 1st Embodiment. The figure which shows vector distribution for every unit area | region when camerawork is expansion or reduction in 1st Embodiment. The figure which shows the relationship between the direction of the straight line which connects a starting point coordinate and a unit area | region, and the vector distribution in a unit area | region in 1st Embodiment. The figure explaining the inner product of the unit vector of the direction of the straight line which connects a starting point coordinate and a unit area, and the vector distribution in a unit area in a 1st embodiment. The figure which shows the example of the image process for analyzing the motion of a target image in 1st Embodiment. The figure which shows the example of the image process for analyzing the motion of camera work in 1st Embodiment. The figure for pinpointing the imaging position of a video camera in a 1st embodiment. The flowchart of the scene division process of the main routine in 2nd Embodiment. The figure which shows the screen which recorded the studio scene of the movie in 2nd Embodiment. The overhead view of the camera work based on the analysis result in 2nd Embodiment. The figure which estimated the actual photography field based on the analysis of the image grammar in 2nd Embodiment. The figure which shows the screen which recorded the location scene of the movie in 2nd Embodiment The figure which shows the location scene of the walk of the movie in 2nd Embodiment. The figure which shows the close-up scene of a person in the movie in 2nd Embodiment The figure which shows the scene in the cockpit of a fighter in the battle scene of the movie in 2nd Embodiment. The figure which shows the camera work according to progress of time in the movie in 2nd Embodiment. The figure which shows the screen at the time of analyzing image grammar by a series of camera work of a movie. The figure which shows the example of a display of the screen division | segmentation in a prior art.

Explanation of symbols

1 CPU
2 ROM
3 RAM
4 Image Input Interface 5 Switch Unit 6 Arithmetic Unit 7 Frame Memory 8 Image Output Interface 9 Display

Claims (4)

Motion estimation means for estimating an optical flow that is a motion vector of a plurality of frame images corresponding to a time-series video screen;
Vector calculation means for calculating a vector distribution based on the optical flow estimated by the motion estimation means;
Display control means for moving and displaying the display position of the image according to the vector distribution calculated by the vector calculation means on the display screen of the predetermined display means;
When the optical flow estimated by the motion estimation unit is a vector distribution of a plurality of frame images corresponding to continuous imaging screens of moving images captured by the predetermined imaging unit, the vector distribution resulting from the movement of the imaging unit Vector determination means for determining
With
The vector determining means parallelizes the vector distribution resulting from the movement of the imaging means when the variance of the optical flow vector distribution in a plurality of unit regions having a predetermined number of pixels in the captured image is within a predetermined threshold. If the variance of the optical flow vector distribution in the plurality of unit regions exceeds a predetermined threshold value, the unit region having the minimum vector amount is detected and the minimum vector amount is determined. When the vector direction of the arbitrary unit region is substantially the same direction or the substantially opposite direction with respect to the direction between the unit region and another arbitrary unit region, the vector distribution resulting from the movement of the imaging means is When the vector direction of the arbitrary unit region is a substantially perpendicular direction, the vector distribution resulting from the movement of the imaging unit is determined. It is determined that the rolling,
The display control means moves and displays the display position of the image based on the relationship between the vector distribution resulting from the movement of the imaging means determined by the vector determination means and the vector distribution of the image in the frame image. An image processing apparatus. 2. The image processing apparatus according to claim 1 , wherein the display control unit displays only an imaging screen when a vector distribution resulting from movement of the imaging unit is determined to be parallel movement by the vector determination unit. . A first step of estimating an optical flow that is a motion vector of a plurality of frame images corresponding to a time-sequential moving image screen;
A second step of calculating a vector distribution based on the optical flow estimated by the first step;
A third step of moving and displaying the display position of the image according to the vector distribution calculated by the second step on the display screen of the predetermined display means;
A vector resulting from movement of the imaging means when the optical flow estimated by the first step is a vector distribution of a plurality of frame images corresponding to continuous imaging screens of moving images captured by the predetermined imaging means. A fourth step of determining the distribution;
Is a program for causing a computer of an image processing apparatus to execute
In the fourth step, when the dispersion of the optical flow vector distribution in a plurality of unit regions having a predetermined number of pixels in the captured image is within a predetermined threshold, the vector distribution resulting from the movement of the imaging unit is calculated. When the variance of the optical flow vector distribution in the plurality of unit regions exceeds a predetermined threshold value, the unit region having the minimum vector amount is detected and the minimum vector amount is determined. Vector distribution resulting from the movement of the imaging means when the vector direction of the arbitrary unit region is substantially the same direction or the substantially opposite direction with respect to the direction between the unit region and any other unit region If the vector direction of the arbitrary unit region is a substantially perpendicular direction, the vector distribution resulting from the movement of the imaging means is calculated. It is determined that,
In the third step, the display position of the image is moved and displayed based on the relationship between the vector distribution resulting from the movement of the imaging means determined in the fourth step and the vector distribution of the image in the frame image. An image processing program characterized by: The third step, an image according to claim 4, characterized in that display only the imaging screen when the originating vector distribution to the movement of the imaging means by said fourth step is determined to be translated Processing program.

Images