JP3411469B2 - Frame multiplex image creation method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system for presenting and browsing stored images or images obtained via a network or broadcast on a display such as a personal computer, and in particular, extracting a plurality of frames from image data and displaying them. The present invention relates to a frame multiplex image creating method for creating one still image by synthesizing.

[0002]

2. Description of the Related Art Conventionally, when a plurality of frames are overlapped to create one still image, a plurality of frames are selected from video data at equal intervals, and pixel values at the same position of each selected frame are selected. A general method is to obtain an average value over frames and use the average value as a pixel value of a multiple still image.

[0003]

In the prior art, the average value of the pixel values is used as the pixel value of the frame-multiplexed image over a plurality of frames to be multiplexed, so that the pixel value of a moving portion becomes small and becomes almost invisible. There is a problem. For example, when a human swings his arm up,
The shape (position) of a certain arm is shown in only one frame, and in other frames, the background is shown in that position. Therefore, if the pixel values at the corresponding positions of these frames are simply taken, they will be buried in the background and the arm will be almost invisible.

A first object of the present invention is to solve the above-mentioned problems of the prior art and to provide a method for producing a frame multiplex image in which an image appearing in only one frame is not faded or invisible. Especially.

A second object of the present invention is to, when determining a frame to be multiplexed, grasp the motion situation of an image or an object and escape a frame having an important meaning such as a location where the motion direction has changed. An object of the present invention is to provide a method of creating a frame-free multiplex image.

A third object of the present invention is to automatically indicate the movement direction of an object in a video without manually drawing an arrow on the image by using CG technology or the like. An object of the present invention is to provide a frame multiplex image creating method for drawing an arrow to shorten the time required for drawing the arrow.

[0007]

In order to achieve the above first object, the present invention relates to the luminance of pixels at the same position on each frame image (called a key frame image) to be multiplexed in video data. The average value and the standard deviation of the components are calculated to determine the degree of variation in pixel values. A large weighting coefficient is given to a pixel with a large variation and a small weighting coefficient is given to a small pixel. A feature is that one still image is created by multiplying and adding pixel values and performing this for all pixels of the key frame image.

In order to achieve the second object,
The present invention, for every frame in the video data, or for each frame included in the specified range,
It is characterized in that a region of an object in an image is extracted, a center of gravity of the region is calculated, and a frame in which the moving direction and the moving amount of the center of gravity are rapidly changed is selected as a key frame.

Further, in order to achieve the above third object, the present invention obtains the center of gravity of an object in an image in the same manner as described above, traces the movement between the frames of the center of gravity, and finds the best track on the trajectory. A feature is that a straight line or a curve that fits well is calculated, and an arrow indicating the operation direction is automatically drawn on the straight line or the curved line on the output still image.

[0010]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a conceptual diagram showing an outline of a frame multiplex image creating method of the present invention. K frames 12 within the specified range from the video data 110
By selecting 0 (key frame image) and superposing these frame images corresponding to pixels, a frame multiplex image in which the motion of one still image 130 can be understood is created. Note that FIG.
Shows that four (K = 4) key frames are selected.

FIG. 2 is a processing flowchart of the first embodiment of the frame multiplex image creating method of the present invention, which corresponds to the invention of claim 1.

The start / end frame of the frame range to be processed and the number K of frames to be selected are set to Start / End Fr.
By specifying the ame number and the number-of-selections instruction section 200, the K data within the specified range from the video data 110
A frame (key frame) 120 is selected and input (step 201). Here, as a method of selecting K frames, for example, K within a designated frame range is selected.
A method or the like in which -1 is equally divided and a frame corresponding to the equally divided points of the interval is selected can be considered.

If the input video data is composed of components other than the YIQ component in the NTSC video signal, the key frame image is converted into the YIQ component (step 20).
2). An example of the conversion formula to the YIQ component when the input video signal is composed of RGB is shown in the following formula (1).

[0014]

[Equation 4]

For each luminance component (Y component) of the K keyframe images after YIQ conversion, the initial pixel position (i,
j) is set (step 203). Here, the notation (i, j) is used to specify pixels on the image, and the notation k is used to specify the selected K frames, and the selected k-th frame is used. The brightness value at the upper position (i, j) is represented as f _Y (i, j; k).

The average value (mean) and standard deviation (σ) of the K luminance values at the position (i, j) are expressed by the following equation (2),
It is calculated by (3) (step 204).

[0017]

[Equation 5]

[0018]

[Equation 6]

Using the luminance value, average value and standard deviation of each pixel (K), the number of pixels satisfying the following equation (4) is obtained,

[0020]

[Equation 7]

A weighting coefficient to be applied to the pixel value of each pixel is determined based on a rule determined in advance by the number of pixels (step 205).

FIG. 3 shows the relationship between the luminance value of each pixel, the average value and the standard deviation when K = 4. Figure 3
In (a), since all K (4) luminance values are within the range of mean _{i, j} ± σ, the number of pixels satisfying the equation (4) is 0.
Is. Similarly, in the case of FIG. 3B, the number is 1, the number in FIG. 2C is 2, the number in FIG. 3D is 3, and the number in FIG. Depending on the number of pixels satisfying the equation (4), for example, the weighting factor w by which the pixel value f (i, j: k) of each pixel is multiplied according to Table 1.
Determine (j, j; k).

[0023]

[Table 1]

The sum of weighting factors w (j, j; k) with respect to k must be 1. That is, it is necessary to satisfy the following expression (5).

[0025]

[Equation 8]

Using the weighting factor w (j, j; k) determined by the processing of step 205, the pixel value at the position (i, j) on the frame multiplex image is calculated as in the following equation (6). (Step 206). In equation (6), g
(I, j) indicates the pixel value at the position (i, j) on the frame multiplex image. The subscripts R, G, B indicate each component in the case of RGB image.

[0027]

[Equation 9]

In step 207, step 204
It is determined whether or not the processing of steps 206 to 206 have been performed for all pixels, and if the processing is not completed, the next pixel position is set (step 208) and then the process returns to step 204 and the position (i,
The process is repeated from the calculation of the average / standard deviation of the K luminance values in j). On the other hand, when it is determined that the process is finished, the frame multiplex image 130 is output and the process is finished.

FIG. 4 is a processing flowchart of the second embodiment of the frame multiplex image creating method of the present invention. This corresponds to the invention of claim 2 and shows a process applied when a key frame is selected by paying particular attention to the movement of the hand in a moving image of a person.

In the Start / End Frame number designating section 400, by designating the frame range to be the subject of this processing, the frame image data within the designated range is sequentially input from the video data 110 frame by frame (step 401). The position of the center of gravity of each of the left and right hands is extracted from the input frame image data (step 40).
2).

FIG. 5 shows an example of the barycentric position extraction processing for the left and right hands. For example, if the color components of the input image are YI such as RGB.
If the image is other than the Q component image, it is converted to the YIQ component image using the above equation (1) or the like (step 501). The Y
Utilizing the fact that the I component image of the IQ components has a high pixel value for the skin color, threshold processing is performed on the I component image to extract a skin color region (step 502), and each of the extracted skin color regions is extracted. Is labeled (step 503).
Then, regions that are considered to be the left and right hand regions are extracted from the plurality of labeled skin color regions (step 504).
For example, it is determined that the skin-colored area having the largest area is the face, and the areas having the second and third largest areas are the left and right hands.

How to determine the left and right hand regions from a plurality of skin color regions will be described with reference to FIG. FIG. 6A is the original image, and FIG. 6B is the skin color region extraction result.
Labeled as 3, 4, 5, and 6. Figure 6
In (b), it is due to noise data that the three or more areas are judged to be skin-colored, and it is not always true skin-colored areas. Of these, the area of the label 2 having the largest area is the face, and the second and third largest areas of the label 3,
It is determined that 4 is the area of the left hand and the area of the right hand, respectively.

Regarding the areas judged to be the left and right hand areas,
The barycentric position (C _i , C _j ) of each area is obtained using the following equation (7) (step 505). In Expression (7), N represents the number of pixels included in the left or right hand region, and i _n and j _n represent coordinates of pixels included in the left or right hand region.

[0034]

[Equation 10]

Returning to FIG. 4, the movement direction and the movement amount between the frames of the calculated left and right center of gravity positions are calculated (step 403). Then, when the movement direction or movement amount exceeds a predetermined threshold value, it is determined that the movement direction or movement speed of the hand in the frame has changed, the frame is determined to be the key frame 120, and the frame image Save the data (steps 404, 40)
5). If it is not determined to be a key frame, the saving process is not performed and the next frame process is performed.

In step 406, it is determined whether or not all the frames have been processed, and if not completed, the next frame number is set (step 407) and then step 401.
The process returns to the frame image data input process of, and the same process is performed for the next frame.

When the processing is completed for all the frames of the video data 110 as described above, the frame multiplex image creation processing is executed using the stored key frame image (step 408), and the frame multiplex image is created. Image 13
Create 0. Note that step 408 is the same as the processing after step 203 in FIG.

FIG. 7 is a processing flowchart of the third embodiment of the frame multiplex image creating method of the present invention. This corresponds to the invention of claim 3 and shows a process applied when automatically drawing an arrow indicating a hand movement in a moving image of a person.

In the Start / End Frame number designating section 700, by designating the frame range to be the subject of this processing, the frame image data within the designated range is sequentially input from the video data 110 in frame units (step). 701). From the input frame image data, the barycentric positions of the left and right hands are extracted by the same method as described with reference to FIG. 5 (step 702), and the barycentric coordinate data 71 is extracted.
After storing 0 (step 703), key frame determination processing is performed, and the key frame 120 is selected and stored (step 704). This step 704 corresponds to steps 403-405 of FIG. The key frames may be selected in the same manner as in the first embodiment, and the barycentric positions of the left and right hands may be extracted from those frame images.

In step 705, the video data 1
It is determined whether or not the above processing has been performed for all 10 frames, and if the processing has not been completed, the next frame number is set (step 706) and the process returns to the frame image data input processing of step 701. , Similar processing is performed for the next frame. When the processing is completed for all the frames, the plurality of stored key frame images 12
The frame multiplex image creation process is executed using 0 to create the frame multiplex image 130 (step 707). This step 707 is the same as the processing after step 203 in FIG.

After that, the barycentric coordinate data 7 saved
10 is used to calculate an equation representing a straight line or a curve that best approximates the connection between the centers of gravity of the left and right hands of each frame (step 708). An example of a straight line or a curve that most closely approximates the connection between the centers of gravity is shown in FIG. In the figure, the left hand center of gravity of each frame is linearly arranged vertically on the right side, and the right hand center of gravity of each frame is lined up on the left side. As shown in FIG. 8A, it is a straight line that best approximates the left-hand center-of-gravity connection, and a curved line is the right-hand one. In some cases, it is best to approximate not only one straight line or curve but also a plurality of straight lines or curves.

The calculated straight line / curve is drawn on the frame multiplex image 130 with an arrow indicating the passage of time, and is output as a frame multiplex image with arrow 720 (step 709). The frame-multiplexed image with the arrow is shown in FIG.
It shows in (b).

The respective embodiments of the present invention have been described above, but these are realized by using a so-called computer. At that time, FIG. 2, FIG. 4, FIG.
The processing flow (procedure) as shown in (1) may be prepared as an application program in a computer in advance, or a recording medium (floppy disk,
Alternatively, an optical disk or the like) may be read (installed) by a computer.

[0044]

As described above, the following effects can be obtained by using the frame multiplex image creating method of the present invention.

(1) The degree of variation of pixel values at the same position on the frame images to be multiplexed is determined by using the average value and the standard deviation, and the weighting coefficient to be applied to each pixel value is determined based on the determination result. By adding, it is possible to prevent the pixel value of a moving portion from becoming smaller and becoming almost invisible as in the conventional case.

(2) When determining a frame to be multiplexed, the motion direction of the object in the image is grasped and the motion direction of the object is selected by selecting the frame in which the motion amount is drastically changed. It is possible to prevent escape of a frame having an important meaning such as a changed part.

(3) When the moving direction of the object in the image is shown, the center of gravity of the object in the image is obtained, the movement between the frames of the center of gravity is tracked, and the trajectory obtained by the tracking is best expressed. By calculating a straight line or curve and automatically drawing an arrow indicating the direction of movement on the output still image,
It is not necessary to manually draw by using CG technology or the like, and the time required for arrow drawing can be shortened.

[Brief description of drawings]

FIG. 1 is a conceptual diagram showing an outline of a frame multiplex image creating method of the present invention.

FIG. 2 is a flowchart showing a process of the first exemplary embodiment of the present invention.

FIG. 3 is a diagram illustrating a relationship among a luminance value of pixels at the same position on each key frame, an average value, and a standard deviation.

FIG. 4 is a flowchart showing a process of the second embodiment of the present invention.

5 is a flowchart showing an example of left and right centroid position extraction processing in FIG. 4. FIG.

FIG. 6 is a conceptual diagram showing an outline of extraction of a skin color area, classing, and labeling as area extraction of an object in an image.

FIG. 7 is a flowchart showing a process of the third embodiment of the present invention.

FIG. 8 is a conceptual diagram showing an example of a locus of the center of gravity of the left and right hands and a straight line / curve that approximates the locus of the center of gravity of the hand as a drawing of the motion of an object in a video.

[Explanation of symbols]

110 video data 120 keyframe images 130 frame multiplex image

Claims (3)

(57) [Claims] 1. A method for extracting a plurality of frame images from video data and superimposing them to create one still image, wherein K frames are selected from all frames in the video data or frames included in a specified range. A frame of (K is an integer of 2 or more) is selected as a key frame, and the same position (i,
For K pixels in j), the average value (mean _{i, j} ) and standard deviation (σ _{i, j} ) in the luminance component are calculated by the following equation, and [Equation 2] f _Y (i, j; k): Position (i, j on the k-th frame image
j) luminance value of pixel, using the luminance value f _Y (i, j; k) of each pixel and the calculated average value (mean _{i, j} ) and standard deviation (σ _{i, j} ), mean _{i, j} + Σ _{i, j} <f _Y (i, j;
k) or mean _{i, j} −σ _{i, j} > f _Y (i, j; k) is given a large weighting coefficient, and mean _{i, j} −σ _{i, j} <f _Y (i, _j
j; k) <mean _{i, j} + σ _{i, j} is given a small weighting coefficient to the pixel value g at the same position (i, j) on the output still image.
(I, j) is calculated by the following equation, and A method for producing a frame multiplex image, characterized in that one still image in which K key frames are superposed is produced by performing the above-mentioned processing for each pixel of the key frame image. 2. The frame multiplex image creating method according to claim 1, wherein for every frame in the video data or for a frame included in a specified range, an area of a predesignated object in the video is set. A method for creating a frame multiplex image, comprising extracting, calculating the center of gravity of the region, and selecting a frame whose moving direction and moving amount of the center of gravity exceed a predetermined threshold value as a key frame. 3. The frame multiplex image creating method according to claim 1, wherein a region of a predesignated object in a video is extracted for each frame, a center of gravity of the region is calculated, and the center of gravity is calculated for each frame. A frame multiplex image characterized by tracing over a frame, calculating a straight line or curve of the traced locus, adding an arrow indicating a motion direction to the straight line or curve, and drawing on an output still image How to make.