JP4317003B2 - Highlight still image generation / distribution device - Google Patents

Description

  The present invention relates to an apparatus for generating and distributing highlight scenes such as sports, movies, and dramas using still images. .

  In recent years, with the widespread use of mobile phone terminals, it has become possible to easily acquire game results, news, etc., such as sports, not only at home but also on the go or in the vehicle of a moving vehicle. Accordingly, there is a need for an apparatus for generating and distributing not only text data but also highlight scene images and videos.

  For example, in order to view or view the highlight scene image or video on a mobile device when the text base confirms that a baseball team player who is cheering on the train is returning home A device for extracting a corresponding highlight scene as a still image and distributing it using a network line is applicable.

Conventionally, moving image editing software has been used as means for extracting a still image from a specific portion of a moving image. In this software, for example, a change in the scene is detected based on the brightness and color difference between the previous and next frames, and the first frame immediately after the scene change is extracted as a still image. For example, the threshold value is compared with the difference in luminance, color information, correlation between blocks, and the like between the previous and next frames, and when the difference is larger than the threshold value, the scene is changed and the cut point of the video is set. A still image is extracted based on the cut point (see, for example, Patent Document 1).
However, since the cut point is not always a highlight scene, the extracted still image cannot be used as a highlight scene image.

Therefore, several methods for extracting still images associated with highlight scenes from moving images have been proposed. For example, in order to quickly search for a desired highlight scene from a moving image, a method for extracting a representative image has been proposed (see, for example, Patent Document 2).
JP, 2000-350156, A In this example, the size of the sound which shows a cheer of a spectator, and the area of the specific color which shows close-up of a person are used as selection criteria of a representative image.

  However, since the representative image is not necessarily a highlight scene, it is necessary to finally select it manually.

When delivering a highlight video to a mobile phone, a thumbnail image, which is a small image sample corresponding to the video, is displayed on the liquid crystal screen of the mobile phone, and a desired scene is selected by selecting the image. (For example, see Patent Document 3).
JP, 2003-250140, A A key frame picture is used for a thumbnail picture, and one key frame picture is made to correspond to each picture.

  However, this does not describe how the highlight video is extracted from the original video. Whether or not the key frame image is a representative still image of the highlight scene moving image cannot be determined because a specific extraction method is not described.

  Accordingly, since it is difficult to reliably extract a highlight still image by the conventional method or a combination of the conventional methods, a service for browsing the highlight still image via a network or the like cannot be implemented.

  The present invention makes it possible to view a highlight still image with certainty by adjusting the cut-out density of frame images, that is, the time interval, when the still image is automatically extracted from the highlight moving image. The present invention provides an apparatus for reliably generating still images.

  In order to solve the above problems, the highlight still image generating / distributing apparatus of the present invention is configured as shown in the following (1) to (5).

(1) in the highlight generating a scene, highlight still image generation and distribution apparatus that distributes a still image, a highlight designating means for designating a portion to be extracted from the moving image obtained by moving image obtaining means as a highlight moving picture, Highlights of extracting a highlight moving image extracting unit for cutting out the highlight moving image designated by the highlight specifying means, on the basis of the cut-out interval, the highlight still picture image from said highlight moving image cut out a still image extracting unit, the previously prepared depending on the type of the moving picture, and cut density template representing the distribution of the cut-out time interval when extracting the highlight moving picture or al the highlight still image cut out, cut out specify clipping density represented by the cut-out interval of the highlight still images with each other when Comprising a cut-out density designating means, extracting a still image display and distribution means for transmitting the highlight still picture images to the user has, the said cut-out density designating means, by using the cut-out density template, cut out the high The cutout density is set to be high in the light moving image, and the cutout density is set to be low in the moving image other than the cut out highlight moving image .

(2) In the above (1), if the subject is used as a marking means of RF tags possession, the marking information from the highlight moving image extracted based on, for obtaining the highlight still image, the cut-out density is set to be high density when the RF tag passes, stay certain position.

In (3) above (1), if the operator specifies the location with the mouse, it acquires the highlight still image from the highlight moving image extracted based on the specified ibid, the cut density , The density is set so as to be high density at the location .

(4) In the above (1), as the cut density template uses distribution with peak formation of the normal distribution, the position of the peak by matching the position, set high and the cut-out density and when there is no still picture which matches the cutout time which is determined on the basis of the cut-out interval, and extracts a still image of nearest of said cut-out time as the highlight still images.

In (5) above (1), or merge the extracted plurality of the highlight still images one still image, and displays and automatically updates the still image sequence, by displaying one by one manually to display and delivering the highlight still images.

  The present invention has the following effects.

  (1) In the range of the highlight moving image cut out according to the present invention, when a still image is cut out according to the specified cut-out density, a large number of moving images after cut-out are adjusted by adjusting the high density in important scenes. Highlight still images can be reliably extracted from a smaller number of still images.

  (2) By using the RF tag possessed by the subject as the mark king means, it is possible to automatically obtain a highlight designation point and automatically obtain a highlight still image to be cut out.

  (3) Since the operator designates the cut-out point with a mouse or the like, there is no need to edit the highlight still image to be cut out.

  (4) When a distribution having a peak shape such as a normal distribution is used as a cutout density template, or when there is no frame image that matches the cutout time by matching the peak position with the highlight designated position, The frame image can be extracted as a highlight still image, and the density of important scenes can be increased, and the density of scenes with lower importance can be adjusted coarsely, that is, the density density can be adjusted.

  (5) Merge multiple extracted still images into a single still image, for example, 2 frames, 4 frames, 9 frames, etc., or automatically update the still image sequence to display like a flip book Highlight scenes can be displayed and distributed by various display methods, such as manually displaying one by one.

  The highlight still image generation / distribution device of the present invention includes highlight moving image extraction means, highlight designation means, cutout density template, cutout density designation means, highlight still image extraction means, and still image display / distribution means.

  The highlight moving image extraction means cuts out the corresponding highlight scene moving image from the video shot by the camera.

  The highlight designating unit designates a location to be extracted as a highlight by a sensor signal or a manual in a moving image being shot.

  The cut-out density template is a template representing a distribution of how many time intervals are extracted when a still image is extracted from a cut-out moving image section. A cutout density template is prepared for each competition. For example, in a marathon goal scene, a template that has a high cutout density when the runner crosses the goal line is used.

  The cut-out density designation means designates a cut-out density represented by a time interval between frame images when cut out. The cut-out density is set so that the density of important scenes is increased and the others are decreased.

  The highlight still image extracting unit extracts a still image from a highlight moving image that has been acquired based on the cut-out time interval obtained by the cut-out density specifying unit.

  The extracted highlight still image is sent to the end user via the still image display / distribution means.

  With such a configuration, it is possible to extract a still image of an important scene from a moving image captured by a camera and provide it to an end user.

  An embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a block diagram of the present invention, FIG. 2 is an explanatory diagram of a highlight moving image clipping state, FIG. 3 is an explanatory diagram of highlight designation means, FIG. 4 is an explanatory diagram of a calculation method of a clipping time interval, and FIG. FIG. 6 is an explanatory diagram of extraction of a highlight still image, FIG. 7 is an explanatory diagram of a configuration state of a compressed moving image, FIG. 8 is an explanatory diagram of a display example of a highlight still image, and FIG. The processing state explanatory view is shown.

  In the figure, 1 is a camera, 2 is a moving image acquisition means, 3 is a highlight moving image extraction means, 4 is a highlight designation means, 5 is a cutout density template, 6 is a cutout density designation means, and 7 is a highlight still image extraction means. , 8 are still image display / distribution means, and 9 is a server.

  The camera 1 converts an optical signal obtained by photographing a scene including a highlight scene into an electrical signal and outputs the electrical signal.

  The moving image acquisition means 2 converts the electrical signal input from the camera 1 into a digital value, and generates and outputs it as a cut-out source video file represented by continuous frame images.

  The highlight moving image extracting unit 3 extracts a highlight moving image from a moving image obtained by the moving image acquiring unit 2 based on a cutting position instructed by a highlight specifying unit 4 described later. For example, as shown in FIG. 2A, when the cutout start position and the end position are designated by the highlight designation means 4, the sandwiched portion is set as a highlight moving image. Alternatively, as shown in FIG. 2B, when only one cutout position is designated as a highlight designation point, a highlight moving image is cut out with a preset start time width and end time width based on the designated position. . The extracted highlight moving image is stored in a temporary storage unit (not shown) of the highlight moving image extraction unit 3.

  The highlight designating unit 4 designates a highlight scene by a sensor signal or manually.

  FIG. 3A shows a method of setting a highlight position using a wireless tag (hereinafter referred to as RF tag) 4-1. The RF tag 4-1 is possessed in advance by a subject to be photographed (for example, a marathon runner in the case of a marathon), and the identification number of the owner (number number in the case of a marathon) is assigned from the RF tag. Called. When the subject carrying the RF tag 4-1 approaches the antenna 4-2, communication is performed between the RF tag 4-1 and the RF reader 4-3. For example, the runner's number number is in the RF reader 4-3. Stored in At the same time, the time tc when the communication is established is recorded, and the highlight position can be set by synchronizing with the shooting time of the moving image being shot.

  FIG. 3B shows a method in which the operator designates a highlight location. When the operator determines that the scene is a highlight scene, the operator notifies that a highlight has occurred using an input means such as a mouse or a keyboard attached to the personal computer. The highlight designating unit 4 generates a time when the highlight occurs and sends it to the highlight moving image extracting unit 3, the cutout density designating unit 6, and the like.

The cut-out density template 5 is a template that represents the distribution of how many time intervals are extracted when a still image is extracted from the cut-out moving image section, and is prepared according to the type of the shooting target. For example, a normal distribution is used as a template for extracting a still image from a marathon goal scene video. It is assumed that the time tc at the moment when the runner (RF tag holding) passes the goal line (antenna) has a relationship as shown in the following equation (1) with the cut-out section (t _S , t _E ).

切り出し密度テンプレートに正規分布を用いると、密度関数ｆ（ｔ）は次式の（２）式で表される。

When a normal distribution is used for the cut-out density template, the density function f (t) is expressed by the following equation (2).

σは分散を表しており、σが小さいときは時刻ｔｃ付近の切り出し密度が高くなり、σが大きいときは切り出し密度が低くなる。正規分布など中心付近で密度が大きくなる分布関数を用い、ＲＦタグの検出時刻やオペレータがマウスを押した時刻を中心付近に配置すると、指定した時刻付近の静止画像が多く抽出され、指定時刻から離れると抽出される静止画像が少なくなるようにすることができる。このように密度を粗密に調節することで、重要なシーンの静止画を確実に抽出する処理を実現できる。

σ represents the variance. When σ is small, the cutting density near time tc is high, and when σ is large, the cutting density is low. Using a distribution function that increases density near the center, such as a normal distribution, and placing the RF tag detection time or the time when the operator presses the mouse near the center, many still images near the specified time are extracted. It is possible to reduce the number of still images that are extracted when they leave. In this way, by adjusting the density roughly, it is possible to realize processing for reliably extracting still images of important scenes.

  In the case of sumo, the time at which the game is decided is tc, the density is adjusted using a normal distribution up to tc, and a function that rapidly decreases after tc.

The cut-out density designating unit 6 designates the cut-out density represented by the time interval between the frame images when cut out. The cut-out density is set so as to increase the density in important scenes and decrease the density in other cases. FIG. 4 shows a method for calculating the cut-out time interval when an appropriate density distribution function f (t) is given. Now, let t _{S be} the start time and t _E the end time of the cut-out section. Also, n still images are extracted from the corresponding section. When the number of images n and the density function f (t) are given, the coefficient k satisfies the following expression (3).

係数ｋを前記（３）式より求めてこれを用いて第Ｉ番目の静止画像の切り出し位置ｔｉを次の（４）式から求める。

The coefficient k is obtained from the equation (3), and the cut-out position ti of the I-th still image is obtained from the following equation (4).

ここでＩｎｔ（ｘ）はｘの整数部分を取り出す関数である。このようにして整数１、２、３・・・ｎが得られるが、この発明ではさらに、切り出し密度指定手段６は整数部分に変化のあった（ｎ−１→ｎ）時刻を取り出し、静止画像の切り出し位置とするものである。これにより整数部分が１、２、３・・・ｎに変化したところを得ることができる。

Here, Int (x) is a function for extracting the integer part of x. In this way, integers 1, 2, 3,... N can be obtained. In the present invention, the cut-out density designation means 6 further takes out the time (n−1 → n) when the integer portion has changed, and It is set as a cut-out position. Thereby, the place where the integer part has changed to 1, 2, 3,... N can be obtained.

  Note that, in the case of the density distribution function of the target form such as the normal distribution, the expression (4) is corrected as the following expression (5) and changed to an integer part because the cut-out frame is biased toward the end time side. You may take out the time when there was.

このようにして（４）式または（５）式から得られた、（６）式で示す時刻が抽出対象のフレーム時刻である。

Thus, the time indicated by the expression (6) obtained from the expression (4) or (5) is the frame time to be extracted.

ハイライト静止画像抽出手段７では、算出された切り出し時間間隔に基づいて、ハイライト動画像の該当する区間から静止画像を抽出する。動画像は、圧縮処理していない場合は、図５に示す如く、適当なサンプリング時間間隔ｔｆ（例えば３３ｍｓ）で並べられたフレーム画像で構成されている。したがって前記（６）式で得られた時刻ｔｉ（ｉ＝１、２・・・）に対応するフレームを抽出するには、図６に示すように、時刻ｔｉに最も近いサンプリング時刻のフレーム画像をハイライト静止画像として取り出すとよい。

The highlight still image extraction unit 7 extracts a still image from a corresponding section of the highlight moving image based on the calculated cut-out time interval. When the moving image is not compressed, the moving image is composed of frame images arranged at an appropriate sampling time interval tf (for example, 33 ms) as shown in FIG. Therefore, in order to extract a frame corresponding to the time ti (i = 1, 2,...) Obtained by the equation (6), a frame image at the sampling time closest to the time ti is obtained as shown in FIG. It should be taken out as a highlight still image.

  In addition, when compression processing is performed, as shown in FIG. 7, a moving image includes a key frame (solid line) having values of all pixels in the image, and a numerical value obtained by an encoding operation without having pixel information. It is composed of compressed frames (dotted lines) represented by A compressed frame cannot be directly extracted as a still image because there is no pixel information. In this case, the compressed frame corresponding to the time ti obtained by the equation (6) is decoded, and the pixel information of the corresponding frame image is recalculated and extracted as a highlight still image. The highlight still image may be acquired by reproducing the highlight moving image from the cutout start time ts and capturing the frame at the time ti obtained by the equation (6).

  The highlight still image extracted in this way is sent to the still image display / distribution means 8 and held.

  The still image display / distribution means 8 provides means for the end user to view the extracted highlight still image by an appropriate method.

  FIG. 8 is a display side of a highlight still image. FIG. 8A shows a case where still images are arranged in the vertical direction. When arranging still images in the vertical direction, the still image display / distribution means 8 generates the following HTML file. Note that img1.gif, img2.gif,... Indicate the file names of the extracted highlight still images.

<HTML>
<HEAD>
<TITLE> ○○○ 〈/ TITLE〉
</ HEAD>
<BODY>
<IMG SRC = ”images / img1.gif”>
<IMG SRC = ”images / img2.gif”>
・
・
・
</ BODY>
</ HTML>
In FIG. 8A, an example in which two highlight still images are displayed on the left side of the display screen has been described.

  FIG. 8B shows a display example when four highlight still images are arranged in a grid pattern in one image. In this case, a process of merging selected still images into one image is performed. For example, when the size of the highlight still image is 160 horizontal pixels × 120 vertical pixels, the image is arranged so that the upper left pixel of the first highlight image is the pixel position of the first row and first column. By sequentially arranging the images so that the upper left pixel of the second highlight image is at the pixel position of the first row and the 161st column, a merge image can be created.

  FIG. 8C is an example in which a pseudo moving image is displayed by displaying an animation that sequentially displays highlight still images at appropriate time intervals. In the case of generating a pseudo moving image, a technique such as animation GIF (Grahic Interchange Format) or FLASH that holds an image in a flipping and manga flipping format is used.

  The server 9 holds such a highlight still image, selects and extracts the highlight still image based on an access request from the end user, and provides it to the end user.

  Next, the processing flow of the highlight still image generation / distribution apparatus of the present invention shown in FIG. 1 will be described with reference to FIG.

  a. The operator selects in advance the cut-out density template 5 that matches the object to be imaged.

  b. The operator operates the camera 1 and starts photographing.

  c. The video shot by the camera 1 is created in a video file as a cutout source in the moving image acquisition means 2. The cut-out source video file is created without interruption until shooting is completed.

  d. When the shooting scene becomes a highlight scene, the highlight designation means 4 determines the highlight scene occurrence time in synchronization with the input of the RF tag or the operator.

  e. The highlight scene generation time is notified to the highlight moving image extraction unit 3 and the cutout density designation unit 6.

f. The cutout start time t _S and the end time t _E are determined based on the highlight occurrence time.

  g. The highlight moving image extraction unit 3 extracts a moving image of the corresponding cutout section.

h. In the cut-out density designating unit 6, the times t ₁ , t ₂ ... Tn of the frame images to be extracted are calculated from the cut-out density template, cut-out start / end times, and the number of still images to be extracted.

  i. The highlight still image extracting means 7 extracts an image corresponding to the frame time calculated by the equation (6) from the highlight moving image.

  j. The still image display / distribution means 8 processes the highlight still image or creates an HTML file for display by a browser in accordance with a display format designated in advance.

  k. The still image display / distribution means 8 uploads the highlight still image and HTML file to the server via the network.

  l. When the end user recognizes the presence of a highlight scene in baseball, soccer, sumo, etc. from text information such as news reports using a mobile terminal, the area-specific highlight area at the corner of the report screen is displayed. When clicked, the access address of the server that holds the highlight still image corresponding to it is automatically sent out. In response to this, a desired highlight still image is transmitted from the server to the end user, and the end user can view the desired highlight still image.

It is a block block diagram of this invention. It is a highlight moving image cutout state explanatory drawing. It is a highlight designation means explanatory drawing. It is explanatory drawing calculation method of extraction time interval. It is a relation explanatory view of a moving picture and a frame picture. It is extraction explanatory drawing of a highlight still image. It is an explanatory view of a configuration state of a compressed moving image. It is explanatory drawing example of a display of a highlight still image. It is processing state explanatory drawing in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera 2 Movie acquisition means 3 Highlight moving image extraction means 4 Highlight designation means 5 Cutout density template 6 Cutout density designation means 7 Highlight still image extraction means 8 Still image display / distribution means 9 Server

Claims (5)

Generating a highlight scene by the still image, the highlight still images generation and distribution apparatus that distributes,
Highlight designation means for designating a portion to be extracted as a highlight moving image from the video obtained by the video obtaining means ,
Highlight moving image extracting means for cutting out the highlighted moving image specified by the highlight specifying means ;
Based on the cut-out interval, and highlights the still image extraction means for extracting a highlight still picture image from said highlight moving image cut out,
Are prepared in advance according to the type of the moving image, the cut density template representing the distribution of the cut-out time interval when extracting the highlight moving picture or al the highlight still image cut out,
A cut-out density designating unit for designating a cut-out density represented by the cut- out time interval between the highlight still images when cut out;
And still image display and distribution means for transmitting the extracted highlight still picture images to the user,
Comprising
The cut-out density designating unit uses the cut-out density template to increase the cut-out density in the cut-out highlight moving image and reduce the cut-out density in the moving image other than the cut-out highlight moving image. A highlight still image generation / distribution device characterized by being set to be Using the RF tag object is possessed as a marking means, the marking information from the highlight moving image extracted based on, when acquiring the highlight still image, the cut-out density, the RF tag is a particular Highlights still image generation and distribution apparatus according to claim 1, wherein the set to be high density when passing-stay position. Operator specifies the location with the mouse, if the acquiring the highlight still image from the extracted based on the specified ibid the highlight moving image, the cut-out density, so that a high density at the point The highlight still image generation / distribution apparatus according to claim 1, wherein As the cut density template uses distribution with peak formation of the normal distribution, the position of the peak by matching the position, set higher the cut-out density, also determined based on the cut-out time interval If there is no still picture which matches the obtained cut time, recently highlighted still image generation and distribution according to claim 1, wherein the still picture beside and extracts as the highlight still picture image of the cut-out time apparatus. Or merge the extracted plurality of the highlight still images one still image, and displays and automatically updates the still image sequence, manually display and distributing the highlight still image by displaying one by one The highlight still image generation / distribution apparatus according to claim 1, wherein:

Images