JP2006054512A - Video image editing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide technology with which a viewer can edit a video image broadcast from a broadcast station, to obtain a layout congenial to the taste of the viewer and display and record the edited image. <P>SOLUTION: The video image edit apparatus includes an input unit 53 with a function of designating a desired part, in a frame image displayed on a display apparatus 56 as a lock image; and a control unit 10 with the function of searching an image region, whose feature is coincident with that of the lock image in a lock image memory 521, a function of storing the designated lock image to the lock image memory 521 to update the locked image in the lock image memory 521 by the image of the searched feature matching region, a function of demarcating a desired image in a way that the searched feature matching region shares a prescribed area ratio at a prescribed part in the desired image to segment the desired image from the frame image, and a function of displaying the segmented desired image on the display apparatus 56 as a frame image, corresponding to the frame image of the segmented source and recording the desired image in a video image memory 541. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a video editing apparatus. For example, a desired subject in a frame image of a received television signal is locked and displayed / recorded in a desired composition, or a desired subject in a frame image of a reproduced recorded signal is locked and displayed in a desired composition. Relates to the device.

(1) A video search system has been proposed that can search at high speed a portion where a subject whose position can be specified is shown (see Patent Document 1).
In the system of Patent Document 1, video data is composed of video raw data, which is a time series set of video frames, and video attribute data including imaging range data indicating the imaging range of each video frame, and is stored in the video storage unit. Is done. The video search unit includes an input unit, a search processing unit, and an output unit. The input means of the video search unit accepts a search request including the position data of the subject and the positional relationship between the subject and the imaging range. The search processing means of the video search unit calculates a mutual positional relationship from the position data and the imaging range data in the video storage unit, and obtains a video frame that satisfies the received search request. The output means of the video search unit outputs the search result to the search application.

(2) There has been proposed a program providing system capable of providing video shot from a direction desired by a user (see Patent Document 2).
The system of Patent Document 2 includes a program production device, a communication line provider device, and a plurality of user terminal devices connected to the communication line provider device via a communication line network. Videos obtained by photographing a certain subject from a plurality of different directions are produced by the program production apparatus, and are distributed as different channels from the communication line provider apparatus to each user terminal apparatus via the communication line network. Thereby, the user can select and display a video channel taken from a desired direction.

Japanese Patent Laid-Open No. 11-282851. JP 2001-309342 A.

  The TV image of each moment broadcast from the broadcasting station is selected from the images shot by multiple cameras, and the composition of the television image is determined by the operator of the camera and the preference of the program director who selected the image. Is reflected. As a result, the broadcast station broadcasts “videos with a composition estimated to meet the majority of preferences”. However, such a composition does not necessarily match the preferences of individual viewers. For example, in a talk program, some viewers are interested in the moderator's performance, while others are interested in other performers. Some viewers want to see the entire view, while others want to see an enlarged image of a certain part.

  An object of the present invention is to enable a viewer to edit and display or record a composition of a video broadcast from a broadcasting station so as to suit his / her preference.

The present invention is configured as the following [1] to [9].
[1] Configuration 1:
Designation input means for designating a desired part in the frame image displayed on the display as a lock image;
Area search means for searching for an image area whose characteristics match the lock image in the lock image memory from the frame image inputted externally;
Storage control means for storing the lock image designated by the designation input means in the lock image memory, and updating the lock image in the lock image memory with the image of the feature matching area found by the area search means;
Image cutout means for demarcating a desired image and cutting out from the frame image so that the feature matching area found by the area search means occupies a predetermined area ratio at a predetermined portion in the desired image;
Image processing means for processing the desired image cut out by the image cut-out means as a frame image corresponding to the frame image of the cut-out source;
A video editing apparatus comprising:
For example, the input device 53 having a function of designating a desired portion in the frame image displayed on the display device 56 as a lock image, and the lock image memory 521 from the frame image input from the antenna 59 or the recording device 54. The control device 10 having a function of searching for an image area whose feature matches the lock image and the specified lock image are stored in the lock image memory 521, and the lock image stored in the lock image memory 521 is searched for the image of the feature match area found. And a control apparatus having a function of defining a desired image and cutting out from the frame image so that the found feature matching region occupies a predetermined area ratio in a predetermined part of the desired image 10 and the desired image clipped is displayed on the display device 56 as a frame image corresponding to the frame image of the clipped source, or recorded. It is a video editing device provided with the ability to record to 541.

The video editing apparatus of configuration 1 can be applied both as an apparatus for editing a television video broadcast from a broadcasting station into a desired composition and as an apparatus for editing a recorded video into a desired composition. Further, the present invention can be applied to a device that edits and displays a television video broadcast from a broadcasting station in real time, or a device that records the edited video.
(A) Designation input means:
For example, by displaying a frame in a frame image displayed on the display, and performing a predetermined locking operation (for example, pressing a lock key) while a desired subject is in the frame, the frame is displayed. It is means for designating the image inside as a lock image. In order to reduce the processing load at the time of pattern matching, an image in the smallest square area including the inside of the frame may be designated as the lock image instead of the image in the frame.
(B) Search method:
As a method for searching for an image region whose characteristics match the lock image from the frame image to be processed, a known pattern matching method such as a shape pattern, color information comparison, neural network, or wavelet is appropriately used. be able to. The processing load may be reduced by starting a search from the vicinity of the center of gravity of the lock image of the previous frame. Further, the lock image may be extracted using the difference image information between the previous frame and the current frame.
(C) Lock image memory:
The lock image is stored in the lock image memory.
As hardware constituting the lock image memory, known hardware such as flash memory, hard disk, and DRAM can be appropriately used. Also, as the data format and encoding method of the lock image, known data formats and encoding methods can be used as appropriate. For example, MPEG (Moving Picture Experts Group) can be used.
The lock image stored in the lock image memory is updated with the image of the feature image area found by the area search means, and is used for the search of the feature image area in the next frame.
(D) Definition of desired image:
The desired image is an area that is cut out from a frame image that is externally input as a processing target, and the image data is set for display or recorded. The desired image is defined so that the feature matching region is located at a predetermined position in the desired image and occupies a predetermined area ratio with respect to the desired image. In other words, the desired image is demarcated so that the feature matching region has a predetermined composition. For example, the feature matching region is defined so as to occupy an area of ¼ at the center position of the desired image. Alternatively, the feature matching area is defined so as to occupy an area of 2/5 at a position 1/3 to the left of the desired image. It is assumed that the position of the center of gravity in the desired image of the feature matching region and the area ratio of the feature matching region in the desired image are appropriately set by initial setting or user setting.

[2] Configuration 2:
In configuration 1,
The image processing means is a display control means for displaying and setting the desired image cut out by the image cut-out means as a frame image corresponding to the frame image of the cut-out source.
A video editing apparatus characterized by that.
The desired image may be displayed and set in place of the externally input image as in the configuration 3, or the desired image may be displayed and set together with the externally input image as in the configuration 4.
[3] Configuration 3:
In configuration 2,
The display control means is configured to display and set the desired image cut out by the image cut-out means instead of the frame image of the cut-out source;
A video editing apparatus characterized by that.
[4] Configuration 4:
In configuration 2,
The display control means sets the display of the desired image cut out by the image cut-out means and the frame image of the cut-out source,
A video editing apparatus characterized by that.
The desired image and the cut-out source image may be displayed on separate displays, or the same display screen may be divided and displayed.

[5] Configuration 5:
In configuration 1,
The image processing means is a recording control means for recording a desired image cut out by the image cutting out means in a recording memory as a frame image corresponding to a frame image as a cutting source.
A video editing apparatus characterized by that.
A desired image may be recorded instead of an externally input image as in configuration 6, or an externally input image and a desired image may be recorded as in configuration 7.
[6] Configuration 6:
In configuration 5,
The recording control means records the desired image cut out by the image cut-out means in a recording memory in place of the cut-out frame image.
A video editing apparatus characterized by that.
[7] Configuration 7:
In configuration 5,
The recording control means records the desired image cut out by the image cut-out means and the frame image of the cut-out source in a recording memory, respectively.
A video editing apparatus characterized by that.

[8] Configuration 8:
In configuration 1,
A tuner that takes out the TV signal received from the program specified by the operation input;
Frame image output means for externally inputting a frame image of a program taken out by the tuner to the area search means;
A video editing apparatus further comprising:
In other words, it is a device for editing television images broadcast from a broadcasting station.

[9] Configuration 9:
In configuration 1,
Playback means for playing back the recorded data of the program specified by the operation input;
Frame image output means for externally inputting a frame image of a program reproduced by the reproduction means to the area search means;
A video editing apparatus further comprising:
That is, it is an apparatus for editing the recorded video that has been reproduced.

  The video editing apparatus of configuration 1 has features of a designation input means for designating a desired part in a frame image displayed on a display as a lock image, and a lock image of a lock image memory from the frame image inputted externally. An area search means for searching for a matching image area, a lock image designated by the designation input means is stored in the lock image memory, and an image of the feature match area found by the area search means is stored in the lock image memory. A storage control means for updating the lock image, and an image cut-out that defines the desired image so that the feature matching area found by the area search means occupies a predetermined area ratio in a predetermined area in the desired image and cuts out from the frame image. And extracting the desired image cut out by the image cutout means as a frame image corresponding to the frame image of the cutout source. For and an image processing means for, it can be processed viewer the composition of an image that is externally input is edited to suit their own tastes. For example, the composition of a video broadcast from a broadcasting station or a recorded video can be edited to suit the user's preference and displayed or recorded.

In the video editing device of configuration 2, the image processing unit of configuration 1 is a display control unit that displays and sets the desired image cut out by the image cutting unit as a frame image corresponding to the frame image of the cutting source. The composition of the video broadcast from the broadcast station and the recorded video can be edited to suit your preference and displayed.
In the video editing apparatus of configuration 3, the display control unit of configuration 2 displays and sets the desired image clipped by the image cropping unit in place of the frame image of the cropping source. The composition of the video can be edited to suit your preference and displayed in place of the video before editing.
In the video editing apparatus of configuration 4, the display control unit of configuration 2 displays and sets the desired image cut out by the image cutting unit and the frame image of the cutting source, respectively. You can edit the video composition to your liking and display it with the video before editing.

The video editing apparatus having the configuration 5 is a recording control unit in which the image processing unit of the configuration 1 records the desired image clipped by the image cropping unit in the recording memory as a frame image corresponding to the frame image of the cropping source. Therefore, it is possible to edit and record a video broadcast from a broadcast station or a composition of a recorded video to suit one's preference.
In the video editing apparatus of configuration 6, the recording control unit of configuration 5 records the desired image clipped by the image cropping unit in the recording memory in place of the frame image of the cropping source, so that the video broadcast from the broadcasting station Or the composition of the recorded video can be edited to suit the user's preference and recorded in place of the pre-edited video.
In the video editing apparatus of configuration 7, the recording control unit of configuration 5 is broadcast from a broadcasting station in order to record the desired image extracted by the image extraction unit and the frame image of the extraction source in the recording memory. You can edit the composition of the video to be recorded and the recorded video to your liking and record each with the video before editing.

  The video editing apparatus having the configuration 8 is configured such that the video editing apparatus having the configuration 1 takes out a tuner that receives a program specified by an operation input from a television signal received, and externally inputs a frame image of the program that is taken out by the tuner to the area search unit. And a frame image output means for allowing the viewer to edit and process the composition of the video broadcast from the broadcast station so as to suit his / her preference. For example, it can be displayed or recorded.

  In the video editing apparatus of configuration 9, the video editing apparatus of configuration 1 uses a playback unit that plays back recorded data of a program specified by an operation input, and a frame image of a program that is played back by the playback unit as the area search unit. Since it further has a frame image output means for external input, the composition of the recorded video can be edited and processed by the viewer so as to suit his / her preference. For example, it can be displayed or recorded.

Hereinafter, specific configuration examples of the present invention will be described with reference to the drawings.
FIG. 1 shows a configuration of a television viewing apparatus having a video editing apparatus function and a recording function of the present invention, and FIG. 2 shows a video editing procedure in the television viewing apparatus. 3 to 6 show details of the video editing procedure shown in FIG.

TV viewing device:
The components of the television viewing device are an antenna 59, a television signal receiving circuit 31, a descrambler 32, a demultiplexer 33, a decoder (eg, MPEG decoder) 34, an audio processing circuit 35, a speaker 55, a video processing circuit 36, and a display device. 56, the input device 53, and the control device 10.

  The input device 53 displays keys for setting / canceling the video editing mode and a processing mode of a desired image to be cut out from the original frame image (frame image) (desired image recording mode / desired image recording desired image). In addition to a key for selecting a desired image display mode), for example, a key for moving a lock frame, which will be described later, by a desired amount in a desired direction, or an image specified by the lock frame is determined as a lock image. Key.

  Television radio waves received by the antenna 59 are selected, demodulated, and error-corrected by the television receiving circuit 31. Next, descrambling is canceled by the descrambler 32, and a transport stream (TS) is taken out. From the transport stream (TS), a demultiplexer 33 extracts a necessary program packet, further obtains service information (SI), and reproduces a clock. The extracted packet is decoded into audio data and video data by the decoder 34, converted into an analog audio signal by the audio processing circuit 35 and output as audio from the speaker 55, and displayed by the video processing circuit 36 as a display device 56. Is converted into an analog video signal in a scanning format conforming to the above, and output from the display device 56 as an image.

  When the desired image display mode is selected in the video editing mode, the desired image area is sequentially cut out by the control device 10 from each frame image decoded in order by the decoder 34, and each desired image thus cut out is extracted. Are sequentially sent to the video processing circuit 36. Thereby, the desired image of each frame cut out in order from each frame image of the received television signal is displayed on the display device 56 in order. At this time, it is assumed that the audio included in the cut-out source video data is output in synchronization.

Recording device:
In a normal recording mode for recording original video data, the recording device 54 stores a packet of a desired program extracted from the transport stream (TS) by the demultiplexer 33 into a built-in recording memory (eg, hard disk) 541. To record. Further, in the desired image recording mode, video data of a desired image composed of a set of each desired image cut out in order by the control device 10 is recorded in the recording memory 541. At this time, as the audio data, the audio data of the cut-out source video data is recorded in synchronization.
It should be noted that the normal recording mode and the desired image recording mode can be executed concurrently.

In the normal reproduction mode for reproducing original video data, the recording device 54 sequentially takes out the packets recorded in the recording memory, sends them to the decoder 34, decodes them, and outputs them as video and audio. In the desired image reproduction mode for reproducing the desired image, the video data of the desired image is read from the recording memory 541 and reproduced.
In addition, you may comprise so that the data after decoding with the decoder 34 may be recorded instead of the packet of a compression state. When the decoded data is recorded, it is output to the audio processing circuit 35 and the video processing circuit 36 without being decoded during reproduction.

Video editing procedure:
The video editing procedures (1) to (9) executed by the control device 10 will be described in accordance with FIG. 2 and with reference to FIGS.
(1) Selection and locking of a partial image of interest:
For example, when a subject to be photographed is in a certain limited range such as a stage, such as a talk program or a music program, the television station broadcasts a video obtained by photographing the entire view with a fixed camera (entire view image). . The resolution of this panoramic video is preferably high in consideration of the clipping process described later. For example, it is about 2000 × 2000 pixels. In the current example, there are 1920 × 1152 pixels, 25 frames.
The panoramic video may be a video broadcast from a television station, or may be a video recorded by the recording device 54 read from the recording memory 541 and reproduced.

The user operates a predetermined key (eg, up / down / left / right movement key, not shown) of the input device 53 while watching a television screen on which a panoramic view image is displayed, to thereby obtain a desired partial image (eg, desired person, etc.). ) And the lock frame (dotted circle in the left figure of FIG. 3).
When the desired partial image enters the lock frame, the user operates a predetermined lock key (not shown) of the input device 53. Thereby, the partial image is locked. In other words, the control device 10 cuts out an image in the smallest square area including the lock frame (the square area in the right diagram in FIG. 3) as a lock image, and lock image memory that is a predetermined storage area in the storage device 52. It stores in 521. At this time, the position of the center of gravity of the locked image in the frame image (the frame in which the partial image is locked in the full-view video) is also stored and used for controlling the start position of the search area in pattern matching (described later). You may make it do. As the storage device 52, known hardware such as a flash memory, a DRAM, and a hard disk can be used.

(2) (3) Original data reception / original frame acquisition:
The next frame image data of the frame from which the lock image is cut out is acquired from the continuously received panoramic video (original data). The data format, encoding method, and the like can be selected as appropriate. For example, MPEG may be used. This image data is temporarily stored in a predetermined area of the storage device 52.

(4) Low resolution image generation for search processing:
In order to reduce the processing load of pattern matching, the control device 10 generates low-resolution image data for area search based on the image data acquired in (3) above. As this method, for example, a known resolution conversion process such as a simple sampling method or an averaging process can be used. In addition, for example, a multi-resolution technique called “Motion-JPEG 2000” wavelet can be used. When the processing capability of the control device 10 is sufficiently high, the original data (high resolution panoramic image data) obtained as described above is used as it is without generating low resolution image data for area search. Also good.

(5) Feature matching region extraction / lock image update:
Next, the control device 10 generates a region whose characteristics match the lock image data stored in the lock image memory 521 (more precisely, “region having the highest degree of coincidence”) generated in the above (4). Extracted from the resolution image data by pattern matching. Here, as a pattern matching method, a known pattern matching method such as comparison of shape patterns, comparison of color information, use of a neural network, use of a wavelet, or the like can be appropriately used.

Further, in order to reduce the processing load at the time of pattern matching, the search may be started from the vicinity of the position of the locked image in the original panoramic image. For this purpose, it is necessary to store in the lock image memory 521 the position of the center of gravity (the position of the center of gravity in the entire view image) together with the image data of the lock image.
Further, instead of extracting the lock image by pattern matching, the lock image may be extracted from the information of the difference image from the previous frame image. For example, a method of extracting a subject using MPEG motion compensation and motion prediction vectors is described in “Information Processing Society Journal, March 2002, Faculty of Science and Engineering, Waseda University, Department of Electronic Information Communication,” “Combining motion compensation and color information. “Extracting Human Region from MPEG Video” ”, and such a method may be used.

  When the feature matching area is extracted, the control device 10 updates the lock image data in the lock image memory 521 with the image data of the feature matching area. At this time, the center of gravity position is also updated. The updated lock image is used for searching for a feature matching region in the next frame image. In order to update the lock image in this way, a minute change on the time axis of a desired partial image (for example, an image of a desired person or the like) (if the partial image illustrated in FIG. 3 is accompanied by movement of the limb) A search can be performed using a lock image that incorporates fluctuations and the like, and the search accuracy can be improved.

(6) Definition of desired image (examination of composition):
When the feature matching region is extracted, the control device 10 determines that the feature matching region is located at a predetermined part in the desired image and that the feature matching region occupies a predetermined area ratio in the desired image. An image is defined within the frame image of the original panoramic video. In other words, the desired image is defined so that the feature matching region is arranged in the desired image in a predetermined composition. Here, it is assumed that the position in the desired image where the feature matching region should be placed and the area ratio that the feature matching region should occupy in the desired image are predetermined by initial setting or user setting. The size of the desired image is also determined in advance by initial setting or user setting.

This will be described with reference to FIG. 4, the outermost frame indicates the frame image of the original panoramic video, and the innermost frame indicates the feature matching area. The intermediate frame indicates the size of the desired image.
As shown in the upper left column of FIG. 4, description will be made on the assumption that the feature matching region is extracted at the position of the approximate center height on the right side in the frame image of the original panoramic image.
In the conventional moving image frame creation method described in Japanese Patent Laid-Open No. 2003-158684, an area of a predetermined size (the area of the middle frame in the upper left column in FIG. 3) is cut out and recorded from the center of the original frame image. ing. In this case, the composition of the moving image frame is a composition in which the feature matching region is arranged at the right end position in the desired image, as shown in the lower left column of FIG.
In contrast, in this apparatus, the position of the desired image in the original frame image is not determined with respect to the frame image, but the position of the feature matching area in the desired image and the feature matching area are within the desired image. It is determined based on the area ratio. For example, in the example of the upper right column in FIG. 4, the center of gravity of the feature matching area is determined as a 5/8 position starting from the left side of the desired image in the left-right direction, and a 1/2 position starting from the upper side in the vertical direction. The area of the feature matching area is determined as 0.28 of the desired image whose size is a predetermined value. Therefore, as shown in the lower right column of FIG. 4, the frame of the desired image to be cut out has a center of gravity of the feature matching region 5/8 to the right starting from the left side of the desired image and 1/2 to the bottom starting from the upper side of the desired image. And the composition occupies an area of 0.28 of the desired image.

In the example shown in FIGS. 4 and 5, the desired image defined as described above does not protrude from the original frame image of the panoramic video. However, as shown in FIG. 6, when the feature matching region is extracted at the end of the original frame image, it may not be demarcated because the desired image is protruded from the original frame image.
Such a case can be corrected as shown in FIG.
FIG. 6 shows a method of demarcating a desired image by shifting it from the original position (the position of the broken line frame in the center diagram of FIG. 6) and placing it in the original frame image. In this case, the composition (the position of the feature matching area in the desired image) is slightly sacrificed.

(7) Digital processing (1. Extraction of desired image, 2. Generation of frame image):
The control device 10 defines and cuts out a desired image as described in (6) (see FIG. 5), and generates a frame image of a frame corresponding to the cut-out source frame.
(8) Set this frame image for display,
(9) Display on the display device 56 and / or record in the recording memory 541.
The above processing is executed for each frame image of the panoramic video input from the antenna 59 or the recording device 54 one after another.

The block diagram which shows the structure of the television viewing-and-listening apparatus provided with the function as a video editing apparatus of this invention, and a video recording function. The flowchart which shows the video editing procedure in the television viewing-and-listening apparatus of FIG. FIG. 3 is an explanatory diagram used for explaining the procedure “(1) Selecting and locking a partial image of interest” shown in FIG. 2. FIG. 3 is an explanatory diagram used for explaining the procedure “(6) Definition of desired image (examination of composition)” shown in FIG. 2. Explanatory drawing used for description of the procedure “(7) Digital processing (1. Extraction of desired image)” shown in FIG. The case without shift is shown. Explanatory drawing used for description of the procedure “(7) Digital processing (1. Extraction of desired image)” shown in FIG. The case with shift is shown.

Explanation of symbols

36 Video processing circuit 56 Display device 52 Storage device 521 Lock image memory 54 Recording device 541 Recording memory 53 Input device

Claims (9)

Designation input means for designating a desired part in the frame image displayed on the display as a lock image;
Area search means for searching for an image area whose characteristics match the lock image in the lock image memory from the frame image inputted externally;
Storage control means for storing the lock image designated by the designation input means in the lock image memory, and updating the lock image in the lock image memory with the image of the feature matching area found by the area search means;
Image cutout means for demarcating a desired image and cutting out from the frame image so that the feature matching area found by the area search means occupies a predetermined area ratio at a predetermined portion in the desired image;
Image processing means for processing the desired image cut out by the image cut-out means as a frame image corresponding to the frame image of the cut-out source;
A video editing apparatus comprising: In claim 1,
The image processing means is a display control means for displaying and setting the desired image cut out by the image cut-out means as a frame image corresponding to the frame image of the cut-out source.
A video editing apparatus characterized by that. In claim 2,
The display control means is configured to display and set the desired image cut out by the image cut-out means instead of the frame image of the cut-out source;
A video editing apparatus characterized by that. In claim 2,
The display control means sets the display of the desired image cut out by the image cut-out means and the frame image of the cut-out source,
A video editing apparatus characterized by that. In claim 1,
The image processing means is a recording control means for recording a desired image cut out by the image cutting out means in a recording memory as a frame image corresponding to a frame image as a cutting source.
A video editing apparatus characterized by that. In claim 5,
The recording control means records the desired image cut out by the image cut-out means in a recording memory in place of the cut-out frame image.
A video editing apparatus characterized by that. In claim 5,
The recording control means records the desired image cut out by the image cut-out means and the frame image of the cut-out source in a recording memory, respectively.
A video editing apparatus characterized by that. In claim 1,
A tuner that takes out the TV signal received from the program specified by the operation input;
Frame image output means for externally inputting a frame image of a program taken out by the tuner to the area search means;
A video editing apparatus further comprising: In claim 1,
Playback means for playing back the recorded data of the program specified by the operation input;
Frame image output means for externally inputting a frame image of a program reproduced by the reproduction means to the area search means;
A video editing apparatus further comprising:

Images