JPH0352070A - Related information reference system for moving image - Google Patents

Abstract

PURPOSE:To execute the reference of related information with regard to a partial image in a frame, and also, to eliminate the need of executing the definition of a related information reference area therefor with respect to all frames by instructing a part of a displayed moving image screen. CONSTITUTION:A reference area definition for related information to a partial image in an arbitrary frame in a moving image is executed only in both ends of a section in which a variation of the partial image is monotonous, and in the case a part of a moving image screen is instructed, a reference area related to the frame is obtained by interpolating from area information in both ends of the section. Therefore, it is unnecessary to execute the reference area definition to all the frames in the section. Also, to area information, a pointer of related information of an object partial image is given, therefore, a related information reference can be executed to a partial image in an arbitrary frame, and a user can refer immediately to the frame in which he is interested without necessitating search of the frame to which he can refer.

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention relates to a method for referring to information related to moving images, and in particular to constructing a multimedia processing system having basic functions such as searching for moving image information and referencing related information. This paper relates to a method for referring to related information of moving images suitable for [Prior Art] Conventionally, there is a method of focusing on a part of a displayed image and referring to information related to that part. Most of them are about still images. Figure 7 is from Apple's hypercard manual rApple Mac. ht-ntosh H
This is an outline of the method explained in yperCardUser's GuideJ. That is, the area surrounding the target partial image 74 of interest is embedded as a button 73 in the still image 71 displayed on the display 70, and the processing 76 related to the button 73 is determined in advance. The button 73 is defined as a rectangular area, and its position and size can be defined interactively. When the button 73 corresponding to the relevant definition area is pointed by the instruction cursor 72, the related information is referenced (for example, accessing the encyclopedia base and acquiring data regarding hats, etc.)
Processing 76 such as the following is executed.

[Problem to be solved by the invention]

Conventional technology is based on the definition of reference areas and related reference processing for still images, so in order to achieve the same thing for moving images, it is necessary to define only the representative frames in the moving images. Either a reference area must be defined/associated/referenced, or a reference area must be defined/associated/referenced for all frame sequences that make up a moving image. However, in the case of the former method, the frame desired by the user does not necessarily match the frame defined for referencing related information, so the user searches before and after the desired frame to define the reference area. We need to find the frame that is On the other hand, with the latter method, users can refer to any related information, but since the number of frames in a video is enormous, it is difficult for humans to define the reference area for every frame. It is not practical unless automatic region recognition comparable to judgment becomes possible. The purpose of this invention is. By pointing to a part of the displayed video screen, the user can refer to the related information prepared in advance for that part. The object of the present invention is to provide an efficient method that allows related information to be referenced and does not require definition of related information reference areas for all frames.

[Means to solve the problem]

In order to achieve the above object, the present invention extracts a plurality of frame sections including a target partial image from a series of frames of a moving image, and converts the section into a plurality of partial sections in which the partial image changes monotonically. The area information surrounding the partial image is stored for both end frames of the partial section, and a pointer of information related to the partial image is attached to the area information, so that the displayed part of the video is If so, list up the area information of the partial section that includes the frame, calculate the reference area related to the frame by interpolation from the area information of both end frames of each partial section, and create the reference area that includes the specified part. , and reference related information using the pointer attached to the area information. [Operation] In the present invention, the reference area definition for the information related to a partial image in an arbitrary frame in a moving image focuses on a section in which the change in the partial image is monotonous, and defines the reference area only at both ends of the section. When a part of the video screen is specified, the reference area related to the frame is obtained by interpolating from the area information at both ends of the section. This eliminates the need to define reference regions for all frames in the interval. In addition, since the area information is attached with a pointer to information related to the target partial image,
It is now possible to refer to related information for partial images in any frame in a video, and users can immediately refer to frames that interest them without having to search for frames that can be referenced. [Example] An example of the present invention will be described below with reference to the drawings. Figure 1 is a schematic block diagram of a system that implements the method of the present invention. The moving image player 11 includes a laser disc player, a video tape recorder, a CD-ROM player, etc., and is connected to the computer 12 via a video signal RIA 15 and a control line 16.

The computer 12 captures a moving image from the moving image player 1l, displays it on the display 13, and executes desired processing in a conversational manner using a pointing device 14 such as a mouse. Related information file 17 contains
Information regarding each partial image in a video image (related information) is stored in advance. When the user indicates a part of the video screen displayed on the display 13, the computer 13 accesses the related information file 17,
The relevant information of the relevant portion is read out and displayed on the display 13. For example, when a certain person in the displayed moving image screen is designated, information related to the person is read from the related information file 17 and displayed. The processing performed by the computer 13 includes processing for defining and pre-registering the correspondence between partial images in a moving image and related information in the related information file 17 (hereinafter referred to as reference area definition processing), and processing for displaying the displayed image. This process is broadly divided into a process of reading out related information from the related information file 17 by specifying a part of the video image (hereinafter referred to as related reference process). Below, these processes will be explained in detail. Figure 2 is a flowchart of reference area definition processing for partial images in a moving image. In process 2l, a partial image during moving image copying is selected as a processing target. This selection of partial images is performed by the registrant inputting a moving image from the moving image player 11 and sequentially selecting partial images suitable for the purpose of registration from among the moving images displayed on the display 13. In this case, the registrant inputs the name, ID, etc. of each target partial image. Next, in process 22, a partial frame sequence in which the selected partial image actually appears is extracted as a target section from the frame sequence forming the moving image. This target section can be extracted by controlling the video player 11 and interactively marking frames in which the target partial image appears while the registrant pans through the video displayed on the display 13. conduct. In process 23, the target section extracted in process 22 is examined, and the target partial image is further divided into partial intervals in which changes in movement and the like are monotonous. In this case as well, the registrant controls the video player l1 and interactively observes changes in the movement of the target partial image in the target section and marks the frames at the dividing points. The registrant also inputs the partial section number, etc. in order to identify each divided partial section. Next, in process 24, the edges of each divided and marked partial section are checked! The gnome is the interval start frame,
This is the end frame of the section, and a reference area regarding the target partial image displayed in the frame is defined. The definition of the reference area is achieved by adding a contour figure surrounding the area to the frame image and allowing the registrant to transform and move it interactively. In process 25, it is determined whether the reference area definition process in process 24 has been completed for all divided partial sections including the target partial image. When the processing in step 24 for all the divided partial sections is completed, in step 26, the target partial image is associated with the related information in the related information file 17. This is done by searching the related information file 17 using the name, ID, etc. of the partial image input when selecting the target partial image in process 21, and searching the related information storage address (pointer) in the reference area created in process 24. This is done by adding it to the definition information. Next, in process 27, when it is determined that the processes 21 to 26 have been completed for all partial images to be defined, the reference area definition process ends. FIG. 3 is a flowchart of a related reference process for acquiring related information by specifying a part of a moving image screen. In process 3 1, a reference instruction from the user to the moving image screen displayed on the display 13 is detected.
Detection of this reference instruction is achieved, for example, by providing a touch screen on the display screen and detecting a touch, or by using an instruction cursor and buttons that are superimposed on the displayed video image in conjunction with a mouse or the like. When a reference instruction is detected in process 3l, in process 32. List all defined sections that include the specified frame from the pre-registered defined sections corresponding to each partial image copy. Next, in process 33, for each of the listed defined sections, reference area definition information defined only for both ends of the partial section, such as the position and size of the area, is read out. moreover. In process 34,
The reference area in the target frame is determined by interpolation from the frame numbers at both ends of the partial interval, the target frame number with the reference instruction, and the reference area definition information obtained in step 33. Next, in process 35, it is determined whether the reference point in the target frame is included in the reference area determined in process 34. If it is not included, perform steps 33 to 35 for another defined interval listed. If it is included, it is determined that the defined section is the target section, and in step 36, the related information file 17 is accessed using the pointer attached to the reference area definition information of that partial section, and the related information of the specified partial image is determined. Read out information. Figures 4 and 5 show specific examples of section division and interpolation processing. In FIG. 4, numeral 40 indicates a moving image consisting of a sequence of frames, and numeral 41 indicates one frame of the moving image. In the example of Fig. 4, if the partial image of interest is a person like 43, the relevant target section is 45, and the change in the partial image is a section in which the position of the person changes monotonically. 1, it shows that it was divided into 46.47 of section 2. Furthermore, the reference definition area in the start point frame of section 1 is 42, the end point of section 1 (start point of section) is 43, and the end point of section 2 is 44.
If the reference instruction from the user is actually the frame indicated by 49, the reference area on the instruction frame is interpolated as shown in FIG. 5, for example. Find it. Figure 5 shows the frames in the target section superimposed. The reference definition areas that are predefined a (registered) are the start point and end point of section 1 (start point of section 2)
, are the three end points of section 2. In this example, the region definition is represented by a rectangle, and the partial image is divided into sections in which the monotony of change is linear.The reference definition region corresponding to the 53 reference instruction frames is simply distributed proportionally from the section width ε frame number difference. I am asking for it by doing. Figure 6 shows an example of the data structure that is registered and registered in the reference area definition process. 61 is a list of target partial images in the moving image, and is indicated by the name of the target partial image, D, etc. Each target image has a pointer to the subinterval list 1.62. The partial interval list 62 is
A list of subsections obtained by dividing the frame section related to the target partial image into a plurality of sections is shown, and each subsection has a pointer to the section foot prosthesis information 63 indicating the start point/end point of the subsection. One section definition report 63 is composed of a set of a starting point frame number and an ending point frame of the partial section. Each section definition information 63 has a pointer to the reference area definition information 64, and one reference area definition information 64 is information for defining the reference area, such as the center of gravity of the area, the spread of the area, and the center of gravity. It consists of horizontal/vertical offsets to determine, and pointers to related information processing when referencing. At the time of reference, all the section definition information 63 including the frame of the designated point is listed, and the reference region definition information 64 is used to find the reference region on the target frame by interpolation, and the designated point is included in the reference definition region. The relevant reference area definition information 64 is recognized.

Then, the related information file 65 is accessed using the related processing pointer attached to the recognized reference area definition information 64, and the related information list is read out. [Effects of the Invention] As explained above, according to the present invention, it is possible to efficiently define a reference area for a moving image consisting of a series of frames, and in actual use, it is possible to define an arbitrary reference area in a moving image. Since it is possible to refer to partial images in a frame, if it is applied to a video database search system, for example, it can be used to refer to information related to people or things appearing in a video, or to search for new related scenes. It has the effect of allowing you to jump quickly.

[Brief explanation of drawings]

11jA is a schematic block diagram of a system that realizes the present invention, FIG. 2 is a flowchart of an embodiment of reference area definition processing of the present invention, FIG. 3 is a flowchart of an embodiment of related reference processing of the present invention, 4 and 5 are diagrams showing specific examples of the present invention, Figure 6 is a diagram showing an example of a data structure registered in the reference area definition process, and Figure 7 is a diagram for explaining the conventional method. 11...Video image player, 12...
・Computer, 13...Display, 14...
・Pointing device, 17...Related information file. Figure 1 Figure 7 Figure 2

Claims (1)

[Claims] (1) By specifying a part of the video display screen,
This is a method that refers to related information set in advance for the relevant part, and extracts a section of multiple frames that includes the target partial image from a series of frames of a video image, and uses this method to calculate changes in the partial image. The video is divided into a plurality of monotonous sub-intervals, the area information surrounding the sub-image is stored for both end frames of the sub-interval, a pointer of information related to the sub-image is attached to the area information, and the displayed video is When a part of the image is specified, list the area information of the partial section that includes the frame,
Find the reference area related to the frame by interpolation from the area information of both end frames of each partial section, find the reference area that includes the specified part, and refer to the related information using the pointer attached to the area information. A video related information reference method characterized by: