JP2001134761A - Method and device for providing related action of object in dynamic image and recording medium with recorded program for the same method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an action related to an object instructed by a user to the image or real time video of a non-storage system while reducing the work burden of the provider of a dynamic image or the like. SOLUTION: In a computer 101 for registered person, concerning a still picture registered by an image registering means 10, a feature amount is calculated by an image feature amount calculating means 120, action information is applied by an action information registering means 130 and a registered information storage means 140 stores these still picture information, feature amount and action information. In a computer 102 for user, the object in a dynamic image displayed on a video display means 150 is instructed by an object instructing means 160, and the area of the object is extracted by an object area extracting means 170. Besides, a feature amount A within the area is calculated by an object feature amount calculating means 180, a distance to the feature amount of a retrieval object narrowed by an image retrieving means 190 is calculated and the action information of the still picture having the minimum distance is retrieved and executed by an action retrieving means 200.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for providing a related action of a subject in a moving image, and more particularly, to a method for providing a hypermedia device having a basic function of referring to related information of a subject in a moving image. The present invention relates to a method and an apparatus for providing a related action of a subject in a moving image.

[0002]

2. Description of the Related Art In a conventional method of this kind, when a related action corresponding to a designated subject in a moving image is provided, a region including the designated subject (hereinafter referred to as an anchor) and a position of the subject are provided. In most cases, information is set and stored in advance. A moving image is considered as a three-dimensional spatio-temporal image consisting of a two-dimensional image and a time axis, and the coordinate information of the anchor and the action when the anchor is designated are set and managed together for each frame, and the user Based on the coordinates in the designated moving image and the frame number at the designated time, it is regarded that the subject has been identified by searching for an anchor including the coordinates, and an action associated with the anchor is executed in advance.

At this time, as for the setting of the anchor, as shown in Japanese Patent Application Laid-Open No. H10-187759, a frame section in which a subject exists in a moving image is determined, and a frame serving as a reference of the movement of the subject in the section is determined. There is a method of estimating an anchor by setting an anchor for the remaining frames and performing linear interpolation between the anchors for the remaining frames. In this method, interpolation calculation is performed in advance, and anchor information for all frames in the section is stored. However, in the method disclosed in JP-A-3-52070, it is only necessary to set an anchor in advance for a reference frame. Only when the user designates a subject, interpolation calculation between the frames is performed, and an anchor including the designated coordinates is estimated.

FIG. 6 shows the above-mentioned Japanese Patent Application Laid-Open No. 10-187759.
FIG. 1 and a diagram illustrating a method disclosed in Japanese Patent Application Laid-Open No. 3-52070. FIG. 6A shows a state in which the subject 20 shown in FIG. 6C moves within the video frame 10 shown in FIG. 6B, and five frames from the video frame at a certain time t0. Frame (Frame F0
To F4) are arranged in the time axis t direction. The subject 20 moves from the lower right to the middle right in the video frame from the frame F0 to the frame F2, and moves slightly from the middle right to the upper right in the direction from the frame F2 to the frame F4. ing. The rectangle enclosing the subject is shown in FIG.
Are shown.

In Japanese Patent Laid-Open No. Hei 10-187759, a provider looks at a video frame and uses it as a reference frame (here,
When the frames F0, F2, and F4 are displayed, the operation of temporarily stopping the video and setting the anchor is performed for all the reference frames, and then the other frames (here, the frames) For F1 and frame F3), an anchor is estimated by linearly interpolating between anchors of the reference frame, and anchor information for all frames in the section is created and stored.
Then, when coordinates are designated by the user, a subject is identified by searching for an anchor that includes the designated coordinates among the stored anchors.

In Japanese Patent Application Laid-Open No. 3-52070, while watching a video frame, when a frame which is considered to be a reference frame is displayed, the operation of temporarily stopping the video and setting an anchor is performed for all the reference frames. Store anchor information. Then, linear interpolation between the reference frames including the frame at the time when the coordinates are designated by the user is used to estimate an anchor, and an anchor including the designated coordinates is estimated, thereby identifying the subject.

Another conventional method is disclosed in Japanese Unexamined Patent Application Publication No.
No. 5131 calculates the likelihood between the position information of the subject stored in advance and the coordinates specified by the user, and uniquely identifies the subject specified by the user based on the likelihood.

[0008]

In the above conventional method,
Since it is necessary to previously set an anchor and position information corresponding to the subject in a section where the subject exists in the moving image, the setting work increases when a large number of anchors need to be set, and Imposes a heavy burden on Further, the conventional method cannot be applied to a non-storage type image or a real-time image in which it is impossible to set an anchor in advance.

[0009] An object of the present invention is to provide an action corresponding to a subject in a moving image in the creation of a hypermedia mainly composed of a moving image, by taking into account the size of the subject, the frame section in which the subject exists, and the like. Provided is a method and an apparatus which can reduce a burden on a provider by omitting an operation and can provide an action corresponding to a subject designated by a user even for a non-storage type image or a real-time image. It is in.

[0010]

The present invention solves the above-mentioned problems by the means listed below.

One of the methods is a method for providing a related action corresponding to a subject designated in a moving image. The method includes extracting a subject region from spatiotemporal information near a designated coordinate position in a moving image. Calculating the feature amount of the subject, searching for an image having a similar feature amount from the images registered in advance based on the calculated feature amount, and assigning the image to the searched image in advance. A related action providing method for a subject in a moving image, wherein a related action is provided by executing a set action.

Another means is a registration information storage means for storing a still image, a feature amount of the still image, an action or an action and an action option given to the still image, and a video display for displaying a moving image. Means, a subject designating means for designating a subject position in the displayed moving image, a subject area extracting means for extracting a region of the designated subject, and a subject feature for calculating a feature amount of the extracted subject Amount calculating means, image searching means for searching the stored still images having similar feature amounts based on the calculated feature amounts, and searching for an action given to the searched still images. A related action providing apparatus for a subject in a moving image, comprising: an action search unit that executes the action.

Alternatively, an image registration means for registering a still image as a search target of a subject in a moving image, and a feature amount of the registered still image for searching for a similar still image from the feature amount of the subject in the moving image An action feature registering means for providing an action or an action for providing an action related to the registered still image and an option of the action to the registered still image; A related action providing apparatus for a subject in a moving image, comprising: a registered still image, a calculated feature amount, and a registered information storage unit that stores the assigned action or the action and an option of the action. .

Further, another means includes a procedure for displaying a moving image, a procedure for designating a position of a subject in the displayed moving image, a procedure for extracting an area of the designated subject, Calculating a feature amount of the obtained subject, searching a still image having a similar feature amount from the registered information storage unit on the basis of the calculated feature amount, and adding a still image to the searched still image. A program for causing a computer for a user to execute a procedure for searching for and executing an action that is stored in a recording medium readable by the computer. Is a recording medium on which is recorded.

Alternatively, a procedure for registering a still image as a search target of a subject in a moving image, and calculating the feature amount of the registered still image for searching for a similar still image from the feature amount of the subject in the moving image Providing the registered still image with an action or action for providing an action associated with the registered still image and an option of the action, and providing the registered still image and the calculated A program for causing a computer for a provider to execute the feature amount and the assigned action or the action and the option of the action in the registration information storage means, in a recording medium readable by the computer. RECORDING MEDIUM CONTAINING PROGRAM FOR PROVIDING METHOD OF PROVIDING ACTION RELATED TO SUBJECT IN MOVING IMAGE A.

According to the present invention, a subject area is extracted from spatiotemporal information near a coordinate position designated in a moving image,
By calculating the feature amount of the subject, searching for an image having a feature amount similar to the subject from among the images registered in advance, and executing an action previously given to the searched image. By providing related actions, it is possible to perform actions that have been given in advance according to the subject specified in the moving image without setting and storing the position information of the anchor and the subject in the moving image in advance. To provide. As a result, it is possible to omit the work of setting the anchor and position information in the moving image in advance in consideration of the size of the subject, the frame section in which the subject exists, and the like, and reduce the time and labor required for creating the hypermedia. It becomes possible to reduce. In addition, the present invention can be applied to a non-storage type moving image or a real-time video in which an anchor or position information in a moving image cannot be set in advance, and a related action corresponding to a designated subject is provided. be able to.

[0017]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a diagram illustrating the principle of the present invention. In the figure, S11 to S16 represent a series of processing steps. In the method for providing an action according to a subject in a moving image according to the present invention, a moving image is displayed (S11), and a user designates a subject in the moving image while watching the displayed moving image (S1).
2), a subject area is extracted from spatiotemporal information near the designated coordinate position (S13), and the feature amount of the extracted subject is calculated (S14), and registered in advance based on the calculated feature amount. An image having a similar feature amount is searched from among the searched images (S15), and an action previously given to the searched image is executed (S1).
6).

An embodiment of the present invention will be described with reference to FIG. The apparatus in the example embodiment of FIG. 2 includes a provider computer 101 for creating hypermedia,
The system includes a user computer 102 for using the created hypermedia and a registration information storage unit 140 such as a database or a file for storing information on a subject in a moving image. User computer 102
Is a video display means 1 such as a display for displaying a moving image.
50 and a subject instructing means 160 such as a mouse for receiving an instruction from the user.

First, the provider computer 101 includes an image registration unit 110 that registers a still image to be searched, an image feature amount calculation unit 120 that calculates a feature amount of a registered still image, and a registered still image. And action information registering means 130 for giving an action and an option of the action to the user.

In the provider computer 101, the provider registers the file name and location of the still image to be searched using the image registration unit 110, and further registers the still image registered using the action information registration unit 130. The action to behave when searched and the action that requires an option are registered together with the option. Further, the pixel value in the registered still image is set to H
The histogram is converted into the SI color system and a histogram related to the hue is created.

The registration information storage means 140 calculates the still image registered by the image registration means 110, the action and action options registered by the action information registration means 130, and the image feature quantity calculation means 120. And the accumulated feature values.

FIG. 5 is a diagram showing an example of storage in the registration information storage unit 140. The ID of the still image registered by the image registration unit 140, the file name of the still image, the feature amount information of the still image, and the like. , Granted actions and action options are accumulated. For example, a still image ID
The still image corresponding to “001” has the file name “D:
\ Picture \ sample1. bmp ”and a plurality of feature amounts, and the feature amount of feature number“ 01 ”is“ 0.12 ”.
3 ”, the feature amount of the feature number“ 02 ”is“ 0.567 ”,
Information that the feature amount of the feature amount number “03” is “0.987”, the action name of the assigned action is “end”, and the option of the action is “none” is accumulated.

FIG. 3 is a flowchart showing a series of processes when the provider registers a still image in advance using the provider computer 101, and the image registration means 1
10, the image feature amount calculation means 120, the action information registration means 130, and the registration information storage means 14
This is performed using 0. In the drawing, S21 to S24 represent the above-described series of processing steps.

In S21, the image registration means 110
Is used to register a still image. In S22, the feature value of the still image is calculated using the image feature value calculation means 120. In S23, the action and the option of the action are set for the still image using the action information registering means 130. In S24, the still image information, the feature amount, and the action information are stored using the registration information storage unit 140. In S25, if there are still still images to be registered, S21 to S
24 are repeated, and when there are no more still images to be registered, a series of processes is terminated.

Next, the user computer 102 includes a subject area extracting means 170 for extracting a subject area designated by the subject designating means 160 in the moving image displayed on the video display means 150; Subject feature calculating means 180 for calculating the feature of the extracted subject
Image search means 190 for searching for stored still images having similar feature amounts based on the calculated feature amounts;
An action search unit 200 for searching for and executing an action assigned to the searched still image.

The subject area extracting means 170 has difference information calculating means for roughly extracting an area, and area correcting means for correcting an area.

The difference information calculating means calculates a pixel value at coordinates within a rectangle having the designated coordinates in the frame at the time point designated by the subject designating means 160 as a barycenter, and the pixel values at two frames next to the frame. The difference between the pixel value of the coordinate and the pixel value is calculated. If the calculated difference is equal to or greater than a certain threshold, the coordinate is determined to be within the region in the frame at the designated time, and if the calculated difference is equal to or less than the threshold. For example, it is determined that the coordinates are out of the area in the frame at the designated time, and this is performed for all the coordinates in the rectangle.

The area correction means calculates a ratio of a pixel determined to be in the area by the difference information calculation means to a set of a pixel in the rectangle and a pixel in the vicinity of the pixel, If the calculated ratio is equal to or more than a certain threshold, all pixels in the set are within the region.If the calculated ratio is equal to or less than a certain threshold, all pixels in the set are outside the region. This is performed for all pixels in the rectangle.

The subject feature value calculating means 180 converts each pixel value in the area extracted by the subject area extracting means 170 into an HSI color system and creates a histogram relating to hue.

The image search means 190 has search target narrowing means and minimum distance calculation means.

The search target narrowing means selects one of the still images stored in the registration information storage means 140
A still image that does not include, as a feature value, a value obtained by converting a pixel value at all coordinates within a rectangle having the designated coordinates as a barycenter in the frame at the time pointed by the subject pointing means 160 into the HSI color system. Remove from search.

The minimum distance calculating means calculates the difference between the histogram created by the subject feature calculating means 180 and the histogram narrowed down by the search object narrowing means among the histograms stored in the registration information storing means 140. The Euclidean distance between them is calculated, and a histogram that minimizes the calculated distance is determined.

The action search means 200 searches the action stored in the registration information storage means 140 for the action assigned to the still image having the histogram determined by the image search means 190 as a feature, and executes the action. .

FIG. 4 shows a case where the user operates the user computer 10.
6 is a flowchart showing a series of processing performed when an instruction is given in a moving image using the image display unit 150;
The subject specifying means 160, the subject area extracting means 170, the subject feature amount calculating means 180, the image searching means 190, and the action searching means 20
This is performed using 0. In the figure, S31 to S38 represent the above-described series of processing steps.

In S31, the image display means 150
To display a moving image. In S32, the subject is designated using the subject designating means 160. In S33, the region of the designated subject is extracted by using the subject region extracting means 170. In S34, the feature amount A in the extracted region is calculated using the subject feature amount calculation means 180. S35, S36, and S3
In step 7, the search target is narrowed down from the still images stored from the image information near the designated position using the image search means 190, and the feature amount A and the feature amounts of all the narrowed down still images are compared. Calculate the distance. In S38, an action associated with the still image having the shortest distance is searched for using the action search means 200, and is executed.

Although the present invention has been described based on the above-described embodiment, the present invention is not limited to the above-described embodiment, and can be variously modified without departing from the gist thereof. The following describes the parts that can be changed.

(1) Providing Computer 101 and User Computer 102 In the above embodiment, the providing computer 101 and the user computer 102 are realized using different computers, respectively. The same computer may be used depending on the area to be used.

(2) Image feature value calculation means 120 The image feature value calculation means 120 converts each pixel value in the still image registered by the image registration means 110 as a feature value into the HSI color system, and Although a histogram is created for the above, another feature amount may be calculated or a plurality of feature amounts may be calculated depending on the region to which the present invention is applied. For example, it may be a shape or contour of a still image, or a color arrangement in a still image or a higher-order local autocorrelation feature between pixels.

(3) Registration Information Storage Unit 140 The storage unit shown in FIG. 5 has been described as the registration information storage unit 140, but another storage unit may be used depending on the area to which the present invention is applied. For example, it may be a linear list of structures.

(4) Subject Area Extraction Means 170 The purpose of the subject area extraction means 170 is to use a difference between frames as a means for extracting a region because the purpose is to extract a region at high speed and roughly. However, depending on the area to which the present invention is applied, another method may be used, or a plurality of methods may be used. For example, a background difference may be used, a velocity field of a pixel may be used, or an active contour model may be used.

(5) Subject Feature Calculator 180 The subject feature calculator 180 converts each pixel value in the subject extracted by the subject region extractor 170 as a feature into the HSI color system, and Although a histogram is created for the above, another feature amount may be calculated or a plurality of feature amounts may be calculated depending on the region to which the present invention is applied. However, the same kind of feature quantity as that of the image feature quantity calculation means 120 is calculated.

(6) Image Searching Unit 190 The image searching unit 190 calculates the Euclidean distance between the histograms as the distance calculating unit. However, another method may be used depending on the area to which the present invention is applied. Alternatively, a plurality of techniques may be used. For example, the shape of the histogram (variance, skewness, or kurtosis) may be calculated as the distance.

Further, application examples of the present invention will be described below.

(Part 1) In a system in which an image of an aquarium in which a plurality of fish swim in an aquarium is taken by a video camera and the image is output in real time to a computer screen, a still image at each point and associated with each fish. The action is stored in advance on the computer side, and when the user indicates a fish of interest on the computer screen, the action associated with the fish, for example,
An action such as displaying a detailed description of the fish is performed. This example shows that the present invention is effective for a real image in which it is difficult to define an anchor in advance.

(Part 2) In a relay video of a car race in which a plurality of race cars appear, a still image of each race car and an action associated with each race car are broadcasted and accumulated to a receiver at the start of reception, When a user indicates a race car of interest in a relay video of a car race, an action associated with the race car, for example, an action of jumping to a new video related to the race car is performed. This example shows that the present invention is effective for a relay video in which it is difficult to define an anchor in advance.

(Part 3) Even in the case of a stored moving image, a large number of objects repeatedly repeat frame-in / frame-out, or a stored moving image that is frequently updated to the latest version. The setting and the setting of the position information impose a great load on the provider. By using the present invention, a still image of each object as well as actions associated with each object are stored in advance, and when the user indicates an object of interest, the action associated with the object, for example, An action such as transition to a screen related to the object is performed. This example shows that the present invention is effective for an accumulated moving image in which an anchor can be set in advance, but a large number of subjects need to be set, and this burdens the provider.

A program for realizing a part or all of the functions of the user computer or the provider computer shown in FIG. 2 or the processing shown in FIG. 1, FIG. 3 and FIG. A program for causing a computer to execute the procedure includes a recording medium readable by the computer, for example, FD (floppy (registered trademark) disk), MO, ROM, memory card, C
It can be recorded, provided, and distributed on D, DVD, removable disk, and the like.

[0049]

As is apparent from the above, according to the present invention, it is possible to provide an action related to a subject in a moving image without setting an anchor or position information on the subject in the moving image in advance. Can be. With the above effects, it is possible to omit troublesome work such as setting an anchor and position information in a moving image in advance in consideration of the size of a subject, a frame section where the subject exists, and the like, so that the time and labor required for hypermedia creation Etc. can be suppressed. Also, by applying the present invention to a real video or a real-time relay video for which an anchor or position information cannot be set in advance, an action corresponding to a subject in a moving image can be provided.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 shows an embodiment of an apparatus for implementing the method according to the invention.

FIG. 3 is a flowchart illustrating an operation principle at the time of image registration according to the present invention.

FIG. 4 is a flowchart illustrating an operation principle at the time of pointing a subject according to the present invention.

FIG. 5 is a diagram showing an example of storage of registration information storage means in one embodiment of the present invention.

FIGS. 6 (a), (b), (c), and (d) are explanatory diagrams of a conventional technique.

[Explanation of symbols]

 101: Computer for provider 102: Computer for user 110: Image registration means 120: Image feature amount calculation means 130 ... Action information registration means 140 ... Registration information storage means 150 ... Video display means 160 ... Subject designation means 170 ... Subject area Extraction means 180 ... Subject feature amount calculation means 190 ... Image search means 200 ... Action search means

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/262 G06F 15/62 340A 380 15/70 410 (72) Inventor Masanori Ishiguro Otemachi, Chiyoda-ku, Tokyo 2-3-1, Nippon Telegraph and Telephone Corporation F-term (reference) 5B050 AA08 BA11 CA07 EA24 FA02 FA19 GA02 GA08 5B057 BA24 CA01 CA08 CA12 CA16 CB01 CB08 CB12 CB16 CC03 CE09 CH20 DA08 DB02 DB09 DC23 DC33 DC36 DC39 5B075 KK25ND ND12 NK37 NK44 PP03 PP13 PP28 PQ02 QM08 QP05 5C023 AA01 AA11 AA32 BA02 CA01 CA06 DA04 DA08 5L096 AA02 AA06 BA18 EA26 EA35 FA05 FA35 GA08 GA41 HA02 HA09 JA11 JA18 KA09 KA15

Claims (5)

[Claims] 1. A method for providing a related action according to a designated subject in a moving image, comprising: extracting a subject region from spatiotemporal information near a designated coordinate position in a moving image; Is calculated, and an image having a similar feature is searched for from images registered in advance based on the calculated feature, and an action previously assigned to the searched image is calculated. And a related action providing method for a subject in a moving image. 2. A registration information accumulating means for accumulating a still image, a feature amount of the still image and an action or an action and an option of the action given to the still image; a video display means for displaying a moving image; Subject instructing means for instructing a subject position in a moving image, subject area extracting means for extracting an area of the designated subject, subject feature calculating means for calculating feature quantities of the extracted subject, Image search means for searching for the stored still image having a similar feature amount based on the calculated feature amount; and action search means for searching for and executing an action given to the searched still image. A related action providing apparatus for a subject in a moving image. 3. An image registration unit for registering a still image as a search target of a subject in a moving image, and a feature amount of the registered still image for searching for a similar still image from a feature amount of the subject in the moving image. An action feature registering means for providing an action or an action for providing an action related to the registered still image and an option of the action to the registered still image; An apparatus for providing a related action of a subject in a moving image, comprising: a registration information storage unit that stores a registered still image, the calculated feature amount, the assigned action or the action, and an option of the action. 4. A procedure for displaying a moving image, a procedure for designating a position of a subject in the displayed moving image, a procedure for extracting a region of the designated subject, and a feature amount of the extracted subject. , A procedure for retrieving a still image having a similar feature amount from the registered information storage means based on the calculated feature amount, and an action attached to the retrieved still image is retrieved and executed. And a program for causing a computer for a user to execute the program. The recording medium stores a program for a method for providing a related action of a subject in a moving image, the program being recorded on a recording medium readable by the computer. 5. A procedure for registering a still image as a search target of a subject in a moving image, and calculating a feature amount of the registered still image for searching for a similar still image from a feature amount of the subject in the moving image. Providing an action or action for providing an action associated with the registered still image and an option of the action to the registered still image; and Storing the feature amount and the assigned action or the action and the option of the action in the registration information storage means, on a recording medium readable by the computer. A recording medium on which a program for providing a related action of a subject in a moving image is recorded.