JPH10187759A - Moving picture anchor displaying, selecting, and stetting device, moving picture hypermedia device using moving picture anchor, and moving picture provision system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device that can save the labor for setting operation for a logical unit (anchor) and facilitates the operation, more concretely, an anchor setting device that automatically calculates or sets anchor information which have been necessary to be set for every frame. SOLUTION: This system has a data operation part 1 for data operation regarding the setting of an anchor and a link, a data storage part 2, which stores their data, a display part 3 which controls display, and a user operation part 4, which inputs an editing command. Once the anchor setting part 11 sets an anchor only for a starting and a final frame, an anchor estimation part 12 estimates anchor information on other frames by interpolative calculation. Consequently, the man-hour of setting operation for an anchor can greatly be reduced.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image anchor setting device and a moving image hypermedia device using the device. The present invention particularly provides a device for inputting a moving image and setting an anchor to a target or the like included in the moving image,
A moving image hypermedia device for linking relevant information to an anchor set by the device and performing a desired search, etc.
The present invention relates to a moving image providing system using this device as a server client system.

[0002]

2. Description of the Related Art Conventionally, information retrieval in a general hypermedia device mainly sets a logical unit for an information link to a text or a still image, and links related information to this logical unit in advance. , The related information is displayed when the user clicks the logical unit. However, various technologies have been proposed in recent years for processing not only still images but also moving images, as represented by MPEG for encoding and decoding of moving images. By handling moving images, the hypermedia device can be used for content creation such as CAI, various presentations, and electronic catalogs. The editing of moving images has been used in some limited industrial fields such as broadcasting stations in the past, but it is expected that it will be rapidly spread as a personal system based on a personal computer in the future.

Japanese Patent Laid-Open No. 4-163589 discloses an image processing apparatus capable of setting a logical unit (called a node in the specification) for a moving image. This device is capable of setting a node in a still image simply by specifying a display range. On the other hand, in the case of a moving image, the validity period of the node is determined from both (1) display range and (2) time. By paying attention to the point that the area range should be designated, these designations are made possible. That is, with respect to (1), the node at that time is set by pointing the region surrounding the subject appearing in the moving image with a mouse or the like, while with respect to (2),
The valid duration of the node is designated by the elapsed time from the output start time of the moving image. Therefore, this node is uniquely determined by the two contents of the area and the elapsed time, and related information can be linked to each node. After the link, when actually reproducing the moving image, if the user clicks a certain area on the screen with a mouse or the like, a node is specified by the position and time, and related information is displayed.

[0004]

In the above-mentioned apparatus, the setting of the area of the logical unit is performed manually. However, a moving image naturally has many frames unlike a still image, and the position and shape of the subject change every moment.
In the case of the NTSC system, 30 frames are required for one second, so even if it is simply calculated, 30 processing operations per logical unit are required when processing a moving image of one second. For example, when creating five minutes of content, if five logical units are set in one frame, the number of times of setting is 45,000.

The present invention has been made in view of the above problem, and an object of the present invention is to provide a device which enables labor saving and simplification of a setting operation of a logical unit (referred to as an anchor in this specification). An object of the present invention is to provide an anchor setting device that automatically calculates or automatically sets anchor information that needs to be performed for each frame. Another object of the present invention is to provide a moving image hypermedia device and a moving image providing system using the anchor setting device.

A moving picture anchor setting apparatus of the present invention is an apparatus for inputting a moving picture and setting an anchor on the moving picture. This device is based on a frame determination unit that determines a start frame and an end frame of a setting target period in which an anchor is to be set from a plurality of frames that form an input moving image, and a reference frame that is taken within the setting target period. When a region is specified, the anchor setting means for setting the anchor information for this reference region as the anchor region and the anchor information in the non-reference frame based on the anchor information set for the reference frame And an anchor estimating means for estimating.

In one aspect of the present invention, when a plurality of reference frames are taken in the setting target period, the frame determining means sets the earliest one of them as the start frame and the latest one as the end frame. Decide

In another aspect of the present invention, the frame determining means detects a frame in which a target included in the reference frame appears and disappears before and after the reference frame, and determines these as a start frame and an end frame, respectively. Decide.

In one aspect of the present invention, when there are a plurality of reference frames, the anchor estimating means estimates the anchor information in the non-reference frames by interpolating the anchor information set for the reference frames.

At this time, in one aspect of the present invention, the anchor estimating means performs interpolation in the following procedure. First, the amount of change in correspondence information between anchors set for two adjacent reference frames is calculated, and then the elapsed time from one reference frame to a certain non-reference frame and the other reference Calculate the ratio of elapsed time to the frame. Thereafter, the change amount is divided by this ratio.

In one aspect of the present invention, the anchor estimating means performs interpolation calculation of anchor information between reference frames located at both ends of the setting target period divided by the reference frame, The anchor information in the non-reference frame included in the section is estimated.

One aspect of the present invention includes anchor information editing means for correcting the anchor information estimated for a non-reference frame. At this time, the anchor estimating means:
The non-reference frame whose anchor information has been modified is promoted to a reference frame.

[0012] In one aspect of the present invention, the anchor information editing means three-dimensionally displays the locus of the anchor based on the estimation of the anchor information.

In one embodiment of the present invention, the anchor information editing means calculates the trajectory of the anchor based on the estimation of the anchor information, and displays a cross-sectional view of the moving image along the trajectory together with the trajectory of the anchor.

In one embodiment of the present invention, the anchor information editing means selects a frame included between the start frame and the end frame, and creates an anchor information display image by superimposing anchor information on the image of the frame. Thereafter, the anchor information display image is displayed in chronological order.

[0015] In one aspect of the present invention, the anchor information editing means includes a moving image reproducing means and an anchor information correcting means. The moving image reproducing means reproduces the moving image on the screen, and when the anchor information correcting operation is performed during the moving image reproduction, the anchor information correcting means specifies the frame reproduced at the time when the operation is performed, If the frame is a non-reference frame, the frame is promoted to the reference frame and the anchor information in the frame is corrected based on the operation.

In one embodiment of the present invention, the anchor estimating means includes an automatic anchor setting means. The automatic anchor setting means selects an arbitrary number of frames from adjacent reference frames, sets them as new reference frames, tracks the anchor, and sets anchor information for these new reference frames. .

In one aspect of the present invention, the automatic anchor setting means tracks the anchor based on the movement of the outline of the target.

In one aspect of the present invention, the anchor automatic setting means tracks the anchor based on the motion vector of the target.

In one embodiment of the present invention, when each section of the setting target period divided by the reference frame is defined as a unit determination period, the automatic anchor setting means sets each of the routes that the anchor can follow in the setting target period. , The degree of coincidence between the path and the motion vector of the anchor is determined for each unit determination period. This judgment result is integrated over the entire set target period. Finally, the route having the highest degree of coincidence in the overall result is regarded as the movement route of the anchor.

In one embodiment of the present invention, the degree of coincidence is determined by a function of an angle formed by the path and the motion vector, the function having no extreme value in the range of 0 ° to 180 °, The total result is obtained by calculating the sum of the values of the function for each unit determination period. The function value gradually increases or decreases as the angle changes from 0 ° to 180 °. Therefore, the degree of coincidence is evaluated by the simple sum of these values.

In one aspect of the present invention, the anchor automatic setting means includes automatic setting reliability determining means and automatic setting reliability displaying means. The automatic setting reliability determination unit determines the reliability of the automatic setting by comparing the anchor information set by the anchor setting unit with the anchor information set by the anchor automatic setting unit. The automatic setting reliability display means displays the reliability.

In one aspect of the present invention, the automatic anchor setting means performs image matching with a new reference frame by using an image of an anchor region of a reference frame in which anchor information has already been set as a model, thereby obtaining an anchor. Track location.

In one embodiment of the present invention, when a similar image is not found in the new reference frame when performing image matching, the automatic anchor setting means further divides the adjacent reference frames into new ones. A reference frame is provided, and image matching is performed again after reducing the interval between the reference frames.

In one aspect of the present invention, the automatic anchor setting means sets, as the reference frame in which the anchor information has already been set, a reference frame close to the new reference frame and a reference frame which is separated by a certain time distance. Select at least two cards. After that, the position of the anchor is tracked based on the matching result using the selected reference frame.

In one aspect of the present invention, the automatic anchor setting means compares the anchor information set by the anchor setting means with the anchor information set by the automatic anchor setting means to determine the reliability of the automatic setting. An automatic setting reliability determination unit for determining is included. If the reliability determined by the automatic setting reliability determination means is low after the anchor information is set by the tracking starting from the first reference frame, the anchor information is determined by the tracking starting from the second reference frame. Is to be reset.

At this time, while tracking is performed with the second reference frame as a starting point, the anchor area set in a certain reference frame and the tracking with the first reference frame as the starting point are set in the reference frame. If the degree of overlap with the anchor area is equal to or more than the predetermined ratio, the tracking with the second reference frame as the starting point ends.

Alternatively, in the tracking using the second reference frame as a starting point, the tracking is stopped when the reference frame specified in advance is reached.

In one aspect of the present invention, the automatic anchor setting means includes a moving image reproducing means, an anchor information displaying means, and an anchor information correcting means. Video playback means,
Play the video on the screen. The anchor information display means displays the anchor information obtained as a result of the image matching sequentially performed on the displayed frame. When an anchor information correction operation is performed during moving image reproduction, the anchor information correction means corrects the anchor information of the frame on which the operation is performed, and invalidates the automatic anchor setting result in a predetermined frame before that frame. Become

In one aspect of the present invention, the anchor setting means includes a reference frame deleting means. The reference frame deletion unit demotes the reference frame to a non-reference frame when the anchor information in a certain reference frame is obtained within a predetermined error tolerance from the interpolation calculation of the anchor information in another reference frame.

In one aspect of the present invention, the anchor setting means includes text anchor setting means. The text anchor setting means, when a text is designated for a reference frame, sets an anchor having the text as the content for the reference frame.

According to an aspect of the present invention, it further includes cursor changing means for changing the display state of the cursor when the cursor enters any anchor area.

At this time, in one aspect of the present invention, the cursor change means changes the display state of the cursor so as to accompany the display of the identification information of the anchor area where the cursor is inserted.

In one aspect of the present invention, the anchor setting means sets the anchor information in units of a group consisting of a plurality of anchors.

An aspect of the present invention further includes anchor search means for searching for an anchor based on characteristics of the movement of the anchor in the anchor information.

An aspect of the present invention further includes anchor search means for searching for an anchor based on the identification of the anchor.

On the other hand, the moving picture hypermedia apparatus of the present invention comprises: frame determining means for determining a start frame and an end frame of a setting target period in which an anchor is to be set from a plurality of frames constituting an input moving image; When an area is designated for a reference frame taken during the period, an anchor setting means for setting anchor information for the reference frame with this as an anchor area,
Anchor estimation means for estimating anchor information in a non-reference frame based on the set anchor information.
The above corresponds to the moving image anchor setting device. The present apparatus further includes link setting means for linking the set or estimated anchor information to any relevant data, and link searching means for searching, from any anchor, related data linked to the anchor.

In one embodiment of the present invention, the moving image processed by the moving image hypermedia apparatus constitutes a video teaching material, and the related data is additional information associated with a desired anchor of the video teaching material in a desired setting target period. The device is used for producing interactive video teaching materials.

On the other hand, the moving picture providing system of the present invention comprises a server and a client. The server stores the moving image, the anchor information set for the reference frame of the moving image, and a means for storing related data linked to the anchor information, and the server based on the anchor information set in the reference frame. The apparatus includes means for estimating anchor information in a reference frame, and means for searching, from an arbitrary anchor, related data linked to the anchor. The client also includes means for determining which anchor area is clicked when the user clicks the target in the moving image.

[0039]

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of the moving picture hypermedia device of the present invention will be described below. This device incorporates the moving image anchor setting device of the present invention. According to the present apparatus, for example, an interactive CAI software capable of displaying a name of a fish, a supplementary explanation, and the like when a user clicks on any swimming fish using a moving image of an aquarium aquarium as a material is easily provided. And it can be created efficiently. In the following embodiments, the “user” mainly refers to the creator of such content, but it goes without saying that this may be a person who personally edits a video tape shot by himself / herself.

Embodiment 1 In this embodiment, the user explicitly specifies a start frame and an end frame, and sets an anchor using these two frames as the first reference frames. A "frame" is a display unit of an image and includes a picture referred to in MPEG. This apparatus automatically calculates anchor information such as the position, shape, and color of the anchor region in another frame by interpolation calculation from the anchor set for the reference frame. The “anchor information” refers to, for example, the position and shape of the anchor area, the color of the anchor area when it is explicitly displayed, and the like. The present invention can be realized even if the first reference frame is one, but an example thereof will be described later.

The configuration of the entire system including the moving picture hypermedia device of this embodiment is the same as that of a personal computer (hereinafter, P
C) and a video playback device that provides moving images to this PC. The PC has a built-in video capture board that captures and digitizes the image provided by the video playback device. The video reproducing apparatus has a function of performing reproduction from a designated frame or time in addition to a normal reproduction start, stop, fast forward, frame advance, and the like. Although such a video reproducing apparatus is widely used in broadcasting business and the like, it is not necessarily limited to this. Control of various functions of the video playback device is controlled by a user interface (hereinafter,
UI), for example, via a "play button" displayed on the screen. In this case, when the user clicks the button, the action is transmitted from the PC to the video playback device via the signal cable. Although the video playback device itself is not an essential component of the moving image hypermedia device, it will be described here as a system including this. FIG. 1 is a configuration diagram of a system including a moving image hypermedia device according to the present embodiment.

This apparatus is roughly classified into a data operating section 1 for operating data relating to anchor information and link information, a data storage section 2 for storing these data, and a display section 3 for displaying these data in a meaningful form. It comprises a user operation unit 4 for receiving and managing user operations, and a moving image input unit 6 for inputting a moving image reproduced by the video reproducing device 5.

(1) Data operation unit 1 The user gives an instruction to the data operation unit 1 through a UI described later. That is, the following internal configuration is a software module.

The frame determination unit 10 determines a start frame and an end frame. In the present embodiment, the frame designated by the user becomes the start frame and the end frame as they are. Examples of the start frame and the end frame are the start and end frames of the scene showing the aquarium in the video of the aquarium described above. If the scene moves to the video of the entrance to the aquarium, it is not necessary to set an anchor for the fish thereafter, so the end frame is specified before the transition of the scene.

The anchor setting section 11 actually sets an anchor between the start frame and the end frame. For example,
When an anchor is set for a certain fish, first, a rectangle surrounding the fish is displayed with a mouse in the start frame, and this is registered as an anchor area. At this time, the moving image is in the stop mode. Then, the moving image is advanced to the end frame, and the anchor area is registered by surrounding the same fish again. Since the fish moves or changes direction between the start frame and the end frame, its position and shape usually change.
In general, the shape and position of the anchor area registered in the start frame do not match the shape and position of the anchor area registered in the end frame. Note that the anchor setting unit 11 includes an anchor information editing unit 11 that is used in the later-described anchor correction.
0, and a text anchor setting unit 111 that sets an anchor for text such as a character string.

The anchor estimation unit 12 performs interpolation calculation based on the anchor information set in the start frame and the end frame, and estimates the position and size of the anchor in an arbitrary frame (non-reference frame). This process will be described in detail later.

The anchor search unit 15 searches for an anchor based on the motion characteristics of the anchor in the anchor information or the identification information of the anchor. The identification information is information that serves as a clue to distinguish the anchor from other anchors, and includes, for example, the name of the anchor, an anchor setting target, and an anchor setting date and time.

The hyperlink setting unit 13 sets a hyperlink for the set anchor, and creates a data structure related to the setting in the form of a table. The hyperlink search unit 14 searches the set link information.
In the case of the above example, a fish anchor and text data indicating the name of the fish are associated by hyperlinks.

(2) Data Storage Unit 2 The data storage unit 2 may be a database, various file devices, or memory devices. This part is mainly hardware.

The moving picture data storage section 20 stores the moving picture data captured by the moving picture input section and digitized. The anchor information storage unit 21 and the link information storage unit 22 store the set anchor information and link information, respectively.

(3) Display Unit 3 The display control unit 30 is a system program of a display system for controlling the display of various images such as a UI and a moving image being edited.
It includes a display circuit such as a VGA controller and this driver. The display control unit 30 is a cursor changing unit 300.
have. The cursor changing unit 300 changes the display state of the cursor when the cursor enters the anchor area. The output data of the display control unit 30 is given to a display device 31 such as a monitor of a PC, and a desired display is performed.

(4) User operation unit 4 This enables a user to input commands, and is composed of hardware such as a keyboard, a mouse, various pointing devices, and a command dispatcher. Examples of commands include setting an anchor, modifying an anchor area, a link, and a link search.

(5) Moving image input section 6 This is hardware equivalent to a video capture board.
It has a D converter and a frame memory (not shown), and digitizes an input moving image. After that, the data is provided to the moving image data storage unit 20.

Based on the above configuration, a procedure for setting an anchor and a link will be described first, and a state of a UI for setting an anchor will be described later.

[1] Anchor Setting FIG. 2 is a flow chart showing an anchor setting and correction procedure according to the present embodiment, and FIG. 3 is a view showing a table of set anchor information. As shown in FIG. 2, first, various initialization processes for hardware and the like are performed (S21).
The moving image data stored in the moving image data storage unit 20 is read (S22). The top frame of the read moving image data is first displayed as a still image on the display device 31. Next, the anchor information already set for the moving image data is read from the anchor information storage unit 21 (S23). If the anchor information exists, the anchor area is actually displayed on the screen (hereinafter, the mode in which the anchor area is displayed on the screen is “anchor display mode”, and the mode in which the anchor area is not displayed is “anchor non-display mode”
That).

Subsequently, the moving image data is advanced to the start frame of the period in which the anchor is newly set this time (S2
4) When the desired frame appears, press the "start frame" button on the screen to register the start frame. In this state, the frame is in a waiting state for setting an anchor area in this frame, and the user provides a rectangular area by clicking the mouse so as to surround another fish, for example. Once the rectangular area is determined, the upper left point (x1, y1) and the lower right point (x2, y1)
The coordinates of 2) are acquired, and this is recorded as anchor information of the fish together with the frame number of the start frame (the serial number from the first frame of the moving image) (S25).

After that, the moving image data is advanced again, and when the desired end frame appears, the moving image data is stopped (S26), and a rectangular area is provided so as to surround the same fish. Here, the setting of the anchor in the end frame is completed (S27). “Anchor1” in FIG. 3 is an anchor ID indicating this fish. Here, the frame numbers of the start frame and the end frame (frames 1 and 100, respectively) and the coordinate information of the anchor area are stored in the table.

When the anchor information in the reference frames at both ends is determined in this way, the anchor information of the frame (non-reference frame) between them is obtained by interpolation calculation (S2).
8). FIG. 4 is a diagram showing an interpolation calculation method of anchor information. Here, the anchor information at the start frame (time t0) is A
(T0)-The anchor information at the end frame (time t1) is A
(T1) -If the anchor information at time t is A (t) -t1-t0 = Δt, A (t) = {A (t1) -A (t0)} t / Δt + {A (t0) t1 −A (t1) t0} / Δt (Equation 1). As A, the x1, y1, x2,
By substituting y2, the outer shape of the anchor area at an arbitrary time is determined. Substituting the barycentric coordinates of the anchor area reveals the rough movement of the anchor area. By substituting the color number for A, the change in color of the anchor area can be tracked. In addition to this, information that can be expressed numerically can also be interpolated by the internal division calculation using Expression 1. The anchor information of the non-reference frame obtained by the interpolation may be added to “anchor1” in the table of FIG. 3, or the table of FIG. The calculation of Expression 1 may be sequentially performed. In the present embodiment, the sequential calculation is assumed hereinafter.

When S28 is completed, the anchor information is actually displayed to confirm the contents (S29). At this time, the moving image data is reproduced by returning to the start frame, and the anchor area is displayed as a rectangle in each frame. This rectangular area moves continuously according to the calculation result.

In the case of "anchor 1", the result is extremely good if the fish makes a uniform linear motion, but if the swimming direction is changed on the way, the anchor area shifts from the fish in the middle frame. Therefore, the anchor information is corrected (S3
0). The user first advances the moving image data to a frame with a large shift, and stops the image here. Next, the end of the anchor area displayed on the screen is clicked, and the shape or position of the area is changed with the mouse. The anchor estimation unit 12 upgrades the frame thus modified to the reference frame (hereinafter, the one that has been promoted to become the reference frame is also referred to as “intermediate reference frame”), and adds this anchor information to the table of FIG. . FIG. 5 shows a table obtained by adding the anchor information of the intermediate reference frame to FIG. On the other hand, FIG. 6 is a diagram showing a method of performing an interpolation calculation based on three frames of an intermediate reference frame and both end reference frames. If the non-reference frame to be estimated exists between the start frame and the intermediate reference frame, interpolation calculation is performed between those frames. If the non-reference frame exists between the intermediate reference frame and the end frame, those frames are calculated. Interpolation calculation is performed between them (S28). After that, when good anchor information is obtained after the display in S29 and the re-correction in S30 (Y in S31), this is saved (S3).
2) End the anchor setting process. If the anchor of another frame is modified in S30, naturally this frame also becomes the intermediate reference frame. When two or more anchors are set in the same frame in S25, the anchor IDs are automatically changed within the apparatus in the setting order, and the anchor IDs are assigned, and a rectangular display of these anchor areas is performed in different colors. You just have to deal with it.

According to the above procedure, the following effects can be obtained. 1. Only setting the anchors in the reference frames on both ends eliminates the need for setting work for a large number of frames existing between them. 2. If a displacement occurs in the position of the anchor in the interpolation calculation, this displacement can be confirmed. Therefore, it is easy to recognize the frame to be corrected, and the frame once corrected is automatically promoted to the intermediate reference frame, so that the user does not need to pay attention to whether or not to change the frame to the reference frame. 3. For example, even in a case where a fish having an anchor set therein draws an arc and swims, sufficiently good anchor information can be obtained by performing correction at most several frames in addition to the both-end reference frames. The above is the outline of the moving image anchor setting device among the moving image hypermedia devices of the present embodiment.

[2] Link setting Next, link setting for the set anchor is performed. FIG. 7 is a flowchart showing a link setting and search procedure according to this embodiment, and FIG. 8 is a diagram showing a table of set link information.

FIG. 7 shows a processing procedure when the setting of the anchor and the setting of the link are performed completely independently. As in FIG. 2, first, various initialization processing (S40) and reading of moving image data (S41) are performed. Do. Subsequently, the anchor information set in [1] is read from the anchor information storage unit 21 (S42), and the already set link information is read from the link information storage unit 22.

Next, while the anchor information of other frames is obtained by interpolation calculation based on the anchor information of both end reference frames and the intermediate reference frame (S44), the anchor information is continuously displayed in accordance with the reproduction of the moving image. Perform (S4
5). In this state, the user operation unit 4 waits for an input from the user (S46).

Here, the user clicks a certain anchor area on the moving picture or after stopping the moving picture once,
If the "Create / Change Link" button is pressed, link information is created for that anchor (S47). For example, when a fish in the aquarium is clicked, text, images, and other candidates to be linked to the fish appear on the screen, and the text or the like selected by the user is anchored to the fish (more precisely, included in the anchor. Object called a fish). If there is no candidate, the user can himself input a character string and link it. In FIG. 8, the text information "anchor1.txt" is added to "anchor1", and similarly "anchor1.txt".
Bitmap image "anchor" as an anchor for "2"
2. “bmp” is shown in a linked state. When the link information is determined in this way, the contents of the link are stored in the link information storage unit 22, and the user waits for an input again.

On the other hand, if the user presses the "link search" button in S46 and specifies the anchor, the link information corresponding to the anchor is searched and displayed (S4).
9). In the case of FIG. 8, for example, for the fish of anchor1, the name, length, characteristics, etc. of the fish are displayed in a character string.
For the fish of nchor2, a picture of the sea where the fish actually lives is displayed. Since the link operation can be confirmed by this display, the user can complete the content creation at this point. By storing the content in a recording medium such as a CD-ROM,
It can also be commercialized. When shipping as a product, generally, the mode is changed to the anchor non-display mode in which the anchor area is not displayed.

Although the setting of the anchor and the link has been described as independent processing here, for example, if a button “return to anchor setting” is provided on the screen during link setting,
Both sides can freely move, and editing becomes easier.

[3] UI for Setting Anchor FIG. 9 is a diagram showing an example of a UI screen for setting an anchor. In the figure, a moving image to be processed is displayed in an image display area 50. The black-painted button group 52 in the upper column is an object button for directly instructing playback, stop, etc. of the video. Next to it, a rectangular button 5 for setting an anchor area for the frame displayed in the image display area.
4. Similarly, a start frame designation button 56 and an end frame designation button 58 for designating the displayed frame as a start frame or an end frame are provided. In the figure, the anchor area 60 for one fish is shown.
Is set.

At the center right of the screen is an anchor-related box group 62 indicating the name, ID, start frame number, and end frame number of the anchor to be set or modified. Below the image display area 50, there is a scene-related box group 64 indicating the number of the scene including the currently displayed frame and the serial number of the frame in the scene. Further below, there is a box 66 for advancing or returning the video by a small amount for editing. Press the button at the right end to advance the movie, and press the left end to return. The position of the currently displayed frame in the scene is indicated by a vertical line 70 in the box 66. Below this box is a box 68 that indicates the position of the start and end frames in the scene. The positions of the start and end frames are indicated by double vertical lines 72, 74, respectively, and the position of the intermediate reference frame therebetween is indicated by the triangular symbol 76.

In the figure, first, using the scene number as a clue, the user advances the video tape to the beginning of the scene for which anchor setting is desired. In this case, for example, among the images of the aquarium composed of a plurality of scenes, the process proceeds to the image of the aquarium with the scene number “5”. Here, the user presses the button at the right end of the box 66 to advance the moving image one frame at a time. When the first frame for which an anchor is to be set appears, the user presses the start frame designation button 56 and registers this. At this time, a double vertical line 72 indicating the position of the start frame appears at a corresponding position in the box 66.
Here, the rectangular button 54 is pressed, and the upper left point and the lower right point of the anchor area to be set in the image display area 50 are clicked with the mouse. This completes the anchor setting of the start frame. Then, the moving image is advanced, and the registration of the end frame and the anchor setting are similarly performed.

When it is detected that the setting in the both-end reference frame is completed, the anchor estimating unit 12 of the present apparatus automatically substitutes the anchor information into Expression 1 and starts the calculation. Here, the user returns to the start frame,
As the frame advances, the anchor estimating unit 12 obtains the time corresponding to the currently displayed frame, and displays the anchor area based on the estimation result corresponding to this time. If the displayed anchor area is displaced, the user presses the rectangular button 54 again to correct the area. After modification, a triangular symbol 76 appears at the location corresponding to that frame.
According to this UI, since the anchor information is actually displayed on the moving image data, the edited result can be confirmed in real time, and the correction can be easily performed.

The above is the outline of the present embodiment. In addition,
Regarding the present embodiment, the following improvements, modifications, and the like can be considered.

(1) Setting of text anchor This is performed by the text anchor setting unit 111 of FIG. First, edit the text data on the screen, overlay it on the video, and set the anchor. The difference from the normal anchor setting is that, instead of specifying a partial area of the reproduced image, the created text is temporarily put on the image, and then the anchor area is set so as to surround the text. Conventionally, for example, a method of directly adding an annotation to a video image has been generally used, but in that case, it is inconvenient at the time of re-editing such as deleting the annotation later. This embodiment solves this.

When a text anchor is set, its anchor information is also stored in the anchor information table. However, in the table shown in FIG. 3, the "frame" portion becomes "text", and the text name is entered in that column.

[0075] The link of the related information is also possible for the text anchor. For example, paste the text “South Sea Fishes” into the aquarium scene in Figure 9 and link this text to the text “South Sea has a lot of brightly colored fish…” Can be done.

(2) Change of display state of cursor This is performed by the cursor change unit 300 of FIG. This feature is especially useful in anchor-hidden modes, such as when content is used on the market. Because of this function, the cursor changing unit 300 includes a position acquisition program that constantly acquires the position of the cursor, a determination program that determines whether the acquired position is included in any anchor region, and an anchor region where the cursor is located. When entering, there is a change program for determining how to change the display state of the cursor, and actually changing the shape and the like of the cursor according to the determination.

The change of the cursor may or may not be changed for each anchor. In the former case, for example, there is a method of changing the cursor, which is usually a + sign, to ◎ or increasing the brightness of the cursor. According to this aspect, it is particularly advantageous when the movement or shape of the target is fast and the anchor region is drastically changed.

On the other hand, in the latter case, it is conceivable that the anchor ID of the anchor area where the cursor is entered is searched from the change program and this is displayed as it is at the position of the cursor instead of the cursor. For example, when the cursor enters the anchor area of a fish, the cursor may indicate the content of the target of the anchor, such as "shark". According to this aspect, the user can know the name of the fish without needing to click the fish.

(3) Explicit Specification of Intermediate Reference Frame In this embodiment, only the both-end reference frames are determined first. However, when the movement of the target is irregular, it can be expected that correction is necessary. There is also. In that case,
From the beginning, it is assumed that designation of an anchor area is also accepted in frames other than the start frame and the end frame. For example, in the UI of FIG. 9, an intermediate frame designation button is provided in addition to the start frame designation button 56 and the end frame designation button 58. Since this frame is used as a reference frame from the beginning, it can be considered that the interpolation calculation is started from the state shown in FIG.

(4) Anchor area other than rectangle The anchor area need not be limited to a rectangle. For example, in the case of a circle or an ellipse, an area may be specified by coordinates of three points, a major axis, a minor axis, and a center. If it is a polygon, the coordinates of each vertex will do. When the outer periphery of the target itself is used as the anchor region, the region can be specified by the coordinates of one point on the outer periphery and the chain code expressed from that point.

(5) Use of Nonlinear Interpolation In this embodiment, linear interpolation is used most simply.
This may of course be a non-linear interpolation. The equation used for interpolation is
It can be determined by experiments or the like according to the characteristics of the moving image to be processed.

(6) Determination of Start and End Frames In this embodiment, these frames are explicitly specified by the user, but the following method is also available. 1. The user does not need to be aware of the start and end frames,
Simply specify the frame and set the anchor. The specified frame becomes the reference frame. Frame determination unit 10
Determines that the frame with the smallest frame number among the frames to which the user has set an anchor is the start frame and the frame with the largest frame number is the end frame. In this case, the start frame designation button 56 and the end frame designation button 58 in FIG. 9 become unnecessary.

2. The user designates one frame, sets an anchor for this, and designates a target for which the anchor is set. This frame becomes the reference frame. The frame determination unit 10 examines frames before and after the reference frame to detect a frame in which the target appears and a frame in which the target disappears, and sets them as a start frame and an end frame, respectively.

The presence or absence of a target is determined by matching images. That is, matching processing is performed on the preceding and succeeding frames using the target specified by the reference frame as a model. The target frame of the search is expanded forward and backward as long as matching can be achieved. Eventually, if the matching cannot be obtained, the start and end frames are determined. According to this method, only one reference frame should be provided initially.

(7) Three-dimensional display of anchor area The anchor setting unit 11 is provided with a function of expanding the set anchor area in the x and y directions and the time t direction, which are the vertical and horizontal directions of the screen, and displaying it in three dimensions. This can be considered that FIG. 4 is displayed on the screen as it is while editing the anchor. As a result of this display, the user can visually grasp the anchor as a whole.

As an application of this technique, the anchor information displayed three-dimensionally may be directly editable. For example, if the anchor area in the intermediate reference frame of FIG. 4 is moved to the left on the screen, the display as shown in FIG. 6 is made. The user can grasp the effect of editing in real time.

(8) Cross-sectional display of moving image In the anchor setting section 11, a horizontal cross-sectional view of the locus of the anchor region from the start frame to the end frame (FIG. 10).
(A)) and a vertical cross-sectional view (FIG. 10 (b)) are created, and the function of displaying this together with the trajectory of the anchor is added. First, the x and y coordinates of the center of gravity G of the anchor area are obtained from the anchor information of the reference frame. In the case of FIG. 10A, a straight line parallel to the x-axis is drawn on the frame from the center of gravity. A plane including these straight lines (hatched portion in the figure) is provided between adjacent reference frames. Next, the moving image is cut on this plane. The obtained cross-sectional view is projected on the xt plane (dotted area in the figure). In the case of FIG. 10B, x and y are exchanged and the same processing is performed. When the trajectory of the anchor is calculated correctly, that is, when the estimation accuracy of the anchor information in the non-reference frame is sufficiently high, the movement path of the anchor should appear in the two projection views. For example, if a red ball is to be set as an anchor, a red streak-like moving path appears in a section of the moving image. Splitting the pencil vertically is the same as making the lead appear linear. If this movement path is cut or thickened on the way, the position of the anchor at that location may be corrected.

(9) Grouping of Anchors The anchor information editing unit 110 is provided with a function of grouping separately set anchor information and virtually treating it as one anchor. For example, the person A is in the frame N1
-N2 and frames N3 to N4 are present in the screen, and frames N2 to N3 are not present, frames N1 to N
2 and the anchor for the person A in the frames N3 to N4 are treated as one. As a result, the work of setting and correcting the anchor information is reduced. In addition,
The person A and the person B appearing in the same frame can be grouped.

(10) List Display of Anchor Information The anchor information editing unit 110 is provided with a function of displaying a list of anchor information set for a moving image currently being processed. For example, along with the video title "AQUARIUM"
A list of anchor names such as “FISH1”, “FISH2”, etc. is displayed on the screen. When the user selects an anchor name whose contents are to be checked, the user may return to the start frame of the anchor and play back the moving image.

(11) Anchor search An anchor information search UI is provided. When character information such as the name of the anchor information to be searched is input as a keyword, the anchor search unit 15 searches the anchor information storage unit 21 for one having the keyword and displays it. Alternatively, the movement of the anchor area may be used as the search key. For example, when the user wants to find an object that moves to the right, the user presses the button "→" in the search UI. The anchor search unit 15 calculates the trajectory of each anchor region, searches for an anchor including an object that moves to the right, and displays the anchor.

(12) List display of anchor information display images The anchor information editing unit 110 sets a frame (either a reference frame or a non-reference frame) included between the start frame and the end frame and the anchor setting unit 11 Set anchor information or anchor estimation unit 12
A function of creating an anchor information display image by superimposing the anchor information estimated by the above, and displaying a list in chronological order. For example, as shown in FIG. 11, first, a start frame 81 and an end frame 82 are arranged at both ends, and frames are selected at a frame interval Δt. Subsequently, the anchor information 80 is superimposed on each of these frames to create an anchor information display image, and these are displayed side by side from the earliest display time. With this configuration, the suitability of the anchor setting can be seen at a glance. Therefore, the position determining operation by the box 66 in FIG. 9 can be omitted, and the anchor can be easily corrected. It is not necessary to select frames at regular intervals.
For example, only the reference frame may be selected. Further, in the displayed anchor information display image, the anchor region may be directly editable by dragging it with the mouse.

(13) Correction of anchor information during moving picture reproduction As shown in FIG. 12, the anchor information editing section 110 displays a moving picture reproducing section 118 for reproducing a moving picture on the screen, and an anchor for displaying anchor information regarding a frame being displayed. When the anchor information correction operation is performed at the information display unit 120 and one time or a plurality of times during moving image reproduction, the frames being reproduced at the time when each anchor information correction operation is performed are specified, and those frames are identified. If is a non-reference frame, an anchor information correction unit 119 is provided which is promoted to a reference frame and corrects the anchor information in those frames based on each anchor information correction operation. With this configuration, first, a moving image is displayed in the image display area 50 of FIG. At this time, simultaneously, the anchor information in the frame being displayed is displayed as the anchor area 60 by the anchor information display unit 120. When the user finds a frame in which the anchor region 60 is displaced from the target, the user clicks the center of the target in the moving image with the mouse. With this action, the anchor information correction unit 119 corrects the anchor information by specifying the frame displayed at the time of the click and generating anchor information centered on the clicked point. The size of the newly set anchor area may be the same as that of the original anchor area, for example. Hereinafter, this frame is treated as a reference frame. With this configuration,
Since the position of the target can be sequentially specified during the playback of the moving image, it is not necessary to check and correct the position later.

Embodiment 2 In the first embodiment, the anchor information is automatically calculated mainly by interpolation, and the correction is manually performed. In the present embodiment, an anchor is automatically set with a certain number of frames as reference frames in advance based on analysis of moving images, and the interpolation method of the first embodiment is used between these reference frames. In this case, since the frame corresponding to the intermediate reference frame of the first embodiment is present from the beginning, the labor of manual correction is reduced.

FIG. 13 is a configuration diagram of the anchor setting unit 11 of the moving image hypermedia apparatus according to the present embodiment. Configurations other than the anchor setting unit 11 are the same as those in FIG. FIG.
In, the automatic anchor setting unit 112 has a nearby frame extracting unit 117. The proximity frame extraction unit 117 extracts non-reference frames at regular intervals between adjacent reference frames and promotes them as reference frames. The automatic anchor setting unit 112 also includes the motion vector use setting unit 11.
3, a contour information use setting unit 114 and a pattern matching use setting unit 116. These three settings are:
Originally, it is sufficient to implement either one, but in the present embodiment, all are implemented, and one of them is selected according to the situation.

As will be described later, the reference frame deletion unit 115 returns a redundant one of the reference frames set by the automatic anchor setting unit 112 to a non-reference frame. Hereinafter, the operation by this configuration will be described.

[1] Automatic Setting of Anchor Using Motion Vector The feature of this process is that the motion vector of the anchor is once obtained from the start frame to the end frame, and then the virtual movement path of the anchor and the motion vector match. It has a two-stage configuration for improving the tracking accuracy of the anchor by determining the degree.

1. Acquisition of motion vector The time of the start frame and the end frame are set to t0
Set to 1. In addition to the above, first, some of the non-reference frames are changed to reference frames by the adjacent frame extraction unit 117. Here, it is assumed that the time is simply changed every five frames.
Is normalized. In order to obtain the motion vector of a specific anchor in the period from the start frame to the end frame, the image area near the center of gravity of the anchor is used as a block,
Perform block matching. A frame corresponding to an arbitrary time t will be referred to as a frame (t).

FIG. 14 is a flowchart showing a procedure for obtaining a motion vector in the present embodiment. As shown in the figure, first, the time counter t is set to t0 (S10).
0). Next, of the anchors set in the start frame, the anchor for which the motion vector is to be acquired is specified. The motion vector utilization setting unit 113 stores the area including the center of gravity of the specified anchor as a block used for block matching (hereinafter referred to as "anchor block") (S101). Subsequently, image data I (t) of frame (t) and I (t +
1) is acquired (S102). I (t) is aggregate data of the pixel value p of each pixel included in the frame.

After this, the anchor block is set to the frame (t
While moving within +1), the optimum matching is searched (S103). Each pixel value of the anchor block itself is I
Since it can be seen from (t), the anchor block is arranged at an arbitrary position of the frame (t + 1), the squared error of the pixel value is calculated between the overlapping pixels, and this is integrated over the entire anchor block. This integration is performed while moving the anchor block little by little, and the position where the integrated value is the minimum is determined to be the destination of the anchor block.

When the movement destination is determined, the movement amount and the movement direction from the anchor block in the frame (t) to the anchor block in the frame (t + 1) are determined, and are obtained as a motion vector V (t) (S1).
04). Here, it is determined whether or not t + 1 has reached time t1 of the end frame (S105). If not, t is incremented (S106), and a motion vector is repeatedly acquired. If t + 1 is equal to t1, then
After saving the obtained V (t) (S107),
Finish the process.

FIG. 15 is a diagram showing an example of motion vectors V (0) to V (2) obtained when t0 = 0 and t1 = 3. As shown in the figure, V (t) is defined by x and y determined by the vertical and horizontal directions on the screen, and t determined by the time direction,
It can be expressed by three components of (x, y, t).

2. Determining the degree of coincidence Each frame is divided into blocks having the same size as the anchor block provided for the motion vector, and all routes that the anchor may have followed are found. FIG. 16 is a diagram showing one of such paths. In the figure, the frame is divided into 16 blocks, and the start point of the route in the start frame and the end point of the route in the end frame coincide with the anchor block in FIG. Under this condition, there are 16 × 16 total routes. Subsequently, a vector (hereinafter referred to as “route vector”) v (t) illustrated in FIG. 16 is defined in each section of this route (hereinafter referred to as “virtual route”). The route vector is determined by the direction of the virtual route from one frame to the next. v
(T) is also described by three components (x, y, t).

Here, V (t) and v
The angle formed by (t) is defined as θt, and f (t) = cosθt is calculated by the following equation using the inner product.

F (t) = (V (t), v (t)) / | V (t) | · | v (t) | (Equation 2) FIG. 17 is compared with v (t) in FIG. FIG. 15 is a diagram in which V (t) is added and displayed, and shows the meaning of θt. Formula 2
Is larger, the degree of coincidence between the virtual path and the motion vector in that section is higher. However, even if the degree of coincidence is maximized in a certain section, if the degree of coincidence in another section is extremely low, the overall Need to be considered low. Therefore, the following evaluation formula is introduced in order to evaluate the degree of coincidence as a whole while considering the degree of coincidence in each section.

G (t) = max {f (t-1) + g (t-1)} (Equation 3) By recursively calculating Equation 3, the degree of coincidence was always the highest until that time. The virtual route is known. If this calculation is performed up to the end frame, the virtual route with the highest degree of coincidence is found throughout, so this route is regarded as the anchor movement route. Thereafter, the anchor is automatically set on the assumption that the anchor at that time exists at the intersection of the movement route and each reference frame. The set anchor information may be added to the anchor information table shown in FIG. For frames other than the reference frame, the anchor information may be sequentially calculated from the interpolation calculation by the same method as in the first embodiment.

[2] Automatic Setting of Anchor Using Contour Information Another method of automatic anchor setting is a method of tracking an anchor based on the movement of a target contour. The contour information use setting unit 114 generates a contour image for each frame by repeating the same processing as in FIG. The contour image is a binarized image in which 1 is on the contour line and 0 is on the contour line, and a compass gradient is added to the image.
It can be generated by applying (nt) type filter. After the contour image is obtained, the anchor may be tracked assuming that the anchor moves exactly the same as the target.

[3] Automatic Setting of Anchor Using Matching As another method of automatic setting of an anchor, there is tracking of an anchor by pattern matching shown in FIG. Also in this method, first, a certain number of reference frames are provided in advance by the adjacent frame extraction unit 117. Next, the anchor area 13 set in the start frame 130
Create a model 134 for pattern matching from 2,
A region 138 having the highest degree of coincidence among the adjacent reference frames 136 is obtained. As a method of pattern matching,
There are a template matching method in which model image data is superimposed as it is, and a structure matching method in which superimposition is performed based on the positional relationship of feature points extracted from an image. Pattern matching is performed mainly in the vicinity of the model.

Thus, in the second reference frame, the area 13
If 8 is found, the same process is repeated using this region 138 as a new model to trace the anchor region. In some cases, good matching cannot be achieved when the target moves or deforms significantly or when the reference frame setting interval is too wide. In such a case, the adjacent frame extracting unit 117 performs the matching process again after narrowing the interval between the reference frames to provide more reference frames.

[4] Elimination of Unnecessary Reference Frames In the above example, the reference frame is provided every 5 frames. However, for example, in the case where the target makes a linear motion at a constant velocity, only the start frame and the end frame are sufficient as the reference frames. Even if the target does not have a uniform linear motion from the start frame to the end frame, for the period of such motion,
Only the reference frames at both ends of the period are needed. As the number of reference frames decreases, the calculation load becomes lighter. In the case of [1], the number of virtual routes is drastically reduced, which is particularly effective.

From this point of view, the reference frame deleting unit 115
Deletes unnecessary reference frames. FIGS. 19 to 21 are diagrams showing a situation in which reference frames are being deleted. In these figures, the horizontal axis is time, and the vertical axis is x- provided on the frame.
The distance from the origin of the y coordinate is shown. In addition, the circles in each figure schematically show the anchor region. Follow the procedure below to delete.

[FIG. 19] Initially, six reference frames are provided including both-end reference frames. Here, the anchor of the start frame and the anchor of the end frame are connected by a straight line, and the distance between the straight line and each anchor is calculated. If there is an anchor whose distance is equal to or less than a predetermined value, the reference frame at that time is deleted. In the figure, it is assumed that no anchor has been deleted. Next, the anchor farthest from the straight line (hereinafter referred to as the farthest anchor) is found. here,
The anchor at t = 3 is the farthest anchor. [FIG. 20] The straight line is deleted, the anchor of the start frame, the farthest anchor, and the anchor of the end frame are connected by a polygonal line in this order, and the distance between this polygonal line and each anchor is obtained again. The anchor at t = 4 whose distance has become equal to or less than the predetermined value is deleted. The farthest anchor is changed to t = 2 anchor.
[FIG. 21] The polygonal line is corrected so as to pass through the new furthest anchor. Here, the reference frame at t = 1, in which the distance from the new polygonal line becomes smaller than the predetermined value, is deleted. This ends the process.

In this example, two reference frames have been deleted. When there are many initial reference frames,
1. 1. Delete reference frames that are less than or equal to a predetermined value. 2. Search for the farthest anchor; Repeat the correction of the broken line.

The above is the outline of the present embodiment. In addition,
The present embodiment can be improved or modified as follows. (1) Modification of Expression 2 In Expression 2, f (t) = cos θt is adopted, but naturally this may be another function. The function of increasing and decreasing θt is f
(T) can be a candidate. (2) Block method In FIG. 15, the anchor block is determined so as to include the vicinity of the center of gravity of the anchor area. However, another method may be used.
For example, the anchor area may be directly used as the anchor block. Similarly, also in FIG. 16, the block may be determined regardless of the size of the anchor block.

(3) Another method of the above [3] (No. 1) Another mode of automatic anchor setting by the above-mentioned matching will be described. When matching is performed one after another using a certain frame region as a model, errors may accumulate and the target may gradually deviate from the target. For this reason, the determination is made in consideration of not only the matching from the adjacent frame but also the matching result from a frame that is distant to some extent.

As shown in FIG. 22, here, a start frame and an end frame are adopted as reference frames separated by time. Now, it is assumed that a new reference frame 404 for specifying the position of the anchor is at time t + Δt. On the other hand, the model A400 of the anchor area of the start frame, the model B401 of the anchor area of the end frame, and the model C403 of the anchor area of the reference frame 402 at time t are all known. Therefore, matching is performed between these three reference frames and the reference frame 404 at time t + Δt. At this time, if all the matching results match, the tracking result of the area is reliable. On the other hand, if the matching results do not match, the anchor position is determined by, for example, the following method. 1. The three regions obtained on the reference frame 404 at the time t + Δt as a result of the matching three times are superimposed, and an anchor region having the same size as the original region is created centering on the center of the overlapping portion.

2. One model, model C403
, A plurality of matching results are obtained, and it is determined in order from the top of the matching results whether or not an area obtained from the matching results of the model A400 and the model B401 is included in the area. If the area is included at a certain ratio or more, an anchor area having the same size as the original area is created around the center of the overlapping portion between the areas obtained from the matching result. Here, the start frame and the end frame are considered in addition to the adjacent reference frame, but the combination has a degree of freedom. For example, an adjacent reference frame and any number of reference frames separated by a certain time distance may be employed.

(4) Alternative method to [3] above (Part 2) As shown in FIG. 23, the automatic anchor reliability setting unit 112 is provided with an automatic reliability setting determination unit 130 and an automatic reliability indicator 131. The automatic setting reliability determination unit 130 determines the reliability of tracking based on the degree of overlap between the anchor area obtained by performing matching in the forward direction up to the last frame and the anchor area specified in the end frame. For example, if the area of the overlapping portion is 70% or more, the tracking result is reliable.
If it is 0% or less, it is determined that it is not reliable. The automatic setting reliability display unit 131 displays the reliability of tracking (whether or not the reliability can be obtained, or% thereof).

For example, as shown in FIG. 24, a tracking result 1 by matching with respect to the original trajectory 140 of the target
When 41 is warped, the tracking is determined to be unreliable because the degree of overlap is low. Therefore, this time, using the anchor region of the end frame as a model, image matching is performed in the reverse direction, and the anchor position is tracked again. At this time, in each reference frame, the anchor area previously set by the forward matching and the anchor area obtained by the current backward matching are compared, and if both overlap by a predetermined ratio or more, the tracking ends. I do. At this time, as another method, tracking may be terminated at a designated frame. The frame for which tracking should be ended may be determined based on the ratio of the distance from the start frame and the end frame. A method of determining overlap and a method of specifying a frame to end tracking may be used together.

As described above, according to this method, even if an error occurs during the tracking, it is possible to avoid the deterioration of the subsequent tracking result, and the final correction work is reduced. Also,
When the matching in the reverse direction is terminated by checking the degree of overlap, only a necessary part is corrected, which leads to a reduction in processing time.

Even when the reliability is low, the matching in the reverse direction may not be started automatically, but may simply be displayed. In this case, the user may cause the matching in the reverse direction to be performed, or may make a desired correction by himself. In any case, the automatic setting reliability display unit 131 allows the user to know whether the tracking is good or not, and can take appropriate measures.

(5) Alternative method of [3] above (Part 3) The respective components of FIG. 12, that is, the moving image reproducing unit 118, the anchor information display unit 120, and the anchor information correction unit 119 are provided in the automatic anchor setting unit 112. . Here, the moving image playback unit 118 chronologically arranges each frame constituting the moving image,
Display at appropriate time intervals. Anchor information correction unit 119
Corrects the anchor information of the frame displayed at that time when the anchor information correction operation is performed at any time during the reproduction of the moving image. At the same time, the automatic anchor setting result of the frame displayed for the predetermined number of frames or the predetermined period immediately before the frame is invalidated.

The operation in this configuration will be described. Here, it is assumed that the moving image is displayed while matching is sequentially performed in the forward direction from the start frame in accordance with the reproduction of the moving image. In the case of this operation, as described above, once the matching becomes unsatisfactory, the anchor information may gradually deviate thereafter. The user clicks the screen when the anchor area deviates from the target while viewing the moving image to be reproduced and the anchor information. At this point, playback stops. Here, for example, when the user clicks the center of the target, the anchor position is corrected so that the point becomes the center. Subsequent matching is performed on the basis of the corrected anchor information, so that it is good.

In this method, since the user clicks the screen during reproduction of the moving image, it is necessary to consider a delay in the action. That is, when the user clicks after recognizing that the anchor region has deviated from the target, it is considered that such a phenomenon has gradually progressed over several frames. Therefore, the anchor information correction unit 119 invalidates the anchor information that is automatically set for a plurality of predetermined frames existing before the frame in which the anchor information is corrected.

FIG. 25 is a diagram showing the relationship between the original trajectory of the target and the tracking result obtained by matching, and the operation of the anchor information correction unit 119. The solid line 150 in FIG. 7 indicates that the trajectory of the target matches the tracking result, and the broken line 151 indicates that the tracking result deviates from the trajectory. As shown in the figure, the tracking results are reliable at times t0 to t1, but start off at time t1. The user notices this and clicks the screen at time t2. As a result, correct tracking is performed again between times t2 and t3.
At times t1 and t2, an incorrect tracking result remains, so this is invalidated. As for the invalidated portion, as shown in the figure, the missing tracking result can be supplemented by linearly interpolating the anchor information at the times t1 and t2. Note that the number of frames for which the anchor information is invalidated may be specified in advance, or may be specified at the time of correction.

Third Embodiment The following devices or systems can be constructed by applying the moving image hypermedia devices of the first and second embodiments.

1. Interactive Video Teaching Material Production Apparatus The moving image hypermedia apparatus according to the present invention is also optimal for producing CAI content. That is, the present apparatus sets an anchor on the video teaching material and links necessary additional information. FIG. 26 is a diagram showing a video teaching material produced by this device. As shown in the figure, the description A is linked to the anchor A, and the description B is linked to the anchor B. While playing the video, the student clicks on the screen the object for which he wants a more detailed description. If the clicked object is associated with anchor A, description A is displayed on the screen.

[0127] 2. Interactive Video Server System The video hypermedia device according to the present invention is also suitable for a video server system. FIG. 27 is a configuration diagram of this video server system. As shown in the figure, this system is composed of a server 200 and a client 250 which almost share the configuration of FIG.

The server 200 stores a moving picture and its anchor information, and a data storage section 204 for storing related data linked to the anchor, an anchor estimation section 206, and a hyperlink searching section for searching related data linked to an arbitrary anchor. 208 is provided. Anchor estimation section 206 estimates anchor information in a non-reference frame.

On the other hand, when the user clicks an object in the moving image, the client 250 includes an anchor determining unit 252 that determines which anchor area has been clicked.

In this configuration, when the user clicks a certain object on the screen with the client 250,
The anchor determining unit 252 specifies the clicked anchor. This information is sent to server 200. Server 20
0 hyperlink search unit 208 is the data storage unit 204
From the relevant data linked to that anchor,
This is sent to the client 250.

As described above, according to this system, moving image data and anchor information are collectively stored on the server 200 side, and a large number of users can view necessary moving images and information linked thereto.

[0132]

According to the moving picture anchor setting device of the present invention, the anchor information of a non-reference frame, that is, a frame other than the reference frame can be estimated by setting the anchor information only for the reference frame. It is not necessary to set anchor information for non-reference frames. As a result, the anchor setting work can be saved.

When a plurality of reference frames are taken and the frame determining means determines the earliest in time among them as the start frame and the latest in time as the end frame, the period for which the user should set the anchor is set. Work efficiency can be improved without having to be aware of it.

When the frame determining means detects the frame in which the target has appeared and the frame in which the target has disappeared and determines these as the start frame and the end frame, respectively, it can be said that there is at least one reference frame, and further labor saving is promoted. be able to.

When the anchor estimating means estimates the anchor information in the non-reference frame by interpolating the anchor information, the anchor information of the non-reference frame can be obtained with a small calculation load.

When the anchor estimating means divides the change amount of the correspondence information between the anchors set in the reference frame according to the elapsed time, the anchor information of the non-reference frame is obtained by a very simple interpolation calculation. be able to.

When the anchor estimating means performs the interpolation calculation of the anchor information between the reference frames at both ends of each section of the setting target period, the anchor for the non-reference frame is estimated using the reference frame located at a close position. Therefore, the accuracy of estimation can be improved.

When the anchor information editing means for modifying the estimated anchor information is included, as a result of the modification, the interpolation calculation using the intermediate reference frame having accurate anchor information is performed. Accuracy can be improved.

When the anchor information editing means three-dimensionally displays the trajectory of the anchor based on the estimation of the anchor information, the editing result can be confirmed on the spot.

When the anchor information editing means calculates the trajectory of the anchor and displays the cross-sectional view of the moving image along this trajectory together with the trajectory of the anchor, the anchor estimated in the non-reference frame is referred to by referring to this cross-sectional view. You can check the accuracy of the information and correct it if necessary.

When the anchor information editing means displays the anchor information display images in time series, the suitability of the anchor setting can be grasped at a glance. Therefore, the confirmation work of the anchor information becomes easy.

When the anchor information editing means includes the moving picture reproducing means and the anchor information correcting means, an appropriate position of the target can be designated while reproducing the moving picture, so that the desirable anchor information is set at the end of the reproduction. As a result, it is possible to reduce the labor of the confirmation and editing work performed later.

When the anchor estimation means includes the anchor automatic setting means, the number of reference frames in which the anchor information should be initially set may be small, so that the setting work can be reduced.

When the anchor automatic setting means tracks the anchor based on the movement of the contour of the target, the accuracy of the anchor tracking can be improved.

The same applies to a case where the automatic anchor setting means tracks an anchor based on a target motion vector. In this case, depending on the type of moving image, a motion vector may be described for each frame during encoding, which is convenient.

The automatic anchor setting means determines, for each route that the anchor can follow in the setting target period, the degree of coincidence between the route and the motion vector of the anchor to search the moving route of the anchor. Can be reduced. That is, although the motion vector should include an error for each unit determination period, the path determination is performed over the entire unit determination period, so that the accuracy of the finally obtained path increases.

When the degree of coincidence is determined by a function of the angle between the path and the motion vector, the degree of coincidence can be evaluated numerically, and a path with high accuracy can be found by relatively simple calculation.

When the anchor automatic setting means includes the automatic setting reliability determining means and the automatic setting reliability displaying means, the user can know whether the automatic setting is good or not, and can take appropriate measures.

When the anchor automatic setting means performs image matching with a new reference frame using the image of the anchor region of the reference frame for which the anchor information has already been set as a model, the accuracy of anchor tracking can be improved.

When a similar image is not found in the new reference frame and the automatic anchor setting means further divides the adjacent reference frames to provide a new reference frame and re-executes the image matching, Matching accuracy increases. In addition, since a new reference frame is provided only at a necessary position, tracking accuracy can be improved while suppressing an increase in calculation load.

In the case where the automatic anchor setting means tracks the position of the anchor based on the matching result of the adjacent reference frame and the reference frame which is separated by a certain time distance, for example, it is possible to improve the situation where the matching result is gradually shifted. It becomes easier and saves the trouble of fixing the anchor.

When the automatic anchor setting means includes the automatic setting reliability judgment means, the anchor information of the deviated portion can be corrected even if the tracking is deviated from the middle.

When re-tracking by the automatic setting reliability determination means, if the tracking is terminated by checking the degree of overlap between the original tracking and the anchor area obtained by the current tracking, re-tracking for an unnecessarily long time is required. The processing time can be shortened.

When designating a frame for which retracking should be terminated, a simple process can produce a corresponding result.

When the anchor information correcting means invalidates the result of automatic anchor setting of a predetermined frame before the frame whose anchor information has been corrected, correction can be made when the tracking result is deteriorated, so that it is confirmed again later. No need. In addition, since there is no need to make corrections while the matching is good, the confirmation work can be saved.

When the anchor setting means includes the reference frame deleting means, the number of reference frames that should hold the anchor information is reduced, and the required storage capacity can be reduced. When the anchor setting means includes a text anchor setting means, the anchor can be set not only in the image data but also in the text data.

When the cursor changing means is further included, even if the anchor area is set invisibly, the existence and position of the desired anchor area can be easily known.

When the display state of the cursor is changed so as to accompany the display of the identification information of the anchor area in which the cursor is inserted, some related information can be known without a click operation.

In the case where the anchor setting means sets the anchor information in units of a group consisting of a plurality of anchors, for example, labor for setting the anchor information many times for the same target can be omitted.

When an anchor search means for searching for an anchor based on the feature of the movement of the anchor is included, the search labor is greatly increased when it is desired to know only one of a large number of objects that moves to the right. It can be reduced.

When an anchor search means for searching for an anchor based on anchor identification information is included, search efficiency can be improved.

On the other hand, according to the moving picture hypermedia device of the present invention, it is possible to link the data required for the anchor and retrieve the linked data while utilizing the advantages of the moving picture anchor setting means of the present invention.

When this moving picture hypermedia device is used for producing an interactive video teaching material, for example, an interactive video teaching material in which additional information can be obtained by clicking an object on the screen can be produced. .

On the other hand, according to the moving picture providing system of the present invention, a necessary moving picture and information linked thereto can be seen from many users.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a system including a moving image hypermedia device according to a first embodiment.

FIG. 2 is a flowchart showing a procedure for setting and correcting an anchor according to the first embodiment.

FIG. 3 is a diagram showing a table of anchor information set in the first embodiment.

FIG. 4 is a diagram showing an interpolation calculation method of anchor information according to the first embodiment.

5 is a diagram showing a table obtained by adding anchor information of an intermediate reference frame to FIG.

FIG. 6 is a diagram illustrating a method of performing an interpolation calculation based on three frames of an intermediate reference frame and both-end reference frames in the first embodiment.

FIG. 7 is a flowchart showing a link setting and search procedure according to the first embodiment.

FIG. 8 is a diagram showing a table of link information set in the first embodiment.

FIG. 9 is a diagram illustrating an example of a UI screen for anchor setting.

FIG. 10A is a lateral cross-sectional view of the locus of the anchor region from the start frame to the end frame, FIG.
Similarly, 0 (b) is a vertical cross-sectional view.

FIG. 11 is a diagram showing an anchor information display image displayed in chronological order by the anchor information editing unit in the first embodiment.

FIG. 12 is a diagram showing an internal configuration example of an anchor information editing unit of the moving picture hypermedia device according to the first embodiment.

FIG. 13 is a configuration diagram of an anchor setting unit of the moving image hypermedia device according to the second embodiment.

FIG. 14 is a flowchart showing a procedure for acquiring a motion vector according to the second embodiment.

FIG. 15 is a diagram illustrating an example of motion vectors V (0) to V (2) obtained when t0 = 0 and t1 = 3 in FIG. 14;

FIG. 16 is a diagram showing one of paths that an anchor may have taken.

FIG. 17 shows V (t) shown in FIG.
It is the figure which added and displayed (t).

FIG. 18 is a diagram showing a method for automatically setting an anchor based on pattern matching in the second embodiment.

FIG. 19 is a diagram showing how reference frames are deleted in the second embodiment.

FIG. 20 is a diagram showing how reference frames are deleted in the second embodiment.

FIG. 21 is a diagram illustrating a state where reference frames are deleted in the second embodiment.

FIG. 22 is a diagram illustrating a method of performing matching using a reference frame that is separated by a certain time distance in addition to an adjacent reference frame in the second embodiment.

FIG. 23 is a diagram illustrating a configuration example of an automatic anchor setting unit of the moving image hypermedia device according to the second embodiment.

24 is a diagram illustrating an erroneous tracking result that may occur without the configuration in order to explain the effect of the configuration in FIG. 23;

FIG. 25 is a diagram showing the relationship between the original trajectory of the target and the tracking result obtained by matching, and the operation of the anchor information correction unit.

FIG. 26 is a diagram showing a configuration of an interactive video teaching material production device according to a third embodiment.

FIG. 27 is a diagram showing a configuration of an interactive video server system according to a third embodiment.

[Explanation of symbols]

1 data operation unit, 2 data storage unit, 3 display unit, 4
User operation unit, 6 moving image input unit, 10 frame determination unit, 11 anchor setting unit, 12 anchor estimation unit, 1
3 hyperlink setting part, 14 hyperlink search part, 15 anchor search part, 20 moving image data storage part,
21 anchor information storage unit, 22 link information storage unit,
Reference Signs List 30 display control unit, 31 display device, 50 image display area, 52 button group, 54 rectangular button, 56 start frame designation button, 58 end frame designation button, 60
Anchor area, 62 Anchor-related box group, 64
Scene-related boxes, 66, 68 boxes, 80
Anchor information, 81 start frame, 82 end frame, 110 anchor information editing part, 111 text anchor setting part, 112 automatic anchor setting part, 113
Motion vector use setting unit, 114 contour information use setting unit, 115 reference frame deletion unit, 116 pattern matching use setting unit, 117 adjacent frame extraction unit, 11
8 moving image display section, 119 anchor information correction section, 120
Anchor information display unit, 130 automatic setting reliability determination unit,
131 automatic setting reliability indicator, 140 original trajectory of target, 141 tracking result by matching, 20
0 server, 204 data storage unit, 206 anchor estimation unit, 208 hyperlink search unit, 250 client, 252 anchor determination unit, 300 cursor change unit, 400 model A, 401 model B, 402 reference frame at time t, 403 model C , 404 Reference frame at time t + Δt.

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] July 23, 1997

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Full text

[Correction method] Change

[Correction contents]

[Document Name] Statement

[Title of Invention] Video anchor display, selection and setting device
And a video hypermedia device using video anchors
Video provision system

[Claims]

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image anchor setting device and a moving image hypermedia device using the device. The present invention particularly provides a device for inputting a moving image and setting an anchor to a target or the like included in the moving image,
A moving image hypermedia device for linking relevant information to an anchor set by the device and performing a desired search, etc.
The present invention relates to a moving image providing system using this device as a server client system.

[0002]

2. Description of the Related Art Conventionally, information retrieval in a general hypermedia device mainly sets a logical unit for an information link to a text or a still image, and links related information to this logical unit in advance. , The related information is displayed when the user clicks the logical unit. However, various technologies have been proposed in recent years for processing not only still images but also moving images, as represented by MPEG for encoding and decoding of moving images. By handling moving images, the hypermedia device can be used for content creation such as CAI, various presentations, and electronic catalogs. The editing of moving images has been used in some limited industrial fields such as broadcasting stations in the past, but it is expected that it will be rapidly spread as a personal system based on a personal computer in the future.

Japanese Patent Laid-Open No. 4-163589 discloses an image processing apparatus capable of setting a logical unit (called a node in the specification) for a moving image. This device is capable of setting a node in a still image simply by specifying a display range. On the other hand, in the case of a moving image, the validity period of the node is determined from both (1) display range and (2) time. By paying attention to the point that the area range should be designated, these designations are made possible. That is, with respect to (1), the node at that time is set by pointing the region surrounding the subject appearing in the moving image with a mouse or the like, while with respect to (2),
The valid duration of the node is designated by the elapsed time from the output start time of the moving image. Therefore, this node is uniquely determined by the two contents of the area and the elapsed time, and related information can be linked to each node. After the link, when actually reproducing the moving image, if the user clicks a certain area on the screen with a mouse or the like, a node is specified by the position and time, and related information is displayed.

[0004]

In the above-mentioned device, the area setting for each logical unit is manually performed. However, naturally, unlike a still image, a moving image has a large number of frames, and the position and shape of the subject change from moment to moment.
In the case of the NTSC system, 30 frames are required for one second, so even if it is simply calculated, 30 processing operations per logical unit are required when processing a moving image of one second. For example, when creating five minutes of content, if five logical units are set in one frame, the number of times of setting is 45,000.

The present invention has been made in view of the above problem, and an object of the present invention is to provide a device which enables labor saving and simplification of a setting operation of a logical unit (referred to as an anchor in this specification). An object of the present invention is to provide an anchor setting device that automatically calculates or automatically sets anchor information that needs to be performed for each frame. Another object of the present invention is to provide a moving image hypermedia device and a moving image providing system using the anchor setting device.

[0006]

A moving picture anchor setting apparatus of the present invention is an apparatus for inputting a moving picture and setting an anchor on the moving picture. This device is based on a frame determination unit that determines a start frame and an end frame of a setting target period in which an anchor is to be set from a plurality of frames that form an input moving image, and a reference frame that is taken within the setting target period. When a region is specified, the anchor setting means for setting the anchor information for this reference region as the anchor region and the anchor information in the non-reference frame based on the anchor information set for the reference frame And an anchor estimating means for estimating.

In one aspect of the present invention, when a plurality of reference frames are taken during the setting target period, the frame determining means determines that the earliest one of them is a start frame and the latest one is an end frame. Decide

In another aspect of the present invention, the frame determining means detects a frame in which a target included in the reference frame appears and disappears before and after the reference frame, and determines these as a start frame and an end frame, respectively. Decide

In one aspect of the present invention, when there are a plurality of reference frames, the anchor estimating means estimates the anchor information in a non-reference frame by interpolating the anchor information set for the reference frames.

At this time, in one aspect of the present invention, the anchor estimating means performs interpolation in the following procedure. First, the amount of change in correspondence information between anchors set for two adjacent reference frames is calculated, and then the elapsed time from one reference frame to a certain non-reference frame and the other reference Calculate the ratio of elapsed time to the frame. Thereafter, the change amount is divided by this ratio.

In one aspect of the present invention, the anchor estimating means performs an interpolation calculation of the anchor information between the reference frames located at both ends of the section in the setting target period divided by the reference frame. The anchor information in the non-reference frame included in the section is estimated.

One aspect of the present invention includes anchor information editing means for correcting the anchor information estimated for a non-reference frame. At this time, the anchor estimating means:
The non-reference frame whose anchor information has been modified is promoted to a reference frame.

In one aspect of the present invention, the anchor information editing means three-dimensionally displays the locus of the anchor based on the estimation of the anchor information.

In one aspect of the present invention, the anchor information editing means calculates the trajectory of the anchor based on the estimation of the anchor information, and displays a cross-sectional view of the moving image along the trajectory together with the trajectory of the anchor.

In one aspect of the present invention, the anchor information editing means selects a frame included between a start frame and an end frame, and superimposes anchor information on the image of the frame to create an anchor information display image. After that, the anchor information display images are displayed in time series.

In one aspect of the present invention, the anchor information editing means includes a moving image reproducing means and an anchor information correcting means. The moving image reproducing means reproduces the moving image on the screen, and when the anchor information correcting operation is performed during the moving image reproduction, the anchor information correcting means specifies the frame reproduced at the time when the operation is performed, If the frame is a non-reference frame, the frame is promoted to the reference frame and the anchor information in the frame is corrected based on the operation.

In one aspect of the present invention, the anchor estimation means includes an anchor automatic setting means. The automatic anchor setting means selects an arbitrary number of frames from adjacent reference frames, sets them as new reference frames, tracks the anchor, and sets anchor information for these new reference frames. .

In one embodiment of the present invention, the automatic anchor setting means tracks the anchor based on the movement of the contour of the target.

In one aspect of the present invention, the anchor automatic setting means tracks the anchor based on the motion vector of the target.

According to an aspect of the present invention, when each section of the setting target period divided by the reference frame is defined as a unit determination period, the anchor automatic setting means determines each route that the anchor can follow in the setting target period. , The degree of coincidence between the path and the motion vector of the anchor is determined for each unit determination period. This judgment result is integrated over the entire set target period. Finally, the route having the highest degree of coincidence in the overall result is regarded as the movement route of the anchor.

In one aspect of the present invention, the degree of coincidence is determined by a function of an angle formed by the path and the motion vector, the function having no extreme value in the range of 0 ° to 180 °, The sum of the values of the function for each unit determination period is calculated to obtain the total result. The function value gradually increases or decreases as the angle changes from 0 ° to 180 °. Therefore, the degree of coincidence is evaluated by the simple sum of these values.

In one aspect of the present invention, the automatic anchor setting means includes automatic setting reliability determination means and automatic setting reliability display means. The automatic setting reliability determination unit determines the reliability of the automatic setting by comparing the anchor information set by the anchor setting unit with the anchor information set by the anchor automatic setting unit. The automatic setting reliability display means displays the reliability.

In one aspect of the present invention, the automatic anchor setting means uses the image of the anchor region of the reference frame in which the anchor information has already been set as a model to perform image matching with a new reference frame, thereby setting the anchor Track location.

In one aspect of the present invention, when a similar image is not found in the new reference frame when performing image matching, the automatic anchor setting means further divides the adjacent reference frames to create a new image. A reference frame is provided, the interval between the reference frames is narrowed, and image matching is performed again.

In one aspect of the present invention, the automatic anchor setting means uses, as the reference frame in which the anchor information has already been set, a reference frame close to the new reference frame and a reference frame separated by a certain time distance. Select at least two. After that, the position of the anchor is tracked based on the matching result using the selected reference frame.

In one aspect of the present invention, the automatic anchor setting means compares the anchor information set by the anchor setting means with the anchor information set by the automatic anchor setting means to determine the reliability of the automatic setting. An automatic setting reliability determination unit for determining is included. If the reliability determined by the automatic setting reliability determination means is low after the anchor information is set by the tracking starting from the first reference frame, the anchor information is determined by the tracking starting from the second reference frame. Is to be reset.

At this time, while tracking is performed with the second reference frame as a starting point, an anchor area set in a certain reference frame and tracking with the first reference frame as a starting point are set in the reference frame. If the degree of overlap with the created anchor region is equal to or higher than a predetermined ratio, the tracking with the second reference frame as a starting point is terminated.

Alternatively, in the tracking using the second reference frame as a starting point, the tracking is stopped when the reference frame specified in advance is reached.

In one embodiment of the present invention, the automatic anchor setting means includes a moving picture reproducing means, an anchor information displaying means, and an anchor information correcting means. Video playback means,
Play the video on the screen. The anchor information display means displays the anchor information obtained as a result of the image matching sequentially performed on the displayed frame. When an anchor information correction operation is performed during moving image reproduction, the anchor information correction means corrects the anchor information of the frame on which the operation is performed, and invalidates the automatic anchor setting result in a predetermined frame before that frame. Become

In one aspect of the present invention, the anchor setting means includes reference frame deleting means. The reference frame deletion unit demotes the reference frame to a non-reference frame when the anchor information in a certain reference frame is obtained within a predetermined error tolerance from the interpolation calculation of the anchor information in another reference frame.

In one embodiment of the present invention, the anchor setting means includes a text anchor setting means. The text anchor setting means, when a text is designated for a reference frame, sets an anchor having the text as the content for the reference frame.

In one embodiment of the present invention, when the cursor enters an arbitrary anchor area, the apparatus further includes a cursor changing means for changing a display state of the cursor.

At this time, in one aspect of the present invention, the display state of the cursor is changed so that the cursor changing means accompanies the display of the identification information of the anchor area in which the cursor is inserted.

In one aspect of the present invention, the anchor setting means sets the anchor information in units of a group consisting of a plurality of anchors.

An aspect of the present invention further includes anchor search means for searching for an anchor based on characteristics of the movement of the anchor in the anchor information.

An aspect of the present invention further includes anchor search means for searching for an anchor based on the identification of the anchor.

On the other hand, the moving image hypermedia apparatus of the present invention comprises: When an area is designated for a reference frame taken during the period, an anchor setting means for setting anchor information for the reference frame with this as an anchor area,
An anchor estimation unit that estimates anchor information in a non-reference frame based on the set anchor information.
The above corresponds to the moving image anchor setting device. The apparatus further includes a link setting means for linking the set or estimated anchor information with any related data, and a link search means for searching the related data linked to the anchor from any anchor.

In one embodiment of the present invention, the moving image processed by the moving image hypermedia device constitutes a video teaching material, and the related data is additional information associated with a desired anchor of the video teaching material in a desired setting target period. , The device is used to produce interactive video materials.

On the other hand, the moving image providing system of the present invention comprises a server and a client. The server stores the moving image, the anchor information set for the reference frame of the moving image, and a means for storing related data linked to the anchor information, and the server based on the anchor information set in the reference frame. The apparatus includes means for estimating anchor information in a reference frame, and means for searching, from an arbitrary anchor, related data linked to the anchor. The client also includes means for determining which anchor area is clicked when the user clicks the target in the moving image.

The moving image anchor display device of the present invention
And anchor information set for the reference frame
And the anchor information set in the reference frame.
Estimate anchor information in non-reference frame based on
Method, and when playing the video, the
Means for displaying anchor information related to the
It is a thing.

Further , the moving picture anchor selecting apparatus of the present invention
Anchor information set for the video and its reference frame
And an anchor set in the reference frame.
The anchor information in the non-reference frame is estimated based on the information.
Setting means, means for playing video, and played screen
Clicking on any of the above points will cause it to be related to that point
Means to identify which anchor is present
It includes.

[0042]

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of the moving picture hypermedia device of the present invention will be described below. This device incorporates the moving image anchor setting device of the present invention. According to the present apparatus, for example, an interactive CAI software capable of displaying a name of a fish, a supplementary explanation, and the like when a user clicks on any swimming fish using a moving image of an aquarium aquarium as a material is easily provided. And it can be created efficiently. In the following embodiments, the “user” mainly refers to the creator of such content, but it goes without saying that this may be a person who personally edits a video tape shot by himself / herself.

Embodiment 1 In this embodiment, the user explicitly specifies a start frame and an end frame, and sets an anchor using these two frames as the first reference frames. A "frame" is a display unit of an image and includes a picture referred to in MPEG. This apparatus automatically calculates anchor information such as the position, shape, and color of the anchor region in another frame by interpolation calculation from the anchor set for the reference frame. The “anchor information” refers to, for example, the position and shape of the anchor area, the color of the anchor area when it is explicitly displayed, and the like. The present invention can be realized even if the first reference frame is one, but an example thereof will be described later.

The configuration of the entire system including the moving image hypermedia device of the present embodiment is a personal computer (hereinafter referred to as P
C) and a video playback device that provides moving images to this PC. The PC has a built-in video capture board that captures and digitizes the image provided by the video playback device. The video reproducing apparatus has a function of performing reproduction from a designated frame or time in addition to a normal reproduction start, stop, fast forward, frame advance, and the like. Although such a video reproducing apparatus is widely used in broadcasting business and the like, it is not necessarily limited to this. Control of various functions of the video playback device is controlled by a user interface (hereinafter,
UI), for example, via a "play button" displayed on the screen. In this case, when the user clicks the button, the action is transmitted from the PC to the video playback device via the signal cable. Although the video playback device itself is not an essential component of the moving image hypermedia device, it will be described here as a system including this. FIG. 1 is a configuration diagram of a system including a moving image hypermedia device according to the present embodiment.

This apparatus is roughly divided into a data operation unit 1 for operating data relating to anchor information and link information, a data storage unit 2 for storing these data, a display unit 3 for displaying these data in a meaningful form, It comprises a user operation unit 4 for receiving and managing user operations, and a moving image input unit 6 for inputting a moving image reproduced by the video reproducing device 5.

(1) Data operation unit 1 The user gives an instruction to the data operation unit 1 through a UI described later. That is, the following internal configuration is a software module.

The frame determination unit 10 determines a start frame and an end frame. In the present embodiment, the frame designated by the user becomes the start frame and the end frame as they are. Examples of the start frame and the end frame are the start and end frames of the scene showing the aquarium in the video of the aquarium described above. If the scene moves to the video of the entrance to the aquarium, it is not necessary to set an anchor for the fish thereafter, so the end frame is specified before the transition of the scene.

The anchor setting section 11 actually sets an anchor between the start frame and the end frame. For example,
When an anchor is set for a certain fish, first, a rectangle surrounding the fish is displayed with a mouse in the start frame, and this is registered as an anchor area. At this time, the moving image is in the stop mode. Then, the moving image is advanced to the end frame, and the anchor area is registered by surrounding the same fish again. Since the fish moves or changes direction between the start frame and the end frame, its position and shape usually change.
In general, the shape and position of the anchor area registered in the start frame do not match the shape and position of the anchor area registered in the end frame. Note that the anchor setting unit 11 includes an anchor information editing unit 11 that is used in the later-described anchor correction.
0, and a text anchor setting unit 111 that sets an anchor for text such as a character string.

The anchor estimating unit 12 performs interpolation calculation based on the anchor information set in the start frame and the end frame, and estimates the position and size of the anchor in an arbitrary frame (non-reference frame). This process will be described in detail later.

The anchor search unit 15 searches for an anchor based on the feature of the motion of the anchor or the identification information of the anchor in the anchor information. The identification information is information that serves as a clue to distinguish the anchor from other anchors, and includes, for example, the name of the anchor, an anchor setting target, and an anchor setting date and time.

The hyperlink setting unit 13 sets a hyperlink for the set anchor, and creates a data structure related to the setting in the form of a table. The hyperlink search unit 14 searches the set link information.
In the case of the above example, a fish anchor and text data indicating the name of the fish are associated by hyperlinks.

(2) Data Storage Unit 2 The data storage unit 2 may be a database, various file devices, or memory devices. This part is mainly hardware.

The moving picture data storage section 20 stores the moving picture data captured by the moving picture input section and digitized. The anchor information storage unit 21 and the link information storage unit 22 store the set anchor information and link information, respectively.

(3) Display Unit 3 The display control unit 30 is a display system program that comprehensively controls the display of various images such as a UI and a moving image being edited.
It includes a display circuit such as a VGA controller and this driver. The display control unit 30 is a cursor changing unit 300.
have. The cursor changing unit 300 changes the display state of the cursor when the cursor enters the anchor area. The output data of the display control unit 30 is given to a display device 31 such as a monitor of a PC, and a desired display is performed.

(4) User operation unit 4 This enables a user to input commands, and is composed of hardware such as a keyboard, a mouse and various pointing devices, and a command dispatcher. Examples of commands include setting an anchor, modifying an anchor area, a link, and a link search.

(5) Video input section 6 A hardware equivalent to a video capture board
It has a D converter and a frame memory (not shown), and digitizes an input moving image. After that, the data is provided to the moving image data storage unit 20.

Based on the above configuration, the procedure for setting the anchor and the link will be described first, and the state of the UI for setting the anchor will be described later.

[1] Anchor Setting FIG. 2 is a flowchart showing an anchor setting and correction procedure according to the present embodiment, and FIG. 3 is a diagram showing a table of set anchor information. As shown in FIG. 2, first, various initialization processes for hardware and the like are performed (S21).
The moving image data stored in the moving image data storage unit 20 is read (S22). The top frame of the read moving image data is first displayed as a still image on the display device 31. Next, the anchor information already set for the moving image data is read from the anchor information storage unit 21 (S23). If the anchor information exists, the anchor area is actually displayed on the screen (hereinafter, the mode in which the anchor area is displayed on the screen is “anchor display mode”, and the mode in which the anchor area is not displayed is “anchor non-display mode”
That).

Next, the moving image data is advanced to the start frame of the period in which the anchor is newly set this time (S2
4) When the desired frame appears, press the "start frame" button on the screen to register the start frame. In this state, the frame is in a waiting state for setting an anchor area in this frame, and the user provides a rectangular area by clicking the mouse so as to surround another fish, for example. Once the rectangular area is determined, the upper left point (x1, y1) and the lower right point (x2, y1)
The coordinates of 2) are acquired, and this is recorded as anchor information of the fish together with the frame number of the start frame (the serial number from the first frame of the moving image) (S25).

After that, the moving image data is advanced again, and stopped when the desired end frame appears (S26), and a rectangular area is provided so as to surround the same fish. Here, the setting of the anchor in the end frame is completed (S27). “Anchor1” in FIG. 3 is an anchor ID indicating this fish. Here, the frame numbers of the start frame and the end frame (frames 1 and 100, respectively) and the coordinate information of the anchor area are stored in the table.

When the anchor information in the reference frames at both ends is fixed in this way, the anchor information of the frames (non-reference frames) between them is obtained by interpolation calculation (S2).
8). FIG. 4 is a diagram showing an interpolation calculation method of anchor information. Here, the anchor information at the start frame (time t0) is A
(T0)-The anchor information at the end frame (time t1) is A
(T1) -If the anchor information at time t is A (t) -t1-t0 = Δt, A (t) = {A (t1) -A (t0)} t / Δt + {A (t0) t1 −A (t1) t0} / Δt (Equation 1). As A, the x1, y1, x2,
By substituting y2, the outer shape of the anchor area at an arbitrary time is determined. Substituting the barycentric coordinates of the anchor area reveals the rough movement of the anchor area. By substituting the color number for A, the change in color of the anchor area can be tracked. In addition to this, information that can be expressed numerically can also be interpolated by the internal division calculation using Expression 1. The anchor information of the non-reference frame obtained by the interpolation may be added to “anchor1” in the table of FIG. 3, or the table of FIG. The calculation of Expression 1 may be sequentially performed. In the present embodiment, the sequential calculation is assumed hereinafter.

When S28 is completed, the anchor information is actually displayed to confirm the contents (S29). At this time, the moving image data is reproduced by returning to the start frame, and the anchor area is displayed as a rectangle in each frame. This rectangular area moves continuously according to the calculation result.

In the case of "anchor1", the result is extremely good if the fish makes a linear motion at a constant speed. However, if the direction of swimming is changed on the way, the anchor area is shifted from the fish in the middle frame. Therefore, the anchor information is corrected (S3
0). The user first advances the moving image data to a frame with a large shift, and stops the image here. Next, the end of the anchor area displayed on the screen is clicked, and the shape or position of the area is changed with the mouse. The anchor estimation unit 12 upgrades the frame thus modified to the reference frame (hereinafter, the one that has been promoted to become the reference frame is also referred to as “intermediate reference frame”), and adds this anchor information to the table of FIG. . FIG. 5 shows a table obtained by adding the anchor information of the intermediate reference frame to FIG. On the other hand, FIG. 6 is a diagram showing a method of performing an interpolation calculation based on three frames of an intermediate reference frame and both end reference frames. If the non-reference frame to be estimated exists between the start frame and the intermediate reference frame, interpolation calculation is performed between those frames. If the non-reference frame exists between the intermediate reference frame and the end frame, those frames are calculated. Interpolation calculation is performed between them (S28). After that, when good anchor information is obtained after the display in S29 and the re-correction in S30 (Y in S31), this is saved (S3).
2) End the anchor setting process. If the anchor of another frame is modified in S30, naturally this frame also becomes the intermediate reference frame. When two or more anchors are set in the same frame in S25, the anchor IDs are automatically changed within the apparatus in the setting order, and the anchor IDs are assigned, and a rectangular display of these anchor areas is performed in different colors. You just have to deal with it.

According to the above procedure, the following effects can be obtained. 1. Just by setting the anchors in the both-end reference frames, the setting work for a large number of frames existing therebetween is not required. 2. If a displacement occurs in the position of the anchor in the interpolation calculation, this displacement can be confirmed. Therefore, it is easy to recognize the frame to be corrected, and the frame once corrected is automatically promoted to the intermediate reference frame, so that the user does not need to pay attention to whether or not to change the frame to the reference frame. 3. For example, even in a case where a fish having an anchor set therein draws an arc and swims, sufficiently good anchor information can be obtained by performing correction at most several frames in addition to the both-end reference frames. The above is the outline of the moving image anchor setting device among the moving image hypermedia devices of the present embodiment.

[2] Link setting Next, a link is set for the set anchor. FIG. 7 is a flowchart showing a link setting and search procedure according to this embodiment, and FIG. 8 is a diagram showing a table of set link information.

FIG. 7 shows a processing procedure in the case where the anchor setting and the link setting are performed completely independently. Similar to FIG. 2, first, various initialization processing (S40) and reading of moving image data (S41) are performed. Do. Subsequently, the anchor information set in [1] is read from the anchor information storage unit 21 (S42), and the already set link information is read from the link information storage unit 22.

Next, based on the anchor information of the both-end reference frame and the intermediate reference frame, the anchor information of the other frames is obtained by interpolation calculation (S44), and the display of the anchor information is continuously performed in accordance with the reproduction of the moving image. Perform (S4
5). In this state, the user operation unit 4 waits for an input from the user (S46).

Here, the user clicks on a certain anchor area after temporarily stopping the moving image or the moving image,
If the "Create / Change Link" button is pressed, link information is created for that anchor (S47). For example, when a fish in the aquarium is clicked, text, images, and other candidates to be linked to the fish appear on the screen, and the text or the like selected by the user is anchored to the fish (more precisely, included in the anchor. Object called a fish). If there is no candidate, the user can himself input a character string and link it. In FIG. 8, the text information "anchor1.txt" is added to "anchor1", and similarly "anchor1.txt".
Bitmap image "anchor" as an anchor for "2"
2. “bmp” is shown in a linked state. When the link information is determined in this way, the contents of the link are stored in the link information storage unit 22, and the user waits for an input again.

On the other hand, if the user presses the "link search" button in S46 and specifies the anchor, the link information corresponding to the anchor is searched and displayed (S4).
9). In the case of FIG. 8, for example, for the fish of anchor1, the name, length, characteristics, etc. of the fish are displayed in a character string.
For the fish of nchor2, a picture of the sea where the fish actually lives is displayed. Since the link operation can be confirmed by this display, the user can complete the content creation at this point. By storing the content in a recording medium such as a CD-ROM,
It can also be commercialized. When shipping as a product, generally, the mode is changed to the anchor non-display mode in which the anchor area is not displayed.

Although the setting of the anchor and the link has been described as independent processing here, for example, if a button "return to anchor setting" is provided on the screen during link setting,
Both sides can freely move, and editing becomes easier.

[3] UI for Setting Anchor FIG. 9 is a diagram showing an example of a UI screen for setting an anchor. In the figure, a moving image to be processed is displayed in an image display area 50. The black-painted button group 52 in the upper column is an object button for directly instructing playback, stop, etc. of the video. Next to it, a rectangular button 5 for setting an anchor area for the frame displayed in the image display area.
4. Similarly, a start frame designation button 56 and an end frame designation button 58 for designating the displayed frame as a start frame or an end frame are provided. In the figure, the anchor area 60 for one fish is shown.
Is set.

On the right side of the center of the screen, there is an anchor-related box group 62 showing the name, ID, start frame number, and end frame number of the anchor to be set or modified. Below the image display area 50, there is a scene-related box group 64 indicating the number of the scene including the currently displayed frame and the serial number of the frame in the scene. Further below, there is a box 66 for advancing or returning the video by a small amount for editing. Press the button at the right end to advance the movie, and press the left end to return. The position of the currently displayed frame in the scene is indicated by a vertical line 70 in the box 66. Below this box is a box 68 that indicates the position of the start and end frames in the scene. The positions of the start and end frames are indicated by double vertical lines 72, 74, respectively, and the position of the intermediate reference frame therebetween is indicated by the triangular symbol 76.

In the figure, the user first advances the video tape to the beginning of the scene for which the anchor setting is desired, using the scene number as a clue. In this case, for example, among the images of the aquarium consisting of a plurality of scenes, the image of the aquarium with scene number “5” is displayed. Here, the user presses the button at the right end of the box 66 to advance the moving image one frame at a time. When the first frame for which an anchor is to be set appears, the user presses the start frame designation button 56 and registers this. At this time, a double vertical line 72 showing the position of the start frame appears at the corresponding position of the box 66.
Here, the rectangular button 54 is pressed, and the upper left point and the lower right point of the anchor area to be set in the image display area 50 are clicked with the mouse. This completes the anchor setting of the start frame. Next, proceed with the video, and similarly, register the end frame and set the anchor.

When it is detected that the setting in the both-end reference frame has been completed, the anchor estimating unit 12 of the present apparatus automatically substitutes the anchor information into Expression 1 and starts the calculation. Here, the user returns to the start frame,
As the frame advances, the anchor estimating unit 12 obtains the time corresponding to the currently displayed frame, and displays the anchor area based on the estimation result corresponding to this time. If the displayed anchor area is displaced, the user presses the rectangular button 54 again to correct the area. After modification, a triangular symbol 76 appears at the location corresponding to that frame.
According to this UI, since the anchor information is actually displayed on the moving image data, the edited result can be confirmed in real time, and the correction can be easily performed.

The above is the outline of the present embodiment. In addition,
Regarding the present embodiment, the following improvements, modifications, and the like can be considered.

(1) Setting of Text Anchor The text anchor setting unit 111 of FIG. 1 is used. First, edit the text data on the screen, overlay it on the video, and set the anchor. The difference from the normal anchor setting is that, instead of specifying a partial area of the reproduced image, the created text is temporarily put on the image, and then the anchor area is set so as to surround the text. Conventionally, for example, a method of directly inserting an annotation into a video image has been common, but in that case, it is inconvenient for re-editing such as deleting the annotation later. This embodiment solves this.

When the text anchor is set, its anchor information is also stored in the anchor information table. However, in the table shown in FIG. 3, the "frame" portion becomes "text", and the text name is entered in that column.

It is also possible to link related information about the text anchor. For example, paste the text "Fish of the South Sea" to the aquarium scene in Fig. 9, and link the text to the text "There are many brightly colored fish in the South Sea ...". Can be made.

(2) Change of display state of cursor This is performed by the cursor change unit 300 of FIG. This feature is especially useful in anchor-hidden modes, such as when content is used on the market. Because of this function, the cursor changing unit 300 includes a position acquisition program that constantly acquires the position of the cursor, a determination program that determines whether the acquired position is included in any anchor region, and an anchor region where the cursor is located. When entering, there is a change program for determining how to change the display state of the cursor, and actually changing the shape and the like of the cursor according to the determination.

The change of the cursor may or may not be changed for each anchor. In the former case, for example, there is a method of changing the cursor, which is usually a + sign, to ◎ or increasing the brightness of the cursor. According to this aspect, there is an advantage particularly when the movement and shape of the target change rapidly and the change in the anchor area is severe.

On the other hand, in the latter case, it is conceivable that an anchor ID of the anchor area where the cursor is entered is searched from the change program, and this is displayed as it is at the position of the cursor instead of the cursor. For example, when the cursor enters the anchor area of a fish, the cursor may indicate the content of the target of the anchor, such as "shark". According to this aspect, the user can know the name of the fish without needing to click the fish.

(3) Explicit Designation of Intermediate Reference Frame In this embodiment, only the reference frames at both ends are initially determined. However, when the movement of the target is irregular or the like, it is necessary to make corrections. There is also. In that case,
From the beginning, it is assumed that designation of an anchor area is also accepted in frames other than the start frame and the end frame. For example, in the UI of FIG. 9, an intermediate frame designation button is provided in addition to the start frame designation button 56 and the end frame designation button 58. Since this frame is used as the reference frame from the beginning, it can be considered that the interpolation calculation is started from the state shown in FIG.

(4) Anchor area other than rectangle It is not necessary to limit the anchor area to a rectangle. For example, in the case of a circle or an ellipse, an area may be specified by coordinates of three points, a major axis, a minor axis, and a center. If it is a polygon, the coordinates of each vertex will do. When the outer periphery of the target itself is used as the anchor region, the region can be specified by the coordinates of one point on the outer periphery and the chain code expressed from that point.

(5) Employment of Nonlinear Interpolation In the present embodiment, linear interpolation is used most simply,
This may of course be a non-linear interpolation. The equation used for interpolation is
It can be determined by experiments or the like according to the characteristics of the moving image to be processed.

(6) Determination of start and end frames In the present embodiment, these frames are explicitly designated by the user, but the following method is also available. 1. The user does not need to be aware of the start and end frames,
Simply specify the frame and set the anchor. The specified frame becomes the reference frame. Frame determination unit 10
Determines that the frame with the smallest frame number among the frames to which the user has set an anchor is the start frame and the frame with the largest frame number is the end frame. In this case, the start frame designation button 56 and the end frame designation button 58 in FIG. 9 become unnecessary.

2. The user designates one frame, sets an anchor for this, and designates a target for which the anchor is set. This frame becomes the reference frame. The frame determination unit 10 examines frames before and after the reference frame to detect a frame in which the target appears and a frame in which the target disappears, and sets them as a start frame and an end frame, respectively.

The presence or absence of a target is determined by matching images. That is, matching processing is performed on the preceding and succeeding frames using the target specified by the reference frame as a model. The target frame of the search is expanded forward and backward as long as matching can be achieved. Eventually, if the matching cannot be obtained, the start and end frames are determined. According to this method, only one reference frame should be provided initially.

(7) Three-dimensional display of anchor area The anchor setting unit 11 is provided with a function of expanding the set anchor area in the x and y directions and the time t direction, which are the horizontal and vertical directions of the screen, and displaying it three-dimensionally. This can be considered that FIG. 4 is displayed on the screen as it is while editing the anchor. As a result of this display, the user can visually grasp the anchor as a whole.

As an application of this technique, the anchor information displayed three-dimensionally may be directly editable. For example, if the anchor area in the intermediate reference frame of FIG. 4 is moved to the left on the screen, the display as shown in FIG. 6 is made. The user can grasp the effect of editing in real time.

(8) Cross-sectional display of moving image In the anchor setting section 11, a horizontal cross-sectional view of the locus of the anchor region from the start frame to the end frame (FIG. 10).
(A)) and a vertical cross-sectional view (FIG. 10 (b)) are created, and the function of displaying this together with the trajectory of the anchor is added. First, the x and y coordinates of the center of gravity G of the anchor area are obtained from the anchor information of the reference frame. In the case of FIG. 10A, a straight line parallel to the x-axis is drawn on the frame from the center of gravity. A plane including these straight lines (hatched portion in the figure) is provided between adjacent reference frames. Next, the moving image is cut on this plane. The obtained cross-sectional view is projected on the xt plane (dotted area in the figure). In the case of FIG. 10B, x and y are exchanged and the same processing is performed. When the trajectory of the anchor is calculated correctly, that is, when the estimation accuracy of the anchor information in the non-reference frame is sufficiently high, the movement path of the anchor should appear in the two projection views. For example, if a red ball is to be set as an anchor, a red streak-like moving path appears in a section of the moving image. Splitting the pencil vertically is the same as making the lead appear linear. If this movement path is cut or thickened on the way, the position of the anchor at that location may be corrected.

(9) Grouping of anchors The anchor information editing unit 110 is provided with a function of grouping separately set anchor information and virtually handling it as one anchor. For example, the person A is in the frame N1
-N2 and frames N3 to N4 are present in the screen, and frames N2 to N3 are not present, frames N1 to N
2 and the anchor for the person A in the frames N3 to N4 are treated as one. As a result, the work of setting and correcting the anchor information is reduced. In addition,
The person A and the person B appearing in the same frame can be grouped.

(10) List Display of Anchor Information The anchor information editing unit 110 is provided with a function of displaying a list of anchor information set for the moving image currently being processed. For example, with the video title "AQUARIUM",
A list of anchor names such as “FISH1”, “FISH2”, etc. is displayed on the screen. When the user selects an anchor name whose contents are to be checked, the user may return to the start frame of the anchor and play back the moving image.

(11) Anchor search An anchor information search UI is provided. When character information such as the name of the anchor information to be searched is input as a keyword, the anchor search unit 15 searches the anchor information storage unit 21 for one having the keyword and displays it. Alternatively, the movement of the anchor area may be used as the search key. For example, when the user wants to find an object that moves to the right, the user presses the button "→" in the search UI. The anchor search unit 15 calculates the trajectory of each anchor region, searches for an anchor including an object that moves to the right, and displays the anchor.

(12) List display of anchor information display image In the anchor information editing unit 110, the frames (between the reference frame and the non-reference frame) included between the start frame and the end frame and the anchor setting unit 11 are used. The set anchor information or anchor estimation unit 12
A function of creating an anchor information display image by superimposing the anchor information estimated by the above, and displaying a list in chronological order. For example, as shown in FIG. 11, first, a start frame 81 and an end frame 82 are arranged at both ends, and frames are selected at a frame interval Δt. Subsequently, the anchor information 80 is superimposed on each of these frames to create an anchor information display image, and these are displayed side by side from the earliest display time. With this configuration, the suitability of the anchor setting can be seen at a glance. Therefore, the position determining operation by the box 66 in FIG. 9 can be omitted, and the anchor can be easily corrected. It is not necessary to select frames at regular intervals.
For example, only the reference frame may be selected. Further, in the displayed anchor information display image, the anchor region may be directly editable by dragging it with the mouse.

(13) Correction of anchor information during reproduction of moving image As shown in FIG. 12, a moving image reproduction unit 118 for reproducing a moving image on a screen and an anchor for displaying anchor information relating to a frame being displayed are provided in the anchor information editing unit 110. When the anchor information correction operation is performed at one or a plurality of times during the reproduction of the moving image, the information display unit 120 specifies the frames that are being reproduced at the time when each anchor information correction operation is performed. If is a non-reference frame, an anchor information correction unit 119 is provided which is promoted to a reference frame and corrects the anchor information in those frames based on each anchor information correction operation. With this configuration, first, a moving image is displayed in the image display area 50 of FIG. At this time, simultaneously, the anchor information in the frame being displayed is displayed as the anchor area 60 by the anchor information display unit 120. When the user finds a frame in which the anchor region 60 is displaced from the target, the user clicks the center of the target in the moving image with the mouse. With this action, the anchor information correction unit 119 corrects the anchor information by specifying the frame displayed at the time of the click and generating anchor information centered on the clicked point. The size of the newly set anchor area may be the same as that of the original anchor area, for example. Hereinafter, this frame is treated as a reference frame. With this configuration,
Since the position of the target can be sequentially specified during the playback of the moving image, it is not necessary to check and correct the position later.

Embodiment 2. In the first embodiment, the anchor information is automatically calculated mainly by interpolation, and the correction is manually performed. In the present embodiment, an anchor is automatically set with a certain number of frames as reference frames in advance based on analysis of moving images, and the interpolation method of the first embodiment is used between these reference frames. In this case, since the frame corresponding to the intermediate reference frame of the first embodiment is present from the beginning, the labor of manual correction is reduced.

FIG. 13 is a configuration diagram of the anchor setting unit 11 of the moving image hypermedia apparatus according to the present embodiment. Configurations other than the anchor setting unit 11 are the same as those in FIG. FIG.
In, the automatic anchor setting unit 112 has a nearby frame extracting unit 117. The adjacent frame extraction unit 117 extracts non-reference frames at regular intervals between adjacent reference frames, and promotes them to reference frames. The automatic anchor setting unit 112 also includes the motion vector use setting unit 11.
3. It has a contour information use setting unit 114 and a pattern matching use setting unit 116. These three settings are:
Originally, it is sufficient to implement either one, but in the present embodiment, all are implemented, and one of them is selected according to the situation.

As will be described later, the reference frame deletion unit 115 returns a redundant one of the reference frames set by the automatic anchor setting unit 112 to a non-reference frame. Hereinafter, the operation by this configuration will be described.

[1] Automatic Setting of Anchor by Using Motion Vector The feature of this process is that the motion vector of the anchor is once obtained from the start frame to the end frame, and then the virtual movement path of the anchor and the motion vector match. It has a two-stage configuration for improving the tracking accuracy of the anchor by determining the degree.

1. Acquisition of motion vector The time of the start frame and the end frame are set to t0
Set to 1. In addition to the above, first, some of the non-reference frames are changed to reference frames by the adjacent frame extraction unit 117. Here, it is assumed that the time is simply changed every five frames.
Is normalized. In order to obtain the motion vector of a specific anchor in the period from the start frame to the end frame, the image area near the center of gravity of the anchor is used as a block,
Perform block matching. A frame corresponding to an arbitrary time t will be referred to as a frame (t).

FIG. 14 is a flow chart showing the procedure for obtaining a motion vector in this embodiment. As shown in the figure, first, the time counter t is set to t0 (S10).
0). Next, of the anchors set in the start frame, the anchor for which the motion vector is to be acquired is specified. The motion vector utilization setting unit 113 stores the area including the center of gravity of the specified anchor as a block used for block matching (hereinafter referred to as "anchor block") (S101). Subsequently, image data I (t) of frame (t) and I (t +
1) is acquired (S102). I (t) is aggregate data of the pixel value p of each pixel included in the frame.

Thereafter, the anchor block is set to the frame (t
While moving within +1), the optimum matching is searched (S103). Each pixel value of the anchor block itself is I
Since it becomes clear from (t), the anchor block is arranged at an arbitrary position in the frame (t + 1), and the square error of the pixel value is calculated between the overlapping pixels, and this is integrated over the entire anchor block. This integration is performed while the anchor block is being moved little by little, and the position where the integrated value is minimum is determined to be the destination of the anchor block.

When the movement destination is determined, the movement amount and the movement direction from the anchor block in the frame (t) to the anchor block in the frame (t + 1) are determined, and are obtained as a motion vector V (t) (S1).
04). Here, it is determined whether or not t + 1 has reached time t1 of the end frame (S105). If not, t is incremented (S106), and a motion vector is repeatedly acquired. If t + 1 is equal to t1, then
After saving the obtained V (t) (S107),
Finish the process.

FIG. 15 is a diagram showing an example of motion vectors V (0) to V (2) obtained when t0 = 0 and t1 = 3. As shown in the figure, V (t) is defined by x and y determined by the vertical and horizontal directions on the screen, and t determined by the time direction,
It can be expressed by three components of (x, y, t).

2. Determining the degree of coincidence Each frame is divided into blocks having the same size as the anchor block provided for the motion vector, and all routes that the anchor may have followed are found. FIG. 16 is a diagram showing one of such paths. In the figure, the frame is divided into 16 blocks, and the start point of the route in the start frame and the end point of the route in the end frame coincide with the anchor block in FIG. Under this condition, there are 16 × 16 total routes. Subsequently, a vector (hereinafter referred to as “route vector”) v (t) illustrated in FIG. 16 is defined in each section of this route (hereinafter referred to as “virtual route”). The route vector is determined by the direction of the virtual route from one frame to the next. v
(T) is also described by three components (x, y, t).

Here, in each section, V (t) and v
The angle formed by (t) is defined as θt, and f (t) = cosθt is calculated by the following equation using the inner product.

F (t) = (V (t), v (t)) / | V (t) | · | v (t) | (Equation 2) FIG. 17 corresponds to v (t) in FIG. FIG. 15 is a diagram in which V (t) is added and displayed, and shows the meaning of θt. Formula 2
Is larger, the degree of coincidence between the virtual path and the motion vector in that section is higher. However, even if the degree of coincidence is maximized in a certain section, if the degree of coincidence in another section is extremely low, the overall Need to be considered low. Therefore, the following evaluation formula is introduced in order to evaluate the degree of coincidence as a whole while considering the degree of coincidence in each section.

G (t) = max {f (t-1) + g (t-1)} (Equation 3) By recursively calculating Equation 3, the degree of coincidence was always the highest until that time. The virtual route is known. If this calculation is performed up to the end frame, the virtual route with the highest degree of coincidence is found throughout, so this route is regarded as the anchor movement route. Thereafter, the anchor is automatically set on the assumption that the anchor at that time exists at the intersection of the movement route and each reference frame. The set anchor information may be added to the anchor information table shown in FIG. For frames other than the reference frame, the anchor information may be sequentially calculated from the interpolation calculation by the same method as in the first embodiment.

[2] Automatic Setting of Anchor Using Contour Information As another method of automatic anchor setting, there is a method of tracking an anchor based on movement of a target contour. The contour information use setting unit 114 generates a contour image for each frame by repeating the same processing as in FIG. The contour image is a binarized image in which the contour line is 1 and the others are 0, and the image has a compass gradient (Compass-gradie).
It can be generated by applying an nt) type filter. After the contour image is obtained, the anchor may be tracked assuming that the anchor moves exactly the same as the target.

[3] Automatic Setting of Anchor Using Matching As another method of automatic setting of an anchor, there is tracking of an anchor by pattern matching shown in FIG. Also in this method, first, a certain number of reference frames are provided in advance by the adjacent frame extraction unit 117. Next, the anchor area 13 set in the start frame 130
Create a model 134 for pattern matching from 2,
A region 138 having the highest degree of coincidence among the adjacent reference frames 136 is obtained. As a method of pattern matching,
There are a template matching method in which model image data is superimposed as it is, and a structure matching method in which superimposition is performed based on the positional relationship of feature points extracted from an image. Pattern matching is performed mainly in the vicinity of the model.

In this way, the area 13 in the second reference frame
If 8 is found, the same process is repeated using this region 138 as a new model to trace the anchor region. In some cases, good matching cannot be achieved when the target moves or deforms significantly or when the reference frame setting interval is too wide. In such a case, the adjacent frame extracting unit 117 performs the matching process again after narrowing the interval between the reference frames to provide more reference frames.

[4] Deletion of Unnecessary Reference Frame In the above example, the reference frame is provided every 5 frames. However, for example, in the case where the target makes a linear motion at a constant velocity, only the start frame and the end frame are sufficient as the reference frames. Even if the target does not have a uniform linear motion from the start frame to the end frame, for the period of such motion,
Only the reference frames at both ends of the period are needed. As the number of reference frames decreases, the calculation load becomes lighter. In the case of [1], the number of virtual routes is drastically reduced, which is particularly effective.

From this viewpoint, the reference frame deleting unit 115
Deletes unnecessary reference frames. FIGS. 19 to 21 are diagrams showing a situation in which reference frames are being deleted. In these figures, the horizontal axis is time, and the vertical axis is x- provided on the frame.
The distance from the origin of the y coordinate is shown. In addition, the circles in each drawing schematically indicate the anchor area. The deletion is performed according to the following procedure.

[FIG. 19] Initially, six reference frames including both end reference frames are provided. Here, the anchor of the start frame and the anchor of the end frame are connected by a straight line, and the distance between the straight line and each anchor is calculated. If there is an anchor whose distance is equal to or less than a predetermined value, the reference frame at that time is deleted. In the figure, it is assumed that no anchor has been deleted. Next, the anchor farthest from the straight line (hereinafter referred to as the farthest anchor) is found. here,
The anchor at t = 3 is the farthest anchor. [FIG. 20] The straight line is deleted, the anchor of the start frame, the farthest anchor, and the anchor of the end frame are connected by a polygonal line in this order, and the distance between this polygonal line and each anchor is obtained again. The anchor at t = 4 whose distance has become equal to or less than the predetermined value is deleted. The farthest anchor is changed to t = 2 anchor. [FIG. 21] The polygonal line is corrected so as to pass through the new furthest anchor. Here, the reference frame at t = 1, in which the distance from the new polygonal line becomes smaller than the predetermined value, is deleted. This ends the process.

In this example, two reference frames have been deleted. When there are many initial reference frames,
1. 1. Delete reference frames that are less than or equal to a predetermined value. 2. Search for the farthest anchor; Repeat the correction of the broken line.

The above is the outline of the present embodiment. In addition,
The present embodiment can be improved or modified as follows. (1) Modification of Expression 2 In Expression 2, f (t) = cos θt is adopted, but naturally this may be another function. The function of increasing and decreasing θt is f
(T) can be a candidate. (2) Block method In FIG. 15, the anchor block is determined so as to include the vicinity of the center of gravity of the anchor area. However, another method may be used.
For example, the anchor area may be directly used as the anchor block. Similarly, also in FIG. 16, the block may be determined regardless of the size of the anchor block.

(3) Another Method of [3] (Part 1) Another mode of automatic anchor setting by the above matching will be described. When matching is performed one after another using a certain frame region as a model, errors may accumulate and the target may gradually deviate from the target. For this reason, the determination is made in consideration of not only the matching from the adjacent frame but also the matching result from a frame that is distant to some extent.

As shown in FIG. 22, here, a start frame and an end frame are adopted as reference frames separated by time. Now, it is assumed that the new reference frame 404 for specifying the anchor position is at time t + Δt. On the other hand, the model A400 of the anchor area of the start frame, the model B401 of the anchor area of the end frame, and the model C403 of the anchor area of the reference frame 402 at time t are all known. Therefore, matching is performed between these three reference frames and the reference frame 404 at time t + Δt. At this time, if all matching results match, the tracking result of the area is reliable. On the other hand, if the matching results do not match, the anchor position is determined by the following method, for example. 1. The three regions obtained on the reference frame 404 at the time t + Δt as a result of the matching three times are superimposed to create an anchor region having the same size as the original region, centered on the center of the overlapping portion.

2. One model, model C403
, A plurality of matching results are obtained, and it is determined in order from the top of the matching results whether or not an area obtained from the matching results of the model A400 and the model B401 is included in the area. If the area is included at a certain ratio or more, an anchor area having the same size as the original area is created around the center of the overlapping portion between the areas obtained from the matching result. Here, the start frame and the end frame are considered in addition to the adjacent reference frame, but the combination has a degree of freedom. For example, an adjacent reference frame and any number of reference frames separated by a certain time distance may be employed.

(4) Alternative method of the above [3] (No. 2) As shown in FIG. 23, the automatic anchor reliability setting unit 112 is provided with an automatic reliability setting determination unit 130 and an automatic reliability display unit 131. The automatic setting reliability determination unit 130 determines the reliability of tracking based on the degree of overlap between the anchor area obtained by performing matching in the forward direction up to the last frame and the anchor area specified in the end frame. For example, if the area of the overlapping portion is 70% or more, the tracking result is reliable.
If it is 0% or less, it is determined that it is not reliable. The automatic setting reliability display unit 131 displays the reliability of tracking (whether or not the reliability can be obtained, or% thereof).

For example, as shown in FIG. 24, the tracking result 1 by the matching with respect to the original trajectory 140 of the target 1
When 41 is warped, the tracking is determined to be unreliable due to the low degree of overlap. Therefore, this time, using the anchor region of the end frame as a model, image matching is performed in the reverse direction, and the anchor position is re-tracked. At this time, in each reference frame, the anchor area previously set by the forward matching is compared with the anchor area obtained by the current backward matching, and if both overlap by a predetermined ratio or more, the tracking is terminated. I do. At this time, as another method, tracking may be terminated at a designated frame. The frame for which tracking should be ended may be determined based on the ratio of the distance from the start frame and the end frame. The method of determining duplication and the method of designating a frame for which tracking should be terminated may be used together.

As described above, according to this method, even if an error occurs during the tracking, it is possible to avoid deterioration of the subsequent tracking result, and the final correction work is reduced. Also,
When the matching in the reverse direction is terminated by checking the degree of overlap, only a necessary part is corrected, which leads to a reduction in processing time.

Even if the reliability is low, matching may be merely displayed instead of automatically starting the matching in the reverse direction. In this case, the user may cause the matching in the reverse direction to be performed, or may make a desired correction by himself. In any case, the user can know whether the tracking is good or bad by the automatic setting reliability display unit 131, and appropriate measures can be taken.

(5) Alternative method of [3] above (Part 3) The respective components of FIG. 12, that is, the moving image reproducing section 118, the anchor information display section 120, and the anchor information correcting section 119 are provided in the automatic anchor setting section 112. . Here, the moving image playback unit 118 chronologically arranges each frame constituting the moving image,
Display at appropriate time intervals. Anchor information correction unit 119
Corrects the anchor information of the frame displayed at that time when the anchor information correction operation is performed at any time during the reproduction of the moving image. At the same time, the automatic anchor setting result of the frame displayed for the predetermined number of frames or the predetermined period immediately before the frame is invalidated.

The operation of this configuration will be described. Here, it is assumed that the moving image is displayed while matching is sequentially performed in the forward direction from the start frame in accordance with the reproduction of the moving image. In the case of this operation, as described above, once the matching becomes unsatisfactory, the anchor information may gradually deviate thereafter. The user clicks the screen when the anchor area deviates from the target while viewing the moving image to be reproduced and the anchor information. At this point, playback stops. Here, for example, when the user clicks the center of the target, the anchor position is corrected so that the point becomes the center. Subsequent matching is performed on the basis of the corrected anchor information, so that it is good.

In this method, since the user clicks the screen during reproduction of the moving image, it is necessary to consider a delay in the action. That is, when the user clicks after recognizing that the anchor region has deviated from the target, it is considered that such a phenomenon has gradually progressed over several frames. Therefore, the anchor information correction unit 119 invalidates the anchor information that is automatically set for a plurality of predetermined frames existing before the frame in which the anchor information is corrected.

FIG. 25 is a diagram showing the relationship between the original trajectory of the target and the tracking result obtained by matching, and the operation of the anchor information correction section 119. The solid line 150 in FIG. 7 indicates that the trajectory of the target matches the tracking result, and the broken line 151 indicates that the tracking result deviates from the trajectory. As shown in the figure, the tracking results are reliable at times t0 to t1, but start off at time t1. The user notices this and clicks the screen at time t2. As a result, correct tracking is performed again between times t2 and t3.
At times t1 and t2, an incorrect tracking result remains, so this is invalidated. As for the invalidated portion, as shown in the figure, the missing tracking result can be supplemented by linearly interpolating the anchor information at the times t1 and t2. Note that the number of frames for which the anchor information is invalidated may be specified in advance, or may be specified at the time of correction.

Third Embodiment The following devices or systems can be constructed by applying the moving image hypermedia devices of the first and second embodiments.

1. Interactive Video Teaching Material Production Apparatus The moving image hypermedia apparatus according to the present invention is also optimal for producing CAI content. That is, an anchor is set in the video teaching material by this device, and necessary additional information is linked. FIG. 26 is a diagram showing a video teaching material produced by this device. As shown in the figure, an explanation A is linked to the anchor A, an explanation B is linked to the anchor B, and so on. While playing the video, the student clicks on the screen the object for which he wants a more detailed description. If the clicked object is associated with anchor A, then explanation A is displayed on the screen.

[0130] 2. Interactive Video Server System The video hypermedia device according to the present invention is also suitable for a video server system. FIG. 27 is a configuration diagram of this video server system. As shown in the figure, this system is composed of a server 200 and a client 250 which almost share the configuration of FIG.

The server 200 includes a data storage unit 204 for storing moving images and their anchor information and related data linked to the anchors, an anchor estimating unit 206, and a hyperlink search unit for searching for related data linked to any anchor. 208. Anchor estimation section 206 estimates anchor information in a non-reference frame.

On the other hand, the client 250 includes an anchor determination unit 252 for determining which anchor area has been clicked when the user clicked on a target in the moving image.

In this configuration, when the user clicks an object on the screen with the client 250,
The anchor determining unit 252 specifies the clicked anchor. This information is sent to server 200. Server 20
0 hyperlink search unit 208 is the data storage unit 204
From the relevant data linked to that anchor,
This is sent to the client 250.

As described above, according to this system, moving image data and anchor information are collectively stored on the server 200 side, and a large number of users can see the required moving image and the information linked thereto.

[0135]

According to the moving picture anchor setting device of the present invention, the anchor information of a non-reference frame, that is, a frame other than the reference frame can be estimated by setting the anchor information only for the reference frame. It is not necessary to set anchor information for non-reference frames. As a result, the anchor setting work can be saved.

When a plurality of reference frames are taken, if the frame determination means determines the earliest one as the start frame and the latest one as the end frame, the frame setting means sets the period in which the user should set the anchor. Work efficiency can be improved without having to be aware of it.

When the frame determining means detects the frame in which the target appears and the frame in which the target has disappeared and determines these as the start frame and the end frame, respectively, it can be said that there is at least one reference frame, and further labor saving is promoted. be able to.

When the anchor estimating means estimates the anchor information in the non-reference frame by interpolating the anchor information, the anchor information of the non-reference frame can be obtained with a light calculation load.

When the anchor estimating means divides the change amount of the correspondence information between the anchors set in the reference frame according to the elapsed time therebetween, the anchor information of the non-reference frame is obtained by a very simple interpolation calculation. be able to.

When the anchor estimation means performs the interpolation calculation of the anchor information between the reference frames at both ends of each section of the setting target period, the anchors with respect to the non-reference frames are estimated by using the reference frames located close to each other. Therefore, the accuracy of estimation can be improved.

When the estimated anchor information includes an anchor information editing means for making a correction, the correction results in the interpolation calculation using the intermediate reference frame having the correct anchor information. The accuracy of can be improved.

When the anchor information editing means displays the trajectory of the anchor three-dimensionally based on the estimation of the anchor information, the edited result can be confirmed on the spot.

When the anchor information editing means calculates the trajectory of the anchor and displays the cross-sectional view of the moving image along this trajectory together with the trajectory of the anchor, the anchor estimated in the non-reference frame is referred to by referring to this cross-sectional view. You can check the accuracy of the information and correct it if necessary.

When the anchor information editing means displays the anchor information display images in time series, it is possible to grasp at a glance whether or not the anchor setting is appropriate. Therefore, the confirmation work of the anchor information becomes easy.

When the anchor information editing means includes the moving picture reproducing means and the anchor information correcting means, it is possible to specify an appropriate position of the target while reproducing the moving picture, so that desirable anchor information is set at the end of the reproduction. As a result, it is possible to reduce the labor of the confirmation and editing work performed later.

When the anchor estimation means includes the anchor automatic setting means, the number of reference frames in which the anchor information should be initially set may be small, so that the setting work can be reduced.

When the automatic anchor setting means tracks the anchor based on the movement of the contour of the target, the accuracy of the anchor tracking can be improved.

The same applies when the anchor automatic setting means tracks the anchor based on the motion vector of the target. In this case, depending on the type of moving image, a motion vector may be described for each frame during encoding, which is convenient.

The automatic anchor setting means determines, for each route that the anchor can follow in the setting target period, the degree of coincidence between the route and the motion vector of the anchor to search the moving route of the anchor. Can be reduced. That is, although the motion vector should include an error for each unit determination period, the path determination is performed over the entire unit determination period, so that the accuracy of the finally obtained path increases.

When the degree of coincidence is determined by the function of the angle formed by the path and the motion vector, the degree of coincidence can be evaluated numerically, and a highly accurate path can be found by a relatively simple calculation.

When the automatic anchor setting means includes the automatic setting reliability determination means and the automatic setting reliability display means, the user can know the quality of the automatic setting and can take appropriate measures.

When the automatic anchor setting means performs image matching with a new reference frame using the image of the anchor area of the reference frame for which anchor information has already been set as a model, the accuracy of anchor tracking can be improved.

When a similar image is not found in the new reference frame and the automatic anchor setting means further divides the adjacent reference frames to provide a new reference frame and re-executes the image matching, Matching accuracy is improved. In addition, since a new reference frame is provided only at a necessary position, tracking accuracy can be improved while suppressing an increase in calculation load.

In the case where the automatic anchor setting means tracks the position of the anchor based on the matching result between the adjacent reference frame and the reference frame which is separated by a certain time distance, for example, it is possible to improve the situation where the matching result is gradually shifted. This makes it easier to fix anchors.

In the case where the automatic anchor setting means includes the automatic setting reliability judgment means, even if the tracking is deviated from the middle, the anchor information of the deviated part can be corrected.

When re-tracking by the automatic setting reliability determination means, if the tracking is terminated by checking the degree of overlap of the anchor area obtained by the original tracking and the current tracking, re-tracking for an unnecessarily long time is required. It does not need to be done and processing time can be shortened.

When a frame for which re-tracking should be terminated is specified, a suitable result can be obtained by simple processing.

When the anchor information correcting means invalidates the result of automatic anchor setting of a predetermined frame preceding the frame whose anchor information has been corrected, correction can be made when the tracking result becomes worse, and thus it will be confirmed again later. No need. In addition, since there is no need to make corrections while the matching is good, the confirmation work can be saved.

When the anchor setting means includes the reference frame deleting means, the number of reference frames for holding the anchor information is reduced, and the required storage capacity can be reduced. When the anchor setting means includes a text anchor setting means, the anchor can be set not only in the image data but also in the text data.

When the cursor changing means is further included, the presence and position of the desired anchor area can be easily known even if the anchor area is set invisible.

When the display state of the cursor is changed so as to accompany the display of the identification information of the anchor area where the cursor is inserted, it is possible to know a certain amount of related information without a click operation.

When the anchor setting means sets the anchor information in units of a group consisting of a plurality of anchors, for example, the labor of setting the anchor information for the same target many times can be omitted.

If an anchor search means for searching for an anchor based on the feature of the movement of the anchor is included, the search effort is greatly increased when it is desired to know only one of a large number of objects that moves to the right. Can be reduced.

When an anchor search means for searching for an anchor based on the identification information of an anchor is included, search efficiency can be improved.

On the other hand, according to the moving image hypermedia apparatus of the present invention, it is possible to link data necessary for an anchor and to search for linked data while taking advantage of the moving image anchor setting means of the present invention.

When this moving image hypermedia device is used for producing an interactive video teaching material, for example, an interactive video teaching material in which additional information can be obtained by clicking a corresponding object on a screen can be produced. .

On the other hand, according to the moving picture providing system of the present invention, a necessary moving picture and information linked thereto can be seen from many users.

According to the moving image anchor display device of the present invention,
While playing the video, the set anchor information is
You can check it in real time.

According to the moving picture anchor selection device of the present invention,
When you click a required part while playing the video,
Know the anchor information associated with the location
It is convenient for editing.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a system including a moving image hypermedia device according to a first embodiment.

FIG. 2 is a flowchart showing a procedure for setting and correcting an anchor according to the first embodiment.

FIG. 3 is a diagram showing a table of anchor information set in the first embodiment.

FIG. 4 is a diagram showing an interpolation calculation method of anchor information according to the first embodiment.

FIG. 5 is a diagram showing a table obtained by adding anchor information of an intermediate reference frame to FIG. 3;

FIG. 6 is a diagram illustrating a method of performing an interpolation calculation based on three frames of an intermediate reference frame and both-end reference frames in the first embodiment.

FIG. 7 is a flowchart showing a link setting and search procedure according to the first embodiment.

FIG. 8 is a diagram showing a table of link information set in the first embodiment.

FIG. 9 is a diagram illustrating an example of a UI screen for anchor setting.

FIG. 10A is a lateral cross-sectional view of the locus of the anchor region from the start frame to the end frame, FIG.
Similarly, 0 (b) is a vertical cross-sectional view.

FIG. 11 is a diagram showing an anchor information display image displayed in chronological order by the anchor information editing unit in the first embodiment.

FIG. 12 is a diagram showing an internal configuration example of an anchor information editing unit of the moving picture hypermedia device according to the first embodiment.

FIG. 13 is a configuration diagram of an anchor setting unit of the moving image hypermedia device according to the second embodiment.

FIG. 14 is a flowchart showing a procedure for acquiring a motion vector according to the second embodiment.

FIG. 15 is a diagram illustrating an example of motion vectors V (0) to V (2) obtained when t0 = 0 and t1 = 3 in FIG. 14;

FIG. 16 is a diagram showing one of paths that an anchor may have taken.

FIG. 17 shows V (t) shown in FIG.
It is the figure which added and displayed (t).

FIG. 18 is a diagram showing a method for automatically setting an anchor based on pattern matching in the second embodiment.

FIG. 19 is a diagram showing how reference frames are deleted in the second embodiment.

FIG. 20 is a diagram showing how reference frames are deleted in the second embodiment.

FIG. 21 is a diagram illustrating a state where reference frames are deleted in the second embodiment.

FIG. 22 is a diagram illustrating a method of performing matching using a reference frame that is separated by a certain time distance in addition to an adjacent reference frame in the second embodiment.

FIG. 23 is a diagram illustrating a configuration example of an automatic anchor setting unit of the moving image hypermedia device according to the second embodiment.

24 is a diagram illustrating an erroneous tracking result that may occur without the configuration in order to explain the effect of the configuration in FIG. 23;

FIG. 25 is a diagram showing the relationship between the original trajectory of the target and the tracking result obtained by matching, and the operation of the anchor information correction unit.

FIG. 26 is a diagram showing a configuration of an interactive video teaching material production device according to a third embodiment.

FIG. 27 is a diagram showing a configuration of an interactive video server system according to a third embodiment.

[Description of Signs] 1 data operation unit, 2 data storage unit, 3 display unit, 4
User operation unit, 6 moving image input unit, 10 frame determination unit, 11 anchor setting unit, 12 anchor estimation unit, 1
3 hyperlink setting part, 14 hyperlink search part, 15 anchor search part, 20 moving image data storage part,
21 anchor information storage unit, 22 link information storage unit,
Reference Signs List 30 display control unit, 31 display device, 50 image display area, 52 button group, 54 rectangular button, 56 start frame designation button, 58 end frame designation button, 60
Anchor area, 62 Anchor-related box group, 64
Scene-related boxes, 66, 68 boxes, 80
Anchor information, 81 start frame, 82 end frame, 110 anchor information editing part, 111 text anchor setting part, 112 automatic anchor setting part, 113
Motion vector use setting unit, 114 contour information use setting unit, 115 reference frame deletion unit, 116 pattern matching use setting unit, 117 adjacent frame extraction unit, 11
8 moving image display section, 119 anchor information correction section, 120
Anchor information display unit, 130 automatic setting reliability determination unit,
131 automatic setting reliability indicator, 140 original trajectory of target, 141 tracking result by matching, 20
0 server, 204 data storage unit, 206 anchor estimation unit, 208 hyperlink search unit, 250 client, 252 anchor determination unit, 300 cursor change unit, 400 model A, 401 model B, 402 reference frame at time t, 403 model C , 404 Reference frame at time t + Δt. ────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] February 9, 1998

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Full text

[Correction method] Change

[Correction contents]

[Document Name] Statement

Patent application title: Video anchor display / selection / setting device and video hypermedia device and video providing system using video anchor

[Claims]

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image anchor setting device and a moving image hypermedia device using the device. The present invention particularly provides a device for inputting a moving image and setting an anchor to a target or the like included in the moving image,
A moving image hypermedia device for linking relevant information to an anchor set by the device and performing a desired search, etc.
The present invention relates to a moving image providing system using this device as a server client system.

[0002]

2. Description of the Related Art Conventionally, information retrieval in a general hypermedia device mainly sets a logical unit for an information link to a text or a still image, and links related information to this logical unit in advance. , The related information is displayed when the user clicks the logical unit. However, various technologies have been proposed in recent years for processing not only still images but also moving images, as represented by MPEG for encoding and decoding of moving images. By handling moving images, the hypermedia device can be used for content creation such as CAI, various presentations, and electronic catalogs. The editing of moving images has been used in some limited industrial fields such as broadcasting stations in the past, but it is expected that it will be rapidly spread as a personal system based on a personal computer in the future.

Japanese Patent Laid-Open No. 4-163589 discloses an image processing apparatus capable of setting a logical unit (called a node in the specification) for a moving image. This device is capable of setting a node in a still image simply by specifying a display range. On the other hand, in the case of a moving image, the validity period of the node is determined from both (1) display range and (2) time. By paying attention to the point that the area range should be designated, these designations are made possible. That is, with respect to (1), the node at that time is set by pointing the region surrounding the subject appearing in the moving image with a mouse or the like, while with respect to (2),
The valid duration of the node is designated by the elapsed time from the output start time of the moving image. Therefore, this node is uniquely determined by the two contents of the area and the elapsed time, and related information can be linked to each node. After the link, when actually reproducing the moving image, if the user clicks a certain area on the screen with a mouse or the like, a node is specified by the position and time, and related information is displayed.

[0004]

In the above-mentioned device, the area setting for each logical unit is manually performed. However, naturally, unlike a still image, a moving image has a large number of frames, and the position and shape of the subject change from moment to moment.
In the case of the NTSC system, 30 frames are required for one second, so even if it is simply calculated, 30 processing operations per logical unit are required when processing a moving image of one second. For example, when creating five minutes of content, if five logical units are set in one frame, the number of times of setting is 45,000.

The present invention has been made in view of the above problem, and an object of the present invention is to provide a device which enables labor saving and simplification of a setting operation of a logical unit (referred to as an anchor in this specification). An object of the present invention is to provide an anchor setting device that automatically calculates or automatically sets anchor information that needs to be performed for each frame. Another object of the present invention is to provide a moving image hypermedia device and a moving image providing system using the anchor setting device.

[0006]

A moving picture anchor setting apparatus of the present invention is an apparatus for inputting a moving picture and setting an anchor on the moving picture. This apparatus, when an area is specified for a reference frame in a plurality of frames constituting the input moving image, an anchor setting means for setting anchor information with this as an anchor area for the reference frame, An anchor estimation means for estimating anchor information in a non-reference frame based on anchor information set for the reference frame.

[0007] In certain embodiments of the invention, the frame determination means, start frame those temporally earliest among the criteria frame, decide to end frame the slowest.

In another aspect of the present invention, the frame determining means detects a frame in which a target included in the reference frame appears and disappears before and after the reference frame, and determines these as a start frame and an end frame, respectively. Decide

In one aspect of the present invention, when there are a plurality of reference frames, the anchor estimating means estimates the anchor information in a non-reference frame by interpolating the anchor information set for the reference frames.

At this time, in one aspect of the present invention, the anchor estimating means performs interpolation in the following procedure. First, the amount of change in correspondence information between anchors set for two adjacent reference frames is calculated, and then the elapsed time from one reference frame to a certain non-reference frame and the other reference Calculate the ratio of elapsed time to the frame. Thereafter, the change amount is divided by this ratio.

In one aspect of the present invention, the anchor estimating means performs an interpolation calculation of the anchor information between the reference frames located at both ends of the section in the setting target period divided by the reference frame. The anchor information in the non-reference frame included in the section is estimated.

One aspect of the present invention includes anchor information editing means for correcting the anchor information estimated for a non-reference frame. At this time, the anchor estimating means:
The non-reference frame whose anchor information has been modified is promoted to a reference frame.

In one aspect of the present invention, the anchor information editing means three-dimensionally displays the locus of the anchor based on the estimation of the anchor information.

In one aspect of the present invention, the anchor information editing means calculates the trajectory of the anchor based on the estimation of the anchor information, and displays a cross-sectional view of the moving image along the trajectory together with the trajectory of the anchor.

In one aspect of the present invention, the anchor information editing means selects a frame included between a start frame and an end frame, and superimposes anchor information on the image of the frame to create an anchor information display image. After that, the anchor information display images are displayed in time series.

In one aspect of the present invention, the anchor information editing means includes a moving image reproducing means and an anchor information correcting means. The moving image reproducing means reproduces the moving image on the screen, and when the anchor information correcting operation is performed during the moving image reproduction, the anchor information correcting means specifies the frame reproduced at the time when the operation is performed, If the frame is a non-reference frame, the frame is promoted to the reference frame and the anchor information in the frame is corrected based on the operation.

In one aspect of the present invention, the anchor estimation means includes an anchor automatic setting means. This automatic anchor setting means selects an arbitrary number of frames from adjacent reference frames, uses them as new reference frames , estimates the target position, and then sets the anchor information for these new reference frames. Set.

In one embodiment of the present invention, the automatic anchor setting means uses the movement of the outline of the target as a clue .
Estimate the position of the get .

[0019] In certain embodiments of the invention, the anchor automatic setting means, position of the target motion-out vector clue
Estimate the location .

In one aspect of the present invention, when each section of the setting target period divided by the reference frame is defined as a unit determination period, the automatic anchor setting means sets each of the paths that the anchor can follow in the setting target period. Then, the degree of coincidence between the path and the motion vector of the anchor is determined for each unit determination period. This judgment result is integrated over the entire set target period. Finally, the route having the highest degree of coincidence in the overall result is regarded as the movement route of the anchor.

In one aspect of the present invention, the degree of coincidence is determined by a function of an angle formed by the path and the motion vector, the function having no extreme value in the range of 0 ° to 180 °, The sum of the values of the function for each unit determination period is calculated to obtain the total result. The function value gradually increases or decreases as the angle changes from 0 ° to 180 °. Therefore, the degree of coincidence is evaluated by the simple sum of these values.

In one aspect of the present invention, the automatic anchor setting means includes automatic setting reliability determination means and automatic setting reliability display means. The automatic setting reliability determination unit determines the reliability of the automatic setting by comparing the anchor information set by the anchor setting unit with the anchor information set by the anchor automatic setting unit. The automatic setting reliability display means displays the reliability.

In one aspect of the present invention, the automatic anchor setting means performs image matching with a new reference frame by using the image of the anchor area of the reference frame in which the anchor information is already set as a model ,
Estimate the position of the get .

In one aspect of the present invention, when a similar image is not found in the new reference frame when performing image matching, the automatic anchor setting means further divides the adjacent reference frames to create a new image. A reference frame is provided, the interval between the reference frames is narrowed, and image matching is performed again.

In one aspect of the present invention, the automatic anchor setting means uses, as the reference frame in which the anchor information has already been set, a reference frame close to the new reference frame and a reference frame separated by a certain time distance. Select at least two. After that, based on the matching result using the selected reference frame, the target
The position of the dot is estimated .

In one aspect of the present invention, the automatic anchor setting means compares the anchor information set by the anchor setting means with the anchor information set by the automatic anchor setting means to determine the reliability of the automatic setting. An automatic setting reliability determination unit for determining is included. If the reliability determined by the automatic setting reliability determination means is low after the anchor information is set by the tracking starting from the first reference frame, the anchor information is determined by the tracking starting from the second reference frame. Is to be reset.

At this time, while tracking is performed with the second reference frame as a starting point, an anchor area set in a certain reference frame and tracking with the first reference frame as a starting point are set in the reference frame. If the degree of overlap with the created anchor region is equal to or higher than a predetermined ratio, the tracking with the second reference frame as a starting point is terminated.

Alternatively, in the tracking using the second reference frame as a starting point, the tracking is stopped when the reference frame specified in advance is reached.

In one embodiment of the present invention, the automatic anchor setting means includes a moving picture reproducing means, an anchor information displaying means, and an anchor information correcting means. Video playback means,
Play the video on the screen. The anchor information display means displays the anchor information obtained as a result of the image matching sequentially performed on the displayed frame. When an anchor information correction operation is performed during moving image reproduction, the anchor information correction means corrects the anchor information of the frame on which the operation is performed, and invalidates the automatic anchor setting result in a predetermined frame before that frame. Become

In one aspect of the present invention, the anchor setting means includes reference frame deleting means. The reference frame deletion unit demotes the reference frame to a non-reference frame when the anchor information in a certain reference frame is obtained within a predetermined error tolerance from the interpolation calculation of the anchor information in another reference frame.

In one embodiment of the present invention, the anchor setting means includes a text anchor setting means. The text anchor setting means, when a text is designated for a reference frame, sets an anchor having the text as the content for the reference frame.

In one embodiment of the present invention, when the cursor enters an arbitrary anchor area, the apparatus further includes a cursor changing means for changing a display state of the cursor.

At this time, in one aspect of the present invention, the display state of the cursor is changed so that the cursor changing means accompanies the display of the identification information of the anchor area in which the cursor is inserted.

In one aspect of the present invention, the anchor setting means sets the anchor information in units of a group consisting of a plurality of anchors.

An aspect of the present invention further includes anchor search means for searching for an anchor based on characteristics of the movement of the anchor in the anchor information.

An aspect of the present invention further includes anchor search means for searching for an anchor based on the identification of the anchor.

On the other hand, the moving image hypermedia apparatus according to the present invention is arranged such that, when an area is designated for a reference frame in a plurality of frames constituting an input moving image, the anchor is used as an anchor area. An anchor setting means for setting information for the reference frame and an anchor estimation means for estimating anchor information in a non-reference frame based on the set anchor information are included. The above corresponds to the moving image anchor setting device. The apparatus further includes a link setting means for linking the set or estimated anchor information with any related data, and a link search means for searching the related data linked to the anchor from any anchor.

In one embodiment of the present invention, the moving image processed by the moving image hypermedia device constitutes a video teaching material, and the related data is additional information associated with a desired anchor of the video teaching material in a desired setting target period. , The device is used to produce interactive video materials.

On the other hand, the moving image providing system of the present invention comprises a server and a client. Server, moving picture and the anchor information set for the reference frame, A linker over function means for storing link information for linking the communication data, based on the anchor information set in the reference frame non-reference A means for estimating anchor information in a frame, and an anchor
And means for retrieving related data linked to the anchor when is selected . The client also
When the user clicks on the target in the moving image, a unit for determining which anchor area is clicked is provided.

[0040] Video anchor display device of the present invention, the anchor in non-reference frame and means for storing anchor information set in <br/> reference frame of the moving image, set the reference frame has been anchor information based on Means for estimating information;
Set or estimated by relating to the frame in the dynamic image reproduction screen
Means for displaying the specified anchor information in a superimposed manner.

Further, the moving picture anchor selection device of the present invention is
Means for storing anchor information set in the reference frame of the moving image, means for estimating the anchor information in the non-reference frame based on anchor information set in the reference frame, when the video play screen is clicked , The frame
Based on anchor information set or estimated for the
And a means for identifying which anchor has been selected .

[0042]

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of the moving picture hypermedia device of the present invention will be described below. This device incorporates the moving image anchor setting device of the present invention. According to the present apparatus, for example, an interactive CAI software capable of displaying a name of a fish, a supplementary explanation, and the like when a user clicks on any swimming fish using a moving image of an aquarium aquarium as a material is easily provided. And it can be created efficiently. In the following embodiments, the “user” mainly refers to the creator of such content, but it goes without saying that this may be a person who personally edits a video tape shot by himself / herself.

Embodiment 1. In this embodiment, the user explicitly specifies the start frame and the end frame, and sets the anchor using the two frames, which are the first frame and the second frame, as the first reference frames. A "frame" is a display unit of an image, and is an MPEG
And the like. This apparatus automatically calculates anchor information such as the position, shape, and color of the anchor region in another frame by interpolation calculation from the anchor set for the reference frame. The “anchor information” refers to, for example, the position and shape of the anchor area, the color of the anchor area when it is explicitly displayed, and the like. The present invention can be realized even if the first reference frame is one, but an example thereof will be described later.

The configuration of the entire system including the moving image hypermedia device of the present embodiment is a personal computer (hereinafter referred to as P
C) and a video playback device that provides moving images to this PC. The PC has a built-in video capture board that captures and digitizes the image provided by the video playback device. The video reproducing apparatus has a function of performing reproduction from a designated frame or time in addition to a normal reproduction start, stop, fast forward, frame advance, and the like. Although such a video reproducing apparatus is widely used in broadcasting business and the like, it is not necessarily limited to this. Control of various functions of the video playback device is controlled by a user interface (hereinafter,
UI), for example, via a "play button" displayed on the screen. In this case, when the user clicks the button, the action is transmitted from the PC to the video playback device via the signal cable. Although the video playback device itself is not an essential component of the moving image hypermedia device, it will be described here as a system including this. FIG. 1 is a configuration diagram of a system including a moving image hypermedia device according to the present embodiment.

This apparatus is roughly divided into a data operation unit 1 for operating data relating to anchor information and link information, a data storage unit 2 for storing these data, a display unit 3 for displaying these data in a meaningful form, It comprises a user operation unit 4 for receiving and managing user operations, and a moving image input unit 6 for inputting a moving image reproduced by the video reproducing device 5.

(1) Data operation unit 1 The user gives an instruction to the data operation unit 1 through a UI described later. That is, the following internal configuration is a software module.

The frame determination unit 10 determines a start frame and an end frame. In the present embodiment, the frame designated by the user becomes the start frame and the end frame as they are. Examples of the start frame and the end frame are the start and end frames of the scene showing the aquarium in the video of the aquarium described above. If the scene moves to the video of the entrance to the aquarium, it is not necessary to set an anchor for the fish thereafter, so the end frame is specified before the transition of the scene.

The anchor setting section 11 actually sets an anchor between the start frame and the end frame. For example,
When an anchor is set for a certain fish, first, a rectangle surrounding the fish is displayed with a mouse in the start frame, and this is registered as an anchor area. At this time, the moving image is in the stop mode. Then, the moving image is advanced to the end frame, and the anchor area is registered by surrounding the same fish again. Since the fish moves or changes direction between the start frame and the end frame, its position and shape usually change.
In general, the shape and position of the anchor area registered in the start frame do not match the shape and position of the anchor area registered in the end frame. Note that the anchor setting unit 11 includes an anchor information editing unit 11 that is used in the later-described anchor correction.
0, and a text anchor setting unit 111 that sets an anchor for text such as a character string.

The anchor estimating unit 12 performs an interpolation calculation based on the first and second anchor information set for the start frame and the end frame, and calculates the position and size of the anchor in an arbitrary frame (non-reference frame). To estimate.
This process will be described in detail later.

The anchor search unit 15 searches for an anchor based on the feature of the motion of the anchor or the identification information of the anchor in the anchor information. The identification information is information that serves as a clue to distinguish the anchor from other anchors, and includes, for example, the name of the anchor, an anchor setting target, and an anchor setting date and time.

The hyperlink setting unit 13 sets a hyperlink for the set anchor, and creates a data structure related to the setting in the form of a table. The hyperlink search unit 14 searches the set link information.
In the case of the above example, a fish anchor and text data indicating the name of the fish are associated by hyperlinks.

(2) Data Storage Unit 2 The data storage unit 2 may be a database, various file devices, or memory devices. This part is mainly hardware.

The moving picture data storage section 20 stores the moving picture data captured by the moving picture input section and digitized. The anchor information storage unit 21 and the link information storage unit 22 store the set anchor information and link information, respectively.

(3) Display Unit 3 The display control unit 30 is a display system program that comprehensively controls the display of various images such as a UI and a moving image being edited.
It includes a display circuit such as a VGA controller and this driver. The display control unit 30 is a cursor changing unit 300.
have. The cursor changing unit 300 changes the display state of the cursor when the cursor enters the anchor area. The output data of the display control unit 30 is given to a display device 31 such as a monitor of a PC, and a desired display is performed.

(4) User operation unit 4 This enables a user to input commands, and is composed of hardware such as a keyboard, a mouse and various pointing devices, and a command dispatcher. Examples of commands include setting an anchor, modifying an anchor area, a link, and a link search.

(5) Video input section 6 A hardware equivalent to a video capture board
It has a D converter and a frame memory (not shown), and digitizes an input moving image. After that, the data is provided to the moving image data storage unit 20.

Based on the above configuration, the procedure for setting the anchor and the link will be described first, and the state of the UI for setting the anchor will be described later.

[1] Anchor Setting FIG. 2 is a flowchart showing an anchor setting and correction procedure according to the present embodiment, and FIG. 3 is a diagram showing a table of set anchor information. As shown in FIG. 2, first, various initialization processes for hardware and the like are performed (S21).
The moving image data stored in the moving image data storage unit 20 is read (S22). The top frame of the read moving image data is first displayed as a still image on the display device 31. Next, the anchor information already set for the moving image data is read from the anchor information storage unit 21 (S23). If the anchor information exists, the anchor area is actually displayed on the screen (hereinafter, the mode in which the anchor area is displayed on the screen is “anchor display mode”, and the mode in which the anchor area is not displayed is “anchor non-display mode”
That).

Next, the moving image data is advanced to the start frame of the period in which the anchor is newly set this time (S2
4) When the desired frame appears, press the "start frame" button on the screen to register the start frame. In this state, the frame is in a waiting state for setting an anchor area in this frame, and the user provides a rectangular area by clicking the mouse so as to surround another fish, for example. Once the rectangular area is determined, the upper left point (x1, y1) and the lower right point (x2, y1)
The coordinates of 2) are acquired, and this is recorded as anchor information of the fish together with the frame number of the start frame (the serial number from the first frame of the moving image) (S25).

After that, the moving image data is advanced again, and stopped when the desired end frame appears (S26), and a rectangular area is provided so as to surround the same fish. Here, the setting of the anchor in the end frame is completed (S27). “Anchor1” in FIG. 3 is an anchor ID indicating this fish. Here, the frame numbers of the start frame and the end frame (frames 1 and 100, respectively) and the coordinate information of the anchor area are stored in the table.

[0061] Thus if confirmed anchor information at both the reference frame, a third anchor information therebetween the third frame (non-reference frame) obtained by interpolation calculation (S28). FIG. 4 is a diagram showing an interpolation calculation method of anchor information. Here, the anchor information at the start frame (time t0) is A
(T0)-The anchor information at the end frame (time t1) is A
(T1) -If the anchor information at time t is A (t) -t1-t0 = Δt, A (t) = {A (t1) -A (t0)} t / Δt + {A (t0) t1 −A (t1) t0} / Δt (Equation 1). As A, the x1, y1, x2,
By substituting y2, the outer shape of the anchor area at an arbitrary time is determined. Substituting the barycentric coordinates of the anchor area reveals the rough movement of the anchor area. By substituting the color number for A, the change in color of the anchor area can be tracked. In addition to this, information that can be expressed numerically can also be interpolated by the internal division calculation using Expression 1. The anchor information of the non-reference frame obtained by the interpolation may be added to “anchor1” in the table of FIG. 3, or the table of FIG. The calculation of Expression 1 may be sequentially performed. In the present embodiment, the sequential calculation is assumed hereinafter.

When S28 is completed, the anchor information is actually displayed and the content is confirmed (S29). At this time, the moving image data is reproduced by returning to the start frame, and the anchor area is displayed as a rectangle in each frame. This rectangular area moves continuously according to the calculation result.

In the case of "anchor1", the result is extremely good if the fish makes a linear motion at a constant speed. However, if the direction of swimming is changed on the way, the anchor area is shifted from the fish in the middle frame. Therefore, the anchor information is corrected (S3
0). The user first advances the moving image data to a frame with a large shift, and stops the image here. Next, the end of the anchor area displayed on the screen is clicked, and the shape or position of the area is changed with the mouse. The anchor estimation unit 12 upgrades the frame thus modified to the reference frame (hereinafter, the one that has been promoted to become the reference frame is also referred to as “intermediate reference frame”), and adds this anchor information to the table of FIG. . FIG. 5 shows a table obtained by adding the anchor information of the intermediate reference frame to FIG. On the other hand, FIG. 6 is a diagram showing a method of performing an interpolation calculation based on three frames of an intermediate reference frame and both end reference frames. The non-reference frame to be estimated is the first
If the frame exists between the start frame and the intermediate reference frame, the interpolation calculation is performed between those frames. If the non-reference frame exists between the intermediate reference frame and the end frame, the interpolation calculation is performed between those frames. (S2
8). Thereafter, when good anchor information is obtained through the display in S29 and the re-correction in S30 (Y in S31), this is saved (S32), and the anchor setting process is ended. If the anchor of another frame is modified in S30,
Naturally, this frame also serves as an intermediate reference frame. In addition, S
When two or more anchors are set in the same frame in step 25, the anchor IDs are automatically changed within the apparatus in the order of setting while the anchor IDs are assigned, and rectangular display of these anchor areas is performed in different colors. Just do it.

According to the above procedure, the following effects can be obtained. 1. Only setting the anchors in the reference frames on both ends eliminates the need for setting work for a large number of frames existing between them. 2. If a displacement occurs in the position of the anchor in the interpolation calculation, this displacement can be confirmed. Therefore, it is easy to recognize the frame to be corrected, and the frame once corrected is automatically promoted to the intermediate reference frame, so that the user does not need to pay attention to whether or not to change the frame to the reference frame. 3. For example, even in a case where a fish having an anchor set therein draws an arc and swims, sufficiently good anchor information can be obtained by performing correction at most several frames in addition to the both-end reference frames. The above is the outline of the moving image anchor setting device among the moving image hypermedia devices of the present embodiment.

[2] Link setting Next, a link is set for the set anchor. FIG. 7 is a flowchart showing a link setting and search procedure according to this embodiment, and FIG. 8 is a diagram showing a table of set link information.

FIG. 7 shows a processing procedure in the case where the anchor setting and the link setting are performed completely independently. Similar to FIG. 2, first, various initialization processing (S40) and reading of moving image data (S41) are performed. Do. Subsequently, the anchor information set in [1] is read from the anchor information storage unit 21 (S42), and the already set link information is read from the link information storage unit 22.

Next, based on the anchor information of the both-end reference frame and the intermediate reference frame, the anchor information of the other frames is obtained by interpolation calculation (S44), and the display of the anchor information is continuously performed in accordance with the reproduction of the moving image. Perform (S4
5). In this state, the user operation unit 4 waits for an input from the user (S46).

Here, the user clicks on a certain anchor area after temporarily stopping the moving image or the moving image,
If the "Create / Change Link" button is pressed, link information is created for that anchor (S47). For example, when a fish in the aquarium is clicked, text, images, and other candidates to be linked to the fish appear on the screen, and the text or the like selected by the user is anchored to the fish (more precisely, included in the anchor. Object called a fish). If there is no candidate, the user can himself input a character string and link it. In FIG. 8, the text information "anchor1.txt" is added to "anchor1", and similarly "anchor1.txt".
Bitmap image "anchor" as an anchor for "2"
2. “bmp” is shown in a linked state. When the link information is determined in this way, the contents of the link are stored in the link information storage unit 22, and the user waits for an input again.

On the other hand, if the user presses the "link search" button in S46 and specifies the anchor, the link information corresponding to the anchor is searched and displayed (S4).
9). In the case of FIG. 8, for example, for the fish of anchor1, the name, length, characteristics, etc. of the fish are displayed in a character string.
For the fish of nchor2, a picture of the sea where the fish actually lives is displayed. Since the link operation can be confirmed by this display, the user can complete the content creation at this point. By storing the content in a recording medium such as a CD-ROM,
It can also be commercialized. When shipping as a product, generally, the mode is changed to the anchor non-display mode in which the anchor area is not displayed.

Although the setting of the anchor and the link has been described as independent processing here, for example, if a button "return to anchor setting" is provided on the screen during link setting,
Both sides can freely move, and editing becomes easier.

[3] UI for Setting Anchor FIG. 9 is a diagram showing an example of a UI screen for setting an anchor. In the figure, a moving image to be processed is displayed in an image display area 50. The black-painted button group 52 in the upper column is an object button for directly instructing playback, stop, etc. of the video. Next to it, a rectangular button 5 for setting an anchor area for the frame displayed in the image display area.
4. Similarly, a start frame designation button 56 and an end frame designation button 58 for designating the displayed frame as a start frame or an end frame are provided. In the figure, the anchor area 60 for one fish is shown.
Is set.

On the right side of the center of the screen, there is an anchor-related box group 62 showing the name, ID, start frame number, and end frame number of the anchor to be set or modified. Below the image display area 50, there is a scene-related box group 64 indicating the number of the scene including the currently displayed frame and the serial number of the frame in the scene. Further below, there is a box 66 for advancing or returning the video by a small amount for editing. Press the button at the right end to advance the movie, and press the left end to return. The position of the currently displayed frame in the scene is indicated by a vertical line 70 in the box 66. Below this box is a box 68 that indicates the position of the start and end frames in the scene. The positions of the start and end frames are indicated by double vertical lines 72, 74, respectively, and the position of the intermediate reference frame therebetween is indicated by the triangular symbol 76.

In the figure, the user first advances the video tape to the beginning of the scene for which the anchor setting is desired, using the scene number as a clue. In this case, for example, among the images of the aquarium consisting of a plurality of scenes, the image of the aquarium with scene number “5” is displayed. Here, the user presses the button at the right end of the box 66 to advance the moving image one frame at a time. When the first frame for which an anchor is to be set appears, the user presses the start frame designation button 56 and registers this. At this time, a double vertical line 72 showing the position of the start frame appears at the corresponding position of the box 66.
Here, the rectangular button 54 is pressed, and the upper left point and the lower right point of the anchor area to be set in the image display area 50 are clicked with the mouse. This completes the anchor setting of the start frame. Next, proceed with the video, and similarly, register the end frame and set the anchor.

When it is detected that the setting in the both-end reference frame has been completed, the anchor estimating unit 12 of the present apparatus automatically substitutes the anchor information into Expression 1 and starts the calculation. Here, the user returns to the start frame,
As the frame advances, the anchor estimating unit 12 obtains the time corresponding to the currently displayed frame, and displays the anchor area based on the estimation result corresponding to this time. If the displayed anchor area is displaced, the user presses the rectangular button 54 again to correct the area. After modification, a triangular symbol 76 appears at the location corresponding to that frame.
According to this UI, since the anchor information is actually displayed on the moving image data, the edited result can be confirmed in real time, and the correction can be easily performed.

The above is the outline of the present embodiment. In addition,
Regarding the present embodiment, the following improvements, modifications, and the like can be considered.

(1) Setting of Text Anchor The text anchor setting unit 111 of FIG. 1 is used. First, edit the text data on the screen, overlay it on the video, and set the anchor. The difference from the normal anchor setting is that, instead of specifying a partial area of the reproduced image, the created text is temporarily put on the image, and then the anchor area is set so as to surround the text. Conventionally, for example, a method of directly inserting an annotation into a video image has been common, but in that case, it is inconvenient for re-editing such as deleting the annotation later. This embodiment solves this.

When the text anchor is set, its anchor information is also stored in the anchor information table. However, in the table shown in FIG. 3, the "frame" portion becomes "text", and the text name is entered in that column.

It is also possible to link related information about the text anchor. For example, paste the text "Fish of the South Sea" to the aquarium scene in Fig. 9, and link the text to the text "There are many brightly colored fish in the South Sea ...". Can be made.

(2) Change of display state of cursor This is performed by the cursor change unit 300 of FIG. This feature is especially useful in anchor-hidden modes, such as when content is used on the market. Because of this function, the cursor changing unit 300 includes a position acquisition program that constantly acquires the position of the cursor, a determination program that determines whether the acquired position is included in any anchor region, and an anchor region where the cursor is located. When entering, there is a change program for determining how to change the display state of the cursor, and actually changing the shape and the like of the cursor according to the determination.

The change of the cursor may or may not be changed for each anchor. In the former case, for example, there is a method of changing the cursor, which is usually a + sign, to ◎ or increasing the brightness of the cursor. According to this aspect, it is particularly advantageous when the movement or shape of the target is fast and the anchor region is drastically changed.

On the other hand, in the latter case, it is conceivable that an anchor ID of the anchor area where the cursor is entered is searched from the change program, and this is displayed as it is at the position of the cursor instead of the cursor. For example, when the cursor enters the anchor area of a fish, the cursor may indicate the content of the target of the anchor, such as "shark". According to this aspect, the user can know the name of the fish without needing to click the fish.

(3) Explicit Designation of Intermediate Reference Frame In this embodiment, only the reference frames at both ends are initially determined. However, when the movement of the target is irregular or the like, it is necessary to make corrections. There is also. In that case,
From the beginning, it is assumed that designation of an anchor area is also accepted in frames other than the start frame and the end frame. For example, in the UI of FIG. 9, an intermediate frame designation button is provided in addition to the start frame designation button 56 and the end frame designation button 58. Since this frame is used as the reference frame from the beginning, it can be considered that the interpolation calculation is started from the state shown in FIG.

(4) Anchor area other than rectangle It is not necessary to limit the anchor area to a rectangle. For example, in the case of a circle or an ellipse, an area may be specified by coordinates of three points, a major axis, a minor axis, and a center. If it is a polygon, the coordinates of each vertex will do. When the outer periphery of the target itself is used as the anchor region, the region can be specified by the coordinates of one point on the outer periphery and the chain code expressed from that point.

(5) Employment of Nonlinear Interpolation In the present embodiment, linear interpolation is used most simply,
This may of course be a non-linear interpolation. The equation used for interpolation is
It can be determined by experiments or the like according to the characteristics of the moving image to be processed.

(6) Determination of start and end frames In the present embodiment, these frames are explicitly designated by the user, but the following method is also available. 1. The user does not need to be aware of the start and end frames,
Simply specify the frame and set the anchor. The specified frame becomes the reference frame. Frame determination unit 10
Determines that the frame with the smallest frame number among the frames to which the user has set an anchor is the start frame and the frame with the largest frame number is the end frame. In this case, the start frame designation button 56 and the end frame designation button 58 in FIG. 9 become unnecessary.

2. The user designates one frame, sets an anchor for this, and designates a target for which the anchor is set. This frame becomes the reference frame. The frame determination unit 10 examines frames before and after the reference frame to detect a frame in which the target appears and a frame in which the target disappears, and sets them as a start frame and an end frame, respectively.

The presence or absence of a target is determined by matching images. That is, matching processing is performed on the preceding and succeeding frames using the target specified by the reference frame as a model. The target frame of the search is expanded forward and backward as long as matching can be achieved. Eventually, if the matching cannot be obtained, the start and end frames are determined. According to this method, only one reference frame should be provided initially.

(7) Three-dimensional display of anchor area The anchor setting unit 11 is provided with a function of expanding the set anchor area in the x and y directions and the time t direction, which are the horizontal and vertical directions of the screen, and displaying it three-dimensionally. This can be considered that FIG. 4 is displayed on the screen as it is while editing the anchor. As a result of this display, the user can visually grasp the anchor as a whole.

As an application of this technique, the anchor information displayed three-dimensionally may be directly editable. For example, if the anchor area in the intermediate reference frame of FIG. 4 is moved to the left on the screen, the display as shown in FIG. 6 is made. The user can grasp the effect of editing in real time.

(8) Cross-sectional display of moving image In the anchor setting section 11, a horizontal cross-sectional view of the locus of the anchor region from the start frame to the end frame (FIG. 10).
(A)) and a vertical cross-sectional view (FIG. 10 (b)) are created, and the function of displaying this together with the trajectory of the anchor is added. First, the x and y coordinates of the center of gravity G of the anchor area are obtained from the anchor information of the reference frame. In the case of FIG. 10A, a straight line parallel to the x-axis is drawn on the frame from the center of gravity. A plane including these straight lines (hatched portion in the figure) is provided between adjacent reference frames. Next, the moving image is cut on this plane. The obtained cross-sectional view is projected on the xt plane (dotted area in the figure). In the case of FIG. 10B, x and y are exchanged and the same processing is performed. When the trajectory of the anchor is calculated correctly, that is, when the estimation accuracy of the anchor information in the non-reference frame is sufficiently high, the movement path of the anchor should appear in the two projection views. For example, if a red ball is to be set as an anchor, a red streak-like moving path appears in a section of the moving image. Splitting the pencil vertically is the same as making the lead appear linear. If this movement path is cut or thickened on the way, the position of the anchor at that location may be corrected.

(9) Grouping of anchors The anchor information editing unit 110 is provided with a function of grouping separately set anchor information and virtually handling it as one anchor. For example, the person A is in the frame N1
-N2 and frames N3 to N4 are present in the screen, and frames N2 to N3 are not present, frames N1 to N
2 and the anchor for the person A in the frames N3 to N4 are treated as one. As a result, the work of setting and correcting the anchor information is reduced. In addition,
The person A and the person B appearing in the same frame can be grouped.

(10) List Display of Anchor Information The anchor information editing unit 110 is provided with a function of displaying a list of anchor information set for the moving image currently being processed. For example, with the video title "AQUARIUM",
A list of anchor names such as “FISH1”, “FISH2”, etc. is displayed on the screen. When the user selects an anchor name whose contents are to be checked, the user may return to the start frame of the anchor and play back the moving image.

(11) Anchor search An anchor information search UI is provided. When character information such as the name of the anchor information to be searched is input as a keyword, the anchor search unit 15 searches the anchor information storage unit 21 for one having the keyword and displays it. Alternatively, the movement of the anchor area may be used as the search key. For example, when the user wants to find an object that moves to the right, the user presses the button "→" in the search UI. The anchor search unit 15 calculates the trajectory of each anchor region, searches for an anchor including an object that moves to the right, and displays the anchor.

(12) List display of anchor information display image In the anchor information editing unit 110, the frames (between the reference frame and the non-reference frame) included between the start frame and the end frame and the anchor setting unit 11 are used. The set anchor information or anchor estimation unit 12
A function of creating an anchor information display image by superimposing the anchor information estimated by the above, and displaying a list in chronological order. For example, as shown in FIG. 11, first, a start frame 81 and an end frame 82 are arranged at both ends, and frames are selected at a frame interval Δt. Subsequently, the anchor information 80 is superimposed on each of these frames to create an anchor information display image, and these are displayed side by side from the earliest display time. With this configuration, the suitability of the anchor setting can be seen at a glance. Therefore, the position determining operation by the box 66 in FIG. 9 can be omitted, and the anchor can be easily corrected. It is not necessary to select frames at regular intervals.
For example, only the reference frame may be selected. Further, in the displayed anchor information display image, the anchor region may be directly editable by dragging it with the mouse.

(13) Correction of anchor information during reproduction of moving image As shown in FIG. 12, a moving image reproduction unit 118 for reproducing a moving image on a screen and an anchor for displaying anchor information relating to a frame being displayed are provided in the anchor information editing unit 110. When the anchor information correction operation is performed at one or a plurality of times during the reproduction of the moving image, the information display unit 120 specifies the frames that are being reproduced at the time when each anchor information correction operation is performed. If is a non-reference frame, an anchor information correction unit 119 is provided which is promoted to a reference frame and corrects the anchor information in those frames based on each anchor information correction operation. With this configuration, first, a moving image is displayed in the image display area 50 of FIG. At this time, simultaneously, the anchor information in the frame being displayed is displayed as the anchor area 60 by the anchor information display unit 120. When the user finds a frame in which the anchor region 60 is displaced from the target, the user clicks the center of the target in the moving image with the mouse. With this action, the anchor information correction unit 119 corrects the anchor information by specifying the frame displayed at the time of the click and generating anchor information centered on the clicked point. The size of the newly set anchor area may be the same as that of the original anchor area, for example. Hereinafter, this frame is treated as a reference frame. With this configuration,
Since the position of the target can be sequentially specified during the playback of the moving image, it is not necessary to check and correct the position later.

Embodiment 2. In the first embodiment, the anchor information is automatically calculated mainly by interpolation, and the correction is manually performed. In the present embodiment, an anchor is automatically set with a certain number of frames as reference frames in advance based on analysis of moving images, and the interpolation method of the first embodiment is used between these reference frames. In this case, since the frame corresponding to the intermediate reference frame of the first embodiment is present from the beginning, the labor of manual correction is reduced.

FIG. 13 is a configuration diagram of the anchor setting unit 11 of the moving image hypermedia apparatus according to the present embodiment. Configurations other than the anchor setting unit 11 are the same as those in FIG. FIG.
In, the automatic anchor setting unit 112 has a nearby frame extracting unit 117. The proximity frame extraction unit 117 extracts non-reference frames at regular intervals between adjacent reference frames and promotes them as reference frames. The automatic anchor setting unit 112 also includes the motion vector use setting unit 11.
3, a contour information use setting unit 114 and a pattern matching use setting unit 116. These three settings are:
Originally, it is sufficient to implement either one, but in the present embodiment, all are implemented, and one of them is selected according to the situation.

As will be described later, the reference frame deletion unit 115 returns a redundant one of the reference frames set by the automatic anchor setting unit 112 to a non-reference frame. Hereinafter, the operation by this configuration will be described.

[1] Automatic setting of anchor by using motion vector The feature of this processing is that the motion vector of the block is once obtained from the start frame to the end frame, and then the target
This is a two-stage configuration that increases the tracking accuracy of the target position estimation by determining the degree of coincidence between the virtual movement path of the target and the motion vector.

1. Acquisition of motion vector The time of the start frame and the end frame are set to t0
Set to 1. In addition to the above, first, some of the non-reference frames are changed to reference frames by the adjacent frame extraction unit 117. Here, it is assumed that the time is simply changed every five frames.
Is normalized. In order to obtain the motion vector of a specific anchor in the period from the start frame to the end frame, the image area near the center of gravity of the anchor is used as a block,
Perform block matching. A frame corresponding to an arbitrary time t will be referred to as a frame (t).

FIG. 14 is a flow chart showing the procedure for obtaining a motion vector in this embodiment. As shown in the figure, first, the time counter t is set to t0 (S10).
0). Next, the block set in the start frame
Among click specifies a block to be acquired motion vector. The motion vector utilization setting unit 113 stores the area including the center of gravity of the specified anchor as a block used for block matching (hereinafter referred to as "anchor block") (S101). Subsequently, image data I (t) of frame (t) and I (t +
1) is acquired (S102). I (t) is aggregate data of the pixel value p of each pixel included in the frame.

[0102] After this, while moving the block frame (t + 1) in a, to search for the optimum matching (S10
3) . Since each pixel value of the block itself to find from I (t), place the blocks in any point of the frame (t + 1), the square error of the pixel values calculated by the pixel each other overlapping, blocks this Integrate over the entire area . While the block is moved little by little performs the integration, with the position where the integrated value becomes minimum, it is determined that the destination block.

[0103] If the destination is known, since the direction and amount of movement from your <br/> only Lube locked to the frame (t) to the lube lock put in a frame (t + 1) is determined, this motion vector It is acquired as V (t) (S104). Where t
It is determined whether +1 has reached the time t1 of the end frame (S105), and if not reached, t is incremented (S106) and the motion vector is repeatedly acquired. If t + 1 becomes equal to t1, then V obtained so far
After saving (t) (S107), the process ends.

FIG. 15 is a diagram showing an example of motion vectors V (0) to V (2) obtained when t0 = 0 and t1 = 3. As shown in the figure, V (t) is defined by x and y determined by the vertical and horizontal directions on the screen, and t determined by the time direction,
It can be expressed by three components of (x, y, t).

2. Each frame is divided by block equivalent to the size of the blocks provided in the matching degree determination motion vector, find all possible routes the target is followed. FIG. 16 is a diagram showing one of such paths. In the figure, a frame is divided into 16 blocks, and the starting point of the route at the start frame, the end point of the path in the end frame of FIG. 14
It is consistent with the block. Under this condition, all routes are 1
There are 6 × 16 patterns. Subsequently, a vector (hereinafter referred to as “route vector”) v (t) illustrated in FIG. 16 is defined in each section of this route (hereinafter referred to as “virtual route”). The route vector is determined by the direction of the virtual route from one frame to the next. v (t) is also (x, y,
It is described by the three components of t).

Here, in each section, V (t) and v
The angle formed by (t) is defined as θt, and f (t) = cosθt is calculated by the following equation using the inner product.

F (t) = (V (t), v (t)) / | V (t) | · | v (t) | (Equation 2) FIG. 17 corresponds to v (t) in FIG. FIG. 15 is a diagram in which V (t) is added and displayed, and shows the meaning of θt. Formula 2
Is larger, the degree of coincidence between the virtual path and the motion vector in that section is higher. However, even if the degree of coincidence is maximized in a certain section, if the degree of coincidence in another section is extremely low, the overall Need to be considered low. Therefore, the following evaluation formula is introduced in order to evaluate the degree of coincidence as a whole while considering the degree of coincidence in each section.

G (t) = max {f (t-1) + g (t-1)} (Equation 3) By recursively calculating Equation 3, the degree of coincidence was always the highest until that time. The virtual route is known. By performing this calculation to the end frame, for the highest degree of coincidence virtual path throughout it is found, with the path ter
It is regarded as the route of the get . After that, the target at that time is displayed at the intersection of this movement route and each reference frame.
As bets are present, it performs automatic setting of the anchor.
The set anchor information may be added to the anchor information table shown in FIG. For frames other than the reference frame, the anchor information may be sequentially calculated from the interpolation calculation by the same method as in the first embodiment.

[2] Automatic Setting of Anchor Using Contour Information As another method of automatic anchor setting, there is a method of estimating the position of the target based on the movement of the target contour. The contour information use setting unit 114 generates a contour image for each frame by repeating the same processing as in FIG. The contour image is a binarized image in which 1 is on the contour line and 0 is on the contour line, and a compass gradient (Compas gradient) is added to the image.
s-gradient) type filter and so on. Once the contour image is obtained, the anchor information may be set thereafter assuming that the anchor moves exactly the same as the target.

[3] Automatic Setting of Anchor Using Matching As another method of automatic setting of an anchor, there is estimation of the position of the target by pattern matching shown in FIG. In the case of this method as well, first, the adjacent frame extracting unit 11
7, some reference frame is provided in advance. Next, a model 134 for pattern matching is created from the anchor area 132 set in the start frame 130, and an area 138 having the highest matching degree in the adjacent reference frame 136 is obtained. As a pattern matching method, there are a template matching method in which image data of a model is superimposed as it is, and a structure matching method in which superimposition is performed based on a positional relationship of feature points extracted from an image. Pattern matching is performed mainly in the vicinity of the model.

In this way, the area 13 in the second reference frame
If 8 is found, the same processing is repeated using this area 138 as a new model, and the target is tracked. In some cases, good matching cannot be achieved when the target moves or deforms significantly or when the reference frame setting interval is too wide. In such a case, the adjacent frame extracting unit 117 performs the matching process again after narrowing the interval between the reference frames to provide more reference frames.

[4] Deletion of Unnecessary Reference Frame In the above example, the reference frame is provided every 5 frames. However, for example, in the case where the target makes a linear motion at a constant velocity, only the start frame and the end frame are sufficient as the reference frames. Even if the target does not have a uniform linear motion from the start frame to the end frame, for the period of such motion,
Only the reference frames at both ends of the period are needed. As the number of reference frames decreases, the calculation load becomes lighter. In the case of [1], the number of virtual routes is drastically reduced, which is particularly effective.

From this viewpoint, the reference frame deleting unit 115
Deletes unnecessary reference frames. FIGS. 19 to 21 are diagrams showing a situation in which reference frames are being deleted. In these figures, the horizontal axis is time, and the vertical axis is x- provided on the frame.
The distance from the origin of the y coordinate is shown. In addition, the circles in each figure schematically show the anchor region. Follow the procedure below to delete.

[FIG. 19] Initially, six reference frames including both end reference frames are provided. Here, the anchor of the start frame and the anchor of the end frame are connected by a straight line, and the distance between the straight line and each anchor is calculated. If there is an anchor whose distance is equal to or less than a predetermined value, the reference frame at that time is deleted. In the figure, it is assumed that no anchor has been deleted. Next, the anchor farthest from the straight line (hereinafter referred to as the farthest anchor) is found. here,
The anchor at t = 3 is the farthest anchor. [FIG. 20] The straight line is deleted, the anchor of the start frame, the farthest anchor, and the anchor of the end frame are connected by a polygonal line in this order, and the distance between this polygonal line and each anchor is obtained again. The anchor at t = 4 whose distance has become equal to or less than the predetermined value is deleted. The farthest anchor is changed to t = 2 anchor. [FIG. 21] The polygonal line is corrected so as to pass through the new furthest anchor. Here, the reference frame at t = 1, in which the distance from the new polygonal line becomes smaller than the predetermined value, is deleted. This ends the process.

In this example, two reference frames have been deleted. When there are many initial reference frames,
1. 1. Delete reference frames that are less than or equal to a predetermined value. 2. Search for the farthest anchor; Repeat the correction of the broken line.

The above is the outline of the present embodiment. In addition,
The present embodiment can be improved or modified as follows. (1) Modification of Expression 2 In Expression 2, f (t) = cos θt is adopted, but naturally this may be another function. The function of increasing and decreasing θt is f
(T) can be a candidate. (2) In-taking view 15 of the block has been decided to Bed B <br/> click to include the vicinity of the center of gravity of the anchor region, which may be another method of determining. For example,
The anchor area may be used as it is as the anchor block. Similarly also in FIG. 16 may be sized and regardless blocks block.

(3) Another Method of [3] (Part 1) Another mode of automatic anchor setting by the above matching will be described. When matching is performed one after another using a certain frame region as a model, errors may accumulate and the target may gradually deviate from the target. For this reason, the determination is made in consideration of not only the matching from the adjacent frame but also the matching result from a frame that is distant to some extent.

As shown in FIG. 22, here, a start frame and an end frame are adopted as reference frames separated by time. Now, it is assumed that the new reference frame 404 for specifying the anchor position is at time t + Δt. On the other hand, the model A400 of the anchor area of the start frame, the model B401 of the anchor area of the end frame, and the model C403 of the anchor area of the reference frame 402 at time t are all known. Therefore, matching is performed between these three reference frames and the reference frame 404 at time t + Δt. At this time, if all matching results match, the tracking result of the area is reliable. On the other hand, if the matching results do not match, the anchor position is determined by the following method, for example. 1. The three regions obtained on the reference frame 404 at the time t + Δt as a result of the matching three times are superimposed to create an anchor region having the same size as the original region, centered on the center of the overlapping portion.

[0119] 2. One model, model C403
, A plurality of matching results are obtained, and in order from the top of the matching results, it is determined whether or not an area obtained from the matching results of the model A400 and the model B401 is included in the area. If the area is included at a certain ratio or more, an anchor area having the same size as the original area is created around the center of the overlapping portion between the areas obtained from the matching result. Here, the start frame and the end frame are considered in addition to the adjacent reference frame, but the combination has a degree of freedom. For example, an adjacent reference frame and any number of reference frames separated by a certain time distance may be employed.

(4) Alternative method of the above [3] (No. 2) As shown in FIG. 23, the automatic anchor reliability setting unit 112 is provided with an automatic reliability setting determination unit 130 and an automatic reliability display unit 131. The automatic setting reliability determination unit 130 determines the reliability of tracking based on the degree of overlap between the anchor area obtained by performing matching in the forward direction up to the last frame and the anchor area specified in the end frame. For example, if the area of the overlapping portion is 70% or more, the tracking result is reliable.
If it is 0% or less, it is determined that it is not reliable. The automatic setting reliability display unit 131 displays the reliability of tracking (whether or not the reliability can be obtained, or% thereof).

For example, as shown in FIG. 24, the tracking result 1 by matching with respect to the original trajectory 140 of the target
When 41 is warped, the tracking is determined to be unreliable because the degree of overlap is low. Therefore, this time, the image is matched in the reverse direction using the anchor region of the end frame as a model, and the estimated position of the target position is tracked again.
At this time, in each reference frame, the anchor area previously set by the forward matching and the anchor area obtained by the current backward matching are compared, and if both overlap by a predetermined ratio or more, the tracking ends. I do. At this time, as another method, tracking may be terminated at a designated frame. The frame for which tracking should be ended may be determined based on the ratio of the distance from the start frame and the end frame. A method of determining overlap and a method of specifying a frame to end tracking may be used together.

As described above, according to this method, even if an error occurs during the tracking, it is possible to avoid deterioration of the subsequent tracking result, and the final correction work is reduced. Also,
When the matching in the reverse direction is terminated by checking the degree of overlap, only a necessary part is corrected, which leads to a reduction in processing time.

Even if the reliability is low, matching may be merely displayed instead of automatically starting the matching in the reverse direction. In this case, the user may cause the matching in the reverse direction to be performed, or may make a desired correction by himself. In any case, the user can know whether the tracking is good or bad by the automatic setting reliability display unit 131, and appropriate measures can be taken.

(5) Alternative method of [3] above (Part 3) The respective components of FIG. 12, that is, the moving image reproducing section 118, the anchor information display section 120, and the anchor information correcting section 119 are provided in the automatic anchor setting section 112. . Here, the moving image playback unit 118 chronologically arranges each frame constituting the moving image,
Display at appropriate time intervals. Anchor information correction unit 119
Corrects the anchor information of the frame displayed at that time when the anchor information correction operation is performed at any time during the reproduction of the moving image. At the same time, the automatic anchor setting result of the frame displayed for the predetermined number of frames or the predetermined period immediately before the frame is invalidated.

An operation in this configuration will be described. Here, it is assumed that the moving image is displayed while matching is sequentially performed in the forward direction from the start frame in accordance with the reproduction of the moving image. In this operation, as described above, once the matching is not good, there is a possibility that the anchor information will gradually deviate thereafter. The user clicks the screen when the anchor area deviates from the target while viewing the moving image to be reproduced and the anchor information. At this point, playback stops. Here, for example, if the user clicks the center of the target, the anchor position is corrected so that the point becomes the center. Subsequent matching is performed on the basis of the corrected anchor information, so that it is good.

In this method, since the user clicks the screen during reproduction of the moving image, it is necessary to consider a delay in the action. That is, when the user clicks after recognizing that the anchor region has deviated from the target, it is considered that such a phenomenon has gradually progressed over several frames. Therefore, the anchor information correction unit 119 invalidates the anchor information that is automatically set for a plurality of predetermined frames existing before the frame in which the anchor information is corrected.

FIG. 25 is a diagram showing the relationship between the original trajectory of the target and the tracking result obtained by matching, and the operation of the anchor information correction section 119. The solid line 150 in FIG. 7 indicates that the trajectory of the target matches the tracking result, and the broken line 151 indicates that the tracking result deviates from the trajectory. As shown in the figure, the tracking results are reliable at times t0 to t1, but start off at time t1. The user notices this and clicks the screen at time t2. As a result, correct tracking is performed again between times t2 and t3.
At times t1 and t2, an incorrect tracking result remains, so this is invalidated. As for the invalidated portion, as shown in the figure, the missing tracking result can be supplemented by linearly interpolating the anchor information at the times t1 and t2. Note that the number of frames for which the anchor information is invalidated may be specified in advance, or may be specified at the time of correction.

Embodiment 3 FIG. The following devices or systems can be constructed by applying the moving image hypermedia devices of the first and second embodiments.

1. Interactive Video Teaching Material Production Apparatus The moving image hypermedia apparatus according to the present invention is also optimal for producing CAI content. That is, an anchor is set in the video teaching material by this device, and necessary additional information is linked. FIG. 26 is a diagram showing a video teaching material produced by this device. As shown in the figure, an explanation A is linked to the anchor A, an explanation B is linked to the anchor B, and so on. While playing the video, the student clicks on the screen the object for which he wants a more detailed description. If the clicked object is associated with anchor A, then explanation A is displayed on the screen.

[0130] 2. Interactive Video Server System The video hypermedia device according to the present invention is also suitable for a video server system. FIG. 27 is a configuration diagram of this video server system. As shown in this figure, this system is composed of a server 200 and a client 2 which are substantially the same as those shown in FIG.
Consists of 50.

The server 200 includes a data storage unit 204 for storing moving images and their anchor information and related data linked to the anchors, an anchor estimating unit 206, and a hyperlink search unit for searching for related data linked to any anchor. 208 is provided. Anchor estimation section 206 estimates anchor information in a non-reference frame.

On the other hand, the client 250 includes an anchor determination unit 252 for determining which anchor area has been clicked when the user clicked on a target in the moving image.

In this configuration, when the user clicks an object on the screen with the client 250,
The anchor determining unit 252 specifies the clicked anchor. This information is sent to server 200. Server 20
0 hyperlink search unit 208 is the data storage unit 204
From the relevant data linked to that anchor,
This is sent to the client 250.

As described above, according to this system, moving image data and anchor information are collectively stored on the server 200 side, and a large number of users can see the required moving image and the information linked thereto.

[0135]

According to the moving picture anchor setting device of the present invention, the anchor information of a non-reference frame, that is, a frame other than the reference frame can be estimated by setting the anchor information only for the reference frame. It is not necessary to set anchor information for non-reference frames. As a result, the anchor setting work can be saved.

When a plurality of reference frames are taken, if the frame decision means decides the earliest one as the start frame and the latest one as the end frame, the user sets the period in which the user should set the anchor. There is no need to be conscious and work efficiency can be improved.

When the frame determining means detects the frame in which the target appears and the frame in which the target has disappeared and determines these as the start frame and the end frame, respectively, it can be said that there is at least one reference frame, and further labor saving is promoted. be able to.

When the anchor estimating means estimates the anchor information in the non-reference frame by interpolating the anchor information, the anchor information of the non-reference frame can be obtained with a light calculation load.

When the anchor estimating means divides the change amount of the correspondence information between the anchors set in the reference frame according to the elapsed time therebetween, the anchor information of the non-reference frame is obtained by a very simple interpolation calculation. be able to.

When the anchor estimation means performs the interpolation calculation of the anchor information between the reference frames at both ends of each section of the setting target period, the anchors with respect to the non-reference frames are estimated by using the reference frames located close to each other. Therefore, the accuracy of estimation can be improved.

When the estimated anchor information includes an anchor information editing means for making a correction, the correction results in the interpolation calculation using the intermediate reference frame having the correct anchor information. The accuracy of can be improved.

When the anchor information editing means displays the trajectory of the anchor three-dimensionally based on the estimation of the anchor information, the edited result can be confirmed on the spot.

When the anchor information editing means calculates the trajectory of the anchor and displays the cross-sectional view of the moving image along this trajectory together with the trajectory of the anchor, the anchor estimated in the non-reference frame is referred to by referring to this cross-sectional view. You can check the accuracy of the information and correct it if necessary.

When the anchor information editing means displays the anchor information display images in time series, it is possible to grasp at a glance whether or not the anchor setting is appropriate. Therefore, the confirmation work of the anchor information becomes easy.

When the anchor information editing means includes the moving picture reproducing means and the anchor information correcting means, it is possible to specify an appropriate position of the target while reproducing the moving picture, so that desirable anchor information is set at the end of the reproduction. As a result, it is possible to reduce the labor of the confirmation and editing work performed later.

When the anchor estimation means includes the anchor automatic setting means, the number of reference frames in which the anchor information should be initially set may be small, so that the setting work can be reduced.

When the automatic anchor setting means tracks the anchor based on the movement of the contour of the target, the accuracy of the anchor tracking can be improved.

The same applies when the anchor automatic setting means tracks the anchor based on the motion vector of the target. In this case, depending on the type of moving image, a motion vector may be described for each frame during encoding, which is convenient.

The automatic anchor setting means determines, for each route that the anchor can follow in the setting target period, the degree of coincidence between the route and the motion vector of the anchor to search the moving route of the anchor. Can be reduced. That is, although the motion vector should include an error for each unit determination period, the path determination is performed over the entire unit determination period, so that the accuracy of the finally obtained path increases.

When the degree of coincidence is determined by the function of the angle formed by the path and the motion vector, the degree of coincidence can be evaluated numerically, and a highly accurate path can be found by a relatively simple calculation.

When the automatic anchor setting means includes the automatic setting reliability determination means and the automatic setting reliability display means, the user can know the quality of the automatic setting and can take appropriate measures.

When the automatic anchor setting means performs image matching with a new reference frame using the image of the anchor area of the reference frame for which anchor information has already been set as a model, the accuracy of anchor tracking can be improved.

When a similar image is not found in the new reference frame, and the automatic anchor setting means further divides the adjacent reference frame to provide a new reference frame and re-matches the image, Matching accuracy is improved. In addition, since a new reference frame is provided only at a necessary position, tracking accuracy can be improved while suppressing an increase in calculation load.

When the anchor automatic setting means tracks the position of the anchor on the basis of the matching results of the reference frames which are close to each other and the reference frames which are apart from each other by a certain time distance, for example, the situation where the matching results are gradually deviated is improved. It becomes easier and saves the trouble of fixing the anchor.

In the case where the automatic anchor setting means includes the automatic setting reliability judgment means, even if the tracking is deviated from the middle, the anchor information of the deviated part can be corrected.

When re-tracking by the automatic setting reliability determination means, if the tracking is terminated by checking the degree of overlap of the anchor area obtained by the original tracking and the current tracking, re-tracking for an unnecessarily long time is required. It does not need to be done and processing time can be shortened.

When a frame for which re-tracking should be terminated is designated, a simple process can produce a corresponding result.

When the anchor information correcting means invalidates the result of automatic anchor setting of a predetermined frame preceding the frame whose anchor information has been corrected, correction can be made when the tracking result becomes worse, and thus it will be confirmed again later. No need. In addition, since there is no need to make corrections while the matching is good, the confirmation work can be saved.

When the anchor setting means includes the reference frame deleting means, the number of reference frames for holding the anchor information is reduced, and the required storage capacity can be reduced. When the anchor setting means includes a text anchor setting means, the anchor can be set not only in the image data but also in the text data.

When the cursor changing means is further included, the presence and position of the desired anchor area can be easily known even if the anchor area is set invisible.

When the display state of the cursor is changed so as to accompany the display of the identification information of the anchor area in which the cursor is entered, some related information can be known without a click operation.

When the anchor setting means sets the anchor information in units of a group consisting of a plurality of anchors, for example, the labor of setting the anchor information for the same target many times can be omitted.

In the case where an anchor search means for searching for an anchor based on the characteristics of the movement of the anchor is included, the search labor is greatly increased when the user wants to know only the right moving object among many objects. Can be reduced.

When an anchor search means for searching for an anchor based on the identification information of an anchor is included, search efficiency can be improved.

On the other hand, according to the moving image hypermedia apparatus of the present invention, it is possible to link data necessary for an anchor and to search for linked data while taking advantage of the moving image anchor setting means of the present invention.

When this moving image hypermedia device is used for producing an interactive video teaching material, for example, an interactive video teaching material in which additional information can be obtained by clicking a corresponding object on a screen can be produced. .

On the other hand, according to the moving picture providing system of the present invention, a necessary moving picture and information linked thereto can be seen from many users.

According to the moving picture anchor display device of the present invention,
While playing the video, the set anchor information can be checked in real time.

According to the moving image anchor selecting apparatus of the present invention,
When you click on a required part while playing the video,
The user can know the anchor information associated with the location, which is convenient for editing and the like.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a system including a moving image hypermedia device according to a first embodiment.

FIG. 2 is a flowchart showing a procedure for setting and correcting an anchor according to the first embodiment.

FIG. 3 is a diagram showing a table of anchor information set in the first embodiment.

FIG. 4 is a diagram showing an interpolation calculation method of anchor information according to the first embodiment.

5 is a diagram showing a table obtained by adding anchor information of an intermediate reference frame to FIG.

FIG. 6 is a diagram illustrating a method of performing an interpolation calculation based on three frames of an intermediate reference frame and both-end reference frames in the first embodiment.

FIG. 7 is a flowchart showing a link setting and search procedure according to the first embodiment.

FIG. 8 is a diagram showing a table of link information set in the first embodiment.

FIG. 9 is a diagram illustrating an example of a UI screen for anchor setting.

FIG. 10A is a lateral cross-sectional view of the trajectory of the anchor area from the start frame to the end frame, and FIG.
0 (b) is a vertical sectional view similarly.

FIG. 11 is a diagram showing an anchor information display image displayed in chronological order by the anchor information editing unit in the first embodiment.

FIG. 12 is a diagram showing an internal configuration example of an anchor information editing unit of the moving picture hypermedia device according to the first embodiment.

FIG. 13 is a configuration diagram of an anchor setting unit of the moving image hypermedia device according to the second embodiment.

FIG. 14 is a flowchart showing a procedure for acquiring a motion vector according to the second embodiment.

FIG. 15 is a diagram illustrating an example of motion vectors V (0) to V (2) obtained when t0 = 0 and t1 = 3 in FIG. 14;

FIG. 16 is a diagram showing one of paths that an anchor may have taken.

FIG. 17 shows V (t) shown in FIG.
It is the figure which added and displayed (t).

FIG. 18 is a diagram showing a method for automatically setting an anchor based on pattern matching in the second embodiment.

FIG. 19 is a diagram showing a situation where reference frames are deleted in the second embodiment.

FIG. 20 is a diagram showing how reference frames are deleted in the second embodiment.

FIG. 21 is a diagram illustrating a state where reference frames are deleted in the second embodiment.

FIG. 22 is a diagram illustrating a method of performing matching using a reference frame that is separated by a certain time distance in addition to an adjacent reference frame in the second embodiment.

FIG. 23 is a diagram illustrating a configuration example of an automatic anchor setting unit of the moving image hypermedia device according to the second embodiment.

24 is a diagram illustrating an erroneous tracking result that may occur without the configuration in order to explain the effect of the configuration in FIG. 23;

FIG. 25 is a diagram showing the relationship between the original trajectory of the target and the tracking result obtained by matching, and the operation of the anchor information correction unit.

FIG. 26 is a diagram showing a configuration of an interactive video teaching material production device according to a third embodiment.

FIG. 27 is a diagram showing a configuration of an interactive video server system according to a third embodiment.

[Description of Signs] 1 data operation unit, 2 data storage unit, 3 display unit, 4
User operation unit, 6 moving image input unit, 10 frame determination unit, 11 anchor setting unit, 12 anchor estimation unit, 1
3 hyperlink setting part, 14 hyperlink search part, 15 anchor search part, 20 moving image data storage part,
21 anchor information storage unit, 22 link information storage unit,
Reference Signs List 30 display control unit, 31 display device, 50 image display area, 52 button group, 54 rectangular button, 56 start frame designation button, 58 end frame designation button, 60
Anchor area, 62 Anchor-related box group, 64
Scene-related boxes, 66, 68 boxes, 80
Anchor information, 81 start frame, 82 end frame, 110 anchor information editing part, 111 text anchor setting part, 112 automatic anchor setting part, 113
Motion vector use setting unit, 114 contour information use setting unit, 115 reference frame deletion unit, 116 pattern matching use setting unit, 117 adjacent frame extraction unit, 11
8 moving image display section, 119 anchor information correction section, 120
Anchor information display unit, 130 automatic setting reliability determination unit,
131 automatic setting reliability indicator, 140 original trajectory of target, 141 tracking result by matching, 20
0 server, 204 data storage unit, 206 anchor estimation unit, 208 hyperlink search unit, 250 client, 252 anchor determination unit, 300 cursor change unit, 400 model A, 401 model B, 402 reference frame at time t, 403 model C , 404 Reference frame at time t + Δt.

──────────────────────────────────────────────────の Continuation of the front page (51) Int.Cl. ⁶ Identification code FI G06F 15/62 340A (72) Inventor Satoshi Tanaka 2-3-2 Marunouchi, Chiyoda-ku, Tokyo Mitsubishi Electric Corporation

Claims (34)

[Claims] 1. A device for inputting a moving image and setting an anchor for the moving image, wherein a start frame and an end frame of a setting target period for which an anchor is to be set are determined from a plurality of frames forming the input moving image. Frame determining means, when a region is specified in a reference frame taken within the setting target period, anchor setting means for setting anchor information for the reference frame with this as an anchor region, An anchor estimation device for estimating anchor information in a non-reference frame based on the set anchor information, and a moving image anchor setting device. 2. The method according to claim 1, wherein, when a plurality of reference frames are taken in the setting target period, the frame determining means determines a temporally earliest one as a start frame and a latest one as an end frame. The described device. 3. The method according to claim 1, wherein the frame determination means detects a frame in which the target included in the reference frame appears and disappears before and after the reference frame, and determines these as a start frame and an end frame, respectively. The described device. 4. The anchor estimation unit according to claim 1, wherein, when there are a plurality of reference frames, the anchor estimation unit estimates the anchor information in the non-reference frame by interpolating the anchor information set for the reference frames. An apparatus according to any one of the above. 5. The anchor estimating means calculates a change amount of correspondence information between anchors set for two adjacent reference frames, and calculates an elapsed time from one reference frame to a certain non-reference frame, and The apparatus according to claim 4, wherein interpolation is performed by calculating a ratio of elapsed time from a non-reference frame to another reference frame, and dividing the change amount by the ratio. 6. The anchor estimation means performs, for each section of the setting target period divided by the reference frame, interpolation calculation of anchor information between reference frames at both ends of the section, and a non-reference frame included in the section. The apparatus according to any one of claims 4 and 5, which estimates the anchor information in (1). 7. An anchor information editing means for modifying the anchor information estimated for the non-reference frame, wherein the anchor estimating means promotes the non-reference frame with the modified anchor information to a new reference frame. The apparatus according to any one of claims 4 to 6, wherein the interpolation is performed between adjacent reference frames including a new reference frame when estimating anchor information in another frame thereafter. 8. The apparatus according to claim 7, wherein said anchor information editing means calculates a trajectory of the anchor based on the estimation of the anchor information and displays the trajectory three-dimensionally. 9. The apparatus according to claim 7, wherein the anchor information editing unit calculates a locus of the anchor based on the estimation of the anchor information, and displays a cross-sectional view of a moving image along the locus together with the locus of the anchor. 10. The anchor information editing means selects a frame included between a start frame and an end frame, creates an anchor information display image by superimposing anchor information on an image of the frame, and generates the anchor information display image. 8. The apparatus according to claim 7, wherein is displayed in a time series. 11. The anchor information editing unit includes a moving image reproducing unit and an anchor information correcting unit, the moving image reproducing unit reproduces the moving image on a screen, and the anchor information correcting unit corrects the anchor information during reproducing the moving image. When an operation is performed, the frame that was being played back at the time the operation was performed is specified, and if the frame is a non-reference frame, the frame is promoted to a reference frame, and anchor information in that frame is determined based on the operation. 8. The apparatus according to claim 7, wherein the correction is performed. 12. The anchor estimating means includes an automatic anchor setting means, and the automatic anchor setting means selects an arbitrary number of frames from adjacent reference frames and sets them as new reference frames, The apparatus according to any one of claims 1 to 11, wherein anchors in these new reference frames are tracked, and anchor information is set for these new reference frames. 13. The apparatus according to claim 12, wherein the anchor automatic setting means tracks the anchor based on the movement of the contour of the target. 14. The apparatus according to claim 12, wherein the anchor automatic setting means tracks an anchor based on a motion vector of a target. 15. When each section of a setting target period divided by a reference frame is defined as a unit determination period, the automatic anchor setting means determines, for each route that the anchor can follow in the setting target period, the route and the anchor. The motion vector matching degree is determined for each unit determination period, the determination results are integrated over the entire setting target period, and the route having the highest matching rate in the overall result is regarded as the anchor movement route. An apparatus according to claim 14. 16. A function of an angle between the path and the motion vector, wherein the degree of coincidence is determined by a function having no extreme value in the range of 0 ° to 180 °, and each unit of the function is determined. The apparatus according to claim 15, wherein the total result is obtained by calculating a sum of values for each determination period. 17. The automatic anchor setting means includes an automatic setting reliability determining means and an automatic setting reliability displaying means, wherein the automatic setting reliability determining means includes an anchor information set by the anchor setting means and the automatic anchor setting means. 17. The apparatus according to claim 12, wherein the reliability of the automatic setting is determined by comparing the anchor information set by the setting unit, and the automatic setting reliability display unit displays the reliability. 18. The anchor automatic setting means tracks the position of the anchor by performing image matching with a new reference frame using the image of the anchor region of the reference frame in which the anchor information is already set as a model. Item 13. The device according to Item 12. 19. When performing similar image matching, if a similar image is not found in the new reference frame, the automatic anchor setting means further divides adjacent frames to provide a new reference frame, 19. The apparatus according to claim 18, wherein the image matching is performed again after reducing the interval between the reference frames. 20. The automatic anchor setting means selects at least two of a reference frame which is close to the new reference frame and a reference frame which is apart from the new reference frame by a certain time as reference frames for which the anchor information has already been set. Then, the position of the anchor is tracked based on the matching result using the selected reference frame.
An apparatus according to claim 8. 21. The anchor automatic setting means compares the anchor information set by the anchor setting means with the anchor information set by the anchor automatic setting means to judge the reliability of the automatic setting. When the reliability determined by the automatic setting reliability determining means is low after the anchor information is set by the tracking including the degree determining means and the starting point is the first reference frame, the second reference frame is the starting point. An apparatus according to claims 18-20, wherein the anchor information is reset by tracking. 22. While tracking is performed with the second reference frame as a starting point, an anchor area set in a certain reference frame and a tracking with the first reference frame as a starting point are performed before the reference frame. 22. If the degree of overlap with the anchor area set in (c) is equal to or more than a predetermined ratio, the tracking with the second reference frame as a starting point ends.
An apparatus according to claim 1. 23. The apparatus according to claim 21, wherein in tracking from the second reference frame as a starting point, the tracking is stopped when a predetermined reference frame is reached. 24. The anchor automatic setting means includes a moving picture reproducing means, an anchor information displaying means, and an anchor information correcting means, the moving picture reproducing means reproduces a moving picture on a screen, and the anchor information displaying means comprises: Anchor information that is obtained as a result of image matching that is sequentially performed for the displayed frame is displayed. When the anchor information correction operation is performed during moving image playback, the anchor information of the frame where the operation is performed is displayed. 20. The apparatus according to claim 18, wherein the correction is made and the result of automatic anchor setting in a predetermined frame before that frame is invalidated. 25. The anchor setting means includes a reference frame deleting means, wherein the reference frame deleting means determines that anchor information in a certain reference frame is within a predetermined error allowable value from interpolation calculation of anchor information in another reference frame. 25. The apparatus of any of claims 12-24, wherein the reference frame is demoted to a non-reference frame if required. 26. An anchor setting means including a text anchor setting means, wherein, when a text is specified for a reference frame, an anchor having the text as the content for the reference frame is provided. The apparatus according to any one of claims 1 to 25, wherein: 27. The device according to claim 1, further comprising cursor changing means for changing a display state of the cursor when the cursor enters an arbitrary anchor area. 28. The apparatus according to claim 27, wherein the cursor changing unit changes the display state of the cursor so as to display the identification information regarding the anchor area in which the cursor is inserted. 29. The apparatus according to claim 1, wherein the anchor setting means sets the anchor information in units of a group consisting of a plurality of anchors. 30. The apparatus according to any one of claims 1 to 29, further comprising anchor search means for searching for an anchor based on a feature of movement of the anchor in the anchor information. 31. The apparatus according to claim 1, wherein the anchor information editing unit displays a list of identification information of the anchor. 32. A frame determining means for determining a start frame and an end frame of a setting target period in which an anchor is to be set from a plurality of frames constituting an input moving image, and a reference frame taken within the setting target period. An anchor setting means for setting anchor information with the anchor area as an anchor area for the reference frame; and an anchor for estimating anchor information in a non-reference frame based on the set anchor information. An animation hyperlink comprising an estimating means, a link setting means for linking the set or estimated anchor information with any related data, and a link searching means for searching the related data linked to the anchor from any anchor. Media device. 33. The moving picture constitutes a video teaching material, the related data is additional information associated with a desired anchor of the video teaching material in a desired setting target period, and the device is used for producing an interactive video teaching material. 33. The device according to claim 32. 34. A moving picture providing system comprising a server and a client, wherein the server stores a moving picture, anchor information set for a reference frame of the moving picture, and related data linked to the anchor information. And means for estimating anchor information in the non-reference frame based on the anchor information set in the reference frame, and means for searching, from any anchor, related data linked to the anchor, When the user clicks on an object in the video, a means for determining which anchor area has been clicked is provided. When the client notifies the server of the clicked anchor, the server transmits relevant data linked to the anchor. Video providing system characterized by searching and sending this to the client Stem.