JP3093987B2 - Partial moving image related information retrieval method and system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for retrieving related information of a partial moving image in a moving image, and more particularly to a multimedia processing system having a basic function of retrieving a moving image and retrieving related information. The present invention relates to a method for retrieving partial moving image related information that is suitable for performing

[0002]

2. Description of the Related Art Conventionally, as a method for searching for information related to a partial moving image appearing in a stored moving image,
As disclosed in Japanese Unexamined Patent Publication No. 3-52070, “Related Information Reference Method of Moving Image”, information on a section of a frame including a target partial moving image, and the partial moving image on each frame in the section Information about an area surrounding an image is accumulated, and a section of a frame including a target partial moving image is extracted from a series of frames of a moving image, and the section is defined as a plurality of monotonous changes in the partial moving image. Is divided into
The region information surrounding the partial moving image is stored in both end frames of the partial section, and a pointer to related information of the partial moving image is added to the region information, and the related information is referred to by the pointer. There is something.

In the method disclosed in this publication, when a part of a displayed moving image is designated, area information of a partial section including the frame is listed up, and the area information of both end frames of each partial section is listed. A reference area relating to the frame is obtained by interpolation by linear prediction, a reference area including the indicated part is found, and related information is referred to by a pointer assigned to the area information.

[0004]

However, in the method disclosed in the above-mentioned publication, since a reference area related to the frame in which a part of the moving image is indicated is obtained by interpolation using linear prediction, the method is complicated. There is a problem that accuracy is poor for a partial moving image. In addition, there is no description of means for searching which frame contains data serving as a reference for interpolation when a search is performed in a certain frame.

A first object of the present invention is to accurately determine the position of a specified partial moving image even if one specified partial image moves in a displayed moving image. It is an object of the present invention to provide a partial moving image related information search method capable of predicting well and searching for information related to the partial moving image.

A second object of the present invention is to provide a partial moving picture related information search method capable of searching related information relating to a tracked partial moving picture at high speed.

[0007]

In order to achieve the first object, the present invention uses a non-linear function such as a spline function to accurately predict the position of a partial moving image.

In order to achieve the second object,
Data for retrieval is managed in a binary tree structure.

More specifically, in a system for retrieving related information related to a partial moving image in a moving image over a plurality of frames, position information in a plurality of frames for one partial moving image is input at arbitrary frame intervals, and A non-linear function for predicting position information of a frame other than the input frame is obtained, and the non-linear function, the number of a frame range using the non-linear function, and related information of a partial moving image whose position is predicted by the non-linear function are taken as one set. Registered and retrieves a nonlinear function paired with the number of the frame range that includes the frame number at the time of the designation operation, for the designation operation of a partial moving image in an arbitrary frame during playback of the moving image. The function predicts the position of the partial moving image in the frame, and the predicted position includes the position coordinates by the designated operation If that is characterized in that so as to output the related information that is on the non-linear function and set.

A plurality of non-linear functions are obtained for one partial moving image, and a binary tree structure based on a frame number in which a frame start number and an end number at which the plurality of non-linear functions are valid is arbitrarily determined. It is characterized in that it is managed.

[0011]

Embodiments of the present invention will be described below in detail with reference to the drawings.

In all the drawings for describing the embodiments, parts having the same functions are denoted by the same reference numerals, and their repeated description will be omitted.

FIG. 1 is a system configuration diagram showing an embodiment of a system for retrieving partial moving image related information in a moving image to which the present invention is applied. The system according to the present embodiment includes an auxiliary storage device 11, a computer 12, a display device 13, a keyboard 1
And a mouse (pointing device) 15.

The auxiliary storage device 11 stores the moving image file 1
1A, related information file 11B, motion information file 11
C is provided, an arbitrary moving image is stored in advance in the moving image file 11A, and related information on a partial moving image in the moving image stored in the moving image file 11A is registered in the related information file 11B. . On the other hand, in the motion information file 11C, information on the motion of the partial moving image generated in the process of executing the processing of the present system and the relevant information in the relevant information file 11B are registered in association with each other. It has become.

On the other hand, the computer 12 includes a central processing unit 12A, a main storage device 12B, and a moving image reproducing device 12C. The processing executed by the computer 12 is as follows.
Information related to the motion of the partial moving image and related information file 11
The motion information generation processing for registering the relevant information in B in association with the relevant information and, when a part of the moving image displayed on the display device 13 is instructed by the mouse 15, refer to the motion information file 11C. Is a process of determining which partial moving image has been instructed, reading related information corresponding to the partial moving image in the related information file 11B, and displaying it on the display device 13 (hereinafter referred to as a related information search process). There is.

The central processing unit 12A stores the moving image file 1
1A is read into the main storage device 12B as appropriate, a moving image is displayed on the display device 13 using the moving image reproducing device 12C, and the keyboard 14 and the mouse 15 are displayed.
The desired motion information generation processing and the related information search processing instructed using are executed, and the results are also displayed on the display device 13.

First, a procedure for inputting motion information of a partial moving image that moves in a moving image will be described with reference to FIGS.

FIG. 2 shows a state in which the butterfly 21 moves from the position A to the position C via the position B on a moving image obtained by shooting the flying of the butterfly 21. The purpose of the motion information input is to link the butterfly 21 with the data on the butterfly 21 in the related information file 11B.

First, the motion of the moving image is temporarily stopped at the first frame when the butterfly 21 is at the position A (step 401).
The circumscribed rectangle 22 surrounding the butterfly 21 is input as shown by the broken line (step 401). Here, ideally, if the circumscribed rectangle 22 of the butterfly 21 is input in every frame, the position of the butterfly 21 can be accurately recorded. However, since the moving image is composed of 30 frames per second, Even for a short moving image of 10 minutes, the number of frames is 1800 frames, and it is not practical to input the position of an individual object (partial moving image) for each frame.

Therefore, the frame of the moving image is moved forward and backward appropriately, and a sample frame (representative frame) on which the butterfly 21 is projected is selected (step 403). In the example of FIG. 2, the butterfly 21 is B,
A frame at the position of C is selected, the motion of the moving image is temporarily stopped at the frame where the butterfly 21 is located at these positions B and C, and the circumscribed rectangles 23 and 24 are input (step 404). ).

The circumscribed rectangles 22, 23 and 24 are shown in FIG.
As shown in the above, it can be represented by two coordinates on a diagonal line of (Xmin, Ymin) and (Xmax, Ymax).

The circumscribed rectangle 2 of the butterfly 21 in the representative frame
When the input of 3, 24 is completed (step 405),
Coordinates Xmin, Ymi of each circumscribed rectangle 22, 23, 24
A natural spline function is obtained for each of n, Xmax, and Ymax (step 406). Hereinafter, this function will be referred to as a trace function.

The coordinates of the circumscribed rectangle of the butterfly 21 in frames other than the representative frame (the frame when the butterfly 21 exists at the positions A, B, and C) are obtained by interpolation using a trace function.

FIG. 3B shows a frame number fn of a moving image.
And the coordinate Ymax. Position A is frame f0
And the position B is the position of the frame f
The position of the butterfly 21 at position i and the position C indicate the position of the butterfly 21 at the frame fn. In the present invention, f
The position of the butterfly 21 (the position of the circumscribed figure of the butterfly 21) in a frame other than 0, fi, and fn is obtained by interpolation using the trace function obtained in step 406.

The butterfly 21 in the f0, fi and fn frames
Once the trace function that satisfies the position of the tracing function is obtained, in order to check whether or not the position prediction accuracy of the tracing function satisfies a predetermined criterion, f0 to f
The position of the circumscribed figure of the butterfly 21 in the n-th frame is predicted and calculated, and the circumscribed figure is sequentially displayed at the predicted and calculated position, and the original moving image is superimposed and displayed (superimposed display).
Then, the accuracy of the interpolation is visually checked (step 40).
7). If the trajectory of the circumscribed figure of the butterfly 21 greatly deviates from the actual trajectory of the butterfly 21 and the accuracy is poor, the number of representative frames is increased and an appropriate trace function is selected again.

However, if the deviation between the trajectory of the circumscribed figure of the butterfly 21 and the actual trajectory of the butterfly 21 obtained by the interpolation operation is within an allowable range, it is determined that a predetermined accuracy is obtained, and the tracing function is determined. And the range of valid frames in which the trace function is valid and the butterfly 2 in the related information file 11B.
A set of IDs (identification numbers) of the related information related to No. 1 is registered in the motion information file 11C. The trace function is
Xmin, Ymin, Xmax, and Ymax form a set of four.
Further, the number of trace functions for one partial moving image is not limited to one. For example, the trace function t1 is appropriate from the 0th frame to the 100th frame, and the trace function t2 is appropriate from the 101st frame to the 500th frame. There are a plurality of trace functions suitable for tracking a target partial moving image as the moving image progresses. The range of the valid frame indicates a range of a frame in which the trace function tj is valid,
It is represented by a start frame number and an end frame number.

As described above, between the start and the end of one moving image, there are a plurality of trace functions suitable for tracking a specific object. If a user watching a moving image designates a specific target, a mechanism that can immediately determine which trace function should be used to track the target in that frame is required.

FIG. 5 shows an example of the effective range of a plurality of trace functions. As shown, trace functions t1 to t7 exist between "frame number 2000" and "frame number 8500". However, for example, when a specific object is designated by “frame number 5000”, the structure must be such that “valid trace function at frame number 5000 is t4”. Figure 6 shows the management structure of the trace function that satisfies the request.
Shown in

FIG. 7 shows the concept underlying the structure of FIG. 6, and the trace function groups t1 to t7 shown in FIG.
Are plotted on a graph by regarding the effective frame range as one point in a two-dimensional space called (start frame number, end frame number). The horizontal axis indicates the start frame number, and the vertical axis indicates the end frame number. Since the end frame is always after the start frame, the distribution is above the 45 ° straight line 701 which rises to the right.

FIG. 6 shows this space divided and expressed as a binary tree. The basis of the division is to repeat dividing the point data into two until the last point is reached. The division is repeated alternately for each axis of the start frame and the end frame. This division is indicated by d0 to d5 in FIG. The division of d0 corresponds to the node 601 in FIG. 6, and is divided depending on whether the start frame number is larger or smaller than "5500". That is, in the left sub-tree of the node 601, only the one whose start frame number is less than “5500” is set, and
Only those with more than 500 "exist.

The second division is for the end frame number. In the area where the start frame number is less than "5500", the area is "4400" and the start frame number is "550".
In the area of "0" or more, the area is divided by "7700". This division corresponds to nodes 602 and 603 in FIG. 6, respectively.

In the following, division is performed until the number of elements in the region becomes one while changing the axis sequentially. In the tree, there are leaf nodes 607 to 613, and this leaf node 60
Records 7 to 613 store the trace functions, their start / end frames, and IDs in the related information file 11B.

The solid arrows in FIG. 6 indicate pointers to lower nodes. In addition, the binary tree structure of FIG. 6 exists one by one for each partial moving image. For example, when the moving image includes n or more (n ≧ 2 integers) target partial moving images, The n binary tree structures are created and registered in the motion information file 11C.

Next, a method of obtaining an effective trace function on a frame when a certain frame number is designated will be described.

If a certain number is N, this search can be rephrased as "find a trace function whose start frame is smaller than N and whose end frame is larger than N". For example, for N = 5000, the shaded area 702 in FIG.

FIG. 8 is a flowchart of an algorithm for tracing the binary tree of FIG. 6 and finding an effective trace function. The algorithm of FIG. 8 is recursive, a node N to be traced, A indicating whether the node N indicates division for a start frame or end frame, and a frame number F As three arguments.

First, in step 801, if the argument node N is a leaf node, that is, a record, step 8
At 09, it is determined whether or not the trace function in the node N is valid at the frame number F of the argument. If valid, the trace function IDtn is output (step 80).
1), end. If not valid, the search ends.

On the other hand, if the node N is not a leaf node in step 801, it is determined in next step 802 whether the argument A is a start frame or an end frame.

If the argument A is the start frame, call itself by using A as the end frame, the frame number F, and the left child pointer of the node N as arguments (step 8).
03). If the boundary value of the division in the node N is equal to or smaller than F (step 804), A is set as the end frame, and the frame itself is called with the frame number F and the right child pointer of the node N as arguments (step 805). ), End the search. If the boundary value of the division in the node N is larger than F in step 804, the search ends.

If the argument A is the end frame in step 802, the self is called using A as the start frame, the frame number F and the right child pointer of the node N as arguments (step 806). Then, if the boundary value of the division in the node N is larger than F (step 8)
08), A is set as a start frame, and the self is called using the frame number F and the left child pointer of the node N as arguments (step 808), and the search ends. Step 807
If the boundary value of the division in the node N is equal to or smaller than F, the search ends.

Next, the trace function described above, its start / end frames, and I
An algorithm for searching for relevant information of a partial moving image in a moving image specified by a user in an arbitrary frame using a data structure storing D will be described with reference to the flowchart of FIG.

First, when the motion of the moving image is temporarily stopped on the screen on which the object to be searched for appears, the frame number F is obtained (step 901). Next, if the user designates a specific object (for example, a partial moving image such as the butterfly 21) on the screen by clicking with the mouse 15, the designated position coordinates are obtained (step 902). .

The algorithm shown in FIG.
= Start frame, F = frame number obtained in step 901, N = call with root node of tree structure managing trace function as arguments (step 903).

Next, a circumscribed rectangle in the frame F is obtained using the trace function obtained in step 903 (step 90).
4) Further, a circumscribed rectangle including the position coordinates acquired in step 902 is obtained (step 905).

Next, the related information file 11B is searched with the ID of the related information linked to the circumscribed rectangle obtained in step 905, and desired data relating to the partial moving image specified by the user in the frame F is extracted. Display on
(Step 906).

The related information includes the name, shape,
Various information such as nature, history, generation process. Also,
The information in this case includes not only characters but also sound and video information.

As described above, for one partial moving image,
A plurality of trace functions are obtained, and a frame start number and an end number for which the plurality of trace functions are valid are managed by a binary tree structure with reference to a frame number arbitrarily determined. For the operation of specifying a partial video, a trace function paired with the number of the frame range including the frame number at the time of the specification operation is extracted, and the position of the partial video in the frame is predicted by the trace function. When the predicted position includes the position coordinates obtained by the specifying operation, the related information paired with the trace function is searched and output, so that the specified moving image is displayed in the displayed moving image. Even if one specified partial image moves in a complicated manner, the position of the specified partial moving image is accurately predicted, and It is possible to search for that information. Further, it is possible to create an index of related information for a partial moving image in a moving image with a small amount of labor.

Furthermore, by registering the index for each partial moving image created in this way in the tree structure, high-speed search can be performed even if the number of target partial moving images increases.

Therefore, for example, as shown in FIG. 10, a moving image viewed from the viewpoint inside a vehicle running on a road is recorded, and road management facilities 101, 102 such as signs, traffic lights, and pedestrian bridges appearing in the moving image are recorded. , Their respective facilities 101,
If the information on the position where 102 is displayed and the related information such as the distance (distance mark) from the reference point of the facility, the construction date, the height, and the like are stored in the related information file 11B, When the equipment 101 is designated with the mouse 15 while the moving image viewed from the inside of the car is being reproduced, the equipment 1 is displayed.
Information such as the construction date related to 01 can be displayed in the window 103 in the display screen. This allows
It is possible to construct a system for checking the related information of a large number of road facilities while watching the video.

One or more moving images are stored, and a function of reproducing an arbitrary moving image in the moving images from an arbitrary frame is provided, so that a partial moving image appearing in the moving image is provided. The information related to the position where each partial moving image is displayed is associated with information indicating which frame of which moving image is to be reproduced next when the partial moving image is designated, and the related information file 11B To be stored. For example, as shown in FIG.
In addition to accumulating a moving image obtained by photographing the scenery in which is displayed, information relating to the position where the building 111 is displayed is associated with a frame in which the details of the building 111 are reflected and accumulated. Then, the building 111
Is designated by the mouse 15, an image showing details of the building 111 can be reproduced and displayed as shown in FIG. 11B.

Thus, it is possible to construct a system that allows a user to refer to a detailed image of a desired object designated by the user himself / herself among buildings, plants, and the like appearing in a landscape image. it can.

Although the related information is registered by ID in the binary tree structure of FIG. 6, the related information itself may be registered.

[0053]

As described above, according to the present invention, even if one specified partial image moves in a displayed moving image, the specified partial moving image Can be accurately predicted and information related to the partial moving image can be searched. Further, it is possible to create an index of related information for a partial moving image in a moving image with a small amount of labor.

Further, by registering the index of each partial moving image created in this way in a tree structure, it is possible to obtain an effect such that a high-speed search can be performed even if the number of target partial moving images is large. This is extremely effective when constructing a multimedia system having image information search and related information search as basic functions.

[Brief description of the drawings]

FIG. 1 is a system configuration diagram showing an embodiment of the present invention.

FIG. 2 is an explanatory diagram showing a specific example of a motion of a partial moving image.

FIG. 3 is an explanatory diagram showing a change in position coordinates over a plurality of frames of the partial moving image in FIG. 2;

FIG. 4 is a flowchart illustrating a user procedure for linking a subject to be captured in a moving image with related information.

FIG. 5 is a diagram showing a relationship between a trace function and its effective range.

FIG. 6 is a diagram showing a specific example of a tree structure for managing a trace function.

FIG. 7 is a diagram showing an effective range of a trace function on a two-dimensional space.

FIG. 8 is a flowchart for obtaining a valid trace function in a frame of a designated number.

FIG. 9 is a flowchart illustrating a user search procedure.

FIG. 10 is a diagram showing an application example of the present invention.

FIG. 11 is a diagram showing another application example of the present invention.

[Explanation of symbols]

11: auxiliary storage device, 11A: moving image file, 11B
... Related information file, 11C ... Motion information file, 12
... Computer, 12A ... Central processing unit, 12B ... Main storage device, 12C ... Moving image reproducing device, 13 ... Display device, 1
4 ... keyboard, 15 ... mouse.

Continuation of the front page (56) References JP-A-3-52070 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 5/76-5/956 G06F 17/30 G06T 1 / 00 G06T 7/20

Claims (5)

(57) [Claims] 1. A system for retrieving related information relating to a partial moving image in a moving image over a plurality of frames, comprising: inputting position information in a plurality of frames for one partial moving image at an arbitrary frame interval ; A non-linear function for predicting position information of a frame other than the frame is obtained, the non-linear function, the number of a frame range using the non-linear function, and relevant information of a partial moving image whose position is predicted by the non-linear function. the registered in to storage means a set
Acquiring a non-linear function paired with a frame range number including a frame number at the time of the designation operation of a partial moving image in an arbitrary frame during the reproduction of the moving image from the storage means. Steps and actions
Predict the position of the partial moving image in the frame using the obtained nonlinear function, and output the relevant information paired with the nonlinear function when the predicted position includes the position coordinates by the designated operation. A method for retrieving partial video related information, characterized by 2. A plurality of non-linear functions are obtained for one partial moving image, and a binary tree structure based on a frame number that arbitrarily defines a frame start number and an end number at which the plurality of non-linear functions are valid is defined. 2. The method for retrieving partial moving image related information according to claim 1, wherein the information is managed. 3. The related information is registered by an identification number, and the related information is searched for from the related information file by the identification number.
Or the partial moving image related information search method described in 2. 4. The non-linear function displays a predetermined symbol at a position of a partial moving image predicted by the non-linear function, and superimposes and displays an original moving image on the symbol, so that prediction accuracy is a predetermined standard. 4. The partial moving image related information search method according to claim 1, wherein the method is registered after checking whether or not the condition is satisfied. 5. A part in a moving image over a plurality of frames.
In a system that searches for related information related to moving images
Te, the position information in the plurality of frames for one partial moving picture
Input means for inputting information at arbitrary frame intervals, and input
Nonlinear predict the location information of other frames other than frame
A function is obtained, and the nonlinear function and a file using the nonlinear function are obtained.
Predict position with frame range number and the nonlinear function
Register a set of related information of partial moving images in the storage unit
Means and partial motion in an arbitrary frame during video playback.
Frame number at the time of specifying operation for image specifying operation
Non-linear paired with the number of the frame range that contains
Reading the shape function from the storage means;
Predict the position of the partial video in the frame with a linear function
And the predicted position contains the position coordinates from the designated operation
The relevant non-linear function
Means for outputting information.
Image related information retrieval system.