JP2002359777A - Time space region information processing method and time space region information processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a time space region information processing method that can efficiently apply input and edit to time space region information. SOLUTION: A video information storage section 1 stores video data. A time space region information storage section 2 stores time space region information describing information able to be identified as to a locus of each representative point in a time base direction for each representative point of a prescribed approximated graphic denoting an optional area as expression of transition of the area in video data over a plurality of frames. A time space region information edit processing section 5 obtains a locus for each representative point based on the time space region information of an edit object, a display section 3 displays a locus of the representative points in a way of a three-dimensional space, and an entry section 4 enters a prescribed correction operation with respect to the shape of the locus in three-dimensional space display to correct the object time space region information based on the entered correction operation.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention displays an arbitrary spatio-temporal region shape in an image in a three-dimensional space of X, Y, and time.
The present invention relates to a spatio-temporal region information processing method and a spatio-temporal region information processing system, which are configured to input and correct spatio-temporal region information by operating a displayed shape in a three-dimensional space.

[0002]

2. Description of the Related Art In recent years, with the rapid development of video image processing technology, it has become common to handle video and images as digital data. This digitization has established a technique for efficiently compressing video image data having a large data amount. Also, with the development of the Internet, network technology such as satellite broadcasting and CATV, it has become possible to handle a large amount of video data, and video and image information is stored and retrieved and used according to needs. Video image databases and video-on-demand are about to enter practical use. In addition, remote monitoring systems from remote locations have become major. When trying to use video and images in this way, there is a demand for recognizing what is on the screen, searching for and extracting video that contains the desired object, and classifying the video. .

[0003] In order to respond to such a demand, a technique for efficiently describing arbitrary spatio-temporal area information over a plurality of frames in a video has been already proposed (for example, Japanese Patent Laid-Open Publication No. HEI 2 (1999) -197686).
001-1118075, JP-A-2001-1119
No. 96).

[0004]

However, when new spatio-temporal domain information is to be input, or when existing spatio-temporal domain information is to be edited, numerical information is input, or input / edit is performed in frame units. Therefore, it has been difficult to input or edit spatio-temporal region information while considering the shape of the preceding and following frames and the spatio-temporal region in all time.

The present invention has been made in view of the above circumstances, and has a spatio-temporal area information processing method capable of efficiently inputting / editing spatio-temporal area information while referring to the whole or a part of the spatio-temporal area shape. And a spatio-temporal area information processing system.

[0006]

SUMMARY OF THE INVENTION According to the present invention, there is provided a spatio-temporal system capable of specifying a locus of a representative point of an approximate figure representing an area described in a video data as representing a transition over a plurality of frames of the area. Targeting region information, a spatio-temporal region information processing method for performing a process of correcting the spatio-temporal region information, based on the spatio-temporal region information of the target region, the representative point of the Obtain a trajectory, the obtained trajectory of the representative point is displayed on a screen, accepts an input of a correction operation of the shape of the trajectory displayed on the screen, based on the input correction operation, the It is characterized in that spatio-temporal area information is corrected.

According to the present invention, the trajectory of the representative point is displayed in the three-dimensional space of X, Y, and time, and the user has the function of operating the displayed trajectory of the representative point in the three-dimensional space. By manipulating the trajectory of each representative point in the dimensional space, spatio-temporal area information can be input and corrected.

Preferably, when the spatiotemporal area information is corrected based on the correction operation, the locus of the representative point is displayed on the screen again based on the corrected spatiotemporal area information. Further, the predetermined correction operation may be input.

Preferably, when displaying the trajectory of the representative point on a screen, a correction operation point is displayed at a predetermined position on the trajectory, and a correction operation performed on the correction operation point is performed. May be received, and the spatiotemporal area information may be corrected based on the input correction operation.

Thus, by manipulating the points in the three-dimensional space, it is possible to input and correct spatio-temporal area information expressed by the locus of the representative points interpolated by the function.

Preferably, the correction operation is a movement in any direction with respect to any of the displayed correction operation points, addition of a new correction operation point, or addition of an existing correction operation point. It may be deleted.

Preferably, when displaying the trajectory of the representative point on a screen, a predetermined frame of the video data may be displayed together.

[0013] This makes it possible to input and correct spatio-temporal region information while referring to video data.

[0014] Preferably, in displaying a predetermined frame of the video data, the approximate figure in the frame to be displayed may be displayed in different display modes inside and outside the approximate figure. Preferably, a display form for the inside of the approximate figure and a display form for the outside may be different from each other in the presence / absence of processing on the video data or the content of the processing on the video data.

Preferably, the information capable of specifying the trajectory of the representative point is such that the trajectory obtained by arranging the position data of the representative point along the progress of the frame is approximated by a predetermined function, and the parameter of the function is used. May be described. Preferably, the information that can specify the trajectory of the representative point is obtained from a position data in a frame serving as a reference of the representative point, a conversion parameter indicating a region conversion to position data in another frame, and the conversion parameter For, the trajectory obtained when the conversion parameters are arranged along the progress of the frame may be approximated by a predetermined function and described using the parameters of the function. Also, preferably, for at least one of the representative points, instead of using the position data for the representative point, a relative value of the representative point with respect to another representative point in the same frame is used. Data indicating the position may be used.

Preferably, when displaying the trajectory of the representative point on a screen, the trajectory is three-dimensionally displayed in a form in which the trajectory is arranged in a three-dimensional space including a two-dimensional coordinate axis and a time axis relating to the frame. It may be.

The present invention relating to the apparatus is also realized as an invention relating to a method, and the present invention relating to a method is also realized as an invention relating to an apparatus. Further, the present invention according to an apparatus or a method has a function of causing a computer to execute a procedure corresponding to the present invention (or causing a computer to function as means corresponding to the present invention, or a computer having a function corresponding to the present invention). The present invention is also realized as a program (for realizing), and is also realized as a computer-readable recording medium on which the program is recorded.

According to the present invention, by displaying an arbitrary spatio-temporal region of a plurality of frames in video data and an image of an arbitrary frame of video data in a three-dimensional space in addition to this, a spatio-temporal region shape is obtained. This makes it possible to easily grasp the relationship between itself and the video data, thereby making it possible to efficiently edit the spatiotemporal area information.

[0019]

Embodiments of the present invention will be described below with reference to the drawings.

A spatio-temporal area information processing system according to an embodiment of the present invention is a system having a function for a user to perform processing such as editing on spatio-temporal area information.

First, spatio-temporal area information will be briefly described.

The contents of the spatiotemporal area information, the generation method,
For the usage method and the like, see, for example, JP-A-2001-11
No. 8075, Japanese Patent Application Laid-Open No. 2001-111996 and the like (the object region data disclosed therein corresponds to one form of the spatiotemporal region information).

First, the contents of the spatio-temporal area information and the method of generation will be briefly described.

The spatio-temporal area information includes a spatio-temporal area (for example, a two-dimensional area in each frame) from appearance to disappearance of a specific area provided on a display screen (for example, a GUI screen) for various purposes. (Time transition). In addition, the spatio-temporal area information is usually composed of video (or moving image) data (the content of the video data itself is a real photograph, a processed real photograph, a CG, an animation, a combination thereof, and the like). Content may be included). The description will be made on the assumption that the moving image data includes a plurality of frames.

This specific area can be used, for example, as an area indicating a specific object existing in video data. In this case, the shape of the specific region in a certain frame represents (or approximates) the specific object region in the frame. Here, the object region is a group of regions in a video, for example, a person, an animal, a plant, a car, a building, a road, a river, the sun, a cloud, or the like (or a part thereof, for example, a human head or a car). Hood, building entrance, etc.), and anything that can be grasped as an object. Further, it may be an independent thing, a part of a thing (for example, a human head, a car hood, a building entrance), or a set of things (for example, a group of birds and fish). .

Next, generation of spatio-temporal region information for one object region will be briefly described with reference to FIGS.

First, the generation of the spatio-temporal region information can be performed roughly by the following procedure, for example.
In addition, any of the procedures can be automated, and it is also possible to intervene manually by a user. (1) Extract an object region of a target object from a predetermined frame of video data. (2) The object area is approximated by a predetermined figure. (3) Extract a representative point specifying the approximate figure. 1 to
The procedure of 3 is performed for all frames or a plurality of sampled frames from a frame in which an approximate figure to the object area appears to a frame to disappear. (4) For each representative point, the time series of the position (or the amount that can specify the position) is represented by a function (approximate function) such as a time t (for example, a time stamp given to a video) or a frame number f. Approximate expression. The value of the parameter of the function when this approximation is expressed is the value to be obtained. This function is separately expressed for each representative point and separately for the X coordinate and the Y coordinate.

Steps 1 to 4 are performed for each object region to be processed.

There are various types of approximate figures, for example, a polygon in which the number of sides is a parameter, a polygon such as a rectangle in which the number of sides is fixed, a circle, an ellipse, and the like. The representative points include, for example, vertices of a polygon, four or three of a rectangle.
One vertex, one point on the center of the circle or both ends of the diameter, four or three vertices of the circumscribed rectangle of the ellipse or 2
There are various types depending on the type of figure, such as one focal point and one point on an ellipse.

By storing the parameters of this function,
The function is obtained from the parameters, the XY coordinate values of each representative point at a desired time t or frame number f are obtained based on the function, and the XY coordinate value of each representative point is used to calculate the desired time. The area of the approximate figure at t or frame number f can be obtained.

FIGS. 1 and 2 show a case where a polygon is used as an approximate figure for an object area, a representative point is a vertex of the polygon, and a quadratic polynomial spline function is used as an approximate function for a locus of each vertex in the time axis direction. This is an example in which spatio-temporal region information for an object “fish” in an image is generated.

In FIG. 1A, reference numeral 200 denotes one frame in a video to be processed. Reference numeral 201 denotes a region of the object to be extracted. 202
Indicates an approximate polygon approximating the region of the object.

FIG. 1 (b) shows a near frame over a plurality of frames.
Representative point of similar figure, that is, approximate polygon 2 in this example
02 and its vertices, one of them
(V ₀) Is an approximation curve.

Reference numeral 204 in FIG. 1C is an example of a function obtained for the representative point V ₀ (here, the reference representative point V _0).
Is shown only for one coordinate axis). This example
The case where the approximation section is divided into t = 0 to 5 and t = 5 to 16 is shown.

FIG. 2 shows an example in which a function that approximates the value of the X coordinate of the representative point V ₀ is obtained. 211 in the figure represents the time interval that exists in the object, black dots (212) is the value of X-coordinate of the representative point V _0. 213 is the approximate function. An approximate function is similarly obtained for the Y coordinate. Since a polynomial spline function is used as the approximation function, a polynomial is defined for each time section divided by a point called a node at 204 in FIG. Here, t = 0, 5, and 16 are the node times, respectively.

The same applies to representative points other than the representative point V ₀ .

On the other hand, the representative point V₀About representative points other than
Means that the representative point is relative to another representative point.
Relations, such as the difference vector
There is also a method of describing by the trajectory of the file. Figure 3 shows an example
As a reference representative point V in the approximate polygon
₀And each of the difference vectors to represent the other representative points
FIG. Black dot sequence (222) in FIG.
Is the vector V at each time _0,1Represents the value of the X component of
ing.

In the above description, the position of the representative point in each frame or the locus of the difference vector is approximated. However, the position or the difference vector of the representative point in a certain reference frame is replaced with the position or the difference vector of the representative point in each frame. There is also a method of approximating the locus of the parameter of the function to be converted. Alternatively, there is a method of approximating a locus of a parameter of a function for converting a position or a difference vector of a representative point in a preceding frame into a position or a difference vector of a representative point in a subsequent frame.

In addition, the form of the spatio-temporal area information includes:
There are various variations, and the present invention is applicable to any form of spatiotemporal domain information. In addition, the spatio-temporal area information includes, for example, for each representative point corresponding to each object area and for each frame, or for each object area and for each frame, or for each representative point of each object area, or for each representative point. Predetermined attribute information or related information may be added in a predetermined unit such as for each object region. The present invention is also applicable to such spatiotemporal domain information.

FIG. 5 shows an example of the data structure of the spatio-temporal area information used in the description of this embodiment. As shown in FIG. 5, the spatio-temporal area information of the present example includes spatio-temporal area information identification information (hereinafter, ID number) 101, start time 102, end time 103, shape flag 104, and trajectory data 105.

The ID number 101 is an identification number given to each space-time area.

The start time 102 and the end time 103 are the start and end times at which the spatiotemporal area of the ID number exists. The start time and the end time may be represented by a time stamp or a frame number.

The shape flag 104 indicates how the region shape is represented. Since the region shape is represented by, for example, a rectangle, an ellipse, and a polygon,
Information that uniquely distinguishes them is described. In the case of a polygon, vertex number information is also added to the shape flag.

The trajectory data 105 is parameter data for expressing a spatio-temporal region. This is such that the spatio-temporal region shape from the first time to the last time can be determined. For example, in the case where the region shape is a rectangle or a polygon, a parameter when the locus of each vertex is approximated by a function,
In the case where the region shape is an ellipse, it is a parameter or the like when a locus of a vertex of a circumscribed rectangle of the ellipse is approximated by a function. 10
As shown in FIG. 6, since the trajectory data 105 describes the X-coordinate trajectory and the Y-coordinate trajectory separately for one representative point, the trajectory data has the number of representative points × 2.

FIG. 6 shows an example of the data structure of each trajectory data 105 in the spatiotemporal area information of FIG. This is for representing a continuous trajectory interpolated by a function such as a spline function, and is for storing the relationship between time and parameters such as X coordinates and Y coordinates.

The number of nodes 151 represents the number of nodes of the spline function, and the number of nodes minus one polynomial data 156
It indicates that there is. Since the spline function is expressed by a different polynomial for each section between adjacent nodes, a number of polynomials corresponding to the number of nodes is required. Therefore, data 156 including a node frame number, a polynomial coefficient, and the like are repeatedly described.

The head node time 152 indicates the time of the first node of the spline function.

The node time 153 indicates the node time at the end of the polynomial data, and indicates that the polynomial data 156 is valid up to this node time. This may be described by a time stamp instead of the frame number. The number of coefficient data of the polynomial varies depending on the highest order of the spline function (where K is the highest order, the number of coefficient data is K + 1). Therefore, the number of coefficient data is stored in the polynomial degree 154. Polynomial degree 15
4 is followed by a number of polynomial coefficients 155 corresponding to the polynomial degree + 1.

Next, a method of using such spatio-temporal area information will be briefly described.

In a system having a function of displaying video data and a function of using spatio-temporal area information attached to the video data, each representative point of an approximate figure of an object area in a target frame based on the spatio-temporal area information (1
There may be a plurality of approximate figures of the object area in one frame), an area of the approximate figure in the frame is obtained from each representative point of the approximate figure of the object area, and the approximate figure of the approximate figure is displayed on the video data. The external shape is displayed, or the range corresponding to the inside of the approximate figure in the video data is displayed in a specific display form, or the user instructs the video being displayed on the screen with a pointing device such as a mouse. In such a case, for example, if an approximate figure of the object area exists at that time (field) and the designated position is inside the approximate figure, it is determined that the target object has been designated and some processing is performed. -A user interface (GUI) can be provided.

For example, when related information (for example, characters, sounds, still images, moving images, etc., or a combination thereof as appropriate, or pointer information indicating the storage address thereof) is added to the spatio-temporal area information, or When there is a database including related information related to the information of each ID number separately from the spatial area information, it can be used for hypermedia and object search. For example, when the user designates the object (or the object and its vicinity) with a mouse or the like, it is determined whether the designated time / place is inside or outside the object area, and is determined to be inside. Then, related information on the object can be searched or displayed. For example, if the video is a content such as a movie, the object is a character such as an actor or other appearing object, and the related information is a description about the actor or its role, etc.
A description of the desired actor can be viewed by simply clicking on the actor's image. Similarly, the present invention can be applied to any electronic contents such as an electronic encyclopedia, an electronic catalog, and the like.

The related information may be, for example, data describing the operation of a program or a computer, or pointer information indicating its storage address. In this case, the user can cause the computer to perform a predetermined operation by designating the object.

For example, in the object search, it is possible to search for an object that satisfies the conditions by using a passing key, a non-passing position, a size at a certain position, a staying time at a certain position or the like as a search key. For each of the search keys, the coordinates of the representative point are sequentially extracted for the time section in which the object exists, and it is determined whether an arbitrary point is inside or outside the figure composed of the representative points, or the area is calculated. Thus, it can be determined whether or not the condition is satisfied. Further, for example, if a keyword is described in the related information, the object can be searched by a keyword. Furthermore, if feature amounts such as shape, texture, activity, and color extracted from the object are described in the related information, an object can be searched based on such feature amounts. Further, for example, it is possible to realize a monitoring system that monitors a suspicious person or the like based on feature amounts such as shape, texture, activity, and color of an object obtained by analyzing spatiotemporal region information. .

Next, a method for providing video data and spatio-temporal area information will be described.

When the spatio-temporal region information created by the creator is provided to the user, it is necessary for the creator to provide the spatio-temporal region information to the user in some way. Various forms of this providing method can be considered as exemplified below. (1) A form in which video data and its spatio-temporal area information are provided simultaneously (2) A form in which video data is provided independently and spatio-temporal area information is separately provided (3) The method of (1) or (2) above To provide video data and its spatio-temporal area information, and to provide related information at the time of providing either of (1) or (2) above, or independently of them. As means for performing this, for example, a method of providing the information by recording it on one or a plurality of recording media, or providing the information via a network or the like (or acquiring by the user)
There are methods and methods of using them in combination.

The spatio-temporal area information processing system according to the present embodiment can be mounted on a creator's system or a user's system. In addition, it can be mounted on a system having a function of generating time domain information, and the time domain information generated by the system can be edited, or a system or a time domain information having a function of generating time domain information can be used. May be installed in a system that does not have a function of generating the time domain information, and time domain information read from outside may be targeted for editing.

Now, the spatio-temporal area information processing system of the present embodiment will be described in detail.

Here, a case where the spatio-temporal area information shown in FIGS. 5 and 6 is to be edited will be described as an example.

FIG. 7 shows a configuration example of the spatiotemporal area information processing apparatus according to the present embodiment. As shown in FIG. 7, the spatio-temporal region information processing apparatus includes a video information storage unit 1, a spatio-temporal region information storage unit 2, a display unit 3, an input unit 4, and a spatio-temporal region information editing processing unit 5. I have.

This spatio-temporal area information processing apparatus can be realized by executing a program on a computer. It is also possible to incorporate the program as a function of another software. If necessary, the computer is equipped with an OS having desired functions, driver software, software for packet communication, encryption software, and the like, or hardware such as a communication interface device, an external storage device, and an input / output device. Alternatively, they can be connected.

The video information storage unit 1 stores video data, for example, a hard disk, an optical disk,
It is composed of a semiconductor memory or the like.

The spatio-temporal area information storage unit 2 stores, for example, a rectangle circumscribing the locus of each vertex of an approximate rectangle or an approximate polygon indicating an object area or an approximate ellipse in the video data stored in the video information storage unit 1. Spatio-temporal region information (represented by a locus of each vertex or the like) is stored. The spatio-temporal area information storage unit 2 is composed of, for example, a hard disk, an optical disk, a semiconductor memory, or the like. As previously mentioned,
Here, as the spatio-temporal area information, information having a data structure as shown in FIGS. 5 and 6 is exemplified.

Note that the video information storage unit 1 and the spatiotemporal area information storage unit 2 may exist on the same physical device or on different physical devices.

The spatio-temporal area information editing processing unit 5 performs processing relating to display of video information and spatio-temporal area information, processing relating to correction of spatio-temporal area information, and the like.

The display unit 3 is for displaying video information, spatio-temporal area information, and the like.
This is for inputting an instruction input, a selection input, and the like from the user. Here, the display unit 3 and the input unit 4 constitute a graphical user interface.

As will be described in detail later, the spatio-temporal area information is
As the trajectory of each representative point of the figure indicating the object area (for example, the trajectory of each vertex of the approximate rectangle or approximate polygon indicating the object area or each vertex of the rectangle circumscribing the approximate ellipse), ｛X,
It is arranged and displayed in a three-dimensional space of Y, time｝. In addition, the video information corresponding to the spatiotemporal area information is appropriately
The images of all or some of the frames are displayed in a three-dimensional space with or without processing (it is also possible not to display).

The spatio-temporal area information processing apparatus may further include a spatio-temporal area information generation processing unit 6 for generating spatio-temporal area information in addition to the configuration shown in FIG. FIG. 8 shows a configuration example in this case. The spatiotemporal area information generation processing unit 6
For example, a configuration disclosed in JP-A-2001-1118075 or JP-A-2001-111996 can be used.

In displaying the spatio-temporal area information on the display unit 3,
Although it is necessary to display a three-dimensional space, a process using a technology capable of displaying a three-dimensional space, such as a holographic technology or a virtual reality technology, can be used for the process for this display. Alternatively, a three-dimensional space may be projected on a two-dimensional plane to be displayed on a normal display or the like. However, in this case, since hiding due to projection or the like may occur, it is preferable to provide an interface for moving a viewpoint or a gazing point.

The input unit 4 can use a keyboard or a mouse. It is also possible to use an input device such as a three-dimensional mouse operated in space.

FIG. 9 shows an example of a display method relating to spatio-temporal area information and video information.

The spatio-temporal area information editing processing unit 5 obtains the trajectory of the representative point of the figure representing the object area based on the spatio-temporal area information to be edited, and displays it as a curve like the representative point trajectory 161. The display form of the representative point trajectory 161 may be, for example, displayed in a different color for each representative point trajectory, or may be displayed by being divided by a line type such as a solid line or a dashed line. It may be set, or the user may set it arbitrarily.)

Further, the spatiotemporal area information editing processing section 5 sets the node coordinate 1 as a point for specifying the representative point locus 161.
62 and the control point 163 are displayed. The node coordinates 162 are those described in the spatio-temporal area information. The control point 163 is a point located on the representative point trajectory 161 other than the nodal coordinates, and is calculated based on the spatio-temporal area information according to how the control point 163 is provided (how the control point 163 is determined). Will be described later).
It is preferable that the node coordinates 162 and the control point 163 are displayed in a color or a shape that can be easily distinguished. FIG.
, The node coordinates 162 are indicated by “•”, and the control points 163 are indicated by “x” (when they are distinguished and displayed, any display form is possible as long as they can be distinguished). Note that there is a method in which all or a part of the node coordinates 162 and the control points 163 are not displayed. In addition, the user may be able to switch whether or not to display the node coordinates 162 and the control points 163, or the display method when displaying them.

The node coordinates 162 and the control points 163 are for the user to input the contents of correction for the spatio-temporal area information. In this spatio-temporal area information processing system, the user wants to correct the spatio-temporal area information. In such a case, drag and
The node coordinates 162 and the control point 163 are arbitrarily selected by a drop operation or the like, and may be arbitrarily selected in any direction (one dimension in the X direction, the Y direction, or the time direction, or any two dimension, (It may be three-dimensional.)
Is moved by an arbitrary amount (that is, an arbitrary movement vector is input), thereby inputting a correction instruction for the spatiotemporal area information. When the node coordinates 162 or the control points 163 are moved by a drag-and-drop operation or the like, the node coordinates 162 or the control points 163 may be selected and moved one by one, or a plurality of the node coordinates 162 and / or It may be possible to select the control points 163 and move them collectively.

If all or part of the node coordinates 162 and the control points 163 are not displayed, for example, the node coordinates 162 or the control points 163 closest to the position specified by the user using a mouse or the like are selected. It should be treated as something.

In the spatiotemporal region information processing system, the node coordinates 162 and the control points 163
Will calculate the spatio-temporal area information corresponding to the position after has been moved by the user.

As shown in FIG. 6, one representative point locus is represented as a set of interpolation functions from one node time to the next node time. Therefore, in order to specify one representative point trajectory, it is only necessary to specify all the node times and the parameters of the interpolation function between the nodes.

A certain representative point trajectory is represented by (X, Y, time) = (fx (t), fy (t), t) (where fx (t), fy (t) is If the node time is t0, t1,..., Tk, the node coordinates 162 of this representative point locus
Are (fx (t0), fy (t0), t0), (fx (t1), fy
(T1), t1), ..., (fx (tk), fy (tk), t
k).

The node coordinates 162 represent the coordinates of the representative point locus at the node time. From the position of the node coordinates 162, the user can know the node time and the coordinates at the node time.

When the interpolation function between the nodes is interpolated by a linear function, the shape of the representative point locus 161 can be uniquely specified by displaying the coordinates of the node coordinates 162 (note that the coordinates of the node coordinates 162 can be uniquely specified). Control point 16
If only the minimum number of 3s is displayed, the control points 163 will not be displayed if the interpolation function is a linear function.)

However, when the interpolation function between nodes is interpolated by a function of second order or higher, the shape of the interpolation function cannot be uniquely specified only by the node coordinates 162. In such a case, the control point 163 is displayed so that the shape of the interpolation function can be uniquely specified.

The control point 163 may be displayed at any coordinates as long as the shape of the interpolation function can be uniquely determined and can be easily understood by the user. For example, when interpolation is performed using an n-th order polynomial, the shape of the complementary function can be uniquely determined if n + 1 coordinates are known. Since the coordinates at the start time and the coordinates at the end time of the interpolation function are indicated by the node coordinates 162, the other n−
By displaying one coordinate as a control point, an interpolation function shape can be uniquely determined.

There are various variations on how to set the (n-1) coordinate positions.
If the control points are too close, not only is it difficult for the user to understand, but also it becomes difficult to operate the control points. For example, the control points may be arranged by equally dividing the start time from the end time.

For example, it is assumed that the time of a frame in which a certain node coordinate exists (hereinafter, referred to as a key frame) is t1, and the time of the next key frame is t2. t1 to t2
, The representative point trajectory is represented by (X, Y, time) = (fx (t), fy (t), t). Where fx (t) and fy (t)
Is an n-order polynomial at time t. At this time, the coordinates 162 in the key frame are (fx (t1), fy (t1), t1) and (fx (t2), fy
(T2), t2). The control point 163 requires n-1 points, and if the coordinates are determined by equally dividing t1 to t2,
The coordinates of the control point 163 are (fx ((t2−t1) * i / n + t1), fy ((t2−t1) * i / n + t
1), (t2-t1) * i / n + t1) (i = 1,2, ..., n
-1)

As a method of arranging the control points, there is a method of arranging the control points more densely in a portion where the curve changes rapidly, in addition to the method of equally dividing the control points.

It is to be noted that the number of control points to be arranged may be set to a number larger than n + 1 that uniquely determines the shape of the complementary function. In this case, for example, the user may select (n + 1) for uniquely determining the corrected complementary function shape, or the system may select (n + 1) It is also possible to use a total of more than n + 1 control points as control points for uniquely determining the complement function shape after correction.)

There are a method of making the time of the control point coincide with the time of the frame, and a method of setting the time without limiting to the time of the frame. In the former case, for example, the time of the control point is calculated by a predetermined algorithm first, and if the time does not match the time of the frame, the time immediately after the calculated or instructed control point time is used. The time may be the time of the control point.

On the other hand, the video information 164 displays image information of a video related to the spatio-temporal area information. For example, an image of all frames or an arbitrary frame of the video is displayed on a plane perpendicular to the time axis in the three-dimensional space and at the time position of the frame to be displayed. Frames to be displayed may be determined by the system according to a predetermined standard such as, for example, every few frames or only frames having nodes, or the user appropriately designates a method of selecting a frame to be displayed. You may be able to. Also,
At least a frame having the same time as the node coordinates and the time of the control point included in the display target on the display screen may be displayed.

In addition to displaying the frame image on the vertical plane in the three-dimensional space on the time axis, the video information is represented by (X,
(Y, time), a sectional view on an arbitrary plane may be displayed. For example, by displaying a cross-sectional view of the (X, time) plane or the (Y, time) plane, it is possible to easily know the temporal movement of the image. As described above, when an arbitrary cross-sectional view is displayed, which cross-sectional view is to be displayed may be determined by the system side or may be designated by the user.

As described above, by displaying the video information 164, it becomes easy to edit the spatiotemporal region while referring to what is drawn in the video. For example, when the area information of a certain object drawn in an image is corrected and input as spatio-temporal area information, the image information 16
By displaying 4, the area information of the object can be accurately extracted.

When displaying the video information 164, the video information may be displayed without being processed, or may be displayed with some processing applied to the video information. For example, the frame image may be made transparent or translucent, painted with a specified color, changed in brightness, displayed in black and white, changed in resolution, or subjected to mosaic or other image processing. Also, at this time, whether or not to perform processing or the content of processing when processing is performed in a range corresponding to the inside of the approximate figure indicating the object region in the frame and a range corresponding to the outside in the video data (For example, the degree of translucency, the color to fill, the brightness, the resolution, etc.) may be made different. Alternatively, the shape of the region in the frame may be displayed by framing according to the shape of the approximate figure indicating the object region in the frame. These image processings may be used in combination.

For example, when many frames of images are displayed on the display screen at the same time, the number of images to be displayed increases, and one image may hide another image. In such a case, by making the image of the frame translucent, it becomes easy to refer to a hidden image. Further, by displaying the outside of the region transparently, an image other than the space-time region is not displayed, so that the shape of the space-time region can be displayed in an easily understandable manner.

[0092] For example, the spatiotemporal region information can be used for a purpose of performing special processing on a part of the video and reproducing the video when viewing the video. When an object which the user does not want to see or which is not shown is included in the video, the image can be displayed by applying a mosaic thereto, or a color tone can be changed to relieve discomfort.
In this way, when editing an area for performing special processing of a video, the video information 164 is subjected to the same special processing as in the case of viewing a video, so that it is possible to refer to how the video is displayed. While editing the spatio-temporal area information.

FIG. 10 shows an example of a screen of a graphical user interface (GUI) when a three-dimensional space is projected onto a two-dimensional plane and displayed.

As shown in FIG. 10, this GUI is
3D space projection unit 31, status unit 32, image special display button 33, image display density button 34, image display position button 35, zoom button 36, gazing point movement button 3
7. It has a viewpoint moving button 38.

The three-dimensional space projection unit 31 is a part for displaying an image obtained by projecting the spatio-temporal region information shown in FIG. 9 in a three-dimensional space and projecting it on a two-dimensional plane by a specified projection method. It is.

The status section 32 is a section for displaying information on the spatio-temporal area and the video, and the coordinate information when the node coordinates / control points are selected. When the information in the status section is rewritten by the input section 4, the spatio-temporal area information is also rewritten in accordance therewith.

The image special display button 33 is
4 is a button for selecting a processing method (for example, a method of displaying the inside or the outside of the area translucent or transparent or performing special processing) when displaying the area 4. This may be a new dialog displayed and selected, or any interface that changes the display method each time the button is pressed, as long as the processing method can be changed.

The image display density button 34 and the image display position button 35 are used by the user to specify which frame image is to be displayed when the video information 164 is displayed. The image display density button 34 is used to specify the number of frames to be displayed, and the image display position button 35 is used to specify the frame position at which display is started. When the display start frame position is the h-th frame and the display is set to display every i-th frame, the displayed frames are the h-frame, the h + i-frame, the h + (i * 2) -frame, and so on. The image display density button 34 and the image display position button 35 may be of any type other than such a setting method as long as a frame to be displayed can be set.

Zoom button 36, gazing point move button 3
7. The viewpoint moving button 38 is a button for setting how to project from the three-dimensional space to the two-dimensional plane when displaying on the three-dimensional space projection unit 31. Viewpoint move button 38
To set the viewpoint in the three-dimensional space, the gazing point moving button 37 to set the gazing point in the three-dimensional space, and the zoom button 36 to set the angle of view in the three-dimensional space. This may be any configuration other than the interface configuration shown in FIG. 10 as long as the projection parameters can be specified.

Next, the correction of the spatio-temporal area information will be described.

As described above, the user operates the node coordinates 162 and the control points 163 of the representative point locus displayed on the GUI to correct (input for) the spatiotemporal area information.

Basic corrections include, for example, (1)
Movement of the control point (including movement in the time axis direction), (2) movement of the node coordinates within a range that does not change the position of the node coordinates in the time axis direction, and (3) node coordinates including a change in the position in the time axis direction , (4) a modification involving an increase or decrease of the order of the approximation function of the trajectory of the representative point (for example, addition or deletion of a control point), (5) addition or deletion of node coordinates (that is, separation or approximation of an approximate interval) Integration), and so on.

That is, the representative point trajectory is composed of two adjacent node coordinates 162 and a predetermined number of control points 163.
Is uniquely determined by the coordinates of Therefore, by adding or deleting the node coordinates 162, the number of nodes of the representative point trajectory can be changed. Also, the node coordinates 16
The node time can be changed by changing the coordinate position of No. 2 in the time direction.

The order of the interpolation function can be changed by adding or deleting control points 163. For example, if one control point is newly added between nodes where one control point is displayed while being interpolated by a quadratic function, it can be changed to a cubic function interpolation.

By moving the node coordinates 162 and the control points 163 on a plane horizontal to (X, Y), the interpolation function shape of the representative point locus can be changed. As described above, by adding or deleting the node coordinates 162 and the control points 163 and changing the coordinate positions,
An arbitrary representative point trajectory shape can be changed.

When a certain node at a certain representative point is moved in the time axis direction, the node having the same time information of another representative point may be treated as having the same correction. . Similarly, when one node is added or deleted with respect to a certain representative point, a node having the same time information may be treated as being added or deleted with respect to the other representative points (however, the representative point may be handled as a representative point). In the case of spatio-temporal domain information having a data structure in which nodes may differ depending on points, in such a case, it is possible to prevent other representative points from being treated as having the same modification.) .

When the user performs an input for correcting the spatio-temporal area information, the range of the spatio-temporal area information to which the correction propagates is specified according to the content of the correction, and the corresponding portion is calculated. , The spatio-temporal area information on which the correction is reflected is obtained.

For example, in the above-mentioned correction (1), the portion to which the correction spreads is the approximate section including the corrected control point in the locus data (105 in FIG. 5) of the XY coordinates of the corresponding representative point. Polynomial coefficient of (155 in FIG. 6)
Becomes

For example, in the above modification (2),
By moving the node coordinates in the XY plane, the correction spreads to two adjacent approximate sections including the corrected node coordinates, and the portion where the correction spreads is the locus data of the XY coordinates of the corresponding representative point. Of (105 in FIG. 5), the polynomial coefficients (155 in FIG. 6) of each approximation section including the corrected nodal coordinates are obtained.

Further, for example, in the above modification (3),
By moving the node coordinates including the component in the time axis direction, the correction spreads to two adjacent approximation sections including the node coordinates.
3 modifications are added.

Further, for example, in the above modification (4),
In the locus data (105 in FIG. 5) of the XY coordinates of the corresponding representative point, the polynomial degree of the approximate section including the corrected control point (1 in FIG. 6)
54), and the polynomial coefficients (155 in FIG. 6).

In each of the cases (1) to (4), the spatio-temporal region information editing processing unit 5 executes the processing of the approximate function of the representative point locus (corresponding to the representative point locus specified by the modified node coordinates 162 and the control point 163. Parameter), and correct the corresponding X-coordinate locus data and Y-coordinate locus data.

Further, for example, in the above modification (5),
In the case of an increase in the node coordinates, of the trajectory data of the XY coordinates of the corresponding representative point (105 in FIG. 5), the data of the approximate section to which the node coordinates are newly added (156 in FIG. 6) is modified. This causes the approximation section to be divided, and the data of the approximation section (156 in FIG. 6) is increased by one. The spatio-temporal area information editing processing unit 5 outputs, before or after the data of the approximate section (156 in FIG. 6) in the spatio-temporal area information,
Data of one approximation section (156 in FIG. 6) is inserted, and two new approximation sections generated by the division correspond to the representative point locus (corresponding to the modified node coordinates 162 and the control point 163, respectively). And the corresponding locus data of the X coordinate and the locus data of the Y coordinate are corrected.

For example, in the above modification (5),
In the case of a decrease in the node coordinates, of the locus data of the XY coordinates of the corresponding representative point (105 in FIG. 5), data of two approximate sections including the deleted node coordinates (1 in FIG. 6).
The correction is propagated to 56), the approximate sections are integrated, and the data of the approximate sections (156 in FIG. 6) is reduced by one. The spatio-temporal region information editing processing unit 5 deletes any one of the data of the approximation section (156 in FIG. 6) from the spatio-temporal region information, and calculates a new approximation section generated by the division. The trajectory of the representative point specified by the modified node coordinates 162 and the control point 163 (the parameter of the approximate function corresponding to the trajectory) is obtained, and the corresponding trajectory data of the X coordinate and the trajectory data of the Y coordinate are corrected.

When the spatiotemporal region information is collectively corrected after a plurality of correction inputs are performed, similarly, the corrected value is calculated for a portion of the spatiotemporal region information to which the correction spreads. The corresponding X coordinate locus data and Y coordinate locus data are corrected.

Note that the start time 102, the end time 103 in FIG. 5, the number of nodes 151, and the start node time 152 in FIG. 6 may be corrected depending on the contents of the correction.

The shape flag 104 shown in FIG. 5 may be instructed to be modified (by menu selection or the like).

In the GUI of FIG. 10, when the spatio-temporal area information changes due to the addition or deletion of the node coordinates 162 or the control point 163 or the change of the coordinate position, the display screen is immediately changed to the display screen. It is preferable to display the spatial area information. By always displaying the latest spatio-temporal area information, intuitive operation becomes possible.

Node coordinates 162 and control point 1
The addition and deletion of 63 and the change of the coordinate position may be performed by any device such as a mouse or a keyboard as long as information can be input. However, since these coordinate changes need to support movement in a three-dimensional space, when input is performed using a pointing device such as a normal mouse that can perform only two-dimensional movement, movement in the XY direction and the time direction is performed. What is necessary is just to allow some sort of three-dimensional movement, such as dividing the operation. For example, when it is desired to change the coordinate position using the mouse, the normal mouse movement is a movement on the (X, Y) plane, and the mouse is moved while pressing a button with a mouse or a button with a keyboard. Sometimes there are methods such as moving in the time direction. Further, a three-dimensional mouse may be used.

By the way, when a node is added (when a node is added to another representative point by adding a node to a certain representative point), the region shape at the node time is estimated as accurately as possible. Thus, the amount of subsequent correction work can be greatly reduced. That is,
The more accurate the initial shape to be added, the more efficiently the work can be performed.
Therefore, it is preferable that the representative point coordinates at the node time are estimated from the representative point trajectory shapes input up to the present time and used as the initial shape. Any method of estimating the coordinates of the representative point at the node time may be used.For example, the coordinates at the node time are calculated from the existing representative point trajectory function closest to the node time to be added, and the representative to add the coordinates is calculated. The initial value of the point coordinates may be used. In addition, other representative point trajectory information may be used to perform processing such as eliminating an extreme error or approaching a luminance edge of video information.

By displaying the spatio-temporal region information in the three-dimensional space, it is easy to grasp the shape of the spatio-temporal region as a whole. This has the advantage that the spatio-temporal region shape can be easily edited while observing the relationship between the front and rear frames.

In the example of the GUI shown in FIG. 10, the trajectory of the temporal transition of each representative point of the approximate figure representing one object area is displayed, and the spatio-temporal area information on one approximate figure is to be corrected. . In the example of the GUI in FIG. 10, when a plurality of spatio-temporal area information exists in the display target range,
For example, the spatio-temporal region information may be selectable by the user, one spatio-temporal region information selected by the user as an editing target may be displayed, and the one spatio-temporal region information may be set as a correction target.

When a plurality of pieces of spatio-temporal area information are present in the display target range, a plurality of pieces of spatio-temporal area information are simultaneously displayed, for example, one or a plurality of pieces of spatio-temporal area information selected by the user or displayed. It is also possible to make all the spatiotemporal domain information that can be modified.

When a plurality of pieces of spatiotemporal area information are displayed at the same time, approximate curves (and video information (eg, key frames)) may be displayed for all pieces of spatiotemporal area information. Displays approximate curves (and video information (eg, key frames)) only for one or more selected spatio-temporal region information, and displays only video information (eg, key frames) for other spatio-temporal region information. It may be.

When a plurality of pieces of spatio-temporal area information are displayed at the same time, the display relating to each piece of spatio-temporal area information may be displayed separately by color or line type.

When displaying a plurality of pieces of spatio-temporal area information at the same time, instead of displaying a plurality of pieces of spatio-temporal area information on the same three-dimensional space projection section 31 as in the past, the three-dimensional space information is displayed. A plurality of projection units 31 may be provided, and the display relating to the plurality of spatio-temporal region information may be performed by separate three-dimensional space projection units 31, respectively.

Of course, the GUI in FIG. 10 is an example,
In addition, various variations are possible.

FIG. 11 shows an example of the processing procedure of the spatiotemporal region information editing processing section 5 of the spatiotemporal region information processing system.

First, the spatio-temporal region information stored in the spatio-temporal region information storage unit 2 is read, and video information related to the spatio-temporal region information is read from the video information storage unit 1 (step S1).

Next, processing for displaying spatio-temporal region information on the three-dimensional space projection unit 31 by a predetermined display method is performed (step S2). For example, calculation and display of the outer shape of the approximate figure, the locus of the representative point of the approximate figure, the coordinates of the nodes, and the control points are performed. The video information is displayed on the three-dimensional space projection unit 31 by a predetermined display method (step S3). That is, special processing or display such as transparent / translucent processing or mosaic processing to be performed as necessary is performed. Step S
Step 2 and step S3 may be performed in the reverse order or may be performed simultaneously.

Next, an input from the user is received (steps S4, S5).

Inputs from the user regarding the display method (for example, the image special display button 33, the image display density button 34, the image display position button 35, and the zoom button 3 in FIG. 10)
6. If there is an input of the gazing point moving button 37, the viewpoint moving button 38, etc. (step S6), the setting of the display method is changed (step S7), and according to the changed setting of the display method, the steps S2 and S2 are performed. Alternatively, step S3 is performed again. Then, the process returns to the state of waiting for input from the user again (steps S4 and S5).

An input relating to correction from the user (for example, an input at the three-dimensional space projection unit 31 in FIG.
If there is an input in the status section 32 (step S8), the process of correcting the spatio-temporal area information is performed (step S9), and step S2 and / or step S3 are performed.
Again. Then, the process returns to the state of waiting for input from the user again (steps S4 and S5).

If there is another input (step S10), the corresponding processing is performed (step S1).
1) If necessary, step S2 and / or step S3
And returns to a state of waiting for input from the user (steps S4 and S5).

Then, when an instruction to end the editing is input (step S10), for example, when the end button is pressed by a mouse or an end command is input (step S10), the corrected spatio-temporal area information is It is stored in the spatiotemporal area information storage unit 2 (step S12).

In the process of correcting the spatiotemporal area information in step S9, when the correction of one node coordinate or control point is input, the corrected spatiotemporal area information is immediately obtained and reflected on the display. How to make
Modification of the node coordinates and control points is repeated arbitrarily (in this case, only the movement of the node coordinates and control points is reflected in the display), and then, when an input indicating a decision is made, the corrections are collectively corrected. There is a method of obtaining spatio-temporal region information and reflecting the corrected curve of the representative point locus on the display.

FIG. 12 shows an example of the processing procedure of step S9 in the former case. In this case, the content of the correction instruction is acquired (step S21), a portion of the spatiotemporal area information to which the correction spreads is specified (step S22), and a corresponding portion of the spatiotemporal area information is calculated (step S23). In this case, in the next step S2 and / or step S3, display based on the corrected spatiotemporal region information is performed.

FIG. 13 shows step S in the latter case.
9 shows an example of the processing procedure of No. 9. In this case, if the instruction is for the correction content (step S31), the content of the correction instruction is acquired and held (step S32). The location to which the correction spreads out of the spatial area information is specified (step S33), and the corresponding part of the spatiotemporal area information is calculated (step S34). In this case, when a determination instruction is input, a display based on the corrected spatiotemporal region information is performed.

Until an instruction for determination is input, for example, the coordinates of the modified nodes and the positions of the control points are displayed on the screen. At this time, the coordinates of the modified nodes and the control points before the modification are also displayed. There is a method to make the contents of the corrections understandable, and a method to delete the node coordinates and control points before the correction on the screen (in the former case, a different form is used so that the node coordinates and the control points before and after the correction can be distinguished) It is preferable to display with.).

Further, for example, when the user selects and moves the node coordinates and the control points with a mouse or the like, the trajectory is calculated in real time when the node coordinates and the control points are moved to the positions. (For example, in a display form different from the current track).

In the above procedure, when a series of processing ends, the spatiotemporal area information after correction is stored in the spatiotemporal area information storage unit 2 in step S12. There is also a method of storing the area information in the spatiotemporal area information storage unit 2.

For this modification, so-called undo and redo may be arbitrarily made possible.

Of course, the procedure described so far is only an example, and various other variations are possible.

As described above, the present invention is applicable to spatio-temporal domain information having various data structures.

The case where the present invention is applied to spatio-temporal area information having a data structure different from those in FIGS. 5 and 6 will be described below.

FIG. 14 shows another example of the data structure of the spatio-temporal area information. As shown in FIG. 14, the spatio-temporal area information of this example includes an ID number 301, a start time 302, an end time 303, a reference area flag 304, and reference area shape information 3
05, a conversion format flag 306, and trajectory data 307.

The ID number 301, the start time 302, and the end time 303 have the same meaning as the ID number 101, the start time 102, and the end time 103 in FIG.

The reference area flag 304 indicates how the shape of the reference area is represented. Since the shape of the reference area is represented by, for example, a rectangle, an ellipse, a polygon, or the like, information that uniquely distinguishes them is described. In the case of a polygon, vertex number information is also added to the reference area flag.

The reference area shape information 305 is for storing a reference area shape. For example, if the reference area shape is a rectangle or polygon, the coordinates of each vertex are stored.If the reference area shape is an ellipse, the parameters such as the vertex coordinates of the circumscribed rectangle, the center point, and the length of the major axis and minor axis are stored. Is done.

The conversion format flag 306 indicates the format in which the conversion parameter indicating the region conversion from the reference region to the spatiotemporal region information in each frame is indicated. Since the format of the conversion parameter indicating the area conversion includes, for example, translation, primary conversion, affine transformation, and the like, information that uniquely distinguishes them is described.

The trajectory data 307 is for describing each trajectory of the conversion parameter indicating the area conversion from the reference area to the spatio-temporal area information in each frame. This has a data structure of the trajectory data as shown in FIG. 6, for example. Since there are many conversion parameters in many cases, the trajectory data is described by the number of conversion parameters. The number of trajectory data is uniquely determined by the conversion format flag 607, and is, for example, two for parallel movement, four for primary conversion, and six for affine conversion. The order of the trajectory data is determined according to the meaning of the conversion parameter, and is, for example, as follows. When the locus data of the parameter as a function of the time t is aligned with z1 (t), z2 (t),..., A point p = (px, py)
Is the trajectory converted by the domain conversion, P (t) = (P
x (t), Py (t)): 1: For parallel movement Px (t) = px + z1 (t) Py (t) = py + z2 (t) 2: For linear transformation Px (t) = px * z1 (t) + py * z2 (t) Py (t) = px * z3 (t) + py * z4 (t) 3: For affine transformation Px (t) = px * z1 (t) + py * z2 (t) + Z3
(T) Py (t) = px * z4 (t) + py * z5 (t) + z6
(T) The locus data z1 (t), z2 (t),.
When represented by an n-th order polynomial, Px (t),
Py (t) can also be represented by an n-th order polynomial.

The configuration example of the spatio-temporal area information processing apparatus in this case is the same as the configuration examples shown in FIG. 7 and FIG. Hereinafter, points different from the configuration example in the case where the present invention is applied to the spatio-temporal region information of the data structure of FIGS. 5 and 6 described above will be described.

The spatio-temporal region information editing processing section 5 has basically the same function as that described above in order to display the spatio-temporal region information of this example, but the data structure of the spatio-temporal region information is different. Therefore, the trajectory is converted into the representative point trajectory according to the above equation and then displayed. The node coordinates 162 and the control points 16 in the key frame of the representative point locus are set by the user.
When 3 has been moved, each conversion parameter is calculated, and all representative point trajectories are changed in accordance with the changed conversion parameter.

Examples of the processing procedure in this case are shown in FIGS.
Or it is the same as FIG.

By doing so, the reference area and the spatio-temporal area information described by the conversion parameter indicating the area conversion from the reference area to the spatio-temporal area information in the frame are also described as the trajectories of the respective representative points. Editing can be performed similarly to the spatio-temporal area information.

Of course, the spatio-temporal region information processing system can also handle spatio-temporal region information having a plurality of types of data structures. In this case, the spatio-temporal domain information is added with identification information for identifying the data structure, and the spatio-temporal domain information processing system refers to the identification information when processing the spatio-temporal domain information, and What is necessary is just to specify a structure and to perform processing according to the data structure.

Now, some variations of the spatio-temporal domain information to which the present invention is applied will be described below.

First, spatio-temporal area information having a data structure with a display flag added will be described.

First, the spatio-temporal area information including the display flag will be described.

This spatio-temporal region information indicates that the object (or a part thereof) of the object in the video is visible on the screen or hidden behind another object. It includes display flag information relating to a display flag indicating whether there is no state.

For example, as shown in FIGS.
When a plurality of objects are present in the video, an object 21
Often 01 is hidden or revealed by another object 2102. To describe this state, display flag information is added to the spatiotemporal area information.

The display flag can be provided for each target object or for each representative point of an approximate figure of the target object.

When a display flag is given for each target object,
When the display flag is set, it indicates that the object is not hidden, and at the time of reproduction, the object is displayed. When the display flag is not set, it indicates that the object is hidden, and the object is not displayed during reproduction.

When the display flags are given for each representative point of the approximate figure of the target object, the same as above if the display flags of all the representative points in the approximate figure of one target object are the same, If the standing representative points and the representative points for which the display flag is not standing are mixed, the object is displayed in consideration of the situation (for example, only a corresponding part of the object is displayed).

The display flag is set for each key point, and is determined at the same time when the representative point trajectory data of the object area is created. The key points may be provided independently of the nodes of the approximation function, but the key points may always be nodes. For example, when a key point occurs, there is a method of forcibly setting that point as a node.

When a display flag is given for each target object,
The key points are set when the object shifts from the appearing state to the hidden state and when the object shifts from the hidden state to the appearing state. In the example of FIG.
01 shows up to frame i, hides from frame i to j, and when it appears again after frame j, puts key points on frames i and j and hides the display flag from frame i to j, Appearance states are set for the display flags of the other frames. The same applies to the case where the display flag is given for each representative point of the approximate figure of the target object.

When the representative point locus data is created, it is created assuming that the object appears over all the frames. If the information of the representative point is not known because the object is hidden, the information of the representative point before and after is known. By complementing the data, representative point trajectory data is created. Then, after creating the representative point trajectory data, a display flag is set according to whether the object is appearing or hiding. Therefore, even if one object is hidden or appears, it can be represented by a series of representative point trajectory data.

Hereinafter, variations of the display flag information will be described.

Normally, the display flag is set for each key point, but a start time stamp and an end time stamp may be added to the display flag itself. in this case,
There is an advantage that the displayed range and the hidden range can be set independently of the key point.

The display flag may have one display flag in one object, or may have it independently for each representative point locus data. For example, when an object is represented by a polygon and each vertex thereof is represented by trajectory data as a representative point, by having a display flag independently for each representative point trajectory data, only a part of the object is represented. It is possible to express hiding and the like.

Further, the display flag may take an integer value as a priority, not only whether it is present or hidden. When the objects overlap, it indicates that the low-priority object is hidden by the high-priority object, and only the high-priority object is displayed. When the priority is 0, it is assumed that the object is hidden irrespective of other objects. By taking the display flag as an integer as described above, there is an advantage that the problem of hiding between objects can be solved even when other objects are combined in the video. When the display flag is set to an integer value, one display flag may be provided in one object, or may be independently provided for each representative point trajectory data.

FIGS. 17 and 18 show examples of spatio-temporal area information obtained by adding a display flag to the data structure of FIG.

FIG. 17 shows an example in which “display flag information” 706 is added to the spatiotemporal area information of FIGS. 5 and 6 when a display flag is added to a target object.

FIG. 18 shows an example of the structure of the "display flag information" 705.

In this example, each display flag (2304)
It has a start time stamp (2302) and an end time stamp (2303). Number of display flags (2301)
Is the total number of display flags and the start time stamp (2302)
When the end flag and the end time stamp (2303) are not used, the display flag number (2301) may be omitted because the display flag has the key point number minus one. The display flag (2304) records whether it appears or is hidden by 0 or 1, but may take an integer value as the priority.

To add a display flag to each representative point of the approximate figure of the object, for example, "display flag information" is added to each representative point locus of the spatiotemporal area information.

FIG. 19 shows an example of the data structure of the representative point trajectory when “display flag information” is added to each representative point trajectory of the spatiotemporal area information of FIG. The configuration example of the “display flag” 900 in FIG. 19 is the same as above.

In this case, when the user makes corrections such as moving, adding or deleting node coordinates or control points, the spatio-temporal area information editing processing section 5 corrects the spatio-temporal area information. As a result, in a certain frame, an object present in the video was not hidden by another object before the correction, but became hidden after the correction, or conversely, by another object before the correction. What was hidden may not be hidden after modification. In such a case, the spatio-temporal region information editing processing unit 5 determines the contents of the flag information to be assigned, for example, by referring to and comparing the flag information of each spatio-temporal region information before and after the frame. What is necessary is just to correct the spatial area information. At this time, for example, the contents of the display flag of FIG. 18 may be simply updated, or the division of the start / end time section of FIG. 18 may occur. Further, the content of the flag information to be assigned may be determined not only by the spatiotemporal region information editing processing unit 5 but also by the user.

The user may be able to directly modify the flag information using the status section 32 or the like.

When displaying the spatio-temporal area information on the GUI screen, the contents of the display flag information may be presented to the user. For example, the outer frame of each approximate figure, the inside of each approximate figure, or each representative point of each approximate figure,
According to the content of the display flag information, the display may be identifiably displayed based on differences in color, luminance, resolution, and the like.

Next, spatio-temporal area information including information indicating a range on the screen from when the object in the video appears on the screen until it disappears (hereinafter referred to as object passing range information) will be described. I do.

First, the object passage range information will be described.

When an object is represented by the representative point trajectory data of the object area, one object is usually represented using a plurality of trajectory data. However, when it is desired to search for an object passing through a specified point, it is convenient if the object passage range can be expressed without calculating an object region from a plurality of trajectory data.

Therefore, in the present spatio-temporal area information, the minimum rectangular or polygonal object passing range information that surrounds the entire trajectory of the object is generated, and is attached to the spatio-temporal area information.

When a rectangle is used, a rectangle having an inclination may be used, or a rectangle having no inclination may be used. When a rectangle having a slope is used, there is an advantage that the trajectory of the object area can be approximated with few errors. When a rectangle having no inclination is used, there is an advantage that the parameter calculation of the rectangle is simple.

In FIG. 20A, reference numeral 2402 denotes the object 2
An example in which the locus area 401 is approximated by a rectangle having no inclination is shown.

In FIG. 20B, reference numeral 2403 denotes the object 2
An example is shown in which a locus area 401 is approximated by a rectangle having an inclination.

In FIG. 20C, reference numeral 2404 denotes the object 2
An example of a locus region 401 approximated by a polygon is shown.

When calculating the smallest rectangle or polygon that surrounds the entire trajectory of the object, after calculating the area in each frame, the logical sum of the area is calculated over all the frames, and the obtained logical sum is calculated. Is approximated by the smallest rectangle or polygon.

When calculating the smallest rectangle or polygon that surrounds the entire trajectory of the object, the smallest rectangle or polygon that surrounds the entire trajectory of the object area with respect to the already calculated frame is newly added. The area of the logical sum with the object area in the frame to be formed may be approximated by a minimum rectangle or polygon.

When calculating the smallest rectangle or polygon that surrounds the entire trajectory of the object, the smallest rectangle or polygon that surrounds the trajectory of each representative point is calculated and obtained over the entire trajectory. The smallest rectangle or polygon surrounding the logical sum of the rectangle or polygon area may be calculated.

FIG. 21 shows a description example of the object passage range information attached to the object area information. Circumscribed figure type 3401
Indicates the type of the circumscribed figure. For example, 0 indicates a rectangle having no inclination shown in FIG. 20A, 1 indicates a rectangle having an inclination shown in FIG. 20B, and 2 indicates a polygon shown in FIG. 20C. When the circumscribed figure type is 0, the number N of vertices is 2,
In the case of 1, the vertex number N is 3, and in the case of 2, the vertex number N is an arbitrary number. When the object has depth information, a three-dimensional circumscribed figure can be considered by similarly adding depth trajectory information. In this case, as shown in FIG. 22, information on Z of the depth information is added.

By adding the information of the minimum rectangle or polygon that surrounds the entire trajectory of the object, the passing range of the object can be expressed efficiently, and it is determined whether or not the object of the object passes a certain coordinate. To do things easily.

By the way, in this case, the object passing range information may be displayed on the GUI screen. Note that the object passage range information may be displayed, for example, in all the frames displaying the approximate figure, or may be displayed only in one or a plurality of selected frames.

When the user makes corrections such as moving, adding, or deleting node coordinates or control points, the spatio-temporal area information editing processing unit 5 corrects the spatio-temporal area information. Can change. In such a case, the spatiotemporal area information editing processing unit 5 may correct the object passage range information.

Further, the user may be allowed to directly modify the displayed graphic of the object passage range information. In this case, the spatiotemporal area information editing processing section 5 may correct the object passage range information as corrected by the user.

Next, spatio-temporal area information related to mosaiking will be described.

First, the spatio-temporal area information will be described.

The mosaicing is a method of combining a plurality of images photographed so that the photographing ranges partially overlap, and synthesizing a single image photographed over a wide range. The image synthesized in this manner is also called a panoramic image or the like. A plurality of proposals have been made for a method of creating a panoramic image from a plurality of images by mosaicking (M. Irani,
P. Anandan, “Video Index
ing Basedon Mosaic Repres
entationss ”, Proceedings o
f the IEEE, Vol. 86, no.
5, pp. 905-921, May 1998.
Such).

The spatio-temporal region information is not based on the position data of the representative point of the approximate figure of the object region in each frame as in the past, but instead of the approximation of the approximate figure of each object region in the panoramic image. The approximation is performed based on the position data of the representative point.

FIG. 23 is a diagram for explaining this method.

Note that the panoramic image itself is one image, and each pixel of each still image before synthesis is coordinated with a reference point (for example, a lower left point in the frame) in the panoramic image as the origin. Will be converted. Therefore, each representative point of the approximate figure of the object area in each still image is a series of the X coordinate and the Y coordinate in the coordinate system of the panoramic image. Here, a series of the X coordinate and the Y coordinate of each representative point of the approximate figure of the object region in each still image is approximated by a function as in the past. At this time, for example, a difference vector is obtained within one still image, or a difference vector is obtained between still images, and the series of the difference vectors is approximated by a function.

First, a panoramic image is created from a plurality of input still images. These input image groups are shown in FIG.
(A) corresponds to 2000 to 2005, and is an image obtained by photographing a moving object while moving the camera. 200
Reference numeral 6 denotes an object in the image, and the same object is photographed in 2000 to 2005. These image groups are often continuous frames in a moving image or still images taken while moving the camera so as to overlap the shooting range. However, any input image that can create a panoramic image may be used.

A panoramic image synthesized from the input image group is 2007 in FIG.

Next, individual object regions existing in the synthesized panoramic image are approximated by figures. Note that the panoramic image creation and the graphic approximation of the object area may be performed in the reverse order. However, depending on the conversion at the time of synthesizing the panoramic image, it may be necessary to change the approximate figure type of the object area. For example, when the object region is approximated by a rectangle, if the panorama image is synthesized by affine transformation, the object region after the synthesis is not necessarily rectangular. In that case, the panorama image is created first, or the correction is performed after the conversion of the panorama image synthesis.

Next, the trajectories of the representative points and characteristic points of the approximate figure of the object region thus obtained are approximated by a function. The trajectory of the object region is determined based on a reference object region and the amount of change of each object region therefrom. For example, in FIG. 23B, a change in the object region subsequent to the object region 2008 of the first input image is defined as a trajectory 2009. In this example, the center of gravity of the object area is used as a representative point. However, the same applies to a case where a representative point of another approximate figure such as a rectangle or an ellipse is used or a case where another feature point is used as a representative point.

The amount of change from the reference point can be obtained by using the difference from the reference point or by using the difference from the immediately preceding object area. These changes can be similarly approximated by a function. . Further, the change from the reference point can be approximated not by the movement of the representative point or the feature point but by a motion model such as parallel / rotational movement or affine transformation, and the movement of the object can be described as the locus of the transformation coefficient. This case can also be applied by approximating the locus of the conversion coefficient by a function.

Next, parameters of a function approximating the obtained trajectory are described in accordance with the format of the data structure.

[0209] Also, various parameters for synthesizing individual images of the input image group as a panoramic image can be similarly described by regarding the entire input image as an object region.

FIG. 24 shows a description example of various parameters attached to the object area information. The various parameters describe the coordinate system of the created panoramic image by the coordinates of the video frame used for the creation and the conversion parameters from the video frame to the panoramic image. Here, the origin may be set at any point, but it is assumed that the coordinates of the video frame have the origin at the lower left. It is assumed that the vertical and horizontal lengths of the frames used for mosaicing are the same and known. The integrated coordinate flag 3601 is a flag indicating whether or not the object is described using the coordinate system of such a panoramic image. When the flag is 0, the integrated coordinate flag 3601 uses the normal coordinate system (that is, the origin at the lower left of all video frames). 1 indicates a coordinate system in which frames are integrated into a panorama. The model type M3602 indicates the type of the conversion model when each frame is converted into a panoramic image. For example, when the model type M is 0, there is no conversion. When the model type M is 2, translation is performed. When the model type M is 4, rotation and scaling are performed.
6 represents affine, 8 represents perspective transformation, and 12 represents secondary projection transformation. In each model, there are as many parameters as the value of model type M.

Parallel movement: Vx (x, y) = a ₁ Vy (x, y) = a ₂ rotation / enlargement / reduction: Vx (x, y) = a ₁ + a ₃ + a ₄ y Vy (x, y) = a ₂ −a ₄ + a ₃ y Affine transformation: Vx (x, y) = a ₁ + a ₃ + a ₄ y Vy (x, y) = a ₂ + a ₅ + a ₆ y Perspective transformation: Vx (x, y) = ( a ₁ + a ₃ + a ₄ y) / (1 + a ₇ + a ₈
y) Vy (x, y) = (a 2 + a 5 + a 6 y) / (1 + a 7 + a 8
y) 2-order projective transformation: Vx (x, y) = a 1 + a 3 + a 4 y + a 7 xy + a 9 x 2 +
^{_{a 10 y 2 Vx (x,}} y) = a 2 + a 5 + a 6 y + a 8 xy + a 11 x 2 +
a ₁₂ y ² Here, the origin for performing the conversion is given by the X origin 3603 and the Y origin 3604. This origin indicates the coordinate system of the original video frame. The reason for giving the origin for the conversion in this way is to reduce errors that occur during the conversion. The number N3605 of conversion parameters is the same as the number N of frames used for panorama. A conversion parameter is required for each frame. The frame time 3606 is the time between frames from the initial frame, and the parameter set 3607 describes M parameters according to the return model type. The trajectory of the object in each frame is also returned and described using this parameter set.

In this way, when the camera is shooting while chasing the object area, a panoramic image is created by mosaicing in which continuous frames are converted by image conversion and joined so as to be connected. By describing the region information of the object on the image, the region information of the object can be uniquely described in a coordinate system based on a certain point of the mosaicing image even when the camera is moving.

[0213] In this case, on the GUI screen, the spatio-temporal area information may be displayed in a three-dimensional manner as before, or instead, as shown in FIG. Panorama display to display spatio-temporal region information.

When the user corrects node coordinates, control points, and the like, the spatiotemporal region information editing processing section 5 may correct the corresponding spatiotemporal region information.

Each of the above functions can be realized as software. In addition, the present embodiment can be implemented as a program for causing a computer to execute predetermined means (or for causing a computer to function as predetermined means, or for causing a computer to realize predetermined functions), The present invention can also be implemented as a computer-readable recording medium on which the program is recorded.

The configuration illustrated in the embodiment of the present invention is an example, and is not intended to exclude other configurations, and a part of the illustrated configuration is replaced with another one or illustrated. Another configuration obtained by omitting a part of the configuration, adding another function or element to the illustrated configuration, or combining them is also possible. Also, another configuration that is logically equivalent to the illustrated configuration,
Another configuration including a portion logically equivalent to the illustrated configuration, another configuration logically equivalent to a main part of the illustrated configuration, and the like are also possible. Further, another configuration that achieves the same or similar purpose as the illustrated configuration, another configuration that achieves the same or similar effect as the illustrated configuration, and the like are also possible. Various variations of the various components exemplified in the embodiments of the present invention can be implemented in appropriate combinations. Further, the embodiments of the present invention relate to an invention as an individual device, an invention about two or more related devices, an invention as a whole system, an invention about components inside an individual device, or a method corresponding thereto. It encompasses and includes inventions related to various aspects, stages, concepts or categories, such as inventions. Therefore, the present invention can be extracted from the contents disclosed in the embodiments of the present invention without being limited to the illustrated configuration.

The present invention is not limited to the above-described embodiments, but can be implemented with various modifications within the technical scope.

[0218]

According to the present invention, it is possible to efficiently input and edit spatio-temporal region information while referring to the whole or a part of the spatio-temporal region shape.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining an outline of a process for describing an object region in a video by object region data;

FIG. 2 is a diagram for explaining an example of obtaining a function that approximates a value of an X coordinate of a reference representative point.

FIG. 3 is a diagram for explaining an example of a difference vector for representing a representative point other than a reference representative point.

FIG. 4 is a diagram for explaining an example of obtaining a function that approximates the value of an X component of a difference vector for representing a representative point other than a reference representative point.

FIG. 5 is a diagram showing an example of a data structure of spatio-temporal area information

FIG. 6 is a diagram showing an example of a data structure of trajectory data.

FIG. 7 is a diagram illustrating a configuration example of a spatio-temporal region information processing apparatus.

FIG. 8 is a diagram showing another configuration example of the spatiotemporal area information processing apparatus.

FIG. 9 illustrates an example of a display method.

FIG. 10 illustrates an example of a user interface.

FIG. 11 is a flowchart illustrating an example of a processing procedure;

FIG. 12 is a flowchart illustrating an example of a processing procedure;

FIG. 13 is a flowchart illustrating an example of a processing procedure;

FIG. 14 is a diagram showing an example of a data structure of spatio-temporal area information according to the second embodiment of the present invention.

FIG. 15 is a diagram illustrating a display flag.

FIG. 16 is a diagram for describing creation of representative point trajectory data;

FIG. 17 is a diagram showing still another example of the data structure of the object area data.

FIG. 18 is a diagram illustrating an example of a data structure of display flag information.

FIG. 19 is a diagram showing still another example of the data structure of the representative point trajectory data in the object area data.

FIG. 20 is a diagram for explaining object passage range information;

FIG. 21 is a diagram illustrating an example of a data structure of object passage range information;

FIG. 22 is a diagram showing another example of the data structure of the object passage range information.

FIG. 23 is a flowchart illustrating an example of a processing procedure of an object area information description method using mosaicing.

FIG. 24 is a diagram for describing a method of describing object region information using mosaicing.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Video information storage part 2 ... Spatiotemporal area information storage part 3 ... Display part 4 ... Input part 5 ... Spatiotemporal area information edit processing part 6 ... Spatiotemporal area information generation processing part 31 ... Three-dimensional space projection part 32 ... Status Part 33: Image special display button 34 ... Image display density button 35 ... Image display position button 36 ... Zoom button 37 ... Point of view moving button 38 ... Point of view moving button

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Toshimitsu Kaneko 1 Toshiba-cho, Komukai-shi, Kawasaki-shi, Kanagawa Inside the Toshiba R & D Center Co., Ltd. No. 1 town Toshiba R & D Center (72) Inventor Koji Yamamoto No. 1 Komukai Toshiba Town, Koyuki-ku, Kawasaki-shi, Kanagawa F-term (reference) 5C023 AA03 AA04 AA10 AA38 AA40 BA01 CA02 DA04 DA08 5L096 CA04 DA01 FA09 HA02

Claims (25)

[Claims] The present invention relates to spatio-temporal area information described as representing a transition of an arbitrary area in a video data over a plurality of frames and capable of specifying a trajectory of a representative point of an approximate figure indicating the area. A spatio-temporal region information processing method for performing a process of correcting spatial region information, based on the spatio-temporal region information of the target region,
Determining the trajectory of the representative point; displaying the trajectory of the determined representative point on a screen; receiving an input of a correction operation of the shape of the trajectory displayed on the screen; A spatio-temporal area information processing method, wherein the spatio-temporal area information is corrected based on 2. When the spatiotemporal area information is corrected based on the correction operation, the locus of the representative point is displayed on the screen again based on the corrected spatiotemporal area information, 2. The spatio-temporal area information processing method according to claim 1, further comprising inputting the predetermined correction operation. 3. When displaying the trajectory of the representative point on a screen, a correction operation point is displayed at a predetermined position on the trajectory, and a correction operation performed on the correction operation point is performed. 2. The spatio-temporal area information processing method according to claim 1, wherein an input is received, and the spatio-temporal area information is corrected based on the input correction operation. 4. The correction operation is a movement in an arbitrary direction with respect to any displayed correction operation point, addition of a new correction operation point, or deletion of an existing correction operation point. The spatio-temporal area information processing method according to claim 3, wherein 5. When the trajectory has been interpolated by a function, the number of points for uniquely determining the interpolation function is displayed as the point for the correction operation, and after the movement or addition or deletion, Based on the interpolation function uniquely determined by the position of the point
5. The spatio-temporal area information processing method according to claim 4, wherein the spatio-temporal area information is corrected. 6. When the trajectory is function-interpolated and its approximate section is composed of a plurality of sections, moving or adding or deleting a point corresponding to a node of the approximate section is performed as the correction operation. 5. The spatio-temporal area information processing method according to claim 4, wherein, when performed, the spatio-temporal area information is corrected based on the approximate section after the movement, addition, or deletion. 7. When the trajectory is function-interpolated and the approximate section is composed of a plurality of sections, and when a point corresponding to a node of the approximate section is added as the correction operation, 5. The spatio-temporal region information processing method according to claim 4, wherein a shape of the approximate figure at a node time of a point to be added is estimated from the spatio-temporal region information. 8. When the trajectory is function-interpolated and its approximate section is composed of a plurality of sections, the number for uniquely determining the interpolation function for each approximate section as the point for the correction operation. And a point corresponding to a node of the approximate section among the points,
The spatio-temporal area information processing method according to claim 4, wherein the other points are displayed in a distinguishable display form. 9. The spatio-temporal area information processing method according to claim 1, wherein, when the trajectory of the representative point is displayed on a screen, a predetermined frame of the video data is also displayed. 10. The spatio-temporal area information processing method according to claim 9, wherein when displaying a predetermined frame of the video data, the frame to be displayed is displayed in a semi-transparent manner. 11. The display device according to claim 9, wherein a predetermined frame of the video data is displayed in a display form capable of recognizing the outline of the approximate figure in the frame to be displayed. Spatio-temporal domain information processing method. 12. The display device according to claim 9, wherein a predetermined frame of the video data is displayed in different display forms between the inside and outside of the approximate figure in the frame to be displayed. Spatio-temporal domain information processing method. 13. The information which can specify the trajectory of the representative point includes:
13. The trajectory obtained by arranging the position data of the representative points along the progress of the frame by approximating the trajectory with a predetermined function, and describing the trajectory using parameters of the function. The spatio-temporal area information processing method according to any one of claims 1 to 7. 14. The information which can specify the trajectory of the representative point is:
From the position data in the frame serving as the reference of the representative point, a conversion parameter indicating a region conversion to position data in another frame is obtained, and the conversion parameter
13. The method according to claim 1, wherein a trajectory obtained by arranging the conversion parameters along the progress of the frame is approximated by a predetermined function, and is described using parameters of the function. The spatio-temporal domain information processing method according to the paragraph. 15. For at least one of the representative points, the relative position of the representative point with respect to another representative point in the same frame is used instead of using the position data of the representative point. The spatio-temporal area information processing method according to claim 13 or 14, wherein data indicating a position is used. 16. When displaying the trajectory of the representative point on a screen, the trajectory is three-dimensionally displayed in a form in which the trajectory is arranged in a three-dimensional space including a two-dimensional coordinate axis and a time axis of the frame. The spatio-temporal area information processing method according to claim 1, wherein: 17. The spatio-temporal area information includes, for each of the approximate graphic or the representative point, flag information for controlling whether or not to display the approximate graphic or the representative point. When modifying the spatio-temporal region information based on the predetermined modification operation input to the shape of the trajectory, if the modification indicates the content indicated by the flag information, the flag information is modified. 17. The spatio-temporal domain information processing method according to claim 1, further comprising: 18. The flag information includes information for specifying a frame section in which the approximate figure or the representative point is visible and a frame section in which the approximate point is not visible, or the flag information or the approximate figure or the representative point. 18. The spatio-temporal area information processing method according to claim 17, wherein the information is information indicating a priority regarding display in each frame section of the information. 19. The spatio-temporal area information also includes information about an existence range approximate graphic indicating an area including a range where the approximate graphic exists in the plurality of frames, and the trajectory displayed on a screen. When correcting the spatiotemporal region information based on the predetermined correction operation input for the shape of the shape, if the content indicated by the approximate existence range graphic is affected by the modification, 17. The method according to claim 1, wherein the information is also corrected.
2. The spatio-temporal domain information processing method according to item 1. 20. The spatio-temporal area information processing method according to claim 19, wherein when displaying the trajectory of the representative point on a screen, the approximate existence range figure is also displayed. 21. A method for receiving an input of a predetermined correction operation on the approximate existence range graphic displayed on a screen, and correcting the approximate existence range graphic based on the input correction operation. 21. The spatio-temporal domain information processing method according to claim 20. 22. The spatio-temporal region information is an arbitrary object which moves in a panoramic image generated by connecting adjacent frames one after another in a plurality of arbitrary continuous frames in video data so that overlapping portions thereof are overlapped. In order to represent information about the area, the area of the target object in the video data is specified by an approximate figure for the area of the target object, and a plurality of representative points of the approximate figure that moves in the panoramic image are extracted, The trajectory obtained by arranging the position data of the representative point of the approximate figure in the panoramic image or the data capable of specifying the position data along the progress of the frame is approximated by a predetermined function, and the target is determined using the parameters of the function. 17. The method according to claim 1, wherein information on an area of the object is described.
2. The spatio-temporal domain information processing method according to item 1. 23. The spatio-temporal region according to claim 22, wherein the video data is displayed in a panoramic manner, and the trajectory obtained based on the spatio-temporal region information is displayed on the panoramic display. Information processing method. 24. Spatio-temporal region information described as representing a transition of an arbitrary region in a video data over a plurality of frames and capable of specifying a trajectory of a representative point of an approximate figure indicating the region is targeted. A spatio-temporal region information processing system for performing a process of correcting the spatial region information, based on the spatio-temporal region information of the target region,
Means for determining the trajectory of the representative point; means for displaying the trajectory of the determined representative point on a screen; and input of a correction operation of the shape of the trajectory displayed on the screen. A spatio-temporal area information input system, comprising: means for receiving; and means for correcting the spatio-temporal area information based on the input correction operation. 25. Spatio-temporal area information described as representing a transition of an arbitrary area in a video data over a plurality of frames and capable of specifying a trajectory of a representative point of an approximate figure indicating the area, A program for causing a computer to function as a spatio-temporal region information processing system for performing a process of correcting spatial region information, based on the spatio-temporal region information of the targeted region,
A function for obtaining the trajectory of the representative point, a function for displaying the trajectory of the obtained representative point on a screen, and an input of an operation for correcting the shape of the trajectory displayed on the screen. A program for causing a computer to realize a function for receiving and a function for correcting the spatio-temporal area information based on the input correction operation.