JPH10164435A - Time-space integration and managing method for a plurality of video images, its device and recording medium recording its program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an integration method and device for a plurality of video images to manage, express and operate them in unified way, in which video information is efficiently acquired from a plurality of video images in a same photographic space. SOLUTION: A photographic state detection section 103 reds an image data string to detect camera on/off information and camera operation information. A video split section 104 splits a video image into shots based on the camera on/off information, and an object background separate section 105 separates an object from a background based on the camera operation information. While an object motion information extract section 106 makes object information mutually correspond between frames, a photographic space re-compositing section 107 composites again the camera operation information and the background and an inter-shot relation calculation section 108 calculates a spatial relation among a plurality of photographic spaces composited again. The camera on/off information, the camera operation information, and each information among the objects, the motion of object, re-composited background and shots are managed and stored in time space and one object or over and the image pickup space are composited again, according to a request of the user or the like and the result is displayed or outputted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for creating a new video by integrating a plurality of videos, and more particularly to a method and apparatus for integrating and managing a plurality of videos in a spatio-temporal manner, and a recording device for recording the program. It is about media.

[0002]

2. Description of the Related Art Computers with higher performance, hard disks with larger capacities, video cameras with smaller sizes, digitalization,
Along with these price reductions, the use of such devices in general households is increasing. With the widespread use of such high-performance computers and large-capacity hard disks, and the establishment of information compression techniques, video digitization has become familiar. Digitization of video has made it possible to handle video on a general-purpose personal computer and output it to a high-resolution personal computer display. This generally means that
Although output is limited to only the TV monitor (640 × 480 pixels), output at a resolution higher than the resolution of the TV monitor is also possible. For example, it is possible to simultaneously display a plurality of images on a display for a personal computer.

With the development of video equipment, several methods for enhancing video have been reported. Mich
al Irani amd Shmuel Peleg
Is described in the document "Motion analysis for
image enhancement: Resolu
, Occlusion and Transp
arency ", Journal of Visual
communication and image
representation, Vol. 4, No.
4, Decembr, pp, 324-335, 199
No. 3 "proposes a method of using a motion information in a video to increase the resolution of the video, a method of interpolating a noisy area of a background caused by a subject, and the like. Also, Laura A. Tadecio reports a method of creating a high-resolution still image from a video (Japanese Patent Application Laid-Open No. 5-304675). recently,
ORAD has developed a system called "Digital Repla," which has realized the reproduction of video with new added value.
The functions of the system include, for example, an emphasis expression, tracking and enlargement function of a subject, and a function of displaying a graphic and information such as a line segment, a distance, and a speed thereof together with an image.

There is also a report on a new video user interface. Mill et al., Reference "A
Magnifier Tool for Video
Data ", Proceedins of CHI'9
2, pp. 93-98 (1992) ", a report was made that allowed frames of video to be arranged in space according to the temporal resolution level, and allowed a new perspective and access style for video from coarse temporal resolution to fine temporal resolution. ing.
Also, E.I. Elliot and A. W. Davis
Is based on the document "Motion Image Processes
sing ”, Striking Possibilit
ies, ADVANCED IMAGING, AUGU
ST (1992) ", and the video image (two-dimensional) + time (1
(3D) as a three-dimensional object, and reports on a new method of expressing time information of video and the realization of intuitive access to time information.

By the way, in an environment where input by a video camera is easy and display form is free, there is a demand to handle a plurality of images. The term "plurality of images" as used herein means, for example, a plurality of images by cameras installed at sports stadiums such as the Olympic Games, or images of different players, even with images taken by one camera. Such as the video that did. The term "handling" used herein refers to simultaneously watching, comparing, searching, and editing a plurality of videos.

[0006]

However, there are problems which have not been solved in the above-mentioned conventional reports for a plurality of videos. In the above-mentioned conventional report, enhancement is realized for one shot, but it is impossible to enhance a plurality of videos using a relationship between the plurality of videos. In addition, it is not possible to integrate a plurality of videos spatiotemporally and manage them in an integrated manner.
Here, "enhancement for multiple images"
Is to associate the original videos with each other, and automatically extracts the video content information (information on the subject actually shown in the video, that is, information on the subject) and visually expresses it. It is to be. It is also to associate a plurality of videos and create one or more videos in which video content information is visually represented. For example, one example is to create a single image in which each player overlaps in a common space from a plurality of images of different players. Further, "manage spatiotemporally" means to integrally manage information extracted based on the spatiotemporal structure of video. For example, the subject and the background are separated, and each is managed, and the time-space related information is also managed. The purpose of the above-mentioned conventional technology does not include the unified management of a plurality of videos by using a spatio-temporal structure. It is impossible to achieve unified management.

[0007] Further, regarding the video user interface that has been conventionally realized, the video is simply re-expressed, and the information of the video and the relationship between a plurality of videos are actively extracted to enhance the video. It was not done. It does not allow for intuitive understanding of video or inter-video content or new enhancements to video information, especially time related information, because nothing is considered about video content or the relationship between multiple videos. . Further, since a plurality of videos are not managed spatiotemporally, it is impossible to interact with the content of the videos.

[0008] As described above, reports on video enhancement, user interface, etc., which are conventionally reported, include enhancements that can provide high added value to a plurality of videos, and management, expression, Nothing can express the operation. That is, there is a problem in the above-mentioned conventional technology.

SUMMARY OF THE INVENTION An object of the present invention is to provide a plurality of videos in a spatio-temporal manner in order for a user to efficiently acquire video information in his / her own style according to his / her interests and purposes for a plurality of videos taken in the same space. A method and apparatus for integrating and managing spatio-temporal video, and a device and a program for recording the program, which can integrate, manage, express and operate multiple videos in a unified manner To provide a medium.

[0010]

In order to achieve the above object, a spatio-temporal integration and management apparatus for a plurality of videos according to the present invention is an apparatus for integrating and managing a plurality of videos in a spatio-temporal manner.
An image data string memory unit that reads video data and saves it as a data string, a shooting state detection unit that reads a data string from the image data string memory unit, and detects shooting state information including camera on / off information and camera operation information, A video dividing unit that divides the video of the data string into shots based on the camera on / off information, and a subject that separates a subject and a background for each frame of the video using the camera operation information and a physical characteristic amount; A background separation unit, a subject motion information extraction unit for associating subject information separated for each frame between frames, and a camera space for capturing the video from the camera operation information and the background separated for each frame; An imaging space resynthesizing unit for synthesizing, and a plurality of images recombined from the plurality of divided shots by the imaging space resynthesizing unit. An inter-shot relationship calculation unit that calculates a spatial inter-shot relationship between spaces; information on the separated subject; information on the associated subject; the shooting state information; the information on the background; And a video structure information management / storage section that manages / accumulates related information.

Further, in the above-mentioned spatio-temporal integration and management apparatus for a plurality of videos, all or a part of the extracted subject information, photographing state information, background information, inter-shot relation information and video data is transmitted or received. A video structured information transmission / reception unit is further provided.

Further, in the above-mentioned spatio-temporal integration and management apparatus for a plurality of videos, information stored and managed in the video structure information management / storage section according to one or both of a condition given in advance and a request from a user. A re-synthesizing unit for re-synthesizing one or a plurality of photographing spaces and one or a plurality of subjects based on, a display unit for displaying an image re-synthesized by the re-synthesizing unit, and displayed on the display unit. A user input unit for inputting the user's request for recomposition based on the video being
And an output unit for outputting a video displayed on the display unit to an external device in a digital or analog format as needed.

Similarly, the spatio-temporal integration and management method for a plurality of videos according to the present invention is a method for integrating and managing a plurality of videos in a spatio-temporal manner, and includes a process of reading video data and storing it as a data sequence. Reading a data string from the image data string memory unit, detecting a shooting state information including camera on / off information and camera operation information, and a video of the data string on a shot-by-shot basis based on the camera on / off information. A video dividing process, a subject for separating a subject and a background for each frame of the video using the camera operation information and the physical feature amount, a background separating process, and subject information separated for each frame to a frame. Extracting the subject motion information to be associated with each other, and capturing the video from the camera operation information and the background separated for each frame. And calculating a spatial inter-shot relationship between the plurality of divided shots and the plurality of shooting spaces re-combined in the shooting space re-synthesis process from the plurality of divided shots. An image structure information management process for managing / accumulating the inter-shot relationship calculation process, the separated subject information, the associated subject information, the shooting / photographing information, the background information, and the shot relationship information. And an accumulating step.

Further, in the above-mentioned spatio-temporal integration and management method for a plurality of videos, all or a part of the extracted subject information, photographing state information, background information, inter-shot relation information and video data is transmitted or received. The method further comprises the step of transmitting and receiving the image structured information.

Further, in the above-mentioned method for integrating and managing a plurality of videos, the video structure information is stored and managed according to one or both of a given condition and a request from a user after the video structure information management and storage process. A re-synthesizing step of re-synthesizing one or a plurality of photographing spaces and one or a plurality of subjects based on the information that has been displayed, and a display or output step of displaying or outputting an image re-synthesized in the re-synthesizing step It is characterized by having.

Similarly, the computer-readable recording medium recording the spatio-temporal integration and management program of a plurality of videos according to the present invention is a computer-readable recording medium recording a program for integrating and managing a plurality of videos in a spatio-temporal manner. An image data string storing step of reading video data and storing the data as a data string; and a shooting state detecting step of reading a data string from the image data string memory unit and detecting shooting state information including camera on / off information and camera operation information. And a video dividing step of dividing the video of the data sequence into shots based on the camera on / off information, and separating a subject and a background for each frame of the video using the camera operation information and a physical feature amount. The subject and background separation process and the subject information separated for each frame are associated with each other between frames. A subject movement information extracting step, a photographing space synthesizing step of recombining a photographing space in which the video is photographed from the camera operation information and the background separated for each frame, and a photographing space from the plurality of divided shots. An inter-shot relation calculating step of calculating a spatial inter-shot relation of a plurality of shooting spaces re-combined in the re-combining step; information on the separated subject; information on the associated subject; A video structure information managing / accumulating step of managing / accumulating information, the background information, and the information on the relationship between shots.

In the above-mentioned computer-readable recording medium on which a spatio-temporal integration of a plurality of images and a management program are recorded, all of the extracted subject information, photographing state information, background information, inter-shot relation information and video data are stored. Alternatively, the method further comprises the step of transmitting or receiving the video structured information for transmitting or receiving a part thereof.

In the above-mentioned computer-readable recording medium on which a spatio-temporal integration of a plurality of videos and a management program are recorded, after the video structure information managing / accumulating process, any one of a condition given in advance and a request from a user is provided. A re-synthesizing step of re-synthesizing one or more photographing spaces and one or more subjects based on the information stored and managed according to one or both, and displaying the video re-synthesized in the re-synthesizing step. Or a display or output process for outputting.

According to the present invention, video data is read, a stored image data sequence is read, shooting state information including camera on / off information and camera operation information is detected, and the video is divided into shots based on the camera on / off information. And
The subject and the background are separated for each frame using the camera operation information and the physical feature amount, the separated subject information is correlated between the frames to extract the subject movement information, and the camera operation information and the frame-by-frame By re-synthesizing the shooting space from the background, calculating the spatial inter-shot relationship between the plurality of shooting spaces re-synthesized from the plurality of shots, and managing and accumulating the information obtained above, It realizes spatio-temporal integration of multiple images that enables spatio-temporal enhancement of images, spatio-temporal and unified management, expression, and operation.

Further, by transmitting or receiving all or a part of the extracted subject information or the like, the user can freely select, search, and retrieve the information at any place, and at any place, at any place. Can be sent to users. Also, any user can send such information from any place.

Further, based on the information managed and stored, one or more photographing spaces and one or more subjects are recombined in accordance with a given condition or a request from a user, and are displayed or externally displayed. By outputting in analog and digital formats, users are interested in the problem that it was difficult to obtain information simultaneously and spatially simultaneously for a plurality of videos taken in the same space. According to the purpose, it is possible to simultaneously and intuitively and efficiently acquire information of a plurality of videos in their own styles.

[0022]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 shows an apparatus configuration diagram of an embodiment of the present invention. Hereinafter, each component and method will be described with reference to the configuration diagram.

The video signal input from the video signal input unit 101 shown in FIG. 1 is temporarily stored in an image data string memory unit 102. Next, the accumulated video signal is
Image processing is performed by the photographing state detection unit 3 to extract camera on / off information and camera operation information.

FIG. 2 shows the configuration of the photographing state detecting unit 103. Also, FIG.
4 shows the flow of processing by the components up to 04. The configuration and processing flow of each component will be described below in detail with reference to these drawings.

Several frames of the video signal read from the image data string memory section 102 are 301 shown in FIG. 3, and this 301 is usually called a spatio-temporal image. With respect to the spatiotemporal image, the horizontal and vertical linear components are calculated for each frame image in a linear component calculation unit 201. The calculation of the vertical straight line is 302 in FIG. 3, and the calculation of the horizontal straight line is 303 in FIG. 304 obtained by each of these calculations is a vertical linear component spatiotemporal image,
305 is called a horizontal linear component spatiotemporal image.

Subsequently, the image data rearranging unit 202 receives the vertical and horizontal linear component spatiotemporal images 304,
305 is rearranged. The rearrangement process here corresponds to 306 in FIG. 3, and is a process of cutting the spatiotemporal image on a plane including the normal line of the screen. The direction perpendicular to the x and y coordinate axes of the frame image is defined as the normal direction of the screen. A plane including the cut time axis is generally called a spatiotemporal sectional image. As an example of the spatio-temporal cross-sectional image, a cut plane obtained by cutting the spatio-temporal image on a plane including a camera traveling direction and a screen normal, which is used in the field of computer vision (Epipolar Pl image)
ane Image)). The three-dimensional position of the subject is estimated from the spatiotemporal sectional image. This is because on the epipolar plane image, the trajectory of the feature point of the object becomes a straight line, and the inclination of the straight line becomes the magnitude of the movement of the feature point of the object [R. C. Bolles, H .; Bak
er, and D.E. H. Marimont, "Epip
polar-plane image analysis
s: An approach to determine
g structurefrom motion ", I
JCV, 1, 1, pp7-55, June 198
9. ]. A spatiotemporal cut image obtained by cutting the spatiotemporal image so as to include the x and y coordinate axes is particularly called an xt spatiotemporal image, and a spatiotemporal cut image including the y and t coordinate axes is similarly called a yt spatiotemporal image. . An arbitrary xt spatiotemporal image is cut out from an arbitrary value of y, and the plurality of xt spatiotemporal images are xt
It is called a spatiotemporal image sequence. The same applies to the yt spatiotemporal image sequence.

Subsequently, the cut planes of the vertical and horizontal linear component spatiotemporal images cut by the image data rearranging section 202 are filtered by the video information filter processing section 203 (first differential, second differential, etc.). ) Process. This processing is performed by the line segment detecting means, and aims at detecting an edge or a line. In FIG. 3, it corresponds to 307. The intensity of an edge or a line is calculated by the filter processing unit 203. The flow pattern along the time axis seen on the cut surface in the cut image is caused by movement in the video. The direction of this flow corresponds to the magnitude of the movement. In the edge detection described above, an edge or a straight line indicating the direction of the flow is detected, and only the motion information is emphasized from the image. The cut image sequence in which the edge is detected is referred to as a vertical or horizontal spatiotemporal edge image sequence.

Subsequently, in the feature extracting unit 204, the vertical and horizontal spatio-temporal edge image sequence is added by the integrating means in the normal direction of the edge image. In FIG. 3, 308
And the addition process is performed in the direction of the broken line 308a. This addition processing is performed for the purpose of further enhancing the movement emphasized in the previous filter processing. That is, if the movement of the feature point of the object is an object due to global movement,
The addition processing emphasizes each other and is significantly reflected in the addition processing result. On the other hand, when the movement of the feature point of the object is a local movement, the addition processing weakens the movement and makes it difficult to be reflected in the addition processing result. Also,
This addition process is a process that is strong against noise, unlike the difference process, and means that motion information can be extracted even from a video that contains a lot of noise. The image obtained as a result of this addition process is called a spatiotemporal projection image. From the xt vertical spatiotemporal image sequence, an xt spatiotemporal projection image 309 is obtained.
Is obtained, and a yt spatiotemporal projection image 310 is obtained from the yt horizontal spatiotemporal image. The meaning of the flow pattern along the time axis of the xt spatio-temporal projection image expresses the horizontal movement of the image, and the pattern of the yt spatio-temporal image expresses the vertical movement.

Subsequently, in the feature statistical analysis unit 205, first, a straight line perpendicular to the time axis is detected from the extracted features expressed as a two-dimensional image having a time axis and a space axis by the camera on / off detection means. The camera on / off information is extracted. Specifically, the xt spatiotemporal projection image is represented by F
The (x, t), yt spatiotemporal projection image is represented by F (y, t). If the value C calculated from the evaluation equation expressed by the following equation is equal to or greater than a predetermined threshold, it is assumed that the camera has been turned on and off at the time t.

[0031]

C (t) = sum (F (x, t)) dx + sum (F (y, t)) dy (1) Next, camera operation information is extracted. FIG. 4 shows camera operation information to be extracted. The camera operation includes seven basic operations and a combination thereof. Basic operations include:
Fixed (fixed camera), pan 401 (operation of shaking the camera left and right), zoom 402 (operation of enlarging or reducing a subject by changing the angle of view), tilt 403 (operation of shaking the camera up and down), track 404 (operation of shaking the camera up and down) There are operations for moving the camera left and right), boom 405 (operation for moving the camera up and down), and dolly 406 (operation for moving the camera back and forth). That is, fix is an operation involving stationary, pan and tilt are changes in the optical axis direction fixed to the camera projection center, zoom is an operation involving a change in the angle of view, and track, boom, and dolly are operations involving a change in the position of the camera projection center. With the change in the position of the camera projection center, the track, boom, and dolly are operations that include the three-dimensional arrangement information of the subject in the motion of the image. The movement of an image captured by a truck, a boom, and a dolly indicates a fast movement when the subject is relatively close to the camera, and a slow movement when the subject is fast.

The xt spatiotemporal projection image calculated by the above method is represented by F (x, t). The xt spatiotemporal projection image is
It is a function of space x and time t. The spatial distribution F (x, t ₀₎ of the x-t spatio-temporal projection images of time t ₀ expressed in, similarly the spatial distribution of the time t ₁ of the x-t spatio-temporal projection images F (x, t ₁₎
Expressed by The global motion parameters calculated below are represented by a, b, and c, where a is a zoom parameter indicating camera operation, b is a pan parameter, and c is a tilt parameter. Hereinafter, a method for obtaining the camera operation parameters will be described.

If there is a global motion in the created xt spatiotemporal projection image, F (x,
t ₀ ) and F (x, t ₁ ) have the following relationship.

[0034]

F (x%, t ₁ ) = F (ax + b, t ₀ ) Similarly, the yt spatiotemporal projection image includes:

## EQU3 ## There is a relationship of F (y%, t ₁ ) = F (ay + b, t ₀ ). First, the association between x% and x and y% and y is performed by the association unit. This is shown in FIG.

In FIG. 5, reference numeral 501 denotes an xt spatiotemporal projection image, and 502 denotes a spatial distribution F (x, T) of time T, 503.
Represents a spatial distribution F (x, T-1) at time T-1. The coordinate values of 504 are associated as shown in the figure,
The corresponding coordinates 505 are calculated. In addition to this calculation method, it is also possible to calculate and associate a correlation function for each minute range. The associated 504 and 505 indicate arbitrary coordinate values, and the relationship between these coordinates is a straight line 506. The inclination of this straight line represents the zoom parameter a, and the intercept represents the pan parameter b.

Subsequently, the camera operation parameters are calculated by the camera operation parameter calculation means using the spatial coordinate values associated with the above. More specifically, in order to calculate a and b of the straight line 506, the maximum value 507 of the projection space is extracted by projecting (voting) to the parameter space using the associated spatial coordinate values according to the following relational expression. , Parameters a and b
Is calculated. This conversion is generally performed by Hough
Conversion [P. V. C. Hough, "Method an
d Means for Recognizing C
omplex Patterns ", US Pat.
nt No. 306954, 1962]. When the associated arbitrary coordinates are represented by x% and x, the relationship between a and b is as follows.

[0037]

B = x% · cos (a) + x · sin (a) The Hough transform is generally established as a method for estimating a straight line constituted by a plurality of points. One point in the image space represents one curve in the (projection space) Hough space, and an intersection 507 of a plurality of projected curves is provided.
Indicate the slope and intercept of the straight line to be extracted.

The computer votes the straight line in the projective space, and calculates the slope and intercept of the straight line to be extracted with the coordinate value indicating the maximum number of votes. The parameters are calculated by voting each of a plurality of sets of coordinate values associated with each other in the projective space.

Similarly, the tilt parameter c can be calculated from the yt spatiotemporal projection image. Further, even if the image is a spatiotemporal projection (integral) image when an operation including three-dimensional information is performed, an image obtained when an operation that does not include a three-dimensional operation is performed microscopically (partially). Therefore, even such an image can be processed by partially (blockwise) performing the above processing. The above is the photographing state detection unit 10
3 is a configuration and processing flow.

Next, returning to FIG. 1, the video dividing unit 104 divides the video into shots based on the camera on / off information calculated by the photographing state detecting unit 103. In the shot divided by turning the camera on and off in this way, the images in the shot are considered to have continuous information of the same space.

Next, the subject and background are separated in the subject / background separation unit 105. Subject / background separation unit 1
FIG. Hereinafter, the subject / background separation unit 1
The configuration of the components 601 to 604 of 05 and the flow of processing by the components will be described in detail.

First, the camera operation canceling unit 601 cancels the camera operation from the image data based on the camera operation information by the video frame deforming means. The image data is changed / displaced by an amount corresponding to the change / displacement generated by the camera operation between the adjacent image data. F (x, y, t), F (x,
y, t + 1). When the camera operation A uses (A is a matrix), the following relationship exists between adjacent image data.

[0043]

F (x, y, t + 1) = AF (x, y, t) Cancellation of the camera operation can be expressed by the following equation.

[0044]

F (x, y, t) = A ⁻¹ F (x, y, t + 1) Subsequently, in the image data comparison unit 602, the adjacent camera whose camera operation has been canceled by the difference processing means as described above. Perform comparison processing between images. The processing performed here is a comparison processing between the images for which the camera operation has been cancelled, and what is calculated is an absolute value of a difference between information such as luminance and color between the images. By this comparison processing, the background is subtracted, and a change in only the movement of the subject is extracted as a difference from the background.

Subsequently, the comparison image data comparison unit 603 compares adjacent comparison image data. The processing performed here is a comparison operation such as a product between images or a smaller value compared between images as a value of a comparison image. Through a series of these processes, the region of the subject (moving object) in the middle image is enhanced based on three continuous image data.

Subsequently, in the area extracting unit 604, first,
The binarization processing of the emphasized subject area is performed by the binarization processing means. Using the threshold value S given in advance, binarization processing is performed under the following conditions. Let the comparison data image be f (x, y) and let the binarized image be F (x, y).

[0047]

F (x, y) = 1: if f (x, y) ≧ S F (x, y) = 0: if f (x, y) <S FIG. 7 shows an example of the binarization processing. Show. 701 is a comparison data image, and 702 is a binarized image. It is assumed that the threshold is set to 9.

Subsequently, the labeling means labels the binarized image F (x, y). The labeling algorithm is shown in FIG. The binarized image F (x, y) is represented by F =
{F _{i, j} } and the label image are represented by L = {l _{i, j} }. l _{i, j}
Is a positive integer representing the label of each connected component. Also, 1 is a variable representing a connected component number, and T (i) represents a label table. In the initial setting of 801, l = 1 is set, and label scanning is started from pixel (2, 2). The current pixel is set to (i, j), and the judgment is made at 802. f _{i, j} = 1
If f _{i, j} = 0, then go to 803 and set l _{i, j} = 0 to 807
Processing proceeds to In the scanning method shown in FIG. 9, the current pixel x ₀ = (i, j) and the pixels which have been operated in the vicinity thereof are shown in FIG.
0, and the label of x _p (the value of image L) is l _p
(P = 1, 2, 3, 4). In 803, ΔT (l
_p), l _p is not 0, and p = 1, 2, 3, 4} different positive value during the there are n type, L them from a small value in order
₁ , L ₂ ,..., L _n . If n = 0, go to 804, n =
If it is 1, the processing proceeds to 805, and if n = 2, the processing proceeds to 806,
After each process, the process proceeds to 807. In 807, it is determined whether or not the processing is completed for all the pixels.
The processing of 809 is performed to complete the labeling.

Next, the physical feature calculating means calculates a physical feature for the labeled area.
The physical characteristics calculated here are luminance, color distribution, texture, and the like. Next, the matching unit compares and compares the feature amount given in advance with the calculated feature amount of the label region to determine a subject region. Next, the background region is separated by subtracting the subject region extracted above from the frame image by the background extraction unit.

The above is the configuration of the subject / background separation unit 105 and the flow of processing.

Returning to FIG. 1, in the subject motion information extraction unit 106, the comparing means compares the physical feature amounts calculated in the area extracted for each frame image between adjacent frames. The comparison amount is compared with a condition given in advance, and a temporal relationship is performed assuming that the compared region is a region having similar physical characteristics, that is, a region of the same subject. This association information is used as the motion information of the subject.

On the other hand, in the photographing space resynthesizing unit 107, the subject / background separating unit 1 is operated by the spatial superimposing means.
The background separated in 05 is overlaid as one continuous space by deforming and displacing the image frame based on the camera operation information calculated by the shooting state detection unit 103, and recombining as a wide shooting space beyond the frame. . FIG.
Fig. 3 shows the re-synthesis. In FIG. 11A, 110
1 and 1102 are temporally continuous frame images,
If the image is shot by panning the camera, 110
2 is shifted by 1103 (pan operation amount per frame) with respect to 1101 and synthesized. Similarly, in the case of tilt, the images are shifted and combined by 1104 (the amount of tilt operation per frame). In the case of the zoom operation, FIG.
As shown in (b), the size of the image is enlarged and reduced in accordance with the zoom amount 1105, and is synthesized. The background image created by this synthesizing method is a general panoramic space, and has a unique distortion that the panoramic space has. This distortion is equivalent to the distortion that occurs when an image is converted into a cylinder.

Next, in the inter-shot relation calculation unit 108,
The photographing space conversion means converts the photographing space for the recombined photographing space for each shot so that the size and position of the photographing space are equal, and then calculates the inter-shot relation information. . Here, the information to be calculated is information relating to the space of the shot. The arrangement relationship of each space recorded in the shot to be related is defined as inter-shot relationship information. The relationship between the shots is calculated by comparing the panoramic space recombined by the photographing space combining unit 197. Panoramic spaces created from images shot at the same focal length have the same cylinder transformation distortion. FIG. 12 shows a state of the cylinder conversion. The line 1201 on the three-dimensional space is transformed into a 1202 on the cylinder 1203 by the cylinder transformation.
Is converted to The point O in FIG. 12 is the projection center of the camera, and the area above the cylinder 1203 is the image plane. The size of the cylinder is uniquely determined by the size of the focal length f at the time of shooting. As a result, the panorama space created from the video shot at the same focal length has the same cylinder transformation distortion.
The relationship between panoramic spaces created from videos shot at the same focal length can be obtained by calculating the amount of spatial translation of those images. One panoramic image can be calculated by matching using a mutual function count as a template. In this calculation, stable matching can be achieved by newly defining an evaluation function from the correlation between the overlapping portions and the overlapping region. In order to relate images having different focal lengths, matching must be performed in consideration of cylindrical distortion. The relationship can be calculated by performing matching while gradually changing the focal length f with reference to one image. It is possible to calculate the relationship between the video images captured without performing the camera operation, by enlarging / reducing the size of the image and performing matching.

The calculated camera on / off information, camera operation information, subject information, subject motion information, recombined background information, and shot relation information are used for a plurality of videos.
The video structure information is managed and stored in the image structure information management and storage unit 109 spatiotemporally. In the past, video has been managed in the storage device as a file or as an RGB time-varying signal. Control at the signal level was effective in mechanical processing, transmission, storage, and display. In the multimedia era, where images can be handled by computers, the handling of images is not limited to conventional signal processing, storage, and display, but rather advanced techniques such as searching, editing, and processing from a large amount of stored video databases. Video is becoming a treatment. In order to enable such advanced handling of video, the video must be managed with information at a level that is deeper into the content of the video than at the signal level. The extracted information is information at this level, and by expressing and managing a video with this information, it is possible to perform a higher-level handling completely different from a video expressed only with the RGB time-varying signal. This expression reflecting the contents of the image is considered to be a semantic expression relating to the time and space of the image, unlike a mere time change of the signal. By accumulating and managing the new expression of the video in a spatiotemporal manner, it becomes possible to realize advanced video handling that is intuitive and easy for humans to understand.

The structure information and video data of the extracted video are compressed and stored. By compressing the data, the storage space can be made more efficient, and data can be transmitted and received via a network. Information that changes with time as a variable in the video structure information is compressed using lossless coding such as Huffman coding. Information relating to a space and relating to an image (for example, an extracted image of a subject or a recombined background image) is compressed as a still image using irreversible coding. A typical method of lossy encoding is
JPEG encoding. An image recombined using the present invention from a video taken by rotating the camera 360 degrees horizontally with a 320 × 240 image has about 5 megabytes of information and 10 minutes using JPEG encoding. Assuming a compression efficiency of 1, the data can be compressed to about 500 kilobytes of data. When the current Internet or the like is used, data transmission of about 5 megabytes is impossible due to time limitation, but transmission of about 500 kilobytes of data is practically possible. In addition, the image of a subject that changes with time, and the like, are also irreversible encoded by H261 or MPEG.
Approximately 1/10 to 1/20 by using encoding such as
Can be compressed to a degree. When sending a video structured by a thin line (line with a low information transmission rate) such as the Internet, a background image compressed by JPEG as a still image,
Similarly, JPEG is used as a still image in which time information is expanded in space (for example, a strobe image in which a subject is expanded in the background).
By sending the minimum information required to respond to the user's request, such as compressed subject information, it is possible to realize network use, time efficiency and good interaction response.

Next, in the re-synthesizing unit 1101, the video structure information management and management is performed in response to a request from the user input unit 112, or in accordance with a previously given condition, or in accordance with both a user request and a previously given condition. The information managed by the storage unit 109 is re-synthesized. It is also possible to filter semantic images. It is also possible to create an image of only the background or only the subject. It is also possible to create an image that expands information information from time and space video information to space and expresses it as information only for space. As an example of this, there is an image in which a subject is strobe-likely expressed in a space where a panorama has been developed as described above. In the prior art, in addition to being able to achieve only a strobed representation that is temporally sampled, a strobed representation that is spatially sampled is also possible.

FIG. 13 shows a strobe representation of time and space sampling. Reference numeral 1301 denotes a recombined panoramic space. Reference numeral 1302 denotes a subject. 1303
Is the spatially sampled strobe representation and 13
04 is a strobe expression sampled temporally. Reference numeral 1304 denotes the arrangement of the subject also expresses the speed of the subject, while reference numeral 1303 denotes a change in the space of the subject. It is also possible to combine a plurality of subjects existing in different shots with one background using the inter-shot relationship information. For example, the subject (player A) photographed in shot A may be strobe-represented in a panoramic space, and the subject (player B) photographed in shot B may be overlapped on the strobe representation and displayed as a moving image. It is possible. Here, shot A and shot B have a spatially common space. This enhancement of the video makes it easy for the user to visually and intuitively grasp the good selection, the difference in form with the poor player, and the like.

In the re-combining unit 110, the user input unit 112
Various expressions are possible according to the user's request. From the video displayed on the display unit 111, the user can give feedback of the expression to the user's request via the user input unit 112. The output unit 113 can perform digital and analog outputs. The digital output is an output to an external printer, a personal computer, or the like, and the analog output is a video signal output to a monitor or the like. Note that the output unit 113 may be provided as needed.

As described above, the present invention has been specifically described based on one embodiment. However, the present invention is not limited to the above embodiment, and various changes can be made without departing from the gist of the present invention. It goes without saying that it is possible.

[0060]

As described above, according to the present invention, video data is read, a stored image data sequence is read,
Detects shooting state information including camera on / off information and camera operation information, divides an image into shots based on the camera on / off information, and uses the camera operation information and physical features to determine the subject and background for each frame. The separated subject information is correlated between frames to extract subject motion information, the shooting space is re-synthesized from the camera operation information and the background for each frame, and a plurality of re-synthesized images are obtained from a plurality of shots. The spatial relationship between shots between the shooting spaces is calculated, and the information obtained above is managed and stored, so that the spatio-temporal enhancement of multiple images, spatio-temporal, unified management, Spatio-temporal integration of multiple images that enables expression and operation becomes feasible.

Further, based on the information managed and stored, one or more photographing spaces and one or more subjects are recombined and displayed according to a given condition or a request from a user. Users can simultaneously and intuitively and efficiently obtain information on multiple videos in the same style according to their interests and purposes for multiple videos that have been shot in the same space by outputting them in analog or digital format to the outside. It is possible to do.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration and a processing flow of an embodiment of the present invention.

FIG. 2 is a diagram illustrating a configuration and a processing flow of a shooting state detection unit according to the embodiment.

FIG. 3 is a flowchart of a process of a shooting state detection unit in the embodiment.

FIG. 4 is a diagram illustrating camera operation information in the embodiment.

FIG. 5 is a diagram illustrating an algorithm for extracting camera operation information in the embodiment.

FIG. 6 is a diagram illustrating a configuration of a subject / background separation unit and a processing flow in the embodiment.

FIG. 7 is a diagram illustrating an example of a binarization process in the embodiment.

FIG. 8 is a flowchart of a labeling process in the embodiment.

FIG. 9 is a diagram showing an image scanning order in the embodiment.

FIG. 10 is a diagram showing target pixels and scanned pixels in the embodiment.

FIG. 11 is a diagram illustrating a method of recombining a shooting space based on camera scanning in the embodiment.

FIG. 12 is a diagram illustrating a cylinder conversion.

FIG. 13 is a diagram illustrating strobe expression by time and space sampling according to the embodiment.

[Explanation of symbols]

 Reference Signs List 101 video signal input unit 102 image data string memory unit 103 shooting state detection unit 104 video division unit 105 subject / background separation unit 106 subject motion information extraction unit 107 shooting space recombining unit 108 shot relation relation calculation unit 109 video structure information management / Storage unit 110 re-synthesis unit 111 display unit 112 user input unit 113 output unit 114 video structured information transmission / reception unit 201 linear component calculation unit 202 image data rearrangement unit 203 video information filter unit 204 feature extraction unit 205 feature statistical analysis unit 301 Spatiotemporal image 302 Vertical straight line calculation 303 Horizontal straight line calculation 304 Vertical method linear component spatiotemporal image 305 Horizontal method linear component spatiotemporal image 306 Rearrangement processing 307 Filter processing 308 Edge image normal direction addition processing 309 xt Spatio-temporal projection image 310 yt spatiotemporal projection image 601 camera operation canceling unit 602 image data comparing unit 603 comparative image data comparing unit 604 region extracting unit

Claims (57)

[Claims] 1. An apparatus for spatially and spatially integrating and managing a plurality of videos, an image data sequence memory unit for reading video data and storing the data as a data sequence, a data sequence read from the image data sequence memory unit, and a camera A shooting state detection unit that detects shooting state information including on / off information and camera operation information; a video division unit that divides the video of the data sequence into shots based on the camera on / off information; A subject and background separating unit that separates a subject and a background for each frame of the video using various feature amounts; a subject motion information extracting unit that associates subject information separated for each frame between frames; A shooting space re-synthesizing unit for re-synthesizing a shooting space in which the video is shot from a background separated for each frame; and An inter-shot relationship calculation unit for calculating a spatial inter-shot relationship between a plurality of imaging spaces recombined by the imaging space recombining unit from a number of shots; information on the separated subjects; And a video structure information managing / accumulating unit for managing / accumulating information of the obtained subject, the shooting state information, the information of the background, and the information of the relation between shots. , Management equipment. 2. A video structured information transmission / reception unit for transmitting or receiving all or a part of the extracted subject information, photographing state information, background information, shot relation information, and video data. The spatio-temporal integration / management apparatus for a plurality of videos according to claim 1, wherein: 3. One or a plurality of photographing spaces and one or more photographing spaces based on information stored and managed in the video structure information management and storage unit according to one or both of a condition given in advance and a request from a user. Or a re-synthesizing unit for re-synthesizing a plurality of subjects; a display unit for displaying an image re-synthesized by the re-synthesizing unit; and a request from the user for re-synthesis based on the image displayed on the display unit. A user input unit for inputting, and an output unit for outputting an image displayed on the display unit in a digital or analog format to an external device as necessary, further comprising: A spatio-temporal integration and management device for a plurality of videos as described in. 4. The image capturing state detection unit includes: a linear component calculation unit that calculates horizontal and vertical linear components for an image frame read from an image data memory unit; and the calculated horizontal and vertical components. An image data arrangement rearranging section for rearranging a temporal and spatial arrangement of an image including a linear component; an image information filter section for performing a filtering process on the image data obtained by the rearranging; and a feature based on a result of the filtering process. A feature extraction unit for extracting the feature information; and a feature statistical analysis unit for statistically analyzing the extracted feature to detect camera on / off information and camera operation information. Or a spatio-temporal integration and management device for a plurality of videos according to 3. 5. The image data rearranging unit includes image data sequence rearrangement means for rearranging the image data sequence into a plurality of spatiotemporal sectional images including a normal line and a time axis of the image. The apparatus for integrating and managing spatio-temporal images of a plurality of videos according to claim 4. 6. The plurality of video information processing apparatuses according to claim 4, wherein the video information filter processing unit includes a line segment detecting unit that detects an edge or a line of video information of the rearranged image data. A spatio-temporal integration and management device for video. 7. The method according to claim 6, wherein the feature extracting unit includes an integrating unit that adds information on the detected edge or line in a normal direction of an image. Spatial integration and management equipment. 8. A camera on / off detecting means for detecting a straight line perpendicular to the time axis of an extracted feature expressed in a two-dimensional image having a time axis and a space axis, and calculating camera on / off information. The spatio-temporal integration / management apparatus for a plurality of videos according to claim 4, comprising: 9. The feature statistical analysis unit compares a spatial distribution of two arbitrary times of an extracted feature expressed in a two-dimensional image having a time axis and a spatial axis, and associates spatial coordinates with each other. The spatio-temporal integration / management apparatus for a plurality of videos according to claim 4, comprising: attaching means; and camera operation parameter calculating means for calculating camera operating parameters by statistically processing the associated spatial coordinates. . 10. A camera operation canceling unit that cancels a camera operation from adjacent image data based on camera operation information detected by a shooting state detection unit, wherein the camera operation information cancels out. An image data comparison unit for comparing the obtained image data, a comparison image data comparison unit for comparing the adjacent comparison image data, and a subject area extracted from the comparison data calculated by the comparison image data comparison unit The spatio-temporal integration and management device for a plurality of videos according to claim 1, 2, or 3, further comprising: 11. The camera operation canceling unit includes a video frame deforming unit that deforms and displaces an adjacent image frame based on the camera operation information so as to cancel the deformation and displacement caused by the camera operation. The apparatus for integrating and managing spatiotemporal images of a plurality of videos according to claim 10. 12. The image data comparison unit according to claim 10, wherein the image data comparison unit includes difference processing means for performing difference processing of luminance and color information between adjacent image data whose camera operations have been canceled. A spatio-temporal integration and management device for multiple images. 13. The binarization processing unit that performs binarization processing on the comparison data, and a label that labels the binarized data that has been binarized. Attaching means; physical characteristic calculating means for calculating physical characteristics of the labeled area; collating for extracting a subject area by collating with a predetermined condition with respect to the calculated physical characteristics. The spatio-temporal integration / management apparatus for a plurality of videos according to claim 10, comprising: 14. The apparatus according to claim 1, wherein the subject / background separation unit includes a background extraction unit that extracts a background by subtracting a subject area for each frame extracted by the area extraction unit from a frame image. 14. The spatio-temporal integration and management device for a plurality of videos according to 13. 15. The subject motion information extraction unit performs a comparison with a condition given in advance to a comparison amount of a physical feature of a region temporally adjacent to a subject region for each frame extracted by the region extraction unit. 14. The spatio-temporal integration / management apparatus for a plurality of videos according to claim 13, further comprising matching means for associating subject information between frames. 16. The image capturing space re-synthesizing unit deforms and displaces adjacent image frames based on camera operation information detected by the image capturing state detecting unit, and superimposes the image frames as one continuous image capturing space. The spatio-temporal integration and management apparatus for a plurality of videos according to claim 1, 2, or 3, further comprising means. 17. The inter-shot relation calculating unit sets the photographing space to each photographing space created by the photographing space re-synthesizing unit so that the photographing space has the same size and position between the photographing spaces. 4. The apparatus according to claim 1, further comprising a photographing space converting means for performing conversion. 18. The image structure information managing / accumulating unit, wherein the extracted subject information, shooting state information, background information, shot relationship information, video data, shooting state information,
Means for compressing all or part of the shooting space of a plurality of videos recombined using background information, inter-shot relation information, and video data, and managing and accumulating them in time and space. The spatio-temporal integration and management device for a plurality of videos according to claim 1. 19. The image structure information managing / accumulating unit expands the extracted subject information in a space and compresses the data as a still image, and converts the shooting space of the recombined plurality of images into a still image as a still image. 19. The apparatus according to claim 18, further comprising: a compression unit. 20. A method for spatio-temporally integrating and managing a plurality of images, comprising: a step of reading image data and storing the image data as a data string; and a step of reading a data string from the image data string memory unit. A photographing state detecting step of detecting photographing state information including on / off information and camera operation information; an image dividing step of dividing an image of the data string into shots based on the camera on / off information; A subject and background separating step for separating the subject and the background for each frame of the video using various feature amounts; a subject motion information extracting step for associating the subject information separated for each frame between the frames; And a photographing space synthesizing step of recomposing a photographing space in which the video is photographed from a background separated for each frame, A shot relation calculation step of calculating a spatial shot relation of a plurality of shooting spaces recombined in the shooting space resynthesis process from the divided shots in the shooting space resynthesis process; information on the separated subject; A video structure information managing / accumulating process for managing / accumulating information of the attached subject, the photographing state information, the background information, and the information of the relation between shots. Integration and management methods. 21. A video structured information transmission / reception step of transmitting or receiving all or a part of the extracted subject information, photographing state information, background information, shot relation information, and video data. 21. The method according to claim 20, wherein the spatiotemporal integration and management of a plurality of videos are performed. 22. After the video structure information managing / accumulating process, one or a plurality of photographing spaces based on the information accumulated / managed according to one or both of a condition given in advance and a request from a user. 22. A re-synthesizing step of re-synthesizing one or a plurality of subjects, and a display or output step of displaying or outputting a video re-synthesized in the re-synthesizing step. Method for integrating and managing spatio-temporal images as described in 1. 23. The photographing state detecting step includes: a straight line component calculating step of calculating horizontal and vertical straight line components for an image frame read from an image data string memory step; Image data arrangement rearranging step of rearranging the temporal and spatial arrangement of the image including, image information filtering processing step of performing a filtering process on the image data obtained by the rearranging, and a characteristic from the result of the filtering process. 22. The method according to claim 20, further comprising: a feature extraction step of extracting; and a feature statistical analysis step of statistically analyzing the extracted features to detect camera on / off operation information and camera operation information. Or 2
2. The method for spatio-temporal integration and management of multiple images according to 2. 24. The image data arrangement rearranging process, wherein the image data sequence is rearranged and rearranged into a plurality of spatiotemporal sectional images including a normal line and a time axis of the image. Method for integrating and managing spatio-temporal images as described in 1. 25. The video information filtering process according to claim 23, wherein edges or lines of video information of the image data obtained by the rearrangement are detected.
Method for integrating and managing spatio-temporal images as described in 1. 26. The method according to claim 25, wherein in the feature extracting step, information on the detected edge or line is added in a normal direction of an image to extract a feature. Spatial integration, management method. 27. The feature statistical analysis process, wherein a straight line perpendicular to the time axis of the extracted feature expressed in a two-dimensional image having a time axis and a space axis is detected, and camera on / off information is calculated. 24. The method for spatio-temporal integration and management of a plurality of videos according to claim 23. 28. The feature statistic analysis step compares the spatial distribution of any two of the extracted features expressed in a two-dimensional image having a time axis and a spatial axis, and associates the spatial coordinates. 24. The spatio-temporal integration of a plurality of videos according to claim 23, further comprising: attaching a camera operation parameter, and statistically processing the associated spatial coordinates to calculate a camera operation parameter. Management method. 29. The subject / background separating step includes: a camera operation canceling step of canceling a camera operation between adjacent image data based on camera operation information detected in a shooting state detecting step; An image data comparison process of comparing the compared image data, a comparison image data comparison process of comparing the adjacent comparison image data, and a subject area extracted from the comparison data calculated from the comparison image data comparison process 22. An area extracting step, comprising:
2. The method for spatio-temporal integration and management of multiple images according to 2. 30. The camera operation canceling step according to claim 29, wherein an adjacent image frame is deformed / displaced based on the camera operation information so as to cancel the deformation / displacement due to the camera operation. Spatio-temporal integration and management method of multiple videos described. 31. The spatio-temporal image processing method according to claim 29, wherein, in the image data comparison process, difference processing of luminance and color information is performed between adjacent image data whose camera operations have been canceled. Integration and management methods. 32. The region extracting step includes: a binarizing process for performing a binarizing process on the comparison data; and a label for labeling the binarized data that has been subjected to the binarizing process. Attaching step, calculating a physical characteristic of the labeled area, and comparing the calculated physical characteristic with a predetermined condition to extract a subject area. 30. The method of claim 29, further comprising the steps of: 33. The plural images according to claim 32, wherein in the subject / background separating step, a background is extracted by subtracting a subject area for each frame extracted in the area extracting step from a frame image. Spatio-temporal integration, management method. 34. In the subject motion information extracting step, a comparison is made with a condition given in advance to a comparison amount of a physical feature of a temporally adjacent area of the subject area for each frame extracted in the area extracting step. 33. The spatio-temporal integration and management method for a plurality of videos according to claim 32, wherein subject information between the frames is associated with each other. 35. In the photographing space re-synthesizing step, adjacent image frames are deformed and displaced based on camera operation information detected in the photographing state detecting step, and are superimposed as one continuous photographing space. 23. The method according to claim 20, 21, or 22, wherein the plurality of videos are spatio-temporally integrated and managed. 36. In the inter-shot relation calculating step, the photographing space is created such that the size and position of the photographing space are equal between the photographing spaces created in the photographing space recombining step. 23. The spatio-temporal integration and management method for a plurality of videos according to claim 20, 21 or 22. 37. The image structure information managing / accumulating process includes:
The extracted subject information, shooting state information, background information,
All or a part of the shooting space of a plurality of videos recombined using the shot relationship information, video data, and shooting status information, background information, shot relationship information, and video data, or a part thereof, is subjected to spatio-temporal processing. 21. The spatio-temporal integration / management method for a plurality of videos according to claim 20, wherein the management / accumulation is performed. 38. The image structure information managing / accumulating process includes:
38. The method according to claim 37, wherein the extracted subject information is expanded in a space, the data is compressed as a still image, and the shooting space of the recombined plurality of videos is compressed as a still image. Spatial integration, management method. 39. A computer-readable recording medium on which a program for integrating and managing a plurality of videos in a spatiotemporal manner is recorded, wherein: an image data sequence storing step of reading video data and storing the data as a data sequence; A photographing state detecting step of reading a data string from a memory unit and detecting photographing state information including camera on / off information and camera operation information; and a video dividing step of dividing an image of the data string into shots based on the camera on / off information. A subject for separating a subject and a background for each frame of the video using the camera operation information and a physical feature amount, a background separation process, and a subject motion for associating the subject information separated for each frame between frames Extracting an image from the camera operation information and the background separated for each frame. A photographing space combining process of recombining the divided photographing spaces; and a shot space calculating a spatial inter-shot relationship of a plurality of photographing spaces recombined from the divided plurality of shots in the photographing space recombining process. A relation calculating step, and a video structure information managing / accumulating step of managing / accumulating the information of the separated subject, the information of the associated subject, the shooting state information, the background information, and the information of the relation between shots. A computer-readable recording medium on which a spatio-temporal integration of a plurality of videos and a management program are recorded, comprising: 40. The apparatus further comprises a video structured information transmission / reception step of transmitting or receiving all or a part of the extracted subject information, photographing state information, background information, shot relation information, and video data. 40. A computer-readable recording medium on which a spatio-temporal integration of a plurality of videos and a management program according to claim 39 are recorded. 41. After the video structure information managing / accumulating process, one or a plurality of photographing spaces based on the information accumulated / managed according to one or both of a condition given in advance and a request from a user. 41. A re-synthesizing step of re-synthesizing one or a plurality of subjects, and a display or output step of displaying or outputting a video re-synthesized in the re-synthesizing step. A computer-readable recording medium on which a spatio-temporal integration of a plurality of videos and a management program are recorded. 42. The photographing state detecting step includes: a linear component calculating step of calculating horizontal and vertical linear components for an image frame read from an image data string memory step; and the horizontal and vertical linear components. Image data arrangement rearranging step of rearranging the temporal and spatial arrangement of the image including, a video information filtering step of performing a filtering process on the image data obtained by the rearranging, and a feature from the result of the filtering process. 41. A feature extracting step of extracting, and a feature statistical analyzing step of statistically analyzing the extracted features to detect camera on / off operation information and camera operation information. Or 4
2. A computer-readable recording medium recording the spatio-temporal integration and management program of a plurality of videos according to 1. 43. The image data arrangement rearranging step, wherein the image data sequence is rearranged and rearranged into a plurality of spatiotemporal sectional images including a normal line and a time axis of the image. A computer-readable recording medium on which a spatio-temporal integration of a plurality of videos and a management program are recorded. 44. The video information filtering process, wherein an edge or a line of video information of the image data obtained by the rearrangement is detected.
A computer-readable recording medium on which a spatio-temporal integration of a plurality of videos and a management program are recorded. 45. The method according to claim 44, wherein in the feature extracting step, information on the detected edge or line is added in a normal direction of an image to extract a feature. A computer-readable recording medium that records a space integration and management program. 46. In the feature statistical analysis step, a straight line perpendicular to a time axis of an extracted feature expressed in a two-dimensional image having a time axis and a space axis is detected, and camera on / off information is calculated. 43. A computer-readable recording medium on which a spatiotemporal integration of a plurality of videos and a management program according to claim 42 are recorded. 47. The feature statistic analysis step compares the spatial distribution of any two of the extracted features represented in a two-dimensional image having a time axis and a spatial axis with each other and associates spatial coordinates. 43. A spatio-temporal integration of a plurality of videos according to claim 42, further comprising: attaching a camera operation parameter to statistically process the associated spatial coordinates to calculate a camera operation parameter. A computer-readable recording medium recording a management program. 48. The subject / background separating step includes: a camera operation canceling step of canceling a camera operation between adjacent image data based on camera operation information detected in a shooting state detecting step; An image data comparison process of comparing the compared image data, a comparison image data comparison process of comparing the adjacent comparison image data, and a subject area extracted from the comparison data calculated from the comparison image data comparison process 41. An area extracting step, comprising:
2. A computer-readable recording medium recording the spatio-temporal integration and management program of a plurality of videos according to 1. 49. The method according to claim 48, wherein, in the camera operation offsetting step, an adjacent image frame is deformed / displaced based on the camera operation information so as to offset a deformation / displacement caused by a camera operation. A computer-readable recording medium on which a spatio-temporal integration of a plurality of videos described above and a management program are recorded. 50. The spatio-temporal image processing method according to claim 48, wherein, in the image data comparison process, difference processing of luminance and color information is performed between adjacent image data whose camera operations have been canceled. A computer-readable recording medium that records an integration and management program. 51. The area extracting step includes: a binarizing processing step of performing a binarizing process on the comparison data; and a label for labeling the binarized binary data. Attaching step, calculating a physical characteristic of the labeled area, and comparing the calculated physical characteristic with a predetermined condition to extract a subject area. 49. The computer-readable recording medium according to claim 48, further comprising the steps of: storing a spatio-temporal integration of a plurality of videos and a management program. 52. The plural images according to claim 51, wherein in the subject / background separating step, a background is extracted by subtracting a subject area for each frame extracted in the area extracting step from a frame image. A computer-readable recording medium that records the spatio-temporal integration and management program. 53. In the subject motion information extracting step, a comparison is made with a condition given in advance to a comparison amount of a physical feature of a temporally adjacent area of the subject area for each frame extracted in the area extracting step. 52. The computer-readable recording medium according to claim 51, wherein the image information is associated with frames, and the spatio-temporal integration of a plurality of videos and a management program are recorded. 54. In the photographing space resynthesizing step, adjacent image frames are deformed and displaced based on camera operation information detected in the photographing state detecting step, and are superimposed as one continuous photographing space. 42. A computer-readable recording medium on which a spatio-temporal integration of a plurality of videos and a management program are recorded according to claim 39, 40 or 41. 55. In the shot relationship calculation step, the photographing space for each shot created in the photographing space re-synthesizing step is set such that the size and position of the photographing space are equal between the photographing spaces. 42. The computer-readable recording medium according to claim 39, 40 or 41, wherein the computer-readable recording medium records a spatio-temporal integration of a plurality of videos and a management program. 56. The video structure information managing / accumulating process includes:
The extracted subject information, shooting state information, background information,
All or a part of the shooting space of a plurality of videos recombined using the shot relationship information, video data, and shooting state information, background information, shot relationship information, and video data, is compressed in time and space. 40. The computer-readable recording medium according to claim 39, wherein the computer-readable recording medium stores a management program. 57. The image structure information managing / accumulating process includes:
57. The method according to claim 56, wherein the extracted subject information is expanded in a space, the data is compressed as a still image, and the shooting space of the recombined plurality of videos is compressed as a still image. A computer-readable recording medium that records a space integration and management program.