JPH0779382A - Moving picture processing unit - Google Patents

Abstract

PURPOSE:To edit a motion of plural operating moving objects on different time spaces to a picture projected to an optional time space by providing a picture conversion means converting a time space map picture of the moving objects into a picture picked up from a different time space. CONSTITUTION:The processing unit is provided with a camera head 1 picking up an object being a moving object to provide an output of a picture picked up together with 3-dimension information, a data recording section 2 recording the picture with the 3-dimension information and time information, and a moving picture edit section 3 executing time space mapping to the obtained picture to convert the picture, and also with a stop watch 5 providing an output of time information being a reference for the management of the time information and a time base unit 4 distributing the time information outputted from the stop watch 5 to the camera head 1 and the data recording section 2. Furthermore, as the configuration to extract the 3-dimension information of the mobile object, the 3-dimension space coordinate of the mobile object is calculated by an arithmetic operation unit provided separately.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is used for editing an image of a moving object. INDUSTRIAL APPLICABILITY The present invention is used, for example, in the case where an image of an athlete such as a sports person is edited into an image from an arbitrary time axis or an arbitrary spatial position for broadcasting or motion analysis.

It should be noted that the moving image referred to in this specification is not only an image of a moving object to be imaged in real time, but also continuous images that are imaged and reproduced at time intervals different from the real time movement. including. For example, an image in which a slow-moving image-capturing target object is captured and recorded at a certain time interval and is subjected to processing such as time-lapse shooting at a different time interval from when the image is captured is also a moving image of the target object. If an image of such a moving object is edited at a time interval different from the time interval at the time of image capturing, it can be handled in exactly the same way as a moving image taken in real time. It also includes a moving image that is the same as a still image captured while moving the spatial position.

[0003]

2. Description of the Related Art When analyzing a sport using a video device, for example, when it is desired to compare player A and player B, a plurality of images may be displayed on the screen at the same time.

In this case, when a conventional video device is used, time management is performed for each frame of a video by a time code or the like, and the video is displayed using a plurality of VCRs by synchronizing the start times. By playing back and displaying these videos on the same screen, the movements of multiple players could be displayed on the same screen for comparison.

[0005]

[Problems to be Solved by the Invention] In the case of athletes moving around a wide range such as track and field sports or swimming, if each athlete is up and photographed in order to see the athlete's movement, where is each athlete? I don't understand. In order to be able to see where the players are, such as the rankings, it is necessary to shoot the entire moving range, or make sure that the landmarks are always in the image. However, doing so makes the player's figure smaller and is not suitable for motion analysis.

Even when a wide range is photographed so that the mark is always in the image, the positional relationship seems to change depending on the viewing location. It was difficult to compare the positions of the players in the images. Moreover, since the lens normally uses a zoom lens, information on the angle of view cannot be obtained, and it is not possible to measure the position of a player or the distances of a plurality of players using an image.

Furthermore, there are cases where it is desired to display images taken by the same athlete at different times on one screen for comparison, but it is easy to combine and display images at different times in this way. There wasn't.

An object of the present invention is to solve such a problem, and a moving image processing apparatus capable of editing the motions of a plurality of moving objects in different spatiotemporal areas into an image projected in an arbitrary spatiotemporal area. To provide.

Another object of the present invention is to display, for example, images of athletes on different courses as adjacent images on the same screen, or images of athletes photographed at different times as adjacent images. An object of the present invention is to provide a moving image processing device that can be displayed on a screen for comparison.

[0010]

SUMMARY OF THE INVENTION The present invention relates to a moving image processing apparatus for picking up a moving object and performing image processing on the moving object.
Alternatively, in a moving image processing apparatus including an image pickup means for picking up images of a plurality of moving objects and outputting together with the three-dimensional spatial information of the moving objects, and a moving image editing means for processing and editing an output image of the image pickup means, The moving image editing means is a spatiotemporal image which is an image obtained by projecting the captured image of the moving object on a spatiotemporal map corresponding to the spatiotemporal space in which the moving object moves based on the three-dimensional space information and the time information of the image. It is characterized by further comprising: a spatiotemporal map image generating means for generating a map image; and an image converting means for converting the spatiotemporal map image of the moving object into an image picked up from a different time or a different spatial position.

The moving image editing means is provided with an image recording means for recording the image of the moving object picked up by the image pickup means together with the three-dimensional spatial information of the moving object and the time information relating to the image. Means for processing and editing the captured image of the moving object or the image of the imaging means may be included.

Further, regarding spatiotemporal map images of a plurality of moving objects on the same spatiotemporal map, the spatial positions on the spatiotemporal map of moving objects at distant spatial positions are moved, and then converted into imaged images. Means may be included.

Further, a plurality of spatiotemporal map images of the same or different moving objects imaged at different times have the same time axis and are arranged on the spatiotemporal map so as to be at different spatial positions. Means for converting can be included.

Means for photographing or synthesizing the background image of the moving object separately from the picked-up moving object, and synthesizing the background image with the converted image of the spatiotemporal map image of the moving object. It is preferable to include.

Further, it is possible to display the moving state of the record of which pace distribution is known in the background. In addition to the captured moving object, a means for synthesizing an image of the virtual moving object whose movement characteristic data is clear and the image of the image-converted moving object may be included.

Further, the image converting means can include means for converting an image of a moving object on the spatiotemporal map into an image picked up so that the camera smoothly moves in one direction.

Further, the spatiotemporal map image generating means reads the image recorded in the image recording means at an interval different from the real time interval or an image read out by reversing the order of the readout time. To a spatio-temporal map to generate a spatio-temporal map image, and the image conversion means converts the spatio-temporal map image into an image picked up at a different time interval from the time of projection or from a different spatial position. Can be included.

Further, the image conversion means may include means for moving one space-time map image of the moving object projected on the spatio-temporal map and converting the spatial position with respect to the moving object into an imaged image. .

[0019]

The three-dimensional spatial position information of the moving object can be obtained together with the picked-up image from the image pickup means for picking up the moving object. Along with this image, three-dimensional spatial position information of the moving object (hereinafter referred to as three-dimensional information) and time information are stored together.

The image editing is performed on the image captured by the image capturing means or the image stored in the recording means based on the three-dimensional information of the moving object to be edited and the time information of the image. This image editing process starts with 3
Based on the dimension information and the time information, the moving object is projected on a spatiotemporal map composed of the three axes of the three-dimensional space and the time axis. For example, when the athletes are moving on a common plane, this is imaged on the common plane as an image projected from the position of the camera that captured the image of each athlete.

Then, the image projected on the spatiotemporal map is converted so that the image is picked up from an arbitrary spatiotemporal position. This is exactly the same as the process of projecting the images of the athletes projected on a common plane with a virtual camera in another spatial position and combining them into one image.

In this way, projection (expansion) on the spatiotemporal map
It is possible to create an image that is edited for the moving object itself by changing the time axis and the spatial position of the created image, and to create an image that is closer to the competitors in space. It is possible to display the images on a suitable course for comparison or to display the images of the competitors photographed at different times side by side in real time for comparison.

[0023]

Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) FIG. 1 is a block diagram showing the arrangement of a moving image processing apparatus according to the first embodiment of the present invention. The moving image processing apparatus of this embodiment captures a target object that is a moving object, extracts three-dimensional spatial coordinate information, and outputs the captured image together with the three-dimensional information.
A data recording unit 2 for recording the image together with three-dimensional information and time information, and a moving image editing unit 3 for performing image conversion by performing spatiotemporal mapping of the obtained image. Further, in order to manage the time information, a stopwatch 5 that outputs reference time information and a time base unit 4 that distributes the time information output by the stopwatch 5 to the camera head 1 and the data recording unit 2 are provided. I have it.

As a structure for extracting the three-dimensional information of the moving object by the camera head 1, a three-dimensional spatial coordinate of the moving object is calculated by a calculation device provided separately from the installation location of the camera device. Of course, you can

In this embodiment, an example of editing a moving image of a swimming competition will be described.

In the swimming competition, it can be considered that the movement of the target object (athlete) is less in the height direction than in the horizontal direction and is moving on a plane. Here, as shown in FIG. 2, the three-dimensional position of the target object 7 is simultaneously measured with the image by the camera head 1. Also, the camera head 1
Automatically follows the movement of the target object 7 and tracks the target object. Since the movement of the target object 7 may be limited to the plane 6, if the position of the camera head 1 with respect to the plane 6 and the direction when the target object 7 is viewed from the camera head 1 are known, the intersection of the plane and the straight line The position of the target object in three dimensions can be found by finding At the same time, the photographed range of the plane 6 can be calculated from the angle of view of the lens. Thereby, the three-dimensional information of the target object 7 can be obtained together with the image. Regarding the method of obtaining such three-dimensional information of the moving target object 7 together with the image, the applicant has filed Japanese Patent Application No.
-82178 and Japanese Patent Application No. 5-82209.

The characteristic feature of this embodiment lies in the image processing in the data recording unit 2 and the moving image editing unit 3, which will be described below.

First, the data recording section 2 will be described.

In the data recording section 2, the image and the three-dimensional information and the time information must be recorded in synchronization with each other. One thing to keep in mind here is time management. For relatively slow-moving subjects such as swimming, the unit of time may be 1/60 seconds of the camera, but if multiple unsynchronized cameras are used, another place, another time If the field of the camera is used as the unit of time, a shift of 1/60 seconds at maximum occurs when the camera field is used as a unit of time. For this reason, the stopwatch 5 for measuring the player's time is used as a time reference, and the time base unit 4 is synchronized with the timing signal from the stopwatch 5. The time at which each field to be photographed corresponds to the time measured by the time base unit 4 is recorded in association with the image. That is, instead of field-based time management such as the order of the field, real-time management is performed such that this field is an image from 4 msec after the start to 20.7 msec after the start. This enables unified editing even if the TV systems such as PAL and NTSC or HDTV are different, or the frequency of the master clock of the camera is varied. Also,
Depending on the elements used in the camera and the circuit configuration of the camera, the video signal output from the camera has a certain delay with respect to the video actually shot. CCD
In a camera using an area image sensor, there is a one-field delay from the light image accumulating charges in the element to the reading of the charges from the element. Normally, this delay is specific to the camera and is a constant value. Therefore, if the camera can be specified, it is necessary to correct it by the recording device and record the time when the image is actually captured on the image sensor. This correction may be performed by the camera or not by the recording device. Further, the correction value may be recorded and corrected at the time of reproduction, or may be performed by the editing unit in the next stage.

The three-dimensional information includes, in addition to the position of the three-dimensional point measured by the camera head, the three-dimensional position of the field of view reflected in the camera, the three-dimensional position of the camera head, and the three-axis of the rotation axis of the platform. Position in dimensional space, deviation between optical axis and rotation axis, etc. 3
The head-specific information related to the dimensional space coordinates is also recorded. However, this recording does not have to be performed for all the images, and the unique information of the head and the like may be in the header portion of the image. Also, 3 measured by the camera head
To be precise, the dimensional position is the position of the target object. Since the target object is not always in the center of the image, the position of the target object in the three-dimensional space as well as the position of the target object in the image must be recorded for each image field.

Further, in addition to the calculated information such as three-dimensional coordinates, raw data such as the rotation angle of the pan head angular axis and the zoom value of the lens are also recorded at the same time, so that recovery in the event of any failure, It is desirable to record as much raw data at the same time as it will be useful information for analysis.

Since the information attached to the image as described above is accessed at the same time as the image, it is preferable to record it on the same recording medium as the image. A normal video signal recording device, such as a video recorder, can be used as a recording device or recording medium by appropriately modulating and adding information to an unused area of a normal video signal during a blanking period. In the present embodiment, digital recording is performed on the optical disk in consideration of the need for random access for editing. In digital recording, incidental information is also recorded as digital information. Therefore, if the file structure is such that information corresponding to each field of the image can be obtained, it is not necessary to modulate and add the blanking period as described above. However, also in this case, considering the handling of the medium, it is preferable to record the image and the incidental information on the same medium.

Next, the moving image editing section 3 will be described. The moving image editing unit 3 performs a process of creating a new image based on the images with three-dimensional information from the plurality of data recording units 2 or the camera head 1. Here, the feature of the present invention is that image processing of projecting a captured or recorded image onto a spatiotemporal map and capturing the image projected onto the spatiotemporal map with a virtual camera to create a moving image. There is a point to do.

A spatiotemporal map is the basis for this moving image processing. The spatiotemporal map has three axes in three-dimensional space,
It is a virtual four-dimensional space with one axis of time, and is actually a data table stored on a high-speed semiconductor memory. In this example, since the output of the moving image editing unit 3 is an image of NTSC timing, the unit of time is 1 field 1/60 seconds, and the spatiotemporal map has a table having three-dimensional data for each field. The required number of lines is lined up. As in this example, if the input and output TV timings are the same and it is not necessary to combine the video of a certain field and the video of the field at another time to produce one field of the output video, As a space map,
It is also possible to rewrite each field by storing only one field in the memory. Further, the data of the three-dimensional space is a necessary range such as a range that may be output and a range in which the target object moves. In this example, the target object is limited to movement within the constraining plane, so the spatial data of the spatiotemporal map may also be a plane. In the case of swimming athletes, the space outside the pool is meaningless, so the range is limited to the size of the pool. At this time, the spatiotemporal map looks like a plan view when the pool is viewed from directly above. FIG. 3 shows an example of this spatiotemporal map.

In order to create this spatiotemporal map, the moving image editing section 3 first uses the three-dimensional information from the data recording section 2 or directly from the camera head 1 to display the input 1-field image on the spatiotemporal map. Map to. This is the opposite of taking an image of the constraining plane with a camera. That is, it corresponds to projecting an image captured by a camera through a lens to a confining plane, like a slide projector.

The actual processing is as follows. Since the image photographed as shown in FIG. 4 is an image of the constraining plane viewed obliquely, the image is deformed and the position is written in the spatiotemporal map in alignment. The position of the target object in the image is measured by the camera head 1. Further, the range in which the image at that time is shown is also calculated based on the state of the camera head 1 and obtained as data together with the image. In this example, in order to make the calculation easier to understand, if the non-linear component such as the lens distortion is ignored, the deformation at this time is expressed by the matrix of the following Expression 1.

[0038]

[Equation 1] The transformation matrix can be obtained from the correspondence between the four corners of the image and the range shown on the constraining plane. Since the camera head 1 moves according to the movement of the target object, this conversion matrix changes in each field, and therefore the data recording unit 2
Each field is calculated based on the three-dimensional data sent from, and the image is converted and mapped using the matrix. This conversion process is performed in parallel for the number of images required for editing. As mentioned above, in this example the spatiotemporal map is considered to be just a plan view of the pool. The spatiotemporal map covers the entire area of the pool, but the images captured in it are limited to the vicinity of the player. The position of the player moves along with the passage of time, but the position of the image moves in the map as well. This is just because you can think of the feeling that the spotlight is shining in the pitch-dark pool only for the player, and the spotlight moves as the player moves.

Next, the image output of the moving image editing section 3 will be described. Unlike conventional image editing, in the present invention, the spatiotemporal map is used as the object of editing, not the input image. The output image is an image obtained when the spatiotemporal map is photographed using a virtual camera. This virtual camera is basically unrelated to the camera head 1 used for input. Therefore, it is possible to shoot from any position at any angle of view. It's just a concept of shooting a pitch-black pool where a spotlight is chasing a player with a new camera. As described above, the spatiotemporal map corresponds to the actual position of the three-dimensional space. Therefore, the virtual camera can be operated in the same manner as a camera head for image input. In other words, the position of the camera can be defined in a three-dimensional space, and the direction and angle of view of the camera from there can be determined. As described above, since the spatiotemporal map has the actual position in the three-dimensional space, if the position and the direction of the virtual camera are known, the position and the range to be photographed can be known. This is just the opposite operation of mapping the image from the camera head. FIG. 5 illustrates the principle, and since the constraining plane is used in this example, the spatiotemporal map is a plane.
The transformation of the image is represented by the matrix shown in the following equation 2 as in the case of mapping.

[0040]

[Equation 2] Further, this virtual camera can also perform an automatic tracking operation like the camera head 1. Since the positions of all the images on the spatiotemporal map are known at the time of mapping, it is sufficient to move the output virtual camera based on them.

Next, the editing operation will be described.

As described above, in the present invention, the object of editing is the spatiotemporal map. For example, in the case of swimming, 1
A different camera head (cam)
Consider the case where the image is taken at 1, cam 2). In the spatiotemporal map at this time, the respective images are located apart from each other as shown in FIG. If you take a picture of this with the above-mentioned virtual camera, both images will become too small when you try to put them on the screen, and you will edit them. In editing, the spatiotemporal map itself can be cut as shown in FIG. In our case,
By cutting out one entire course of the pool and moving it to another location as in Figure 6, it is as if two athletes were swimming on adjacent courses. Since this corresponds to the actual space position in the spatiotemporal map, performing this operation is the same as replacing the actual pool course location on the data. This editing operation is the movement of a conditional image on the spatiotemporal map, in which the figure in the range of coordinates corresponding to the course to be moved is moved, but not in the other range. This is also expressed by the matrix as shown in Expression 3. However, since the movement is conditional as described above, this matrix E is located at the position (x, z) on the map.
Depends on

[0043]

[Equation 3] After this editing, shooting with a virtual camera produces an image as shown in FIG. 7, and the player's image does not become too small. This operation gives an image of a player swimming on a course far from the original course, swimming on the next course.

In the present example, the editing is such that the spatially distant one is brought closer, but when the spatially distant one is brought near, the editing can be similarly brought near. For example, if you want to compare the first day and the second day of a certain player, you can do as follows. Record the first and second days with the camera head. Although actual recording is performed on an optical disk or the like, when reflected in the spatiotemporal map, the spatiotemporal map is a plane in this example, so that it can be considered that the maps in field units are stacked as shown in FIG. In the editing operation, this block is cut and moved as a block as shown in FIG. This will allow you to compete on the adjacent course for the first and second days. Positioning at the time of fitting may be spatially performed at the position of the pool, but attention should be paid to the temporal alignment. Although they must start at the same time to compete, the timed stopwatch is usually out of sync with the television field. In the case of relatively slow movement such as swimming, it may not be so problematic even if it is adjusted in the unit of about 16.7 msec of the field period of the television, but in the case where 1/100 sec is a problem, the accuracy is reasonable. You have to adjust the position with. Therefore, in the present invention, time data is managed in real time as described above. The processing using the real time is as follows.

When editing using images from a plurality of camera heads, first, the relationship between the clock and the field is determined by using one of the cameras as a reference timing. An output image is also obtained at this timing. Since the image from the camera head 1 includes the data of the three-dimensional coordinates and the time of the stopwatch 5, the time of each field can be known. Each field has a width because it accumulates photocharges during that time, but the field switching is used as a typical timing to facilitate calculation.
Considering an electronic shutter in which charge accumulation occurs in the second half of the field, it is preferable to use the end time of the field as the timing. For example, as shown in FIG. 10, in the camera 1, the timing of the stopwatch is −4.7 msec to +12 msec, which is the nth field, and +12 msec to the n + 1th field. On the other hand, it is assumed that the camera 2 has a field from -12.7 msec to the kth field and a field from +4 msec to the kth field. If the first camera is used as a reference, a second camera image with a timing matched to the field is required. For example +
At 12 ms, the image from the second camera is the k + th image.
It is in the middle of field 1. In this example, it is considered that there is not much difference between the images of adjacent fields because the movement is relatively slow during swimming. In such a case, the previous field can be used to simplify the hardware. At the previous 12 msec, the image of the k-th field from -12.7 msec is used as the image of the camera 2. In this example, the previous video is simply diverted, but a plurality of videos before and after the video may be used to synthesize the video at the required time.

Similar to the video, the three-dimensional coordinate values used for mapping on the spatiotemporal map are obtained corresponding to the field, so the time when there is a value is limited. Therefore, in the spatiotemporal map matched to the camera 1, the three-dimensional data of the camera 2 is the same as that of the image, and the timing is shifted. When the nature of the motion of the target object is known, the value can be interpolated by an equation that depends on the nature of the motion, as was done when the camera head was automatically tracked. In this example, linear interpolation is used in the sense of simplifying the hardware, but in this case, it is interpolation interpolation from two adjacent fields instead of extrapolation as in the case of automatic tracking. Strictly speaking, the high accuracy is obtained only in one point of the target object, and the other parts are not so high in accuracy. Therefore, even the linear interpolation can obtain a practical accuracy. FIG. 11 shows the mapping on the spatiotemporal map with the time axis aligned as described above. Once mapped, edit the spatiotemporal map as in the previous example and shoot with a virtual camera.

In the case of swimming, the size of the pool is limited. Therefore, when swimming a long distance, the turn is repeated. At this time, for example, you may want to compare before and after the turn. In such a case, edit as follows. At the time of shooting, the traveling direction is reversed on the spatiotemporal map before and after the turn as shown in FIG. In the editing operation, as shown in FIG. 13, the spatiotemporal maps after the turn are switched to the left and right in order to reverse the traveling direction after the turn, and are aligned with those before the turn. At this time, attention should be paid to the position and time adjustment as described above. By this operation, as shown in FIG. 14, it is possible to obtain an image as if the players before and after the turn just started the adjacent courses at the same time and are competing. The positional relationship of the images compared by any of the above editing operations is displayed at the position based on the coordinates of the actual three-dimensional space. Therefore, it is possible to measure the position based on the positional relationship in the image and the distance measurement. Become.

The above operation is represented by equation 4. That is, first, the captured image is projected on the spatiotemporal map by the matrix A. Next, the matrix E is edited in the spatiotemporal map. Finally, conversion is performed in the matrix B corresponding to shooting with a virtual camera.

[0049]

[Equation 4] FIG. 15 shows an example of the hardware configuration of the moving image editing unit 3 that performs these edits. The moving image editing section 3 is provided with an input 1 (3
From 1) and input 2 (32), three-dimensional data and time data are input together with the image data. Two matrix coefficient operation circuits 33 and 34 to which the three-dimensional data of input 1 and input 2 and time data are respectively input, and frame memories (FM1 and FM) corresponding to input 1 and input 2
2) 35 and 36, a write address counter 37 that gives write addresses to the frame memories 35 and 36,
38, a virtual camera controller 39, a matrix coefficient calculation circuit 40 for calculating a matrix coefficient by the output of the virtual camera controller 39, an editing controller 41,
Matrix calculation circuits 42 and 44 for performing matrix calculation by outputs of the editing controller 41 and matrix coefficient calculation circuits 33, 34 and 40, read address counter 44, read address counter 44, and output of matrix coefficient calculation circuit 40 for virtual camera Matrix calculation circuit 45 for performing matrix calculation and outputting to the edit controller 41, address calculation circuits 46 and 47 for performing address calculation for each image, and read address output and frame memory selection signal from outputs of both circuits 46 and 47. An address manager circuit 48 for outputting, a data selector circuit 49 for selectively outputting the outputs of the frame memories 35, 36 based on the selection signal,
A background data table 51 for giving background data and a background data controller 50 are provided.

The operation will be described. Since the output image is obtained by raster scanning as shown in FIG. 16, this hardware is designed to obtain an arbitrary point (u ', v') of the output image. Inversely, the following expression 5 is obtained from the expression 4.

[0051]

[Equation 5] This gives an input image corresponding to the points in the output image. The input 1 (31) and the input 2 (32) may be the output of the data recording unit 2 described above or the camera head 1. Of the input data, the three-dimensional data and the time data are used to obtain the matrix A ₁ ^-1 when projecting onto the space-time map. Since the inputs are different parameters, the matrix A ₁ ^-1 is necessary for each. On the other hand, the input image data is stored in the frame memory (FM) 3 respectively.
5, 36. The address at this time is (u,
It corresponds to v). This frame memory has 2 each
There are systems, and writing and reading are performed alternately. The virtual camera controller 39 produces three-dimensional parameters of the virtual camera. For automatic tracking, calculation is performed based on the input three-dimensional data. The matrix B ^-1 is obtained from this data by the matrix coefficient calculation circuit 40. The read address counter 44 creates a read address (u ', v') corresponding to the raster scan of the output. Based on this, the matrix calculation circuit 45 obtains the address (x, z) on the space-time map. Based on this, the edit controller 41 edits the edit matrix E ₁ ^-1 ,
Make E ₂ ^-1 . Further, from the read addresses and their matrix, the read address (u ′,
The write address (u, v) corresponding to v ′) is calculated. As described above, there are a portion with an image and a portion without an image on the spatiotemporal map, which can be determined by examining the calculated write address value. That is, if a write address corresponding to a certain read address has a value that actually exists, it is a portion where an image exists. This comparison is performed for each of the inputs by the address manager circuit 48 composed of a comparator and a selector to calculate which input corresponds to which address. The frame memory 35, 3 described above is calculated using the calculated read address.
An image of the virtual camera can be obtained by reading 6 and selecting it with the data selector circuit 49. In this example, the input is 2
It is a system, but this can be calculated similarly for any number.

(Second Embodiment) In the first embodiment, only the image of one virtual camera is handled as the output image.
In some cases, it may be better to use several virtual cameras to display several images at the same time. Also, in the case of soccer etc., since the positional relationship between the players is important, it corresponds to just looking from above. It is also possible to output the video of each field of the spatiotemporal map as it is. In that case, it is necessary to use a high-definition standard or a computer screen standard because the resolution of a normal TV is not sufficient. Further, instead of outputting the image of the spatiotemporal map as it is, the trajectory of the player may be left by displaying, for example, a dot at the position of the player using the coordinate values.

(Third Embodiment) In the first embodiment, the automatic tracking method is used for the operation of the virtual camera, but this operation may be performed by a human. As the operation at this time, it is desirable to use a data input device having a shape similar to the operation of an actual camera, not a mouse or the like. For example, it is a camera parameter input device as shown in FIG.

In this device, the tilt axis corresponding to the vertical direction of the camera can be controlled by moving the handle up and down. Similarly, the left and right movements of the steering wheel can be controlled by the pan axis of the camera, by pushing and pulling the steering wheel back and forth, zoom control corresponding to the angle of view and by rotating the steering wheel focus control. The focus is used, for example, when the effect of the virtual camera is intentionally blurred to obtain a staging effect. Also, in order to mimic the actual camera operation, there is a finder in which a virtual camera image is displayed. In addition, since there are many cases where there is no video, it may be difficult to see wherever you look. Therefore, as shown in FIG. 6, the field of view of the camera is displayed together with the spatiotemporal map to facilitate the operation. is necessary.

Further, instead of matching the tracking of the virtual camera with the target object, a predetermined operation can be performed. In the example of swimming, for example, by controlling the virtual camera so that the center of the screen is set to the pace of Japan record, whether the record is likely to appear at the position of the screen shown to the athlete or whether the pace distribution is You can see at a glance.

Further, since the position of the virtual camera can be freely moved, it is possible to control the virtual camera so that it is always above the ceiling of the leading player. In addition, when the rail is pulled and the truck is placed on the rail, the movement can be performed while moving and the virtual camera can be given to the virtual camera, as is used for shooting television and movies.

(Fourth Embodiment) When the spatiotemporal map is photographed by the virtual camera, there is no problem if the range is narrower than the image obtained by the camera head, but normally the visual field of the virtual camera as shown in FIG. Is wider than the mapped video range. Therefore, in the visual field of the virtual camera, the part 21 having no image is captured. There is no problem if the purpose is measurement, but when used as an image such as broadcasting, the part 21 without the image is unnatural. Therefore, in this example, the image is embedded in the entire space-time map in advance as the background. Since the spatiotemporal map is updated on a field-by-field basis, the background can also be a moving image.
However, it is not efficient because it is necessary to always shoot the entire range in order to make it into a moving image. Also, since it is the target object that is required and not the background, it is preferable to use the background as a still image. The background data controller 50 of the moving image editing unit shown in FIG. 15 performs this processing. Since the background is a still image, it is basically not changed once it is captured. A high-resolution camera such as a high-definition camera may be used for capturing,
If the range of the spatiotemporal map is large, the resolution may still be insufficient. Therefore, it is better to use the automatic tracking head to scan the entire range and capture the image. The image from the head is converted and mapped to a map by using three-dimensional coordinates as usual, and this is used as a background still image. This still image is stored in the background data table 51.

The map is clipped at the time of editing, but the background processing at that time is set so that some selections can be made separately from the object map. For example, in the case of swimming, if you want to bring a player from a different course to a different course, leave the background map as it is and edit only the object map that contains the player's image. You can get an image of moving from one course to another. Also, if you make a transformation such as editing the turn, it may be more confusing to turn over the background so that you can see that you turned it over.

When the background is captured, it is usually performed before the measurement, and further, the entire range is scanned. By utilizing this, it is possible to create a white balance value or brightness map. The white balance setting does not need to be changed if the illumination light is uniform, but if the ambient light from the window is mixed with the illumination, or if several types of lamps are mixed with the illumination, the color temperature of the illumination will vary depending on the location. Because there may be different. Regarding brightness, you can also obtain brightness data from the image while shooting, but when you chase a moving target object, depending on the direction of the camera, the illumination light may enter directly or become backlit, so only from the image. Sometimes it is difficult to get brightness information. It is also possible to calibrate when capturing the background. However, since three-dimensional coordinates are required for mapping, it is necessary to perform minimum calibration once before this calibration. The calibration when capturing the background improves the accuracy by checking whether there is a camera shift after the calibration before installation, for example, and by statistical processing by repeating the calibration. It makes a lot of sense.

In this example, each target object has its own map, and the captured background is mapped to the map dedicated to the background. These are displayed in context when displayed. As shown in FIG. 18, the target object 1
Is in front of the target object 2, and when overlapping, the target object 1
Can be seen. The background is placed at the back and is displayed only when there is no target object.

Next, the boundary between the background and the image obtained by the camera head will be described. In the above-mentioned example, the range of the image from the camera head is displayed as it is in the output image. If there is no image in the background, it is an unnatural image, so the border does not matter. However, as in this example, when the quality of the image is secured by adding the background, the boundary between the background of the still image and the image of the moving image becomes quite dirty as it is. Therefore, the following three methods are used properly according to the nature of the object. First, the first method is shown in FIG.
In this method, only the target object is extracted from the image and the boundary of the target object is the boundary with the background. In this case, the image becomes natural, but the condition is that the target object can be extracted from the image. If the background is relatively uniform, such as lawn, and has a different color from the target object, it can be extracted with a different color, such as chroma key. If the target object cannot be extracted from the image even if the difference in color or the difference in brightness is used, the second method is to use the background information to extract the target object. Since both the image from the camera head and the background photographed before that are mapped based on the three-dimensional position, the difference is small when the same portion is photographed. When the background is subtracted from the image from the camera head, only the target object can be extracted as a difference as shown in FIG. However, in this case, it is important that the conditions for capturing the background and the conditions for capturing the target object are the same as much as possible. Not very suitable for changing things such as pool backgrounds. Further, when editing, it is necessary to make a set of the map of the target object and the map of the background and perform the same operation. If the target object cannot be extracted by the above two methods, such as when the image is taken in another place, the third method is to blur the boundary as shown in FIG. Blurring can be obtained by averaging the data of the two images while changing the ratio. As the blurring shape, a rectangular shape or an ellipse along the shape of the edge of the screen is selected according to the object. Although this method is not so beautiful as an image, it can be applied even if it is not possible with the above two methods. As a background, in addition to the actual image captured by the camera as described above, an image created by calculation like computer graphics can be used. As shown in FIG. 22, by adding a grid as a mark, which does not actually exist, to the background captured by the camera, the movement of the player can be easily understood as shown in FIG. Also, in order to enhance the image effect, it is possible to synthesize the background of a famous venue or the like.

(Fifth Embodiment) The computer graphics used in the fourth embodiment can be used not only as a background but also as a target object. Similar to the target object from the camera head, the target object of the computer graphic can be processed in exactly the same way as the target object from the camera head by mapping it on the spatiotemporal map of the target object. By moving the virtual target object of computer graphics, it becomes possible to compete with the player who has only data and the player who was actually photographed. If only the total time is recorded, make a prediction based on the player's pace distribution such as drive-in type or lead type. FIG. 24 shows an example of a swimming race editing system using this prediction. An image monitor from a CRT1 to CRT3 camera head. The camera head is controlled by SW7 on the console. In addition to automatic tracking, this device can also be moved in accordance with the expected pace. The CRT 5 is a control screen and corresponds to the spatiotemporal map. In this case, a moving image shot with a wide-angle lens is used as the background so that the states of other players can be seen.
In this editing apparatus, the virtual target object is displayed in a line, and the console has various changeover switches related to the movement of the virtual target object. In this figure, a player with a standard pace distribution that produces a time equivalent to a Japanese record is set as a virtual target object and its position is displayed as a line. Since this virtual target object may be plural, the world record equivalent may be displayed simultaneously. The edited image can be monitored by CRT4.

(Sixth Embodiment) In the case of automatic tracking, it is often the case that an object that can be beautifully extracted from an image is selected as a target object. For example, when a running person is tracked by a motion analysis and a shoe or the like is selected as the target object, the image obtained as shown in FIG. 25 makes more movement than necessary. The desired image is a smooth movement as when a human operates the camera as shown in FIG. The tracking algorithm can be changed to obtain smooth movement, but the characteristics of the camera head to be tracked must be taken into consideration, and this is a dedicated program. In this case, the constraining plane is a plane perpendicular to the ground through the line running on the constraining plane, and the movement of the virtual camera is limited to the horizontal direction, and by filtering the movement, it is as smooth as human operation. You can get an image of camera work. Further, in order to determine the movement of the virtual camera at this time, unlike the camera head, it is not necessary to consider the actual physical quantity such as the characteristic of the inertia motor of the camera, which is easy. Also, the head does not have to be smoothly tracked, and a general-purpose head can be used as it is because it simply tracks the target object. The design of the head can also be pursued by pursuing the ability to chase after all, and the characteristics can be tuned to the maximum. It is also easy to change the camera work after recording once.

(Seventh Embodiment) A normal image recording apparatus is used,
When the output from the image recording device is edited, stop motion and reverse reproduction are possible in addition to normal moving image reproduction. You can use the jog dial or shuttle dial to search for the desired frame and reverse playback, as you would with a normal editing machine. As described above, according to the present invention, when an image is mapped to the spatiotemporal map, the image is mapped on a field-by-field basis, so that a still image or reverse reproduction can be performed on a field-by-field basis. In this case, a still image can be obtained by stopping writing the image in the field memory and replaying the same data repeatedly. Further, the image of the required field and the additional information can be read out by designating the time or field number to the recording device. Further, since the supplementary information such as three-dimensional information is attached to each field of the image, there is no problem in mapping on the spatiotemporal map even when an arbitrary field is designated. In a normal video, fields are mapped and read in order, but by changing this order arbitrarily, you can reverse the order and play backwards, or change the interval to perform slow motion, fast forward, etc. it can.

Further, according to the present invention, the mapping of the image onto the spatiotemporal map and the calculation of the conversion matrix by the virtual camera can be controlled independently, so that, for example, the image remains a still image and only the virtual camera is used. You can move around. This will move the video in slow motion,
By changing the position, direction, zoom, etc. of the virtual camera to emphasize the positional relationship and the like, it is possible to obtain a so-called “showing place” effect. For example, with respect to the above-mentioned swimming, the image projected on the spatiotemporal map for the image of the last touch is converted to an image taken by moving the position of the virtual camera or changing it from a different position or zooming. It is possible to show it.

[0066]

As described above, according to the present invention, by using the concept of the virtual camera for the image projected on the spatiotemporal map, the image is projected on the spatiotemporal map at any spatial position or at any time axis. It is possible to convert moving images.

As a result, according to the present invention, (1) it is possible to obtain a moving image at an arbitrary cut point that the observer desires.
It makes it easy to analyze the moving object that is the target of motion analysis, (2)
It is easy to compare multiple moving objects, (3) Moving objects at different times or different spatial positions can be compared at the same time, and (4) Pace allocation such as Japanese or world records to be compared. Compared with the above, it is possible to easily recognize the position of the current target object. (5) By using the background, the moving image captured and edited by the virtual camera can be used for broadcasting as an image that is not unnatural. it can,
And so on.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a moving image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is a diagram showing how a moving image to be processed by the first embodiment is photographed.

FIG. 3 is a diagram illustrating a spatiotemporal map of the present invention.

FIG. 4 is a diagram illustrating a principle of projecting an image captured by a camera head on a spatiotemporal map of the present invention.

FIG. 5 is a diagram illustrating the principle of capturing an image projected on a spatiotemporal map of the present invention with a virtual camera.

FIG. 6 is a diagram for explaining space cutting of the spatiotemporal map according to the first embodiment.

FIG. 7 is a diagram illustrating a state in which target objects having different courses according to the first embodiment are photographed by a virtual camera.

FIG. 8 is a diagram illustrating a spatiotemporal map of images at different times according to the first embodiment.

FIG. 9 is a diagram for explaining space cutting of a spatiotemporal map at different times according to the first embodiment.

FIG. 10 is a diagram illustrating a time shift of video signals from different camera heads.

FIG. 11 is a diagram illustrating a manner in which images on different time axes by a virtual camera are aligned with each other on the spatiotemporal map.

FIG. 12 is a diagram showing how the camera head photographs before and after a turn.

FIG. 13 is a diagram for explaining space cutting of a spatiotemporal map before and after a turn.

FIG. 14 is a diagram showing a state in which a virtual camera images the front and the rear of a turn.

FIG. 15 is a block diagram showing a hardware configuration of a moving image editing unit.

FIG. 16 is a diagram illustrating raster scanning of an output image.

FIG. 17 is a diagram showing an example of a camera on which a spatiotemporal map of the third embodiment is displayed.

FIG. 18 is a diagram illustrating a spatiotemporal map including a background image according to the fourth embodiment.

FIG. 19 is a diagram showing an example of an image combined with the background image of the fourth embodiment.

FIG. 20 is a diagram illustrating a spatiotemporal map for explaining composition with a background image according to the fourth embodiment.

FIG. 21 is a diagram showing an example of an image combined with a background image with blurring according to the fourth embodiment.

FIG. 22 is a diagram illustrating a spatiotemporal map in which a grit background image according to the fourth embodiment is created.

FIG. 23 is a diagram showing an example of a screen in which a grit background image and computer graphics of the fourth embodiment are combined and combined.

FIG. 24 is a diagram illustrating image composition of the fifth embodiment.

FIG. 25 is a diagram showing an example of an image on the spatiotemporal map captured by the camera of the sixth embodiment.

FIG. 26 is a diagram showing an example of an image converted into an image by a smooth-motion virtual camera according to the sixth embodiment.

[Explanation of symbols]

 1 Camera Head 2 Data Recording Section 3 Moving Image Editing Section 4 Time Base Unit 5 Stopwatch 6 Plane 7 Target Object (Athlete) 31, 32 Input 33, 34, 40 Matrix Coefficient Calculation Circuit 35, 36 Frame Memory 37, 38 Write Address Counter 39 Virtual camera controller 41 Editing controller 42, 43 Matrix operation circuit 44 Read address counter 45 Matrix calculation circuit 46, 47 Address calculation circuit 48 Address manager circuit 49 Data selector circuit 50 Background data controller 51 Background data table

Claims (10)

[Claims] 1. A moving image including an image pickup means for picking up one or a plurality of moving objects and outputting together with three-dimensional spatial information of the moving objects, and a moving image editing means for processing and editing an output image of the image pickup means. In the image processing device, the moving image editing means projects the captured image of the moving object on a spatiotemporal map corresponding to a spatiotemporal space in which the moving object moves based on the three-dimensional space information and time information of the image. A spatio-temporal map image generating unit that generates a spatio-temporal map image that is a captured image, and an image converting unit that converts the spatio-temporal map image of the moving object into an image captured from different time or different spatial position. And a moving image processing device. 2. An image recording means for recording an image of a moving object picked up by an image pickup means together with three-dimensional spatial information of the moving object and time information relating to the image, and the moving image editing means records the image of the image recording means. 2. The moving image processing apparatus according to claim 1, further comprising means for processing and editing the captured image of the moving object or the image of the imaging means. 3. Regarding the spatiotemporal map images of a plurality of moving objects on the same temporary space map, the spatial positions on the spatiotemporal map of moving objects at distant spatial positions are moved, and then converted into imaged images. Claim 1 including means for
Alternatively, the moving image processing device according to item 2. 4. A plurality of spatio-temporal map images of the same or different moving objects imaged at different times have the same time axis and are arranged on the spatio-temporal map so that they are at different spatial positions. The moving image processing apparatus according to claim 1, further comprising a converting unit. 5. A means for capturing or synthesizing a background image of a moving object separately from the captured moving object, and synthesizing this background image with the transformed image of the spatiotemporal map image of the moving object. The moving image processing apparatus according to claim 1, further comprising: 6. The moving image processing apparatus according to claim 5, wherein a moving state of a record whose pace distribution is known is displayed on the background. 7. The method according to claim 1, further comprising means for synthesizing an image of a virtual moving object whose movement characteristic data is distinct from the imaged moving object and the image of the image-converted moving object. Video processing device. 8. The moving image processing according to claim 1, wherein the image converting means includes means for converting an image of a moving object on the spatiotemporal map into an image captured so that the camera smoothly moves in one direction. apparatus. 9. The spatiotemporal map image generation means reads an image recorded in the image recording means at an interval different from the actual time interval or an image read out by reversing the order of the readout time. To a spatiotemporal map to generate a spatiotemporal map image, and the image conversion means converts the spatiotemporal map image into an image picked up at a different time interval from the time of projection or from a different spatial position. The moving image processing apparatus according to claim 2, further comprising: 10. The image conversion means includes means for converting one spatiotemporal map image of a moving object projected on a spatiotemporal map into a captured image by moving a spatial position with respect to the moving object. The moving image processing apparatus described.