JP2000020728A - Image processor and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate and display simulation images from the arbitrary line of a sight direction by generating object expression in a three-dimensional computer model according to image data from first and second cameras. SOLUTION: An image with the movement of an object along a background is recorded by video cameras 12a and 12b. A CPU 4 projects those objects to a 3D world space (three-dimensional computer model), generates the expression (model) of each object in the three-dimensional work space and stores two sets of time stamped 3D object data in a memory 6. The CPU 4 renders those object data to a frame buffer 16, and combines the rendered image data to generate images which are not viewed discontinuously by a user. Then, the sequence of the simulation moving images is displayed from a desired visual field direction to the user.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus and method for use in a three-dimensional computer modeling system for generating an object representation of a three-dimensional computer model using a moving image such as a video image. .

[0002]

2. Description of the Related Art At present, the content of an image generated from video data or other moving image data is determined by the visual field characteristics of a camera that has captured the data. In particular, from which position the observer views the object is determined by the camera's field of view position and gaze direction with respect to the scene.

One way to overcome this limitation is to use video data to generate a dynamic three-dimensional computer model of the scene, based on which a simulated image from any desired gaze direction is generated. An interactive system for displaying to a user has been suggested. The present invention aims to provide an apparatus or a method for use in such an interactive system.

[0004]

According to the present invention, image data from cameras having different fields of view of objects in a scene is processed so that the image data identified for one object is associated with one or more objects. An image processing apparatus or method for determining whether or not it is actually related is provided.

Further, the present invention processes image data,
Image processing apparatus or method for identifying image data for an object, processing the identified data using image data from different cameras, and dividing the identified image data for use in representing a plurality of objects I will provide a.

According to the present invention, image processing for identifying image data for an object in a scene and image data for a shadow of the object using image data from at least two cameras having different views of the object. An apparatus or method is provided.

Further, the present invention converts the image data of the object and its shadow from the first camera and the image data of the object and its shadow from the second camera into a common modeling space, and converts the data. An image processing apparatus and method for identifying a part of image data related to an object and a part of image data related to a shadow thereof based on data are provided.

According to the present invention, there is provided an image processing apparatus or method for processing image data relating to an object in a scene and defining a model of the object based on a footprint of the object on the ground.

Further, the present invention processes image data from a camera having a different field of view of an object in a scene to determine a ground surface profile of the object, and an image representing the object in a three-dimensional model according to the ground surface profile. A processing device or method is provided.

In accordance with the present invention, a first and second rendering techniques are performed, in which a three-dimensional computer is used to generate data defining an image representing a model from a predetermined viewing direction. Render the model and the second
Provides an image processing apparatus or method that generates data to represent a schematic image of the location of an object in a model.

Further, the present invention renders a three-dimensional computer model with respect to the selected line-of-sight direction, unless the selected line-of-sight direction is within a predetermined angle range, and renders the selected line-of-sight direction within the predetermined angle range. In some cases, an image processing apparatus or method is provided that generates data that schematically represents the position of an object in a model and outputs the data to a user.

These features are useful when modeling objects without a top surface, for example using one or more vertical planes in a computer model.
In such a case, a realistic image of the object cannot be obtained unless the object is looked down on. Also, the features described above are independent of whether a top surface is provided or not.
It is also effective when modeling an object on one or more vertical planes. This is because it is considered that a realistic image of the object cannot be obtained when the plane edge is turned on and viewed.

According to the present invention, there is provided an image processing apparatus or method for processing image data of an object in a scene and generating a three-dimensional model of the object according to a surface plane of the object identified from the image data.

The present invention also processes image data from at least two cameras having different fields of view of the object to identify a flat surface of the object where the feature points are located, and according to the identified plane. An image processing apparatus or method for representing an object in a three-dimensional manner is provided.

According to the present invention, there is provided an image processing apparatus or method for rendering a three-dimensional computer model in a selected gaze direction and generating information indicating the accuracy of the model with respect to the selected gaze direction.

According to the present invention, image data from at least two cameras having different fields of view of an object are received, the object is modeled in a three-dimensional computer model, and the model is rendered according to the selected viewing direction. An image processing apparatus or method is provided. The selection uses the selected line-of-sight direction, the viewing parameters of each camera, and the parameters that affect the image quality of the image data, to determine which image data to use for processing, the images associated with the different cameras. Done between data.

According to the present invention, when rendering a three-dimensional computer model for a sequence of images using image data available from two or more cameras,
An image processing apparatus or method for performing a test to determine whether an image looks discontinuous to an observer when changing a line of sight direction and generating image data to correspond to the determined discontinuity Is provided.

[0018]

Embodiments of the present invention will be described below in detail with reference to the drawings.

[First Embodiment] FIG. 1 is a block diagram showing the general configuration of an image processing apparatus according to the first embodiment. This device includes a computer 2 having a central processing unit (CPU) 4.
It is connected to a memory 6 that stores a program that defines the operations to be performed by the PU 4 and is operable to store objects and image data processed by the CPU 4.

Disk drive 8 coupled to memory 6
Is operable to accept a removable data storage medium such as a floppy disk 10 and to transfer stored data to the memory 6. Central processing unit 4
May be input from the removable data storage medium to the memory 6 using the disk drive 8.

Image data to be processed by the CPU 4 may be input to the computer 2 from a removable data storage medium using the disk drive 8. Alternatively or additionally, image data to be processed may be directly input to the memory 6 from a plurality of cameras (shown schematically as video cameras 12a and 12b in FIG. 1). In this embodiment, the camera has a digital image data output terminal. The image data may be stored in the cameras 12a and 12b before being input to the memory 6, or the image data may be transferred to the memory 6 in real time while the data is being collected by the camera. Also, image data can be input from a non-digital video camera instead of the digital cameras 12a and 12b. In that case, a digitizer (not shown) is used to digitize the image captured by the camera and generate digital image data to be input to the memory 6 based on the digitized image. Further, the image data may be downloaded from a local database or a remote database storing the image data to the memory 6 via a connection unit (not shown).

Connected to the input port of the CPU 4 is a user command input device 14, which is, for example, a keyboard and / or a position sensitive input device such as a mouse, tracker ball or the like.

A frame buffer 16 is also connected to the CPU 4. In this embodiment, the frame buffer 16 stores image data associated with an image generated by the central processing unit 4, for example by providing one (or more) storage locations for pixels of the image. It comprises three frame buffers. The value stored in the frame buffer for each pixel defines the color or intensity of that pixel in the image.

A display device 18 coupled to the frame buffer 16 displays image data stored in the frame buffer 16 in a conventional manner. The frame buffer 16 is also connected to another image recording device such as a video tape recorder (VTR) 20 or a paper printer or a 35 mm film recorder.

A mass storage device, such as a hard disk drive, having a large data storage capacity is coupled to the memory 6 (typically via the CPU 4) and to the frame buffer 16. Mass storage device 2
2 can receive the data processed by the central processing unit 4 from the memory 6 or the data to be displayed on the display device 18 from the frame buffer 16. CP
The data processed by U4 may be stored in a removable storage device via the disk drive 8 or by transmitting data such as a signal to a receiving device via a communication link such as the Internet, for example. It may be output from the computer 2.

The CPU 4, the memory 6, the frame buffer 16, the display device 18, and the mass storage device 22
It may form part of a complete commercial system such as a personal computer (PC).

As one embodiment of the present invention, the computer 2
May be supplied as a commercial product in the form of a program stored on the floppy disk 10 or another data storage medium, but the receiving computer 2 embodies the present invention. It may be transmitted as a signal to the computer 2 via a data link (not shown), for example, to be reconfigured into a device. The operation command may be input via the user input device 14.

FIG. 2 schematically shows a state of collecting image data to be processed by the CPU 4 in the embodiment.

By way of example, FIG. 2 shows two automobiles 30, 32 traveling on a road 34 toward a pedestrian crossing 36 where two people 50, 52 are crossing. Road 34
And the pedestrian crossing 36 and the car 30 running on it,
32 and the movement of the people 50 and 52 are two video cameras 12
a and 12b. In the present embodiment, the video camera is mounted at a fixed visual field position, the direction of the line of sight is constant, and the setting of zoom (magnification) is also constant. The cameras 12a and 12b are positioned so that their fields of view overlap at least a portion of the road 34 and pedestrian crossing 36 where the cars 30, 32 and the people 50, 52 are moving.

FIG. 3 shows an image processing operation executed in this embodiment. In FIG. 3, the same steps, but processing steps that are performed separately for data from video camera 12a and for data from video camera 12b,
Depending on the video data to be processed, the same reference numerals in the figure are followed by letters "a" or "b" (representing camera 12a or camera 12b, respectively) for identification.

Referring to FIG. In step S2, the user generates a three-dimensional computer model of a stationary background (ie, a portion that does not move), which is a background where the object moves. That is, in the example shown in FIG. 2, the road 34, the pedestrian crossing 36, and the surrounding area are modeled. This uses commercially available modeling packages for modeling the background as a separate object, for example, or EOS Systems Inc. to facilitate the modeling of the background from images. It is performed in a conventional manner using a modeling package such as a photo modeler manufactured by the company.

In steps S3a and S3b, image parameters relating to the background scene of each camera are set.

FIG. 4 shows steps S3a and S3 shown in FIG.
3b shows the processing steps performed in 3b. Since this process is the same for each camera, only the process related to the camera 12a will be described.

In FIG. 4, in step S22, a plurality of reference images of a still background are recorded using the camera 12a. In the present embodiment, 10 frames of video are recorded. The plurality of reference images, as described further below,
Background lighting conditions change over time, noise, and "stillness"
It is recorded to take into account unwanted movements in the background (such as may be caused by movements of tree foliage, etc.).

In step S24, the image space (that is, the image recorded by the camera 12a) and the three-dimensional (3D)
Calculate the transformation to and from the world space (ie, the space when the three-dimensional computer model was generated in step S2).

This transformation defines the mapping between the ground plane in image space (the plane on which the object is moving) and the ground plane in 3D world space (3D computer model). In the present embodiment, the absolute position of the camera, that is, the position of the camera with respect to the scene being observed is not determined in advance, and the imaging parameters (focal length, charge-coupled device size, zoom setting, etc.) of the camera are also not determined. Similarly, since it has not been determined in advance, this transformation is calculated. By the conversion, as described later,
The representation of the object based on the position and range of the object in the image space can be reliably and efficiently generated by the 3D computer model.

In order to calculate the conversion in step S24, one of the background images recorded in step S22 is displayed to the user on the display device 18 so that the user
In the prompting by, a plurality of points (in the present embodiment, in the present embodiment, in the image located on the plane in which the object in the scene is moving and located in the field of view of both the cameras 12a and 12b) 4 points). That is, in the example shown in FIG. 2, the points 38, 40, 42 and 44 at the four corners defined by the marks of the crosswalk.
(These points are located on the road surface 34 on which the cars 30, 32 and the people 50, 52 move). A point corresponding to the point identified in the video image in the three-dimensional computer model generated in step S2 is also defined by the user. For example, the field of view of the three-dimensional computer model with respect to a predetermined gaze direction may be displayed on the display device 18 for the user, and the user may use the input device 14 to specify a corresponding point.

Using the positions of the points specified in the video image and the positions of the corresponding points specified in the three-dimensional computer model, the CPU 4 performs conversion between the image space and the 3D world space using, for example, the following equation. Calculate as before using

[0039]

(Equation 1)

Where n = 1... 4; Xn, Yn are points in the world space, xn, yn are points in the image space, and AH is obtained by the following equation.

[0041]

(Equation 2)

This corresponds to the ground plane of the image space, 3
Defines the conversion between the D computer model (3D world space) and the ground plane.

When executing step S24 for the camera 12b in step S3b, the four reference points selected from the image recorded by the camera 12b and the four reference points of the three-dimensional computer model are determined in step S24 for the camera 12a. Select so that it is the same reference point as you selected. In this way, the camera 12a and the camera 12b have the same scale in the 3D world space.

In step S26, the CPU 4 calculates a reference image pixel parameter relating to a still background. this is,
This is performed by calculating the average gray level μ for each pixel from the plurality of images recorded in step S22.
In other words, this is a process of calculating the average of the gray levels of the corresponding pixels in every 10 frames while considering the gray levels. The variance σ of the determined average is also calculated. Next, the gray level “window” of each pixel is set as μ ± (2σ + F). In the equation, F is an error coefficient set to take variables such as gain and noise of the video camera 12a into consideration. In the present embodiment, the total number of gray scale levels is 2
56, and the error coefficient F is set to 5 gray scale levels.

The "window" set for each pixel in step S26 represents the spread of the gray scale value to be taken when the pixel forms a part of the image of the still background (the pixel forming the part of the background). The viewing position and viewing direction of the video camera 12a are constant so that the grayscale value changes only in response to illumination changes and errors due to noise.) As will be described below, these "windows" are used to identify objects that are not part of the background (thus moving pixel values recorded by camera 12a out of the defined window).

Returning again to FIG. 3, steps S4a and S4b
Then, an image of the "action", that is, the movement of the object along the background (for example, the automobiles 30, 3 on the road surface 34)
2 and the images of the people 50 and 52 on the crosswalk 36) are recorded by the video cameras 12a and 12b. Video frames recorded by cameras 12a and 12b are time-stamped so that temporally corresponding frames can be used in subsequent processing.

In steps S6a and S6b, the CPU 4 sets the time synchronous image, that is, the camera 1 in step S4a.
The image data relating to the image recorded by 2a and the image simultaneously recorded by the camera 12b in step S4b are processed, and objects in the image that are not part of the "still background", that is, along the background Identify objects that are moving or stationary with respect to the background. Next, the CPU 4 projects these objects onto the common world space (three-dimensional computer model) defined in step S2.

FIG. 5 shows steps S6a and S6b.
Shows a processing operation executed by the CPU 4. Since this process is the same for both cameras, only the process of the camera 12a will be described.

In FIG. 5, in step S30, the CP
U4 compares the gray level of each pixel in the image data to be processed with the gray scale "window" previously set for the corresponding pixel in the image in step S26. If any pixel has a gray level that deviates from a predefined window for that pixel, that pixel is a "foreground" pixel, i.e., a pixel that forms part of an object moving or stationary along the background Is potentially considered to be Therefore, in step S30, the CPU 4
Keeps a record of which pixels have grayscale levels outside the corresponding pre-computed window.

In step S32, the CPU 4 processes the image data to remove noise. Such noise may include, for example, quantum effects when the video camera 12a is a charge coupled device (CCD) camera, data compression techniques used to compress data from the camera 12a, processing by the CPU 4, etc. It can also be introduced into image data by a number of sources, such as a frame grabber used to capture frames of data,
Alternatively, the noise may often occur in the image data near the boundary of the moving object.

FIG. 6 shows an operation performed by the CPU 4 when processing image data to remove noise in step S32 of FIG.

In FIG. 6, at step S50, the CP
U4 is, for example, a "C" by RMHaralick and LGShapiro.
omputer and Robot Vision Volume 2 '' (Addison-Wesley
Publishing Company, 1993, ISBN 0-201-56943-4 (v.
2) Apply a "shrinking" mask to the image data in a conventional manner as described in P583). This operation applies a 3 × 3 pixel mask to the image data and “foreground” pixels in each of a set of nine pixels defined by the mask (identified in step S30)
And the number of "background" pixels. If most of the pixels in the mask are background pixels (even those previously identified as foreground pixels), the center pixel is defined as a background pixel. If most of the pixels in the mask are foreground pixels, do not change. This operation is repeated until the shrinking mask is applied to the entire image data.

In step S52, the CPU 4
Computer and Ro by RMHaralick and LGShapiro
bot Vision Volume 2 '' (Addison-Wesley Publishing Co
mpany, 1993, ISBN 0-201-56943-4 (v.2), P583), applying a "growth mask" to the image data in a conventional manner. If most of the pixels in the mask are foreground pixels (even those previously identified as background pixels), this operation defines the center pixel as the foreground pixel and If most of the pixels are background pixels, step S5 is performed except that no change is made.
Performed in the same way as 0. Step S52 has an effect of returning a pixel erroneously set as a background pixel to a foreground pixel by the shrinking mask operation in step S50.

Referring again to FIG. 5, in step S34, C
PU4 processes the data to identify clusters of foreground pixels. It operates on image data to identify foreground pixels,
It is then performed in a conventional way to identify clusters of pixels having the same characteristics by interactively considering neighboring pixels and identifying all connected foreground pixels.

In step S36, the CPU 4 considers the next cluster of the foreground pixels identified in step S34 (this is the first cluster in which step S36 is executed first), and determines the number of pixels in the cluster. It is determined whether it is more than “30”.

If the number of pixels is 30 or less, the cluster forms a relatively small portion with respect to the entire image (768 × 512 pixels in this embodiment).
Think of clusters as representing noise. In this case, there is no further processing of the cluster. On the other hand, if the number of pixels in the cluster exceeds “30”, it is considered that the cluster represents a foreground object, and further processing is performed.

In step S38, the CPU 4 determines the extent of the spread of the pixel cluster. In the present embodiment, C
The PU 4 performs this operation by determining a circumscribed rectangle of a cluster in the two-dimensional image having sides parallel to the sides of the image.

In step S40, the CPU 4 projects the circumscribed rectangle determined in step S38 onto the three-dimensional world space when the computer model was formed in step S2, using the conversion calculated in step S24. This creates a single plane at the position of the three-dimensional computer model determined by the position of the object in the video image. In the present embodiment, the plane in the three-dimensional computer model is defined as vertical (the object in the scene to be observed is moving on the corresponding real world surface, that is, on the road surface 34 in the example of FIG. 2). ), The base of the plane is the step S24 in the 3D model.
Above the plane defined by the point selected by the user.

FIG. 7 shows an operation executed by the CPU 4 when converting the circumscribed plane in step S40 of FIG.

In FIG. 7, at step S62, the CP
U4 projects the two corners of the base of the circumscribed rectangle from image space to three-dimensional world space by transforming the coordinates using the transform previously calculated in step S24. Each corner of the base of the circumscribed rectangle is transformed into one point in the three-dimensional world space of the computer model located on the plane defined by the point previously selected in step S24.

In step S64, the CPU 4 calculates the width of the circumscribed rectangle in the three-dimensional world space by determining the mutual distance between the corners converted in step S62.

In step S66, the CPU 4 determines the ratio between the width and the height of the circumscribed rectangle in the image space and the step S6.
The height of the circumscribed rectangle in the three-dimensional world space is calculated using the width in the three-dimensional world space calculated in 4 (that is, the aspect ratio of the circumscribed rectangle in the image space and the three-dimensional world space is kept the same). ).

Returning to FIG. 5, in step S42, the CPU
4 together with texture data relating to the circumscribed rectangle extracted from the circumscribed rectangle in the video image and a "foreground mask", i.e., a mask for identifying which pixel in the circumscribed rectangle corresponds to the foreground pixel, S4
The position and size of the circumscribed rectangle in the three-dimensional world space calculated by 0 are stored. The extracted texture data effectively provides a texture map for a circumscribed rectangle in the 3D world space.

In step S44, the CPU 4 determines whether there is another unprocessed cluster among the clusters of the foreground pixels identified in step S34. Steps S36 to S44 are repeated until all clusters of foreground pixels of the video frame to be considered have been processed as described above. When the processing is completed, the generation of the three-dimensional computer model of the object captured by the camera 12a ends. In the model, one flat surface (circumscribed rectangle) is arranged to represent the position of each moving object, and texture image data on those moving objects is stored. Thus, this data corresponds to a three-dimensional computer model of a single two-dimensional image (video frame) from camera 12a (the corresponding three-dimensional computer model of a temporally corresponding frame of image data recorded by camera 12b). Is generated in step S6b).

Returning to FIG. 3, in step S8, the CPU 4
Processes the three-dimensional computer model generated in steps S6a and S6b to identify the portion of the object in the three-dimensional computer model that corresponds to the shadow of the first video image. Next, the shadow associated with each object is stored as a separate object in the computer model. Hereinafter, this process will be described.

As described above, in the present embodiment, it is assumed that each foreground object identified in the video image is at its lowest point in contact with the real world surface, and the corresponding flat surface of that object is Circumscribed rectangle has 3 bases
It is placed in the 3D world space while being located on the ground surface in the D model. However, since the shadow is attached to the object in the image and moves with the object, the shadow created by the object is identified as a part of the object when performing the above steps S6a and S6b. Thus, the circumscribed rectangle contains both the shadow and the object, so if the shadow is below the object in the video image, the base of the circumscribed rectangle is determined by the lowest point of the shadow. In this situation, the object will effectively stand on the shadow in the three-dimensional model. Therefore, the processing executed by the CPU 4 in step S8 addresses this problem.

FIG. 8 shows a processing operation executed by the CPU 4 when the shadow processing is executed in step S8 in FIG.

In FIG. 8, in step S100, C
PU4 is the next pair of corresponding objects (this is the first pair when step S100 is executed for the first time)
Is converted from the image space to the 3D world space. That is, the CPU 4 maps each pixel of the image data stored in step S42 (FIG. 5) relating to the object recorded by the camera 12a from the image space of the camera 12a to the 3D world space, and the corresponding time frame by the camera 12b. Is mapped from the image space of the camera 12b to the 3D world space, for each pixel of the image data stored for the corresponding object recorded in. In the present embodiment, the corresponding object is 3D
It is identified from the coordinates of the corners of the base of the circumscribed rectangle in world space. That is, the predetermined object has angular coordinates that are within a predetermined distance from the angular coordinates of the corresponding object,
The mapping of the pixels of the image data is first performed in step S24 using the conversion specified for each camera.

The mapping between the image space of each camera and the 3D world coordinates of the computer model calculated in step S24 is effective only for points on the ground plane of the real world (road surface in the example of FIG. 2). is there. Therefore, after applying this transformation in step S100, the transformed image data related to the shadow of the object recorded by one camera is converted to the transformed image data related to the shade of the same object recorded by the other camera. And 3D world space. This is because the shadow is located on the ground plane of the real world. In contrast, the object itself is not on the real world ground plane, so the transformed image data for different cameras will not be aligned in 3D world space (this image data will be "inaccurate when applying the mapping transformation" To 3D world space).

Therefore, even in step S100, the CPU 4
Compares the converted image data related to the corresponding object. In the present embodiment, the comparison is performed in a conventional manner, for example, by “Computer” by RMHaralick and LGShapiro.
and Robot Vision Volume 2 '' (Addison-Wesley Publish
ing Company, 1993, based on the method described in Chapter 16 of ISBN 0-201-56943-4 (v.2), the pixel values after conversion are compared pixel by pixel, and the gray scale value (and And / or color values) are within a predetermined amount range (10 gray levels out of a total of 256 levels in this embodiment),
It does this by identifying the pixels as identical. C
PU4 includes an alignment part (shading) of the object data,
Data defining the boundary with the unaligned portion (object) is stored for later processing (this boundary is the “footprint” of the object on the ground, ie, the outline of the point where the object touches the ground). Is effectively represented.

In step S102, the CPU 4 takes out the corresponding object portion identified as being aligned in step S100. As described above, this portion corresponds to the shadow of the object, and is stored by the CPU 4 as a separate object on the ground plane of the 3D model for each camera.

In step S104, the CPU 4 determines whether the camera 12 has been identified as being out of alignment in step S100.
Consider the image data recorded by a with the converted image data from the corresponding object recorded by camera 12b. The CPU 4 determines a circumscribed rectangle of this image data in the image space of the camera 12a (this step corresponds to step S38 in FIG. 5,
This time about objects without shading).

In step S106, the CPU 4 projects the new circumscribed rectangle determined in step S104 from the image space of the first camera to the 3D world space. This step corresponds to step S40 in FIG. 5, and is executed in the same manner as step S40.

In step S108, the CPU 4 stores the position and size of the converted circumscribed rectangle as object data relating to the 3D model of the first camera, together with the image data and the related "foreground mask" (this step is shown in FIG. 5 corresponds to step S42).

The object data (related to the shadow) stored in step S102 and the object data stored (related to the object excluding the shadow) stored in step S108 are previously stored in step S42 for the object and its shadow. Replaced by composite object data.

In steps S110 to S114, the CP
U4 repeats steps S104 to S108 for the second camera (camera 12b). In this case as well,
Steps S110 and S1 for shadows and objects
The data stored as a separate object at 14
The second camera is replaced with the synthesized data previously stored in step S42.

In step S116, the CPU 4 determines whether or not there is still a corresponding object pair to be processed in the world space. As described above, steps S100 to S116 are repeated until all corresponding objects have been processed.

Referring back to FIG. 3, in step S10, C
The PU 4 processes object data relating to different cameras, determines whether or not an object composed of two objects to be separately represented actually exists, and performs an appropriate correction. This process is performed on objects, not shadows.

As an example, FIG.
1 schematically shows persons 50 and 52 who may represent two objects as one object in the D world space, and one configuration of cameras 12 and 12b.

In the example of FIG. 9A, the viewing direction of the camera 12 is set such that the person 50 is immediately behind the person 52. In this situation, as shown in FIG.
0 and 52 appear as one object in the image recorded by the camera 12a at a specific point in time when the two maintain this alignment. Therefore, step S6a
The CPU 4 identifies the image data relating to the two people as a single foreground object, defines a circumscribed rectangle surrounding this image data, and defines the circumscribed rectangle as the camera 12a.
3D world space. In contrast, as shown in FIG. 9c, the image data recorded by the camera 12b for the person arrangement of the example shown in FIG. 9a clearly shows the two persons as separate objects. Thus, in step S6b, the CPU 4 identifies the two persons as separate foreground objects, defines another circumscribed rectangle, and projects the separate circumscribed rectangles into the 3D world space of the camera 12b.

FIG. 10 shows that the CPU in step S10 of FIG.
4 shows the processing operation executed by

In FIG. 10, in step S200,
The CPU 4 determines the height in 3D world space of the circumscribed rectangle of the next object from the camera 12a (this is the first object executed first in step S200) and the 3D of the circumscribed rectangle of the corresponding object from the camera 12b. Compare with height in world space. In this embodiment, the corresponding objects are identified using the coordinates of the corners of the base of each circumscribed rectangle in 3D world space, since their coordinates are within a predetermined distance with respect to the corresponding objects. is there.

In step S202, the CPU 4 determines whether the heights of the circumscribed rectangles are the same or within a predetermined error limit (10% in this embodiment). In step S24 (FIG. 4), 3
Since the same reference point has been selected to determine the mapping to D world space, and since each camera has a constant zoom setting, the object data for each camera has the same scale. Therefore, as shown in FIG.
The circumscribed rectangle 100 generated from the image data of 2a and representing the composite object composed of the two persons 50 and 52 is larger than the circumscribed rectangle 102 generated from the image data of the camera 12b and representing the person 52 in the 3D world space. Have a height. Therefore, if it is determined in step S202 that the circumscribed rectangles have the same height, it is determined that each circumscribed rectangle represents a single object, and the process proceeds to step S214. On the other hand, when it is determined that the heights are not the same, the CPU 4
It is determined that the tallest circumscribed rectangle represents the composite object to be divided into a plurality of constituent objects, and the process proceeds to step S204.

In step S204, the CPU 4 sets the tallest circumscribed rectangle to the bottom (106 in FIG. 11) having the same height as the circumscribed rectangle 102 of the corresponding object.
The original tall circumscribed rectangle 100 is divided into the uppermost part (108 in FIG. 11) composed of the remaining part. Since the bottom portion 106 and the image data related thereto represent one object (person 52) in front of the other object (person 50) in the composite object, they are stored as a single object. Therefore, as described below, the CPU 4 executes the processing of steps S206 to S212 in order to represent the rear object (that is, the object farthest from the camera 12a) at the correct position in the 3D model.

In step S206, the CPU 4 divides the corners of the bases of all the circumscribed rectangles of the camera that has generated the circumscribed rectangle (102 in FIG. 11) of the correct height from the world space in step S204, and divides them in step S204. (100 in FIG. 11) is projected onto the image space of the camera (camera 12a) that has generated it. This conversion is performed for the camera (camera 12a) that generated the tall circumscribed rectangle in step S2 in FIG.
Performed using the inverse of the transformation calculated in 4.

In step S208, the CPU 4 determines which of the bases of the circumscribed rectangle projected in step S206 overlaps with the "too large" circumscribed rectangle surrounding the image data of the composite object in the image space of the camera where the composite object is recorded. Identify. In the present embodiment, this is performed by comparing the coordinates of the corners of the projected base with the coordinates of the four corners of the circumscribed rectangle that is too large (in steps S208 and S210, the base is If you belong to a circumscribed rectangle that encloses at least a portion of the projected base is located behind the `` too large '' circumscribed rectangle, just project the coordinates of the vertices of the base of the circumscribed rectangle and compare Is good).

In step S210, the camera (camera 12a) that has generated the circumscribed rectangle that is too large is converted to the image in step S2 using the conversion previously calculated in step S24.
The base of the circumscribed rectangle identified at 08 is projected into the original 3D world space. The base of the reprojected circumscribed rectangle represents the correct position in 3D world space with respect to the back object in the composite image. Accordingly, the CPU 4 runs along the reprojected base of the top of the circumscribed rectangle 108 divided from the tall circumscribed rectangle in step S204, and the center of the reprojected base is the same in the 3D world space. Reposition top 108 to occupy position.

In step S212, the CPU 4 generates the initially large circumscribed rectangle of the camera (camera 12).
As a separate object in the object data relating to a), the repositioned top part is set in step S2.
04 together with the image data identified.

After executing steps S200 to S212, CPU 4 completes the operation of separating the composite objects into their constituent objects and storing them as separate objects in the object data relating to the associated camera.

This process is intended to prevent a sudden change in the height of an object that has been found to be very noticeable, even if it lasts only a few image frames reproduced from the 3D computer model. It is valid.

Referring back to FIG. 9B, the person 5
It can be seen that not all of the zeros, but some of them are over the person 52. Therefore, when the uppermost part 108 of the circumscribed rectangle 100 is separated in step S204, image data relating to only a part of the person 50 is separated. Therefore, step S
Steps 206 to S212 generate an expression such that the image data relating to the object (person 50) farthest from the camera in the composite object only corresponds to the top of the actual object. Further, in step S204,
The image data held at the bottom 106 of the rectangle 100 for the object (person 52) closest to the camera is actually the image data corresponding to the object farthest from the camera (when this is located at the bottom of the circumscribed rectangle). It is thought to contain. However, this "inaccurate" representation has been found to be relatively inconspicuous in images generated from the three-dimensional computer model. This is especially because composite objects only exist over a relatively small number of video frames (the alignment of the objects changes in the scene).

In step S214 in FIG.
Determines whether there is another object in 3D world space for the first camera. Steps S200 to S214 are repeated until all such objects have been processed as described above.

Returning again to FIG. 3, steps S11a and S11a
In 11b, the CPU 4 generates a representation (model) of each object in the 3D world space. That is, step S
In 11a, the CPU 4 generates a representation in the object data relating to the first camera of each foreground object identified in step S6a. Similarly, step S11
In b, the CPU 4 generates a 3D model related to the 3D object data of the second camera. Step S1
In 1a / S11b, the CPU model is an object, not its shadow. This is because the shading is already in step S
8 (FIG. 3) is accurately modeled.

FIG. 12 shows steps S11a and S11b.
Shows a processing operation executed by the CPU 4. Since this process is the same for each camera, only the process related to the camera 12a executed in step S11a will be described.

In FIG. 12, in step S230,
The CPU 4 determines the ground profile (“footprint”) of the next object in the object data of the camera 12a, which is stored in step S100 (FIG. 8) earlier.
(I.e., the contact profile of the object on the ground, representing the point at which the object touches the ground plane in 3D world space) is read. Next, the CPU 4 executes, for example, RMHaralick and LGShapiro
`` Computer and Robot Vision Version 1 '' (Addison-
Wesley Publishing Company, 1992, ISBN 0-201-10877-
Apply a pixel mask to the circumscribing pixels to calculate the pixel-by-pixel gradient value, then apply the "t-statistic" test, based on the method described in Section 11.4 of 1 (v.1), etc. The "footprint" is approximated by fitting a plurality of straight lines to the shape of the boundary in a conventional manner. Step S230
13a and 13b show examples of a ground boundary shape 150 representing the ground profile of the vehicle 32 (the boundary shape in the example shown in FIG. 13a already contains a substantially straight line, but is curved. Footprint may be approximated).

In step S232, the CPU 4 uses the footprint approximation 160 defined in step S230 to define a model for expressing an object in the 3D world space. In the present embodiment, the CPU 4 defines a plurality of flat surfaces 170 and 172 (in the present embodiment, rectangles) in the 3D world space, and the base of each rectangle corresponds to one of the straight lines defined in step S230. , The vertical side is the height of the single circumscribed rectangle previously used as a representation of the object in the 3D world space in the 3D world space (stored in step S42 of FIG. 5 and then the step of FIG. 3) S8 and / or modified as needed in step S10). Step S
232 is shown in FIG. 13c.

In step S234, the CPU 4 maps each rectangle defined in step S232 from the 3D world space to the image space of the camera 12a. This applies the inverse of the transform previously calculated in step S24 (FIG. 4) for camera 12a to the coordinates of the corners of the base of each rectangle, and the aspect ratio of the transformed rectangle in the image space of camera 12a (ie, high Is performed by allowing the height of the transformed rectangle in image space to be determined by assuming that the aspect ratio of the rectangle in 3D world space is the same as the aspect ratio of the rectangle in 3D world space.

In step S236, the CPU 4 extracts pixel data located in each of the converted rectangles in the image space of the camera 12a.

In step S238, the CPU 4 determines the position and size of the flat surface in the 3D world space (defined in step S232) and the image data associated with each rectangle (step S232) as the object data of the camera 12a.
236) and the foreground mask of the image data (based on the data previously stored in step S42).

In step S240, CPU 4 determines whether there is another object to be modeled for camera 12a. Step S until all objects have been modeled as described above.
Steps S230 to S240 are repeated.

Returning to FIG. 3, steps S12a and S12a are executed again.
In step 12b, the CPU 4 stores the object data generated in steps S11a / S11b as time-stamped 3D object data for each camera.

In step S14, the CPU 4 determines whether or not an unprocessed image remains among the images recorded by the cameras 12a and 12b. Steps S6a / S6b, S8, S10, S11a / S11 until all such images have been processed in the manner described above.
b, S12a / S12b and S14 are repeated. When the processing is completed, two sets of the time-stamped 3D object data are stored in the memory 6, one set for each of the cameras 12 a and 12 b.

In step S16, the CPU 4 displays an image on the display device 18 for the user at a desired viewpoint selected by the user. The image displayed by the CPU 4 in this step is a simulated video image generated using the previously generated three-dimensional model object data.

FIG. 14 shows a processing operation executed by the CPU 4 when displaying an image in step S16.

In FIG. 14, in step S296,
The user specifies the direction in which the object should be observed by the input device 14.
Stipulated using.

In step S298, the CPU 4 determines whether or not the view direction selected by the user is within a predetermined angle of the vertical view direction overlooking the object. In the present embodiment, the CPU 4 determines whether or not the selected viewing direction is within ± 15 ° from the perpendicular. If it is determined in this step that the user has selected a viewing direction that falls within a cone having a vertical axis and a half-angle of 15 °, the process proceeds to step S300, where the CPU 4 determines all the directions on the ground plane. Is rendered in the frame buffer 16 to roughly represent the position of the object.
In the present embodiment, this uses object data relating to a predetermined camera (for example, camera 12a), and sets the position of each object on the ground plane determined from the position of the object in the object data relating to the selected camera. This is performed by rendering the vertical field of view of the stationary background scene using the common 3D model generated in step S2, with a graphic or other visible sign, such as a cross indicated at the corresponding position.

Steps S298 and S300 comprise a process for giving the user an aerial chart showing the positions of all objects. Such an aerial chart may be useful in some situations. For example, if the recorded video data is related to an accident or a crime, the police will use the image data generated in steps S298 and S300 to determine the relative positions of cars and / or people at different times. You can gather evidence to do it. Similarly, if the object is a person participating in a team competition, the image data generated in steps S298 and S300 can be used for training purposes in order to analyze the relative positions of the players.

If it is determined in step S298 that the viewing direction selected by the user is not within the predetermined angle range, C
The PU 4 executes a process for rendering a real image of the object to the user from the selected viewing direction, as described below.

In step S302, CPU 4 should use the object data (stored in step S12a) generated from the image data of camera 12a to generate image data to be displayed to the user, or It is determined whether to use the object data (stored in step S12b) generated from the image data of the camera 12b.

FIG. 15 shows an operation executed by the CPU 4 when selecting object data in step S302.

In FIG. 15, in step S400,
The CPU 4 compares the viewing direction selected by the user in step S296 with the viewing direction (optical axis) of the camera 12a and the viewing direction of the camera 12b, and identifies a camera having a viewing direction closer to the viewing direction selected by the user.

In step S402, the CPU 4 sets the view direction of the camera not identified in step S400 (that is, the camera having a view direction farther from the view direction selected by the user) to the predetermined view direction of the view direction selected by the user. It is determined whether or not it is within the angle range (cone). In the present embodiment, the CPU 4 determines whether the axis is within ± 30 ° of the selected viewing direction. If the axis of the camera deviates from this predetermined angle, the image quality of the image generated by the camera for the viewing direction selected by the user is significantly inferior to the image generated by the camera identified in step S400. Will be. Accordingly, in this case, the processing is performed in step S42.
Proceeding to 8, the CPU 4 selects object data from a camera having a viewing direction closer to the viewing direction selected by the user.

On the other hand, if it is determined in step S402 that the viewing direction of the other camera is within the range of the predetermined angle, the camera relating to the camera having a viewing direction farther from the viewing direction selected by the user is involved. The object data may actually show better image quality than the image in the viewing direction selected depending on the situation. Therefore, in this case, C
PU4 uses a camera having a viewing direction that is farther from the user-selected viewing direction than using the camera data identified in step S400 that has a viewing direction closer to the user-selected viewing direction. In order to determine whether or not good image quality can be obtained, a further test is performed in steps S404 to S426.

At steps S404 to S414, the CP
U4 compares the characteristics of cameras 12a and 12b which affect the image quality of the image reproduced from the camera's three-dimensional object data, as described below.

In step S404, CPU 4 compares the method used to transfer image data from camera 12a to computer 2 with the method used to transfer image data from camera 12b to computer 2. . This information may be input by the user prior to processing, or may be transmitted from the camera along with the image data.

In step S406, the CPU 4 determines whether the transfer method is the same. If they are not the same, in step S408, the CPU 4 selects the object data from the camera with the higher image quality (that is, the transfer method with the smaller number of errors usually introduced into the image data). In the present embodiment, the image data is transferred via a cable and / or a recording medium (for example, after the image data is recorded on the internal recording medium of the camera, the image data is It is conceivable that the error introduced is smaller when the data is transferred to the computer 2). Therefore, when one camera is transferring data by wireless transmission, the CPU 4 selects object data from the other camera in step S408.

On the other hand, if it is determined in step S406 that the image data transfer methods are the same, the process proceeds to step S410, where the CPU 4 compares the resolutions of the cameras. In the present embodiment, this is performed by comparing the number of pixels in the image from the camera.

In step S412, the CPU 4 determines whether or not the resolutions of the cameras are the same. If the resolutions are not the same, the process proceeds to step S414,
Selects object data from the camera with the higher resolution (the one with the larger number of pixels).

On the other hand, if it is determined in step S412 that the cameras have the same resolution,
Compares the characteristics of the image data generated by the camera. That is, the process proceeds to step S416, and the CPU 4 compares the stability of the images generated by the cameras 12a and 12b. In the present embodiment, this is performed by comparing the optical image stability parameters calculated by each camera according to the amount of camera shake and transmitted with the image data.

In step S418, the CPU 4 determines whether the stability compared in step S416 is the same. If the stability is not the same, in step S420,
The CPU 4 selects object data from a camera having higher image stability.

On the other hand, if it is determined in step S418 that the image stability is the same, the process proceeds to step S422, where the CPU 4 compares the number of closed objects in the object data from each camera. I do. In the present embodiment, this is performed by comparing the number of circumscribed rectangles for each set of object data divided when the processing is executed in step S10 (the divided circumscribed rectangles are respectively closed or closed. Represent).

At step S424, CPU 4 determines whether or not the number of closed objects is the same for each camera. If the numbers are the same, step S426
Then, the CPU 4 selects object data from the camera with the least number of blockages.

On the other hand, if it is determined in step S424 that the number of obstructed objects is the same, the process proceeds to step S428, in which the CPU 4 determines whether the camera having a viewing direction closer to the viewing direction selected by the user. Select object data from. This data is
In steps S406, S412, S418, and S424, the CPU 4 determines that the intrinsic camera characteristics and the image data characteristics that affect the image quality are the same for both cameras, and therefore, are arranged closer to the viewing direction selected by the user. Is selected because it has been determined that the best image quality will be generated using object data from the failed camera.

Returning to FIG. 14, in step S304,
The CPU 4 renders the selected object data on the frame buffer 16.

FIG. 16 shows an operation executed by the CPU 4 when rendering object data in step S304.

Referring to FIG. 16, in step S500,
3D of the object data selected in step S302
The world space is converted to the visual field space according to the visual field position and the visual field direction selected in step S296 (FIG. 14). This transformation identifies a particular field of view that is not large enough to typically cover the entire modeling space. Therefore, in step S502, the CPU 4 executes a clipping process for removing a surface or a part thereof out of the field of view.

Up to this stage, the object data processed by the CPU 4 defines a three-dimensional coordinate location. In step S504, the vertices of a triangular surface forming the 3D computer model are projected to define a two-dimensional image.

After projecting the image in two dimensions, a triangular surface “facing the front”, ie, a surface facing the observer, and a “rearward” surface not visible to the observer are identified. There is a need to. Therefore, in step S506, the face facing backward is identified and selected by a conventional method. Thus, after step S506, the vertices are defined in two dimensions that identify the surface of the visible polygonal triangle.

In step S508, the CPU 4 scan-converts the two-dimensional data defining the plane to generate pixel values. In this step, in addition to rendering the surface representing the background in the image, the shadow (stored as a separate object) is rendered, and the surface defined in step S11a or S11b is used to model each object. Also render with the video texture data previously stored for those surfaces. Only the foreground pixels in the plane are rendered with the stored texture data, and those pixels are defined by the stored "foreground mask". Other pixels are rendered with the background texture data. The drawing data generated in step S508 represents a simulated video frame, in which
The background is generated from the computer model generated in step S2, and each object is generated in steps S11a / S11.
The image data is represented by a model defined by b, and image data of an object extracted from a video image is projected thereon.

In step S510, step S508
The data of the entire two-dimensional image is generated by writing the pixel values generated in step 2 into the frame buffer 16 for each plane.

Returning to FIG. 14, in step S306, the CPU 4 extracts an image to be displayed to the user for the current data frame being processed and an image to be displayed to the user from the previous data frame from different cameras. It is determined whether or not it is. That is, CPU4
Determines whether the camera has changed in the sequence of images that the user should see. In the present embodiment, the CPU 4 determines in step S30 for the current frame.
It is determined whether or not this change has occurred by determining whether or not the object data selected in step 2 has been extracted from a camera different from the object data selected in step S302 for the previous frame.

If it is determined in step S306 that a camera change has occurred, then in step S308, the CPU 4
Renders the object data of the other camera related to the current frame in the frame buffer 16. This is the rendering in step S304 (see FIG.
6 described above), but the data is written to a different frame buffer 16 to prevent overwriting on the data rendered in step S304.

In step S310, the CPU 4 outputs the image data rendered in step S304 to step S310.
Compare with the image data rendered in 308. This comparison is performed to determine the similarity of the images, for example,
Computer and Robot by RMHaralick and LGShapiro
Vision Volume 2 '' (Addison-Wesley Publishing Compan
y, 1993, ISBN 0-201-56943-4 (v.2), pp. 293-378)
Performed using conventional techniques such as those described in US Pat.

In step S312, the CPU 4 determines whether or not the user sees "interruption" in the image when rendering in step S304 but displaying the total data (ie, whether or not the image looks discontinuous). Is determined. In the present embodiment, such a discontinuity (discontinuity) is an approximation of the model generated for each object in steps S11a / S11b, and the position of each model in the 3D world space is completely accurate. It is considered that this is caused because the judgment is not made and the object data of each camera is different. In the present embodiment,
The CPU 4 determines whether or not the image becomes discontinuous by determining whether or not pixels exceeding a predetermined number (for example, 20%) have failed the similarity test performed in step S310. I do.

If it is determined in step S312 that a continuous image looks discontinuous, then in step S314, C
The PU 4 combines the image data rendered in step S304 and the image data rendered in step S308 to generate an image that does not appear discontinuous for the user. In the present embodiment, the combination of images is, for example, George Wolberg's "Digital Image Warping" (IEEE
Computer Society Press, ISBN 0-8189-8944-7, P222-24
(Page 0) using conventional morphing techniques.

In step S316, the CPU 4 determines the image quality information relating to the image data generated in step S304 or S314 for display to the user thereafter. That is, in the present embodiment, the CPU 4 determines the following information indicating the reference of the image to be displayed, that is, a value indicating the degree of reliability (accuracy) of the image. This information is useful in situations where a user is trying to use a computer model to determine what happened in the real world (e.g., a police officer uses a video recording of a crime to generate a computer model). And use the computer model to gather evidence by looking at crime from various directions). Since each image generated from the computer model is a simulated image, the accuracy value determined in this step is a value indicating the reliability of the image. In the present embodiment, the CPU 4 ranges from 100% (when the viewing direction selected by the user is the same as the axis of the camera that generated the object data) to 0% (when the viewing direction deviates from the camera axis by 30 °). Assign the value of CPU4 for a departure angle from 0 ° to ± 30 °
Varies from 100% to 0%. In the case of a departure angle exceeding 30 °, the CPU 4 assigns a value of 0%. Further, in the present embodiment, when the image generated by combining the image data in step S314 is to be displayed to the user, the CPU 4 sets the assigned percentage value to 1
Reduce by 0% (0% is the minimum limit).

The CPU 4 may further add a graphic such as an arrow or other visible sign to be displayed to the user to show how the viewing direction can be changed to improve the image quality value. Generate.

In step S318, the CPU 4 determines the pixel value rendered in step S304 or S314,
A signal that defines the image quality information generated in step S316 is generated. This signal is used to generate an image of the object on the display device 18 and / or is recorded, for example, on a video tape of a video tape recorder 20. The signal may be sent to a remote receiver for display or recording. It goes without saying that recording may be performed further from the master recording. The image quality information may be combined with the pixel value so that the image quality information is always visible in the image together with the object, or the pixel information may be selectively displayed and deleted according to a command from the user.

In step S320, the CPU 4 determines whether or not there is another time-stamped “frame” of the three-dimensional object data created in steps S6a and S6b, which is not yet displayed to the user. judge. By repeating steps S296 to S320 until all the frames of such object data are displayed as described above, the sequence of the simulated moving image is displayed to the user from a desired visual field direction. Of course, the user can change the viewing direction at any time during the display. In the present embodiment, when step S306 is performed on a frame subsequent to the first frame, if the previous frame of the image data has been generated from the combined image data (generated in step S314), the CPU 4 Determine that a change has occurred. In this case, in step S308, the CPU 4 renders the combined image of the current frame into the frame buffer (that is, the CPU 4 renders the image generated using the object data on the current frame from the first camera into the current image from the second camera). An image generated by combining with an image generated using object data regarding a frame). Next, in step S310, C
The PU 4 compares the image data rendered in step S304 for the selected camera with the combined image data rendered in step S308. If the previous frame is a schematic diagram of the object position generated in step S300 in step S306, the CPU 4
Determines that there has been no camera change (because the user would expect to see a “break” in the image when switching between a schematic view of the position and a real image).

[Second Embodiment] Next, a second embodiment of the present invention will be described.

In the present embodiment, steps S11a / S11b are used to model an object in the 3D world space.
Are the same as those of the first embodiment except for the processing operations executed by the CPU 4.

In the second embodiment, the modeling of the object in the 3D world space is executed integrally by each camera (instead of being executed separately in steps S11a and S11b as in the first embodiment).

FIG. 17 shows a processing operation executed by the CPU 4 to model an object in the 3D world space. As in the first embodiment, these operations are performed to model the object rather than the shadows (already modeled in step S8 of FIG. 3).

Referring to FIG. 17, in step S250,
The CPU 4 considers the next object to be processed and identifies the plane in image space of each camera where the point on the object is located.

FIG. 18 shows the processing operation executed by CPU 4 in step S250.

Referring to FIG. 18, in step S264,
The CPU 4 compares the image data of the object recorded by each camera and identifies a coincident point in the image data.
The CPU 4 compares the coordinates of the corner of the base of the circumscribed rectangle of the object in the object data related to the first camera with the coordinates of the corner of the base of the circumscribed rectangle of the object in the object data from the second camera. , Of the circumscribed rectangle of the object data related to the second camera,
By determining which circumscribed rectangle having a base corner located within a predetermined distance from the base corner of the circumscribed rectangle in the object data relating to the first camera in the 3D world space, Identify what should be compared. Next, the CPU 4 stores the previously stored image data (step S6a in FIG. 3, and then makes corrections in step S8 or S10 as necessary) with respect to the circumscribed rectangle in the object data of the first camera. The circumscribed rectangle of the corresponding object in the object data of the second camera is compared with the previously stored image data (step S6b in FIG. 3 and then, if necessary, modified in step S8 or S10). In this embodiment, the CPU 4 performs this comparison by comparing the vertex identified in the image data from the first camera with the vertex identified in the image data of the second camera.

FIG. 19 shows the processing executed by the CPU 4 in step S264.

Referring to FIG. 19, in step S282,
The CPU 4 calculates, for each pixel in the image data of the object from the first camera, a value indicating the “side” and “corner” of the pixel. This is done, for example, by applying a conventional pixel mask to the image data and moving the mask such that each pixel is taken into account. Such techniques are:
Computer and Robot by RMHaralick and LGShapiro
Vision Volume 1 '' (Addison-Wesley Publishing Compan
y, 1992, ISBN 0-201-10877-1 (v.1)). In step S284, “side” exceeding a predetermined threshold
And the pixel having the value of “corner” is identified as a possible corner in the image data of the first camera as in the related art.

In step S286, the CPU 4 executes the operation previously performed on the image data from the first camera in step S282 on the image data from the second camera. In step S288, the CPU 4 executes the operation in step S28.
Using the same technique as performed in step 4, identify the likely corners in the image data from the second camera as well.

In step S290, the CPU 4 determines each influential corner identified in the image data from the first camera in step S284 as each influential angle identified in the image data from the second camera in step S288. A similarity is generated for a corner in the image data from the first and second cameras as compared to the corner. In the present embodiment, this is, for example, the AWGuen “Adaptive Least Squares Correlati
on: A Powerful ImageMatching Technique '' (Photogram
Published in metry Remote Sensing and Cartography, 1985
, Pp. 175-187).

In step S292, the CPU 4 identifies a matching vertex and stores it. This is performed using "relaxation" techniques, as described below. Step S
290 generates a similarity between each dominant corner in the image data from the first camera and a plurality of dominant corners in the image data from the second camera. In step S292, for example, the CPU 4 arranges all possible corners in the image data from the first camera in a column, and arranges all possible corners in the image data from the second camera in a row. By finding the similarity for a pair of angles at the intersection of the table,
The values are effectively arranged in a table array. In this manner, the rows of the table array define the similarity between a given vertex in the image data from the first camera and each vertex in the image data from the second camera. Similarly, the columns of the array correspond to predetermined vertices in the image data from the second camera and the first vertices.
Stipulates the similarity between each vertex in the image data from the camera. Next, the CPU 4 considers the value of the first row, selects the highest similarity value in that row, and determines whether this value is the highest value in the column where it is located. . If the value is the highest value in both the row and column, it is the vertex in the image data from the second camera that is the best match for the point in the image data from the first camera;
In addition, the vertex in the image data from the first camera is the best matching point with the point in the image data from the second camera. In this case, the CPU 4 sets all the values of the rows and columns to zero (so that those values are not considered in the subsequent processing), and the highest similarity value becomes a predetermined threshold (0.1 in this embodiment). Is determined. If the similarity value exceeds the smart value, the CPU 4 stores the vertex in the image data from the first camera and the corresponding vertex in the image data from the second camera as coincidence points. If the similarity value does not exceed a predetermined threshold, it is determined that the degree of similarity is not enough to store the point as a match point, even though those points are the best match points with each other. I do.

Next, the CPU 4 repeats this process for each row of the table array until all rows have been considered. If it is determined that the highest similarity of one row is not the highest measure of the column in which it is located, CPU 4 proceeds and proceeds
Consider the following line:

If a match is identified when all rows are considered first, the CPU 4 repeats the above process, again considering each row of the table. The CPU 4 continues to execute such an iterative process until no matching point is identified in the repetition.

Referring back to FIG. 18, in step S266,
The CPU 4 considers the next four pairs of coincidence points identified in step S264.

In step S267, the CPU 4 determines whether or not the four points in the image space of the first camera are located on one plane and the corresponding four points in the image space of the second camera. Is also determined on one plane. If any one set of four points is not on the plane, the process proceeds to step S276 without further considering the point selected in step S266. In contrast, both sets of 4
If the points are on a plane in each image space, the process proceeds to step S268.

In steps S268 and S267, four pairs of coincidence points are used instead of the minimum number of three pairs required to ensure that a plane can always be defined via points in each image space. Reduces the number of times to consider “false” planes (ie, planes that do not correspond to the plane of the object),
Thereby, processing requirements are also reduced. Step S268
Then, the CPU 4 determines between the plane including the four points considered in step S266 in the image space of the first camera and the plane including the corresponding four coincident points in the image space of the second camera. Calculate the transformation of This transformation is conventionally calculated, for example, using the following equation:

[0157]

(Equation 3)

In the formula, n = 1. . . . 4, x2n, y2n
Are points in the image space of the second camera, x1n, y1n are points in the image space of the first camera, J, K, L, M, N,
P, Q, and R are obtained by the following equations.

[0159]

(Equation 4)

In step S270, the CPU 4 proceeds to step S2 for each pair of remaining matching points (that is, the pair of matching points not used when calculating the conversion in step S268).
Test the conversion calculated at 68. That is, the CPU 4 considers each of the remaining pairs of points, and uses the coordinates of the points (by determining whether or not the conversion calculated in step S268 is valid for the coordinates of the points) to convert the calculated conversion. It is determined whether or not these points are valid.

In step S272, the CPU 4 determines whether or not the number of pairs of points determined to satisfy the conversion is larger than a predetermined number (eight in this embodiment). If it is determined that the transformation is valid for more than a predetermined number of pairs of points, the CPU 4 has identified the plane of the object in the image data from the first camera and the second camera. judge. Accordingly, the process proceeds to step S274, where the CP
U4 stores data defining a plane in the image data of the first camera and the image data of the second camera, together with the coordinates of an identified point located on the plane.

On the other hand, if it is determined in step S272 that the conversion calculated in step S268 is not valid for a predetermined number of pairs of points, the CPU 4 uses the conversion in step S268 to calculate the conversion. It is determined that the point is located on an unreal plane, that is, a plane that does not actually form a part of the surface of the object. Therefore, the plane is ignored.

In step S276, the CPU 4 determines whether or not there are four further pairs of coincidence points. Until the number of remaining matching point pairs not yet considered is less than 4,
Steps S266 to S276 are repeated.

Referring back to FIG. 17, in step S252, the CPU 4 calculates the boundaries of the planes identified in step S250, and extracts image data relating to those planes from image data of an appropriate camera. The CPU 4 considers the points stored in step S274 (FIG. 18) for all the planes, and identifies those points that are located on two or more planes and that are located on the boundary between the planes. Perform the action. Since the point that matches in the image data from the first camera and the second camera in step S264 (FIG. 18) is the vertex, the processing executed by the CPU 4 generates such a boundary point. .

After identifying the points located on the boundary between the planes, the CPU 4 defines the boundary by connecting the points identified for each boundary (these points are located on the same two surfaces). If there are three or more points that cannot be connected by a straight line, CPU 4 defines the boundary by drawing a straight line between the points using the conventional "least squares" method. Next, the CPU 4 determines the intersection of the defined boundaries to define a flat surface in the image space (if the boundary is not defined by a defined boundary of a part of a certain plane, the spread is determined, for example, in step S42). (Specified by the identification pixel data of the object within the entire circumscribed rectangle of the object in the same manner as determined by the “foreground mask” stored earlier in (2)). After defining the boundaries of each plane in the image space of each camera in order to define a flat surface having a finite extent, the CPU 4 extracts and stores the pixel data in each flat surface.

In step S254, the CPU 4 uses the surface defined in the image space of the camera to generate a 3D image for each camera.
Model objects in world space. To execute this, the CPU 4 calculates the position of the plane model in the 3D world space for each camera, and converts the plane model defined in the image space of the camera into the 3D world space of the camera. That is, in the present embodiment, the CPU 4 converts the conversion from the image space of the first camera to the 3D world space previously calculated in step S24 to the base of each plane identified in the image data of the first camera in step S252. Apply to the coordinates of the corner of. Similarly, the CPU 4 converts the conversion between the image space of the second camera and the 3D world space previously calculated in step S24 into the corner of the base of each plane in the image data of the second camera determined in step S252. Apply to coordinates.
The transformation previously calculated in step S24 is based on image space and 3D
Since the points located on the ground plane of the world space are also effective, when converted from the image space of the first camera and the image space of the second camera, the vertices relating to the plane tangent to the ground plane are 3D world space. (Within a certain allowable distance), but the vertices of other planes not touching the ground plane do not align. Thus, CPU 4 identifies which vertices are aligned, and in terms of the aligned points, the 3D world space of each camera has a flat vertical plane (ie, a flat vertical plane with the same aspect ratio as the flat plane in the appropriate camera image space). Place a corresponding flat surface with vertices on the ground). Next, CPU
4 is a step S in the 3D world space for each camera.
At 252, a flat surface having the same intersection and aspect ratio as the surface identified in image space for the appropriate camera is defined.

In step S256, the CPU 4 stores, as object data for each camera, data defining a flat surface in the 3D world space, image data extracted for each surface, and a foreground "mask" for each surface. .

At step S258, CPU 4 determines whether or not another object to be modeled exists. Steps S250 to S258 are repeated until all objects have been modeled as described above.

The rendering step in this embodiment is executed in the same manner as in the first embodiment, and renders the image data stored in step S256 on an appropriate surface.

The modeling of an object in the 3D world space described with respect to the second embodiment is particularly effective for an object composed of many planes, such as the cars 30 and 32 in the example of FIG. The processing operation executed in the present embodiment models the top surface of the object more accurately than the modeling operation in the first embodiment.

[Third Embodiment] In the first embodiment,
The processing operation executed to model the object in the 3D world space in steps S11a / 11b is performed when the object to be modeled is in contact with the ground over a wide area (for example, the vehicles 30, 3 in the example of FIG. 2).
This is particularly effective in 2). In the second embodiment, the modeling technique is particularly effective when the object is composed of many planes.

However, not all objects have these characteristics (for example, person 5 in the example of FIG. 2).
0,52), such objects can be modeled by the techniques of the first or second embodiment, but those techniques may also include unnecessary processing operations. .

In the third embodiment, steps S11a / S1 are used to model an object in a 3D world space.
This is the same as the first embodiment except for the processing operation executed in 1b.

In the third embodiment, the CPU 4 determines in steps S6a and S6b in step S11a / 11b.
, And then model the object using the vertical plane in 3D world space modified in steps S8 and S10. Thus, each object is modeled as a single vertical plane (one plane per camera) in the 3D world space of each camera. As in the previous case, no shadow is modeled in steps S11a / S11b.

Various modifications can be made to the embodiment described above.

Returning to FIG. 3 again, in the above embodiment, steps S6a and S6b (process the image data to identify foreground objects and generate object data therefrom) consist of steps S4a and S4.
Executed after all images are recorded in b. Similarly,
Step S16 (displaying an image) includes steps S4 and S4.
S6a / S6b, S8, S10, S11a / S11b,
This is executed after S12a / S12b and S14 are completed. However, these steps may be performed to allow a user to display in real time from a desired viewing direction. That is, while the data of the next frame is recorded by the video camera, steps S6a / S6b, S6
8, S10, S11a / S11b, S12a / S12
It is possible to execute b, S14 and S16. This real-time operation is performed in steps S6a / S6b, S
8, S10 and S11a / S11b are processed by the CPU
4 is not particularly troublesome, and is possible because it can be executed in 1/30 second, which is the time between the recording of one video frame and the recording of the next video frame.

In the above embodiment, steps S6a / S
At 6b, foreground objects are identified based on the grayscale values. However, in addition to or instead of setting a window of color values and / or infrared values,
It is also possible to identify foreground objects using their image characteristics.

In the above embodiment, step S8
Then, the image data relating to the corresponding foreground object is mapped (converted) to the ground plane of the 3D computer model defined in step S2, the converted data is compared, and the aligned part (shadow) and the non-aligned part (object) are compared. The shadow is identified by identifying the boundary with. Then, in the first embodiment, the object is modeled using the identified boundary (the ground surface “footprint”) (steps S11a / S11b). However, the mapping and comparison need not be performed on the ground plane of the 3D computer model. That is, the image data relating to the corresponding foreground object is identified by applying a transformation that identifies the shadow and applies a mapping that defines the mapping of the image data of each camera from the ground plane to a surface in modeling space for object modeling. The ground surface “footprint” may be determined by mapping to one surface in an arbitrary modeling space and comparing the converted data. Then, according to the result, a model (expression) of the object and its shadow is generated by a 3D computer model.

Similarly, in step S10, 3 is set to determine whether the object is a composite object.
Instead of comparing the heights of the circumscribed rectangles of the corresponding objects in the D computer model, the circumscribed rectangles may be converted from the image data of each camera into some common modeling space, and the heights may be compared in the modeling space.

In the above embodiment, the CPU 4
In order to display an aerial representation of the position of the object when the user selects a viewing direction close to the vertical, step S
The processing is executed in 298 and S300 (FIG. 14). Further processing is performed on image (video pixel) data stored as object data for each object in the 3D world space in order to determine the color of the object and indicate this color in the aerial representation of the position. It would be possible to do so. Such a process may be particularly useful where the objects are people participating in team sports. In this case, it would be possible to identify the color of each player's head from the object data and show a visual indication on the aerial chart to indicate which team each player belongs to. . Further, step S2
The apparatus may be configured such that the user can selectively turn on and off the processing operations executed by the CPU 4 in 98 and S300. That is, the user does not show an aerial view in any view direction, but instead can instruct the CPU 4 to always render a realistic view of the object from the selected view direction. This on / off function may be useful when modeling the top of an object, for example, in the second embodiment.

In the above embodiment, step S302
When selecting which set of object data to use to generate a frame of image data in FIG. 14, the CPU 4 determines the unique camera characteristics, the method of image data transfer between the camera and the processor, Camera resolution,
Priority is given to image stability and image data error characteristics that affect image quality in the order of the number of blocked objects in the image (FIG. 15). However, it is possible to use different priorities. Also, it is not necessary to use all of the characteristics of the camera and image data described above. For example, (Steps S404 to S408, S410 to S414, and S4, respectively,
It is possible to omit one or more of the four tests (defined by 16-S420 and S422-S426). Further, other camera characteristics (a more suitable illumination condition such as a shutter speed transmitted together with image data from each camera can be obtained, so that a camera with a faster shutter speed is selected), and whether the image is color or Image data characteristics indicating whether the image is black and white (a camera that generates color image data is selected in preference to a camera that generates black and white image data) may be considered.

In the above embodiment, step S11a /
Step S3: selecting data used to generate an image to be displayed to a user after modeling each object using image data from each camera in S11b
02 is executed. However, if the user has already defined the viewing direction, step S302 of selecting data may be executed before modeling the object in steps S11a / S11b. That is, for example, a camera is selected as described above, and then an object is modeled with a 3D computer model using image data from the selected camera. Such processing (because it does not generate models that are not used for images)
Prevent unnecessary modeling of objects, thereby saving processing time.

Even when an object is individually modeled for each camera, step S302 for selecting data used for generating an image may be executed (that is, steps S8 and S10 are omitted).

In the above embodiment, in step S310, the CPU 4 compares the rendered images using a difference technique. However, Daniel Huttenlocher et al. `` Tracking Non-Rigid Objects in Complex Scenes ''
(Proc. 4th International Conference on Computer Vis
ion, 1993, Berlin, IEEE Computer Society Press IS
Other techniques may be used, such as the Hausdorff distance as described in BN 0-8186-3870-2). Similarly, in step S314, another image combination technique (such as averaging) is used.
May be used.

In the above embodiment, steps S304 to S314 (FIG. 14) are executed for the image data of the entire frame. However, it is also possible to perform these steps so that the image data is compared for each object to determine whether the objects appear discontinuous when viewed individually. As a result, the frame of the generated image data is composed of image data from different cameras (different objects can use image data from different cameras).

In the above embodiment, in step S416, the stability of an image is determined using data from a camera or a sensor attached to its stand. Instead, the stability of the image may be determined by processing the received image data itself. For example, using optical flow techniques or S. Mann and RWPica
rd's `` Virtual Bellows: Constructing High Quality S
tills From Video '' (MIT Media Laboratory Perceptual
Computing Section Technical Report No.259, Proc. F
irst IEEE Int. Conf. on Image Proc., Austin TX, 1994
Image data may be processed using techniques such as those described in US Pat.

In the above embodiment, input image data is generated using two cameras. However, the processing operations performed in the above embodiment can be similarly performed for more cameras. In fact, a larger number of cameras will help some processing operations. For example, the shading process executed by the CPU 4 in step S8 (FIG. 3) can be improved. This is because, when there are only two cameras, one of the cameras cannot completely capture the shadow of the object, and thus the aligned image data portion in the 3D world space that is extracted and stored in step S102 (FIG. 8) is May not support full shading. Increasing the number of cameras increases the likelihood of capturing a completely shaded image with at least two cameras, so this problem may be alleviated. Thus, if necessary, the extracted complete shadow could be added to the image data belonging to the camera where the complete shadow did not actually appear in the image data.

In the above embodiment, the viewing direction of the camera is fixed. However, the view direction may be changed, and a conventional direction sensor or angle sensor may be arranged on the camera or the stand to provide information defining the view direction of each camera for each frame of image data to be recorded. good.
In that case, for example, Simon Rowe and Andrew Blake's "St
atistical Background Models for Tracking with a Ca
In order to project image data onto a fixed virtual image plane, processing will be performed in a conventional manner as described in "mera" (British Machine Vision Conference 1995).

In the above embodiment, the foreground object in the image is identified by comparing the value of each pixel with a set of values based on a plurality of images recorded only for the background (steps S3a / S3b, S4a / S4b and S6a
/ S6b). However, conventional optical flow techniques may be used to identify foreground objects. Such optical flow techniques can be used, for example, when the viewing direction of a camera changes.

In the above embodiment, the zoom (magnification) of each camera is constant. However, the zoom may be changed. In this case, for example, the transformation between the camera image space and the 3D world space calculated in step S24 (FIG. 4) would be calculated for the same four reference points for each zoom setting. This calculation is performed in steps S4a / S4b.
Can be executed prior to the recording of the frame of the image data, and by transmitting the zoom setting of the camera for each frame from the camera to the computer, the correct conversion can be selected for the image data of one frame.
Alternatively, the transformation may be calculated in real time as the camera's zoom setting changes (where the four reference points used for calibration are selected to always be visible in the camera's field of view, and It is assumed that the four points are selected to be easily recognizable by conventional image recognition techniques so that CPU 4 can identify those points from the image data to be used to calculate the transform).

The information determined by the CPU 4 for display to the user in step S316 (FIG. 14) may be provided in an embodiment where one camera is used to generate the input video image data. In such an embodiment, processing operations (such as shading processing in step S8 and complex object processing in step S10) that require image data from two or more cameras are not executed, and the third embodiment described above. Similarly, the foreground object can be modeled (step S11a).

Similarly, in the above embodiment, step S298 is performed to display the aerial representation of the object position when the viewing direction selected by the user is close to the vertical line.
The processing operation executed by the CPU 4 in S300 (FIG. 14) may be executed in an embodiment in which input image data is generated using one camera.

In the above embodiment, the CPU 4 and the memory 6
Forms part of a computer separate from the display device 18 and the cameras 12a and 12b. However, by providing an appropriate processor and memory in the camera or the display device, the processing described in the above embodiment can be executed in one or more cameras or display devices.

In the first embodiment, step S11 is performed.
In a / S11b (FIG. 3), the CPU 4 applies straight lines to the ground "footprint" of the object and defines a flat vertical plane having those straight lines as a base, thereby using the object in 3D world space. Specify the model to be used. In addition, the CPU 4 may execute a process for defining the uppermost part of the object by combining the upper sides of the flat vertical surface. Further, the CPU 4 may use the ground surface “footprint” to define the 3D model in different ways. For example, CP
U4 may use a spline curve (such as a Bezier curve) to represent the ground "footprint", representing the object as a vertical surface in 3D world space with a horizontal cross section corresponding to the defined spline curve. May be.

In the second embodiment, in step S254 (FIG. 17), the CPU 4 determines the 3D image of the camera for each plane determined to be in contact with the ground in the image space of the camera.
It defines a plane in world space. However, in order to be able to define a model of an object in 3D world space (since the position of one plane in 3D world space fixes the position of all planes in the model), only the object that touches the ground in image space. It is sufficient to define a plane in 3D world space with respect to one plane.

In the second embodiment, step S26 is performed.
In FIG. 4 (FIG. 18), the CPU 4 identifies and compares the vertices in the image data. Additionally or alternatively, the minimum, maximum, or saddle point of the color or intensity values of the image data may be identified and compared. For example, Haralick and Shapiro's "Computer and Robot Vision Volume 1"
(Addison-Wesley Publishing Company, ISBN 0-210-108
77-1 (v.1)), a technique for detecting points as described in Chapter 8 may be employed. As described above, the detected points may be collated by the adaptive least squares correlation.

In the above embodiment, step S2
In FIG. 4 (FIG. 4), for the unknown camera position and camera parameters, a transformation that maps the ground plane from image space to world space is calculated. However, it is also possible to arrange one or more cameras in a known relationship to the scene to be observed, with known camera parameters, so that different transformations can be used.

In the above embodiment, in steps S296 to S300 (FIG. 14), the direction of the field of view selected by the user is determined, and if this direction is within a range of a predetermined angle from the vertical line, the outline of the position of the object is determined. Execute processing for displaying an image. Additionally or alternatively, determine whether the viewing direction is parallel to or within a predetermined angle with respect to the flat vertical plane that makes up the model of the object, and so on. If so, a process may be executed to render image data to display a schematic image of the position of the object by some method as in step S300. This is particularly advantageous when modeling objects using one flat vertical plane, as in the third embodiment.
This is because, in this case, the object itself does not appear in the true image when the flat surface is in the viewing direction such that the edge is on, and therefore, it is considered that the schematic view of the object position becomes more useful to the user. It is. By determining whether the view direction is within a predetermined angle range with respect to the flat surface of the object, it is possible to identify a case where the object is to be observed with the top edge on, and therefore, in the previous step S298, It goes without saying that this test can be used in place of the test for determining the direction of the view for a given fixed angle range with respect to the vertical direction.

[Brief description of the drawings]

FIG. 1 is a diagram schematically showing components of an embodiment.

FIG. 2 schematically illustrates collection of video data from a dynamic environment in an embodiment.

FIG. 3A illustrates, at a top level, processing operations performed in embodiments to process signals defining a moving image, generate a three-dimensional computer model, and display an image to a user from a desired viewing direction. FIG.

3A and 3B illustrate, at a top level, processing operations performed in embodiments to process signals defining a moving image, generate a three-dimensional computer model, and display an image to a user from a desired viewing direction. FIG.

FIG. 4 is a diagram showing a processing operation executed in step S3a or step S3b shown in FIG.

FIG. 5 is a diagram showing a processing operation executed in step S6a or step S6b shown in FIG.

FIG. 6 is a diagram showing a processing operation executed in step S32 shown in FIG. 5;

7 is a diagram showing a processing operation executed in step S40 shown in FIG. 5 (and steps S106 and S112 shown in FIG. 8).

FIG. 8 is a diagram showing a processing operation executed in step S8 shown in FIG. 3;

FIG. 9 is a diagram schematically illustrating an example of an arrangement of an object and a camera and an image recorded by each camera in the illustrated arrangement.

FIG. 10 is a diagram showing a processing operation executed in step S10 shown in FIG. 3;

11 is a diagram schematically illustrating a circumscribed rectangle in a 3D world space generated with respect to the arrangement of the object and the camera illustrated in FIG. 9;

FIG. 12 shows a step S11a or a step S shown in FIG.
FIG. 11b is a diagram showing a processing operation executed in the first embodiment at 11b.

FIG. 13 shows steps S230 and S232 shown in FIG.
FIG. 3 schematically illustrates the formation of a model of an object according to FIG.

FIG. 14A is a diagram showing a processing operation executed in step S16 shown in FIG. 3;

FIG. 14B is a diagram showing a processing operation executed in step S16 shown in FIG. 3;

FIG. 15A is a diagram showing a processing operation executed in step S302 shown in FIG. 14;

FIG. 15B is a diagram showing a processing operation executed in step S302 shown in FIG. 14;

FIG. 16 is a diagram showing a processing operation executed in step S304 or step S308 shown in FIG.

FIG. 17 shows step S11a or step S11 shown in FIG.
FIG. 11b is a diagram showing a processing operation executed in the second embodiment in 11b.

FIG. 18 is a diagram showing a processing operation executed in the second embodiment in step S250 shown in FIG.

FIG. 19 is a diagram showing a processing operation executed in the second embodiment in step S264 shown in FIG. 18;

[Explanation of symbols]

 2 Computer 4 Central processing unit (CPU) 6 Memory 12a, 12b Camera 14 User command input device 16 Frame buffer 18 Display device 20 Video tape recorder 22 Mass storage device

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme court ゛ (Reference) H04N 13/02 G06F 15/72 450A (31) Priority claim number 98076610 (32) Priority date January 1998 14th (1998.1.14) (33) Priority claim country United Kingdom (GB) (31) Priority claim number 98800677 (32) Priority date January 14, 1998 (1998.1.14) (33) Priority country United Kingdom (GB) (31) Priority claim number 9807651 (32) Priority date January 14, 1998 (Jan. 14, 1998) (33) Priority claim country United Kingdom (GB) (31) Priority Claim number 98076644 (32) Priority date January 14, 1998 (Jan. 14, 1998) (33) Priority claiming country United Kingdom (GB) (31) Priority claim number 98076628 (32) Priority date 1998 January 14 (1 998.1.14) (33) Priority Country United Kingdom (GB) (72) Inventor Alan Divison BIG2 GU2 5 YJ Surrey, Guildford, Surrey Research Park, Occam Road, Occam Court 1 Canon Research Center Within Europe Limited

Claims (236)

[Claims] 1. A method for processing image data defining a sequence of a plurality of images obtained from respective cameras of a plurality of objects moving in a scene to define a representation of the object in a three-dimensional computer model. An image processing method for processing image data from a first camera to identify image data for an object in a scene; and processing image data from a second camera to Identifying image data for the objects; processing the identified image data from the first camera for each object; and in a modeling space having a height corresponding to the image data of the objects from the first camera. Defining an object representation, and identifying each object from a second camera Processing the captured image data to define an object representation in a modeling space having a height corresponding to the image data of the object from the second camera; and each object generated according to the image data from the first camera. Comparing the height of the representation of the object with the height of the representation of the corresponding object generated according to the image data from the second camera; and generating the object representation in the three-dimensional computer model according to the height comparison. An image processing method characterized by the following. 2. A modeling space in which an object representation is defined using the image data from the first camera and the image data from the second camera is a three-dimensional computer model. 2. The image processing method according to 1. 3. The method according to claim 2, wherein the step of generating the object representation in the three-dimensional computer model modifies the higher representation if the height of the corresponding representation is not within a predetermined amount of each other. The image processing method described in the above. 4. If the height of the corresponding expression is not within a predetermined amount of each other, the higher expression is modified by obtaining an expression having a height based on the height of the lower expression. The image processing method according to claim 3. 5. If the height of the corresponding representation is not within a predetermined amount of each other, define another representation in the three-dimensional model using part of the image data from which the higher representation is defined. The image processing method according to claim 3, wherein the image processing is performed. 6. The first part having a height corresponding to the height of the lower representation and the remainder of the higher representation if the height of the corresponding representation is not within a predetermined amount of each other. 6. The image processing method according to claim 5, wherein the image is divided into a second part having a second part and another expression is defined by repositioning the second part in the three-dimensional model. 7. The image processing method according to claim 6, wherein the second portion is repositioned according to a defined expression based on image data from a camera that has generated a lower expression. 8. The image used to define a higher representation in the image space of the camera that generated the higher representation, wherein any of the defined representations based on image data from the camera that generated the lower representation. 8. The image processing method according to claim 7, wherein the second part is repositioned by identifying whether the second part overlaps with the data and repositioning the second part according to the position of the identified representation in the three-dimensional model. Method. 9. A method of mapping at least a portion of each of the defined expressions based on image data from the camera that generated the lower expression into the image space of the camera that generated the higher expression from the three-dimensional model. Determine which representations overlap image data related to higher representations in the image space of the camera that generated the higher representations, and projected the higher representations to the image space of the camera that generated the higher representations, and 9. The image processing method according to claim 8, wherein the second part is repositioned by repositioning the second part according to the position in the three-dimensional model of the representation overlapping with the data. 10. The repositioning of the second portion so that the center of the base of the second portion is at the same position as the center of the base of the expression overlapping the image data related to the higher expression. The image processing method according to claim 9. 11. Each object representation is defined as a flat surface having a base located on a predetermined surface of the modeling space and having a position and a size determined according to a polygon circumscribing image data related to the object. The image processing method according to claim 1. 12. The image processing method according to claim 11, wherein the polygon is a rectangle. 13. The image processing method according to claim 12, wherein sides of the rectangle are parallel to sides of the image. 14. The flat surface width is determined by the width of a circumscribed polygon in the image data, and the height of the flat surface is calculated using the aspect ratio of the circumscribed polygon in the image data. An image processing method according to any one of claims 11 to 13. 15. The image processing method according to claim 11, wherein the flat surface is in a vertical plane. 16. The method of claim 1, further comprising the step of generating image data by rendering an image of a three-dimensional computer model in which texture data based on the processed image data is rendered into a representation of each object. The image processing method according to claim 1. 17. The image processing method according to claim 16, further comprising a step of generating a signal for carrying image data. 18. The image processing method according to claim 17, further comprising the step of recording a signal. 19. The image processing method according to claim 16, further comprising the step of displaying an image of the object using the generated image data. 20. The method according to claim 1, further comprising the step of directly or indirectly creating a record of the image data.
The image processing method according to any one of claims 6 to 19. 21. Process image data from the first and second cameras to identify image data for each object, and use the identified image data to determine the height of each object in a modeling space. And the first
The height of the object determined using the image data from the second camera is compared with the height of the object determined using the image data from the second camera. An image processing method characterized by determining if there is image data related to the above object. 22. Processing the image data of the objects in the scene from the first camera to identify image data for each object and processing the image data of the objects in the scene from the second camera. By comparing the size parameter of each object determined from the image data of the first camera with the corresponding size parameter determined from the image data of the second camera,
If any of the identified image data from the first camera is 1
An image processing method characterized in that it is determined whether or not the object is related to the object. 23. A signal for processing image data defining a sequence of a plurality of images obtained from respective cameras of a plurality of objects moving in a scene to define a representation of the object in a three-dimensional computer model. Means for processing image data from the first camera to identify image data relating to objects in the scene; and processing image data from the second camera to generate Means for identifying image data for the object, processing the identified image data from the first camera for each object, and in a modeling space having a height corresponding to the image data of the object from the first camera. Means for defining the object representation, and knowledge from the second camera for each object. Means for processing the generated image data and defining an object representation in a modeling space having a height corresponding to the image data of the object from the second camera; and each object generated according to the image data from the first camera. Means for comparing the height of the representation of the object with the height of the representation of the corresponding object generated according to the image data from the second camera; and means for generating the object representation in the three-dimensional computer model according to the height comparison An image processing apparatus characterized by the above-mentioned. 24. The modeling space in which an object representation is defined using the image data from the first camera and the image data from the second camera is a three-dimensional computer model. 23. The image processing device according to 23. 25. The means for generating an object representation in a three-dimensional computer model is configured to modify a higher representation if the height of the corresponding representation is not within a predetermined amount of each other. The image processing apparatus according to claim 24, wherein 26. If the heights of the corresponding expressions are not within a predetermined amount of each other, processing is performed to modify the higher expressions by obtaining an expression having a height based on the lower expression height. 26. The system of claim 25, wherein
The image processing apparatus according to any one of the preceding claims. 27. If the heights of the corresponding representations are not within a predetermined amount of each other, define another representation in the three-dimensional model using a part of the image data from which the higher representation is defined. 27. The image processing apparatus according to claim 25, wherein the image processing apparatus is configured to execute a process to perform the processing. 28. A first portion having a height corresponding to a lower representation height and a remainder of a higher representation if the corresponding representation heights are not within a predetermined amount of each other. And processing is performed to define another representation by repositioning the second part in the three-dimensional model with a second part having a second part. 28. The image processing device according to 27. 29. The apparatus of claim 28, wherein the processing is performed to reposition the second portion according to a representation defined based on image data from the camera that generated the lower representation. The image processing apparatus according to any one of the preceding claims. 30. An image used to define a higher representation in the image space of the camera that generated the higher representation, wherein any of the defined representations based on image data from the camera that generated the lower representation. 30. The image processing of claim 29, further comprising: repositioning the second portion by identifying whether it overlaps the data and repositioning the second portion according to the position of the identified representation in the three-dimensional model. apparatus. 31. Mapping at least a portion of each of the defined expressions based on image data from the camera that generated the lower expression into the image space of the camera that generated the higher expression from the three-dimensional model. Determine which representations overlap image data related to higher representations in the image space of the camera that generated the higher representations, and when projected into the image space of the camera that generated the higher representations, 31. The image processing apparatus according to claim 30, wherein the second part is repositioned by repositioning the second part according to the position in the three-dimensional model of the representation overlapping with the image data. 32. The method according to claim 30, wherein the processing is performed such that the center of the base of the second portion is at the same position as the center of the base of the expression overlapping with the image data relating to the higher expression. Item 32. The image processing apparatus according to Item 31, 33. Each object representation is defined as a flat surface having a base on a predetermined surface in the modeling space, and having a position and a size determined according to a polygon circumscribing image data related to the object. 3. The processing according to claim 2, wherein
The image processing device according to any one of claims 3 to 32. 34. The image processing apparatus according to claim 33, wherein the polygon is a rectangle. 35. The image processing apparatus according to claim 34, wherein sides of the polygon are parallel to sides of the image. 36. A process wherein the width of the flat surface is determined by the width of the circumscribed polygon in the image data, and the height of the flat surface is calculated using the aspect ratio of the circumscribed polygon in the image data. The image processing apparatus according to any one of claims 33 to 35, wherein the image processing apparatus is configured to execute the processing. 37. The image processing apparatus according to claim 33, wherein the processing is performed so that the flat surface is on a vertical plane. 38. The image processing apparatus further comprises means for generating image data by rendering an image of a three-dimensional computer model in which texture data based on the processed image data is rendered into a representation of each object. An image processing apparatus according to any one of claims 23 to 37. 39. The image processing apparatus according to claim 38, further comprising means for displaying an image of the object using the generated image data. 40. Process image data from the first and second cameras to identify image data for each object, and use the identified image data to determine a height of each object in a modeling space. And the first
The height of the object determined using the image data from the second camera is compared with the height of the object determined using the image data from the second camera. An image processing apparatus operable to determine if there is image data on the above object. 41. Process image data of an object in a scene from a first camera to identify image data for each object and process image data of an object in a scene from a second camera. By comparing the size parameter of each object determined from the image data of the first camera with the corresponding size parameter determined from the image data of the second camera,
If any of the identified image data from the first camera is 1
An image processing apparatus operable to determine whether or not the object is related to the object. 42. A programmable processing device comprising:
23. A storage medium for storing instructions for executing the method according to claim 22. 43. A programmable processing device comprising:
A signal carrying instructions for performing a method according to any of claims 22 to 23. 44. Processing image data defining a sequence of a plurality of images from respective cameras of an object moving in a scene to generate a signal defining a representation of the object in a three-dimensional computer model. An image processing method comprising: processing image data from a first camera to identify image data for an object in a scene; and processing image data from a second camera to generate an image for an object in a scene. Identifying the data; applying a transform to the identified image data from the first camera, the transform defining a mapping from a ground plane in the space of the image data of the first camera to a surface in the modeling space; Defines the mapping from the ground plane to the surface of the modeling space in the space of the image data of the second camera Applying the transformed image data to the identified image data from the second camera; comparing the transformed image data from the first and second cameras on a plane of the modeling space; An image processing method comprising: determining which part of image data represents a shadow; and generating at least an object representation using a three-dimensional model. 45. The image processing method according to claim 44, further comprising the step of generating a shade expression using a three-dimensional model. 46. The image processing method according to claim 44, wherein the surface of the modeling space is a ground plane in a three-dimensional model. 47. A method according to claim 44, wherein it is determined that the aligned part of the converted image data represents a shadow.
47. The image processing method according to claim 46. 48. The method of claim 48, further comprising the step of generating image data by rendering an image of a three-dimensional computer model in which texture data based on the processed image data is rendered into a representation of the object. 48. The image processing method according to claim 44. 49. The method according to claim 48, wherein the image data rendered into the representation is determined according to the result of the comparison. 50. The method according to claim 48, further comprising the step of generating a signal for carrying the image data.
9. The image processing method according to 9. 51. The image processing method according to claim 50, further comprising a step of recording a signal. 52. The image processing method according to claim 48, further comprising a step of displaying an image of the object using the generated image data. 53. The method according to claim 4, further comprising a step of directly or indirectly creating a record of the image data.
The image processing method according to any one of claims 8 to 52. 54. An image processing method for generating a model of an object by a three-dimensional computer model by processing images of the object from a plurality of cameras, wherein the image data from the first camera is processed to Identify the image data for the shadow together,
Using the image data from the second camera, the identified image data for the shadow from the first camera and the first
An image processing method for determining object image data from an object camera. 55. An image processing method for generating a model of an object using a three-dimensional computer model, the method comprising: mapping one image data of an object and its shadow onto a plane; Applying a transformation to the data and mapping one image data of the object and its shadow to a surface to image data from the second camera for the object and its shadow, and transforming the object according to a portion of the transformed image data. An image processing method characterized by modeling. 56. Process image data defining a sequence of a plurality of images from respective cameras of an object moving in a scene to generate signals defining a representation of the object in a three-dimensional computer model. An image processing apparatus for processing image data from a first camera to identify image data relating to an object in a scene; and processing image data from a second camera to generate an image relating to an object in a scene. Means for identifying data; means for applying a transform defining a mapping from a ground plane to a surface in modeling space in the space of image data of the first camera to the identified image data from the first camera; Defines the mapping from the ground plane to the surface of the modeling space in the space of the image data of the second camera Means for applying a transformation to the identified image data from the second camera, means for comparing the transformed image data from the first and second cameras on a plane of the modeling space, and according to the result of the comparison. An image processing apparatus comprising: means for determining which part of image data represents a shadow; and means for generating at least an object representation using a three-dimensional model. 57. The image processing apparatus according to claim 56, further comprising means for generating a shade expression using a three-dimensional model. 58. The image processing apparatus according to claim 56, wherein the surface of the modeling space is a ground plane in a three-dimensional model. 59. The image processing apparatus according to claim 56, wherein it is configured to determine that an aligned portion of the converted image data represents a shadow. 60. The apparatus of claim 60 further comprising means for generating image data by rendering an image of a three-dimensional computer model in which texture data based on the processed image data is rendered into an object representation. The image processing device according to any one of claims 56 to 59. 61. The apparatus according to claim 60, wherein the image data rendered into the representation is determined according to the result of the comparison. 62. The image processing apparatus according to claim 60, further comprising means for displaying an image of the object using the generated image data. 63. An image processing apparatus for generating a model of an object by a three-dimensional computer model by processing images of the object from a plurality of cameras, wherein the image data from the first camera is processed to Operable to identify image data for the shadow together, and using the image data from the second camera to identify the image data associated with the shadow from the first camera; An image processing apparatus operable to determine identified image data from the camera of the object. 64. An image processing apparatus for generating a model of an object by using a three-dimensional computer model, wherein a transformation for mapping one image data of the object and its shadow to a surface is performed by using an image from the first camera regarding the object and its shadow. Means for applying a transformation mapping one image data of the object and its shading to a surface to image data from the second camera relating to the object and its shading; Means for modeling an object according to a section. 65. The programmable processing device according to claim 44.
A storage medium storing instructions for executing the method according to any one of claims 55 to 55. 66. The programmable processing device according to claim 44.
56. A signal carrying instructions for performing the method of any one of claims 55. 67. Processing image data defining a sequence of multiple images from respective cameras of an object moving in a scene to generate a signal defining a representation of the object in a three-dimensional computer model. An image processing method comprising: processing image data from a first camera to identify image data for an object in a scene; and processing image data from a second camera to generate an image for an object in a scene. Identifying the data; processing the identified image data from the first camera and the identified image data from the second camera to determine a footprint of the object on the surface of the earth. Defining the model of the object in the three-dimensional computer model according to the footprint An image processing method characterized in that it comprises a. 68. Processing the identified image data to determine a footprint of the object on the ground surface comprises transforming a ground plane to a modeling space surface in the first camera image data. Applying the identified image data from the first camera to the identified image data from the second camera; and transforming the image data of the second camera defining a mapping from a ground plane to a surface in modeling space. 68. The image processing method according to claim 67, further comprising: a step of applying to the data; and a step of comparing the image data converted on a plane of the modeling space. 69. The image processing method according to claim 68, wherein the surface of the modeling space is a ground plane in a three-dimensional computer model. 70. The image processing method according to claim 68, wherein the contour of the image on the ground is determined according to the aligned part and the non-aligned part of the converted image data in the plane of the modeling space. 71. The image processing method according to claim 67, wherein the step of defining a model of the object uses a plurality of flat vertical planes to define the model. 72. The image processing method according to claim 71, wherein the flat vertical surface is defined such that its base substantially matches the contour of the object on the ground surface. 73. The image processing method according to claim 71, wherein each flat surface is rectangular. 74. Each of the flat surfaces is defined by a height determined according to image data identified from the first camera or image data identified from the second camera. 74. The image processing method according to claim 71. 75. The height of each flat surface is defined according to a rectangle circumscribing some or all of the image data associated with the object identified from the first camera or the second camera. The image processing method according to claim 74, wherein 76. The image processing method according to claim 74, wherein each flat surface is defined to have the same height. 77. The image processing method according to claim 71, further comprising a step of generating an uppermost part of the model of the object according to an upper edge of the vertical flat surface. . 78. The method further comprises generating image data by rendering an image of a modeled object, wherein texture data based on the identified image data from at least one camera is rendered into a model. Claims 67-77.
The image processing method according to any one of the above. 79. Each flat surface is mapped to image data of a first camera or a second camera, and the image data encompassed by each mapped surface is rendered to a flat surface in the model. 79. The image processing method according to claim 78, wherein: 80. The method according to claim 78, further comprising the step of generating a signal for carrying the image data.
9. The image processing method according to 9. 81. The image processing method according to claim 80, further comprising a step of recording a signal. 82. The image processing method according to claim 78, further comprising the step of displaying an image of the object using the generated image data. 83. The method according to claim 7, further comprising the step of directly or indirectly creating a record of the image data.
The image processing method according to any one of claims 8 to 82. 84. An image processing method for generating a model of an object by a three-dimensional computer model by processing images of the object from a plurality of cameras, wherein the image data from the first camera is processed and Identify the image data and use the image data from the second camera to determine which portion of the identified image data from the first camera relates to a portion of the object on or near the ground. And an image processing method for expressing an object in a computer model according to the determination. 85. Processing image data defining a sequence of multiple images from respective cameras of an object moving in a scene to generate a signal defining a representation of the object in a three-dimensional computer model. An image processing apparatus for processing image data from a first camera to identify image data relating to an object in a scene; and processing image data from a second camera to generate an image relating to an object in a scene. Means for identifying the data; and means for processing the identified image data from the first camera and the identified image data from the second camera to determine a footprint of the object on the ground. Of defining the model of an object in a three-dimensional computer model according to the footprint The image processing apparatus characterized by having and. 86. The means for processing the identified image data to determine a footprint of the object on the ground comprises: a transformation defining a mapping from a ground plane to a surface of the modeling space in the image data of the first camera. Means for applying the mapping to the identified image data from the first camera; and a transform defining a mapping from a ground plane to a surface in modeling space in the image data of the second camera. 86. The image processing apparatus according to claim 85, further comprising: means for applying to image data; and means for comparing image data converted on a plane of a modeling space. 87. The image processing apparatus according to claim 86, wherein the surface of the modeling space is a ground plane in a three-dimensional computer model. 88. An image according to claim 86 or 87, characterized in that the contour of the image on the ground is determined according to the aligned and non-aligned parts of the transformed image data in the plane of the modeling space. Processing equipment. 89. The apparatus of claim 85, wherein the means for defining a model of the object comprises means for defining the model using a plurality of flat vertical planes.
The image processing device according to any one of the above. 90. The method of claim 89, wherein the processing is performed to define a flat vertical surface such that a base of the flat vertical surface substantially matches the contour of the object on the ground. Image processing device. 91. An image processing apparatus according to claim 89, wherein each flat surface is rectangular. 92. An arrangement configured to perform processing such that each flat surface is defined by a height determined according to image data identified from the first camera or image data identified from the second camera. The image processing apparatus according to any one of claims 89 to 91, wherein the processing is performed. 93. Processing is performed such that the height of each flat surface is defined according to a rectangle circumscribing some or all of the image data associated with the object identified from the first camera or the second camera. The image processing apparatus according to claim 92, configured to execute. 94. An image processing apparatus according to claim 92, wherein each flat surface is defined to have the same height. 95. The image processing apparatus according to claim 89, further comprising means for generating an uppermost part of the model of the object according to an upper edge of the flat vertical plane. 96. The apparatus further comprises means for generating image data by rendering an image of a modeled object in which texture data based on the image data from the identified at least one camera is rendered into a model. Claims 85 to 95
The image processing device according to any one of the above. 97. Each flat surface is mapped to image data of a first camera or a second camera, and the image data surrounded by each mapped surface is rendered to a flat surface in a model. 97. The image processing apparatus according to claim 96, wherein the image processing apparatus is configured to execute a process so as to be performed. 98. The image processing apparatus according to claim 96, further comprising means for displaying an image of the object using the generated image data. 99. An image processing apparatus for generating a model of an object by a three-dimensional computer model by processing images of the object from a plurality of cameras. Identifying the image data, using the image data from the second camera to determine which portion of the identified image data from the first camera relates, and representing the object in a computer model according to the determination. An image processing apparatus operable to perform 100. The programmable processing device according to claim 6,
A storage medium storing instructions for executing the method according to any one of claims 7 to 84. 101. A programmable processing device comprising:
A signal carrying instructions for performing a method according to any of claims 7 to 84. 102. Processing image data defining a sequence of images of a plurality of objects moving in a scene to generate a signal defining a representation of the object in a three-dimensional computer model, and selecting a signal selected by a user. An image processing method for generating image data by rendering an image of a three-dimensional computer model according to a viewing direction, wherein the image data is processed to identify image data for each object in a scene. Defining the representation of each object in the three-dimensional computer model according to the image data, and generating image data by rendering an image of the three-dimensional computer model according to the viewing direction selected by the user, The viewing direction selected by the user is the predetermined viewing direction If within the range, render the texture data based on the identified image data into an object representation; if the viewing direction selected by the user is not within the predetermined viewing direction range, the position of the object in the scene An image processing method characterized by rendering an outline of the image processing. 103. The object of claim 1, wherein the representation of each object comprises only a plurality of flat vertical surfaces.
02. The image processing method described in 02. 104. The representation of each object is comprised of one flat vertical plane.
The image processing method described in the above. 105. The image processing method according to claim 102, wherein the predetermined viewing direction range is a range for a fixed direction in the computer model. 106. The image processing method according to claim 105, wherein the predetermined viewing direction range is a range in a vertical direction in the computer model. 107. The predetermined range in the viewing direction is a range for expressing an object.
5. The image processing method according to any one of 4. 108. The representation of the object is at least one
2. The method according to claim 1, wherein the predetermined vertical direction range is a range with respect to the flat surface.
07. The image processing method according to item 07. The image processing method according to any one of claims 102 to 108, wherein an approximate position of the object is rendered from a predetermined viewing direction. 110. The image processing method according to claim 109, wherein an outline is rendered from a vertically downward viewing direction. 111. Processing the image data to determine at least one color for each object;
120. The image processing method according to claim 102, further comprising the step of: generating image data for indicating the determined color on the outline of the position of the object. 112. The image processing method according to claim 102, further comprising a step of generating a signal for carrying image data. 113. The image processing method according to claim 112, further comprising the step of recording a signal. 114. The image processing method according to claim 102, further comprising the step of displaying an image of an object using the generated image data. 115. The image processing method according to claim 102, further comprising the step of directly or indirectly creating a record of image data. 116. A three-dimensional computer model comprising a representation relating to an object and associated texture data extracted from image data recorded by at least one camera, according to a viewing direction selected by a user. An image processing method for rendering the selected image, wherein when the view direction selected by the user is within a predetermined view direction range, rendering texture data to an expression of an object according to the view direction selected by the user; Rendering the outline of the position of the object when the selected viewing direction is not within the predetermined viewing direction range. 117. An image processing method for processing object data defining a three-dimensional computer model of a plurality of objects in a scene to generate image data of an image of the object and the scene, wherein the image processing method is responsive to a first user input. And rendering the object and the scene according to the selected viewing direction, and rendering the image data with respect to an image representing the position of the object in the scene according to a second user input. Method. 118. Processing image data defining a sequence of images of a plurality of objects moving in a scene to generate a signal defining a representation of the object in a three-dimensional computer model, and selecting a signal selected by a user. An image processing apparatus for generating image data by rendering an image of a three-dimensional computer model according to a direction of view, wherein the image processing unit processes the image data to identify image data relating to each object in the scene. Means for defining the representation of each object in the three-dimensional computer model according to the image data, and means for generating image data by rendering an image of the three-dimensional computer model according to the viewing direction selected by the user, The viewing direction selected by the user is the predetermined viewing direction When within the range, the texture data based on the identified image data is rendered into an object representation, and if the user selected viewing direction is not within the predetermined viewing direction range, the position of the object in the scene is determined. An image processing device operable to render an overview. 119. The image processing apparatus according to claim 118, wherein the image processing apparatus is operable to execute a process such that a representation of each object is composed of only a plurality of flat vertical planes. 120. The apparatus according to claim 118, operable to perform processing such that a representation of each object is comprised of one flat vertical plane. 121. An apparatus according to any of claims 118 to 120, operable to execute a process such that the predetermined viewing direction range is a range for a fixed direction in the computer model. Image processing device. 122. The apparatus of claim 1 operable to perform the processing such that the predetermined range of the viewing direction is a range for a vertical direction in the computer model.
22. The image processing device according to 21. 123. An apparatus according to claims 118 to 120, operable to execute processing such that the predetermined viewing direction is a range for the representation of the object.
The image processing device according to any one of the above. 124. The representation of the object is at least one
124. The image processing apparatus according to claim 123, wherein the image processing apparatus is constituted by two flat vertical planes, and is operable to execute the processing such that the predetermined viewing direction range is a range for the flat plane. 125. The image processing apparatus according to claim 118, wherein the image processing apparatus is operable to execute a process so that an outline of the object position is rendered from a predetermined viewing direction. . 126. An image processing apparatus according to claim 125, operable to perform processing such that the outline is rendered from a vertically downward viewing direction. 127. A means for processing the image data to determine at least one color for each object;
The image processing apparatus according to any one of claims 118 to 126, further comprising: means for generating image data for indicating the determined color on the outline of the object position. The image processing apparatus according to any one of claims 118 to 127, further comprising means for displaying an image of the object using the generated image data. 129. A three-dimensional computer model comprising a representation relating to an object and associated texture data extracted from image data recorded by at least one camera, according to a viewing direction selected by a user. Means for rendering texture data to an object representation in accordance with the viewing direction selected by the user when the viewing direction selected by the user is within a predetermined viewing direction range. Means for rendering an outline of the position of the object when the selected viewing direction is not within a predetermined viewing direction range. 130. Operate to process object data defining a three-dimensional computer model of a plurality of objects in a scene to generate image data for an image of the object using first and second techniques. In a possible image processing device, a first technique renders objects and scenes according to certain viewing directions, and a second technique renders image data with respect to a schematic image in which the positions of the objects in the scene are represented. An image processing apparatus comprising: 131. The programmable processing device according to claim 1,
A storage medium storing instructions for executing the method according to any one of claims 02 to 117. 132. A programmable processing device comprising:
118. A signal carrying instructions for performing a method according to any of claims 02 to 117. 133. Processing image data defining a sequence of a plurality of images obtained from respective cameras of an object moving in a scene to generate a signal defining a representation of the object in a three-dimensional computer model. An image processing method for processing image data from a first camera to identify image data for an object in a scene; and processing image data from a second camera for processing an image data for an object in a scene. Identifying the image data; and processing the identified image data from the first camera and the identified image data from the second camera to obtain feature points of the identified image data from the first camera. Is compared with the feature point of the image data from the identified second camera, and the flat surface on which the matched feature point is located Identifying a flat surface on which a point on the object is located, and defining a model of the object with a three-dimensional computer model according to the identified flat surface. Image processing method. 134. The image processing method according to claim 133, wherein the vertex of the image data from the identified first camera is compared with the vertex of the image data from the identified second camera. 135. A flat surface is identified by identifying the plane on which the matched feature points are located and determining the boundaries of the matched feature points located on one or more planes. 135. The image processing method according to claim 133 or 134. 136. Each plane identifies a plane on which at least a predetermined number of feature points of the identified image data from the first camera is located, and the image data from the identified second camera is identified. Identify the plane on which the matched feature points are located, calculate the transformation between the plane of the image data from the first camera and the plane of the image data from the second camera, and calculate other pairs of matched features. 136. The method of claim 135, wherein the points are identified by testing the transformation using points. 137. The image processing method according to claim 136, wherein the predetermined number is four. 138. The step of defining a model of the object comprises forming a model of a flat surface with a three-dimensional computer model, wherein each flat surface in the model comprises image data from at least one camera. The image processing method according to any one of claims 133 to 137, wherein the image processing method corresponds to a flat surface identified by: 139. The step of defining a model of the object includes identifying a flat surface in contact with the ground surface in the image data of the camera, and a flat vertical surface in the three-dimensional computer model according to the identified flat surface in contact with the ground surface. And defining another flat surface in the three-dimensional computer model for each other flat surface of the camera image data, such that the flat surface of the three-dimensional computer model and the image data have the same aspect ratio. 139. The image processing method according to claim 138, further comprising a step. 140. A plane that is in contact with the ground in the image data of the predetermined camera is converted from a ground plane to a surface of the modeling space in the image data of the first camera by a conversion from the first camera. A transformation that defines the mapping from the ground plane to the surface of the modeling space in the image data of the second camera by applying to the base vertex of the flat surface in the image data of the second camera. 139. The image processing method of claim 139, wherein the method is applied to a base vertex and is identified by comparing the transformed vertices to determine which vertices are within a predetermined distance of each other. . 141. The image processing method according to claim 140, wherein the surface of the modeling space is a ground plane in a three-dimensional computer model. 142. A flat vertical surface defined by the three-dimensional computer model is a converted vertex from a given camera located a predetermined distance from a corresponding converted vertex from the other camera. 142. The image processing method according to claim 141, wherein the image data is defined by an aspect ratio corresponding to an aspect ratio of a flat surface in image data of a predetermined camera to which a vertex after conversion belongs and a base defined by: . 143. The method further comprising generating image data by rendering an image of a modeled object, wherein texture data based on the image data from the identified at least one camera is rendered into a model. The image processing method according to any one of claims 133 to 142. 144. The image processing method according to claim 143, wherein image data surrounded by each flat surface is rendered on a corresponding flat surface of the object model. 145. The image processing method according to claim 143, further comprising the step of generating a signal for carrying image data. 146. The image processing method according to claim 145, further comprising the step of recording a signal. 147. The image processing method according to claim 143, further comprising the step of displaying an image of the object using the generated image data. 148. The image processing method according to claim 143, further comprising a step of directly or indirectly creating a record of image data. 149. An image processing method for generating a model of an object by a three-dimensional computer model by processing images of the object from a plurality of cameras, wherein image data from the first camera and the second camera are processed. Then, the feature points of the image data from the first camera are compared with the feature points of the image data from the second camera, and using the resulting match, the flat surface forming the object is identified. An image processing method comprising: judging and expressing an object by a computer model according to the judgment. 150. Processing image data defining a sequence of a plurality of images obtained from respective cameras of an object moving in a scene to generate a signal defining a representation of the object in a three-dimensional computer model. Means for processing image data from a first camera to identify image data relating to an object in a scene; and processing image data from a second camera to process image data relating to an object in a scene. Means for identifying image data; and processing of the identified image data from the first camera and the identified image data from the second camera to identify a flat surface on which a point on the object is located. Means for identifying feature points in image data from the identified first camera from the identified second camera. Means for matching feature points in the image data; means for identifying a flat surface on which the matched feature points are located; An image processing apparatus comprising: means for defining 151. An apparatus operable to perform processing such that a vertex of image data from the identified first camera is matched with a vertex of image data from the identified second camera. The image processing apparatus according to claim 150, wherein: 152. A flat surface is identified by identifying the plane on which the matched feature points are located and determining the boundaries of the planes using the matched feature points located on one or more planes. The image processing apparatus according to claim 150, wherein the image processing apparatus is operable to execute a process to perform the processing. 153. A plane on which at least a predetermined number of feature points are located in the image data from the identified first camera, and a plane in the image data from the identified second camera is identified. Identify the plane on which the matched feature points are located, calculate the transformation between the plane in the image data from the first camera and the plane in the image data from the second camera, and calculate the other matched pairs. The method of claim 1 operable to perform a process to identify each plane by testing the transformation using the feature points.
52. The image processing device according to 52. 154. The image processing apparatus according to claim 153, wherein the predetermined number is four. 155. The means for defining a model of the object comprises means for forming a model of a flat surface in a three-dimensional computer model, wherein each flat surface in the model comprises image data from at least one camera. The image processing apparatus according to any one of claims 150 to 154, wherein the image processing apparatus corresponds to a flat surface identified by: 156. The means for defining a model of the object comprises: means for identifying a flat surface in contact with the ground in camera image data; and means for flattening the three-dimensional computer model in accordance with the identified flat surface in contact with the ground. Means for defining a vertical plane and another flat surface in the three-dimensional computer model for each other flat surface in the camera image data such that the flat surface and the image data in the three-dimensional computer model have the same aspect ratio. 155. The image processing apparatus according to claim 155, further comprising: means for defining a surface. 157. A flat surface in contact with the ground surface in the image data of the predetermined camera is converted from a ground plane in the image data of the first camera to a surface in the modeling space by a conversion from the first camera. A transformation that defines the mapping from the ground plane to the surface in modeling space in the image data of the second camera by applying to the base vertex of the flat surface in the image data of Applying to the base vertex, operable to perform the processing as identified by comparing the transformed vertices and determining which vertices are within a predetermined distance of each other. 157. The image processing apparatus according to claim 156, wherein 158. The image processing device according to claim 157, wherein the surface in the modeling space is a ground plane in a three-dimensional computer model. 159. The flat vertical plane defined by the three-dimensional computer model is a transformed vertex from a given camera located within a given distance from a corresponding transformed vertex from the other camera. , And that the vertices are operable to perform processing as defined by the aspect ratio corresponding to the aspect ratio of the flat surface in the image data of the predetermined camera to which the transformed vertex belongs. 158. The image processing apparatus according to claim 158, wherein 160. The apparatus further comprises means for generating image data by rendering an image of a modeled object wherein texture data based on the image data from the at least one identified camera is rendered into a model. 160. The image processing device according to claim 150, wherein: 161. The apparatus of claim 160, operable to perform processing such that image data surrounded by each flat surface is rendered to a corresponding flat surface of the object model. Image processing device. 162. The image processing apparatus according to claim 160, further comprising means for displaying an image of the object using the generated image data. 163. An image processing apparatus for generating an object model by a three-dimensional computer model by processing images of an object from a plurality of cameras, wherein the image data from the first camera and the second camera is processed. Then, the feature points in the image data from the first camera are compared with the feature points in the image data from the second camera, and the resulting matching is used to determine a flat surface constituting the object. An image processing apparatus operable to represent an object by a computer model according to the determination. 164. A programmable processing device comprising:
150. A storage medium storing instructions for executing the method according to any one of claims 33 to 149. 165. The programmable processor according to claim 1,
150. A signal carrying instructions for performing a method according to any of claims 33 to 149. 166. Processing image data defining a sequence of images of at least one object moving in the scene to generate signals defining a representation of each object in the three-dimensional computer model, and An image processing method for generating image data by rendering an image of a three-dimensional computer model according to a selected viewing direction, wherein the image data is processed to identify image data for each object in a scene; Defining a representation of each object in the three-dimensional computer model according to the identified image data; and three-dimensionally in response to a viewing direction selected by a user such that texture data based on the identified image data is rendered into the object representation. Computer model images An image processing method comprising the steps of: generating image data by rendering; and generating image quality information of image data indicating image quality of image data determined according to a viewing direction selected by a user. . 167. The step of generating image quality information of image data generates information indicating reliability of image data according to an angle formed between a viewing direction selected by a user and a viewing direction when input image data is recorded. 166. The image processing method according to claim 166, comprising a step. 168. The information indicating reliability is generated according to a linear relationship between an image quality and an angle difference between a viewing direction selected by a user and a viewing direction when input image data is recorded. The image processing method according to claim 167. 169. The method according to claim 1, further comprising the step of generating information for instructing how to change the viewing direction in order to improve the reliability of the generated information.
67. The image processing method according to 67 or 168. 170. Image data relating to a sequence of images recorded by the first camera and a sequence of images recorded by the second camera is associated with each object in the scene by processing the image data. Identifying image data from a second camera associated with each object in the scene, and defining a representation of each object, as well as identifying image data from the first camera. Defining a first representation of each object according to image data from the second camera, and defining a second representation of each object according to image data from the identified second camera, generating image data; At least one identified
166. The image processing apparatus according to claim 166, wherein texture data based on image data from one camera is rendered into an object representation.
The image processing method according to any one of the above. 171. The image processing method according to claim 166, wherein in the step of defining the expression of each object, each object is expressed as a flat surface. 172. The image processing method according to claim 166, wherein the image quality information is generated as pixel data in the generated image data. 173. The image processing method according to any one of claims 166 to 172, further comprising the step of generating a signal carrying image data and image quality information. 174. The image processing method according to claim 173, further comprising the step of recording a signal. 175. The image processing according to any one of claims 166 to 174, further comprising a step of displaying an image using the generated image data and displaying image quality information. Method. 176. The image processing method according to claim 166, further comprising a step of directly or indirectly creating a record of image data and image quality information. 177. A three-dimensional computer model comprising a representation relating to at least one object and associated texture data derived from image data recorded by at least one camera.
In an image processing method for rendering an image according to a view direction selected by a user, by rendering an image of a three-dimensional computer model according to the view direction selected by the user, such that the texture data is rendered to a representation to be written. An image processing method, comprising: generating image data; and generating image quality information of image data indicating image quality of image data determined according to a viewing direction selected by a user. 178. Object data defining a three-dimensional computer model of at least one object in the scene is rendered according to a user-selected viewing direction using image data recorded by the camera. An image processing method, comprising: generating an indicator indicating the quality of generated image data for rendering an object and outputting the object to a user. 179. Processing image data defining a sequence of images of at least one object moving in the scene to generate signals defining a representation of each object in the three-dimensional computer model, and An image processing apparatus that generates image data by rendering an image of a three-dimensional computer model according to a selected viewing direction, a unit that processes the image data and identifies image data regarding each object in a scene. Means for defining a representation of each object in the three-dimensional computer model according to the identified image data; and, in accordance with a viewing direction selected by the user, such that texture data based on the identified image data is rendered into an object representation. 3D computer model drawing An image, comprising: means for generating image data by rendering an image; and means for generating image data image quality information indicating image quality of the image data determined according to a viewing direction selected by a user. Processing equipment. 180. A means for generating image quality information of an image,
179. The image processing apparatus according to claim 179, further comprising: means for generating information indicating the reliability of the image data according to an angle formed by the viewing direction selected by the user and the viewing direction when the input image data is recorded. . 181. A process is performed so that information indicating reliability is generated in accordance with a linear relationship between image quality and an angle difference between a viewing direction selected by a user and a viewing direction when input image data is recorded. 181. The image processing apparatus according to claim 180, wherein the apparatus is operable to operate. 182. The apparatus according to claim 1, further comprising means for generating information for instructing how to change the viewing direction in order to improve the generated reliability.
181. The image processing apparatus according to 80 or 181. 183. An apparatus operable to process image data associated with a sequence of images recorded by a first camera and a sequence of images recorded by a second camera, wherein the means for processing the image data comprises: Identifying image data from the first camera associated with each object in the scene, and identifying a second image associated with each object in the scene.
Operable to identify image data from the first camera, the means for defining a representation of each object, defining a first representation of each object according to the identified image data from the first camera, and Operable to define a second representation of each object according to image data from the identified second camera, the means for generating image data comprises:
179. The method of claim 179, further comprising operable to render texture data based on image data from one camera into an object representation.
The image processing device according to any one of the above. 184. The method according to claim 179, wherein the means for defining the representation of each object is configured to represent each object as a flat surface.
83. The image processing apparatus according to any one of 83s. 185. An apparatus according to any of claims 179 to 184, wherein the apparatus is operable to execute processing such that image quality information is generated as pixel data in the generated image data. Image processing device. 186. The image processing apparatus according to any one of claims 179 to 185, further comprising means for displaying an image using the generated image data and displaying image quality information. apparatus. 187. A three-dimensional computer model comprising a representation relating to at least one object and associated texture data derived from image data recorded by at least one camera.
In an image processing device that renders an image according to a viewing direction selected by a user, by rendering an image of a three-dimensional computer model according to a viewing direction selected by a user, such that texture data is rendered into each expression. An image processing apparatus, comprising: means for generating image data; and means for generating image quality information of image data indicating image quality of image data determined according to a viewing direction selected by a user. 188. The image data recorded by the camera is operable to render object data defining a three-dimensional computer model of at least one object in the scene according to a viewing direction selected by a user. And an operable to generate an image quality indicator of the generated image data for output to a user. 189. The programmable processing device according to claim 1
178. A storage medium storing instructions for performing the method of any of claims 66-178. 190. A programmable processing device comprising:
178. A signal carrying instructions for performing a method according to any of claims 66 to 178. 191. Image data defining a sequence of multiple images from respective cameras of at least one object moving in a scene to define a representation of each object in a three-dimensional computer model. An image processing method for generating image signals by generating a signal and rendering an image of a three-dimensional computer model according to a viewing direction selected by a user, comprising: processing input image data from at least one camera; Defining at least one representation of each object in the three-dimensional computer model; and a three-dimensional computer according to a viewing direction selected by the user such that texture data based on the input image data is rendered into a representation of each object. Render an image of the model Generating image data, wherein the representation of each object to be rendered is in accordance with a viewing direction selected by the user, a viewing direction of each camera, and at least one camera characteristic that affects image quality of the image data. An image processing method characterized by being determined. 192. The representation of each object to be rendered includes: a viewing direction selected by the user, a viewing direction of each camera, a method of transferring image data from each camera, a resolution of each camera, and a resolution of each camera. 192. The method of claim 191, wherein the determination is made according to at least one of a shutter speed, stability of image data from each camera, and whether image data from each camera is color or black and white. Image processing method. 193. The view direction selected by the user is input prior to the step of defining the object representation, and the representation of each object is determined by the view direction selected by the user, the view direction of each camera, and the image quality of the image data. 192. The image processing method according to claim 191 or 192, wherein the image processing method is defined using image data from one camera selected according to at least one camera characteristic to influence. 194. processing the image data from the first camera to define a first representation in the three-dimensional computer model, processing the image data from the second camera,
A second representation of each object in the three-dimensional computer model is defined and rendered according to a user-selected viewing direction, viewing directions of the first and second cameras, and at least one camera characteristic that affects image data image quality. 192. The image processing method according to claim 191 or 192, wherein the first expression or the second expression is selected to perform the operation. 195. An image according to any of claims 191 to 194, wherein a plurality of camera characteristics affecting image quality are taken into account to determine the representation of each object for rendering. Processing method. 196. Camera characteristics that affect image quality are considered in a predetermined order, and the values of each camera characteristic are compared, and the determination of an expression to be rendered differs by more than a predetermined amount for a predetermined camera. 195. The image processing method according to claim 195, wherein the method is executed when a certain characteristic is identified by a test. 197. The image processing method according to any one of claims 191 to 196, further comprising a step of generating a signal for carrying image data. 198. The image processing method according to claim 197, further comprising the step of recording a signal. 199. The image processing method according to any one of claims 191 to 198, further comprising the step of displaying an image of the object using the generated image data. 200. The image processing method according to claim 19, further comprising the step of directly or indirectly creating a record of image data. 201. Process image data from each sequence of images, each obtained from a different camera, to define a representation of at least one object in the three-dimensional computer model, and to select a user-selected viewing direction, An image processing method comprising: selecting a representation of each object for rendering according to at least one camera parameter related to a camera viewing direction and image quality of image data. 202. A plurality of images, each recorded by a corresponding camera, are defined using a user-selected viewing direction that is a direction in which to render an image of at least one object in the three-dimensional computer model. From among the image data, image data to be used to define an object in the three-dimensional computer model is selected, wherein the selection is at least one related to the viewing direction of each camera and the image quality of the image data.
An image processing method characterized by being executed according to a view direction selected by a user together with two camera parameters. 203. Process image data defining a sequence of a plurality of images obtained from respective cameras of at least one object moving in the scene to define a representation of each object in a three-dimensional computer model. An image processing apparatus for generating image data by generating a signal to be processed and rendering an image of a three-dimensional computer model according to a viewing direction selected by a user, wherein input image data from at least one camera is processed. Means for defining at least one representation of each object in the three-dimensional computer model; and tertiary according to the viewing direction selected by the user such that texture data based on the input image data is rendered into a representation of each object. Render image of original computer model Means for generating image data by performing image processing, wherein the representation of each object to be rendered influences a view direction selected by a user, a view direction of each camera, and an image quality of the image data. An image processing apparatus operable to execute processing as determined according to the following. 204. A representation of each object to be rendered is a view direction selected by a user, a view direction of each camera, a method of transferring image data from each camera, a resolution of each camera, each camera. Operable to perform a process such that the image data from each camera is determined in accordance with at least one of a shutter speed, image data stability of each camera, and color or black and white. 203. The image processing apparatus according to claim 203, wherein: 205. When the viewing direction selected by the user is input prior to defining the object representation, 1
One representation of each object is defined using the image data from the one camera, the one camera affecting the user-selected viewing direction, the respective viewing directions of the cameras, and the quality of the image data. 210. The image processing apparatus according to claim 203 or 204, operable to perform a process as selected according to at least one camera characteristic. 206. Process the image data from the first camera to define a first representation of each object in the three-dimensional computer model, process the image data from the second camera, A second representation of each object in the model is defined for rendering according to a user-selected viewing direction, viewing directions of the first and second cameras, and at least one camera characteristic that affects image quality of the image data. The image processing apparatus according to claim 203, wherein the image processing apparatus is operable to execute a process to select either the first expression or the second expression. 207. The apparatus of claim operable to perform processing such that a plurality of camera characteristics that affect image quality are taken into account to determine a representation of each object for rendering. 203 to claim 20
7. The image processing apparatus according to any one of 6. 208. Camera characteristics that affect image quality are considered in a predetermined order, and the values of each camera characteristic are compared, and the determination of an expression to be rendered differs by more than a predetermined amount for a predetermined camera. 207. The image processing apparatus of claim 207, operable to perform processing to be performed when a characteristic that is present is identified by a test. 209. The image processing apparatus according to claim 203, further comprising means for displaying an image of an object using the generated image data. 210. Processing image data from each sequence of images, each obtained from a different camera, to define a representation of at least one object in the three-dimensional computer model, and to select a user-selected viewing direction, An image processing apparatus operable to perform processing to select a representation of each object for rendering according to at least one camera parameter related to a camera viewing direction and image quality of image data. 211. A plurality of images of an object, each recorded by a single camera, using a user-selected viewing direction that is a direction in which to render an image of at least one object in a three-dimensional computer model. From the defining image data, select the image data to be used to define the object in the three-dimensional computer model, the selection comprising:
An image processing apparatus operable to be executed according to a view direction selected by a user, together with at least one camera parameter related to a view direction of each camera and image quality of image data. 212. A programmable processing device comprising:
A storage medium storing instructions for executing the method according to any one of claims 91 to 202. 213. The programmable processing device according to claim 1,
A signal carrying instructions for performing a method according to any of claims 91 to 202. 214. Processing image data defining a plurality of image sequences obtained from respective cameras of an object moving in a scene to generate a signal defining a representation of the object in a three-dimensional computer model. And generating image data relating to the first and second images in the sequence of images of the object by rendering the images of the three-dimensional computer model according to the viewing directions selected by the first and second users. Processing the image data to define at least one representation of the object in the three-dimensional computer model; and converting the image data from at least the first camera according to a first user-selected viewing direction. Rendering based texture data into object representations Generating image data for use in a first image in a sequence, and rendering texture data based on image data from a second camera into a representation of an object according to a second user-selected viewing direction. Generating image data for use in a second image in the sequence, wherein the image data of the object in the second image in the sequence differs from the predetermined image data by more than a predetermined amount. Testing whether the first image and the second image of the object displayed from the generated image data are discontinuous by testing whether the object in the second image Generating modified image data of an object in the second image if the image data of the second image differs by more than a predetermined amount An image processing method characterized in that it comprises a. 215. A process for processing image data, comprising the steps of:
Processing image data from the first camera to generate a first representation of the object in the three-dimensional computer model, and processing image data from the second camera to generate a second representation of the object in the three-dimensional computer model. Generating a representation, wherein image data of the object in the first image is generated by rendering the first representation, and image data of the object in the second image is generated by rendering the second representation. The image processing method according to claim 214, wherein: 216. Rendering texture data based on image data from the first camera into a first representation of an object in response to a direction of view selected by a second user, so that the second image in the sequence is Another image data of the object is generated, and the image data of the object in the second image generated using the image data from the second camera is generated using the image data from the first camera. 215. The image processing method according to claim 215, wherein the comparison is performed with image data of an object in the second image. 217. In the step of generating corrected image data,
The modified image data is used to generate the image data of the object in the second image generated using the image data from the second camera and the image data of the object in the second image generated using the image data from the first camera. The image processing method according to claim 216, wherein the image processing method is generated according to image data of the object. 218. The step of generating image data related to the first image includes the step of rendering a three-dimensional computer model according to a viewing direction selected by the first user, and the method related to the image related to the second image. Generating the data comprises rendering a three-dimensional computer model in response to a viewing direction selected by a second user; and testing comprises rendering rendered image data relating to the second image to a predetermined image. 228. The image processing method according to claim 214, further comprising a step of comparing the image data with image data. 219. The apparatus according to claim 21, further comprising the step of generating a signal for carrying the corrected image data.
The image processing method according to any one of claims 4 to 218. 220. The image processing method according to claim 219, further comprising a step of recording a signal. 221. The image processing method according to any one of claims 214 to 220, further comprising the step of displaying an image of the object using the modified image data. 222. The image processing method according to claim 214, further comprising the step of directly or indirectly creating a record of the modified image data. 223. A representation relating to at least one object and associated texture data extracted from image data recorded by a first camera and texture data extracted from image data recorded by a second camera. Rendering the three-dimensional computer model composed of the texture data having the first and second users in accordance with the respective viewing directions selected by the first and second users, thereby relating to the first and second images in the sequence of images. An image processing method for generating image data, comprising: rendering texture data based on image data from at least a first camera into a representation of each object in accordance with a viewing direction selected by a first user; Generate image data for use in one image And rendering the texture data based on the image data from the second camera into a representation of each object according to the viewing direction selected by the second user for use in the second image in the sequence. Generating image data of the second image in the sequence by testing whether the image data of the object in the second image differs from the predetermined image data by more than a predetermined amount. Testing whether the first image and the second image of the object displayed from are discontinuous, and if the image data of the object in the second image differs by more than a predetermined amount Generating corrected image data of an object in the second image. 224. A three-dimensional computer model including at least one representation of the object is processed a first time and rendered using image data recorded by the first camera as a basis for texture data for the representation. Thereby generating image data for the first image in the sequence of images, processing the three-dimensional computer model again, and providing a basis for texture data for representing the image data recorded by the second camera. Generates image data for the next image in the sequence by rendering as, and if the image data comparison test indicates that the objects in the image in the sequence appear to be discontinuous, then Generating modified image data for an object in the image. Image processing method. 225. An image processing method for generating image data related to successive images in a sequence by rendering a representation of an object in a three-dimensional computer model using image data from a plurality of cameras. Performing a test on the image data to determine whether the image of the object appears discontinuous in the image to be processed, and processing the image data to reduce the discontinuity. Processing method. 226. Processing image data defining a plurality of image sequences obtained from respective cameras of an object moving in a scene to generate a signal defining a representation of the object in a three-dimensional computer model. And an image processing apparatus for rendering image data of a three-dimensional computer model according to first and second user-selected viewing directions, thereby generating image data relating to the first and second images in the sequence of images. Means for processing image data to define at least one representation of an object in a three-dimensional computer model; and generating texture data based on image data from at least a first camera according to a first user-selected viewing direction. During the sequence by rendering to a representation of the object Means for generating image data for use in the first image; and rendering the texture data based on the image data from the second camera into a representation of the object in response to the second user-selected viewing direction, thereby providing a sequence. Means for generating image data for use in the second image in the sequence; and whether the image data of the object in the second image in the sequence differs from the predetermined image data by more than a predetermined amount. Means for testing whether the first image and the second image of the object displayed from the generated image data become discontinuous, and image data of the object in the second image Means for generating modified image data of the object in the second image if the values differ by more than a predetermined amount. The image processing apparatus according to symptoms. 227. The means for processing image data comprises:
Processing image data from the first camera to generate a first representation of the object in the three-dimensional computer model, and processing image data from the second camera to generate a second representation of the object in the three-dimensional computer model. Operable to generate a representation, wherein rendering the first representation generates image data of the object in the first image, and rendering the second representation causes the object in the second image to be rendered. 226. The image processing apparatus according to claim 226, wherein the image processing apparatus is configured to execute a process so as to generate the image data. 228. The means for testing comprises: rendering texture data based on image data from the first camera into a first representation of the object in response to a second user-selected viewing direction, thereby providing a second representation in the sequence. Image data of the object in the second image is generated using the image data from the first camera, and the image data of the object in the second image generated using the image data from the second camera is generated using the image data from the first camera. 228. The image processing apparatus according to claim 227, wherein the image processing apparatus is operable to execute processing so as to be compared with image data of an object in the generated second image. 229. The means for generating modified image data includes:
According to the image data of the object in the second image generated using the image data from the second camera and the image data of the object in the second image generated using the image data from the first camera. 229. The apparatus according to claim 228, wherein the apparatus is configured to perform processing to be generated. 230. The means for generating image data relating to the first image includes means for rendering a three-dimensional computer model in accordance with a first user-selected viewing direction, and the means for generating image data relating to the second image. The means for generating the second
Means for rendering a three-dimensional computer model according to the user-selected viewing direction, wherein the means for testing comprises means for comparing rendered image data associated with the second image with predetermined image data. 229. The image processing device according to claim 226. 231. The image processing apparatus according to claim 226, further comprising means for displaying an image of the object using the modified image data. 232. A representation relating to at least one object and associated texture data extracted from image data recorded by a first camera and texture data extracted from image data recorded by a second camera. Rendering image data related to the first and second images in the sequence of images by rendering a three-dimensional computer model composed of the texture data having the first and second user-selected viewing directions. An image processing device that renders texture data based on at least image data from a first camera into a representation of each object in accordance with a first user-selected viewing direction, for use in a first image in a sequence. Means for generating image data for Means for generating image data for use in the second image in the sequence by rendering texture data based on image data from the second camera into a representation of each object according to the selected viewing direction; By testing whether the image data of the object in the second image in the sequence differs from the predetermined image data by more than a predetermined amount, the first of the objects displayed from the generated image data is tested. Means for testing whether or not the image of the second image and the second image are discontinuous; and if the image data of the object in the second image differs by more than a predetermined amount, Means for generating corrected image data. 233. A three-dimensional computer model including at least one representation of the object is processed and rendered using image data recorded by a first camera at a first time as a basis for texture data for the representation. Thereby generating image data for a first image in a sequence of images and rendering using image data recorded by a second camera as a basis for texture data for a second time. Is operable to generate image data for the next image in the sequence, and if the image data comparison test indicates that objects in the images in the sequence appear to be discontinuous, the next image Operable to generate modified image data for the object Image processing apparatus. 234. Image processing for a method of generating image data for successive images in a sequence by rendering a representation of an object in a three-dimensional computer model using image data from multiple cameras. The apparatus is operable to perform a test on the image data to determine whether an image of the object appears discontinuous in a sequence of images, and to process the image data to reduce the discontinuity. An image processing apparatus, comprising: 235. A programmable processing device, comprising:
220. A storage medium storing instructions for executing the method according to any one of claims 14 to 225. 236. The programmable processor according to claim 2
226. A signal carrying instructions for performing a method according to any of claims 14 to 225.