JP3339980B2 - Image synthesis device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus.

[0002]

2. Description of the Related Art In order to present an image with a sense of reality as if the user actually saw it on the spot, a viewpoint change that allows the user to look around when the user moves his / her head. It is important to present an image corresponding to

Conventionally, if an image is synthesized from a three-dimensional model of an object or a reflection model of a surface used in computer graphics or the like, it is possible to synthesize an image from an arbitrary viewpoint. However, it is often impossible to synthesize a high-quality image in real time in response to a user's instruction to move the viewpoint, due to the performance of the device and the like. There is also a problem that an image generated by computer graphics has an image quality that is more artificial and uncomfortable than a natural image captured by a camera or the like.

In order to solve the above problem, there is a method in which a CG image created by taking a long processing time in advance or a natural image captured from a camera or the like is captured and used as a composite image.

Conventionally, as such a technique, a wide-angle planar image photographed by a wide-angle lens, or a cylindrical projection image photographed by rotating 360 degrees in a horizontal direction around one viewpoint called a panoramic image is used. There has been an image synthesizing apparatus that cuts out an image in accordance with a viewpoint direction and a viewing angle specified by a user.

[0006] These techniques will be described below with reference to the drawings. FIG. 14 is an explanatory diagram of a method of cutting out a composite image from a wide-angle planar image. Reference numeral 1101 denotes a plane image taken by a wide-angle lens or the like, with the center at O, the X axis in the horizontal direction, and the Y axis in the vertical direction. Viewpoint 1
Consider a case where an image is rotated from the image 102 and rotated by θ degrees in the horizontal direction. In this case, the center of the cutout region 1103 is P, and the right and left ends of the X axis are R and L. Point R,
The X coordinate value of L is Rx, Lx is the horizontal viewing angle α degrees,
The distance from the viewpoint 1102 to the center O of the plane screen, that is, the focal length at the time of photographing is F, and is obtained by the following equation.

[0007]

## EQU1 ## Rx = F.tan (.theta .-. Alpha./2) Lx = F.tan (.theta. +. Alpha./2) Here, the rotation angle is positive with respect to the Y axis in the direction of the left screw.

Similarly, the upper end of the vertical cutout coordinates,
Assuming that the Y coordinate of the lower end is Uy and By, the vertical viewing angle is β and can be calculated by the following equation.

[0009]

## EQU2 ## Ux = Ftan (-β / 2) Bx = Ftan (β / 2) Accordingly, the area specified by Rx, Lx, Uy, and By is cut out according to the rotation angle θ to obtain the viewpoint. An image corresponding to the horizontal rotation of can be synthesized.

Next, a method for obtaining a composite image in a certain viewing direction from a panoramic image will be described with reference to the drawings. FIG.
a is a schematic diagram showing the relationship between a panoramic image and a composite image viewed from above. Reference numeral 1201 denotes a panoramic image obtained by projecting a 360-degree image in the horizontal direction onto a cylinder. As a method of creating this panoramic image, the light receiving element of the vertical slit is set to 360
There is a method of imaging while rotating by degrees, and a method of geometrically deforming and joining planar images taken while rotating a camera or the like by a fixed angle around a light receiving surface. The panoramic image 1201 is a two-dimensional image having a rotation angle θ around the viewpoint and a height h in the vertical direction. FIG. 15B shows this panoramic image developed on a plane. In FIG. 15A, the image viewed from the viewpoint 1202 in the θ0 direction is an image 1203 obtained by projecting a panoramic image on a tangent plane of a cylinder. If the horizontal and vertical axes of the composite image are U and V, and the focal length is F, the point (θ1, h1) of the panoramic image corresponding to the point (u1, v1) in the composite image is given by the following equation. Is required.

[0011]

## EQU3 ## θ1 = arctan (u1 / F) + θ0 h1 = v1 · cos (θ) Assuming that the viewing angle is α, this is θ0−α / 2 to θ0 + α /
By calculating for u1 corresponding to 2, a desired composite image can be obtained. FIG. 15c shows an example of a composite <image. FIG.
Reference numeral 204 denotes an area in the panoramic image corresponding to the clipped image in FIG. 15C.

[0012]

However, in the above-described image synthesizing apparatus, since the images are all on the same plane and cylinder, it is impossible to perform processing using information on the unevenness on the image. There was a point.

That is, in the above-described image synthesizing apparatus, when a background image is handled, it is possible to synthesize a sufficiently distant image, but it is not possible to synthesize an image having mutual interference with a foreground.

For example, when an object by computer graphics or the like is simultaneously synthesized as a foreground other than the background, there is no unevenness information in the background image, so that a part of the foreground object is hidden by a part of the background, or When a foreground object is moving and collides with the background, it is impossible to change the collision state according to the shape of the background.

For example, when dealing with a background having a short distance from the viewpoint, such as a room or a street corner, mutual interference between the background and the foreground has the effect of increasing the sense of reality of the synthesis result.

According to the present invention, there is provided a method for removing the above-mentioned problem, in which an object in the foreground is hidden by a part of the background image due to unevenness of an image serving as a background, or reproduction of a direction of reflection when an object collides with the background. It is an object of the present invention to provide an image synthesizing apparatus which can synthesize images.

[0017]

SUMMARY OF THE INVENTION The present invention comprises a background image storage means for storing an image of a wide range of background as a two-dimensional array, in the same two-dimensional array with respect to the image of the background, the
A background depth storage means for storing depth information whose elements are depth values, and a three-dimensional shape of a foreground object moving in a three-dimensional space
Before moving to generate foreground object information including direction, speed, and speed
A scene object generating unit, the foreground object information, the background image, and the depth information, the image synthesizing apparatus including an image synthesizing unit that synthesizes the background and the foreground object.
The image synthesizing means includes the foreground object information and the depth
Determining contact between the background and the moving foreground object from the information
And outputs as the movement restriction information that has been contacted, the transfer
The moving foreground object generation means uses the movement restriction information to
Image synthesizing apparatus characterized by changing foreground object information
It is.

[0018]

According to the present invention, as described above, since the unevenness information of the background image is used, not only the shape such as a plane or a cylinder but also the hidden background and foreground which is a process requiring a complicated shape of the background are required. It is possible to perform surface processing and image synthesis having an effect that a foreground object collides with a background image and is reflected according to the inclination of the unevenness of the background object.

On the other hand, with regard to the rotation of the viewpoint, it is only necessary to change the cut-out position of the image, so that the processing is simpler than the computer graphics processing in which the background image is synthesized from its complicated three-dimensional shape. Background images can be combined.

[0020]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a schematic block diagram of an image synthesizing apparatus showing a first embodiment of the present invention. In FIG. 1, 101 is a background image storage unit that stores a wide range of background images, 102 is a background image cutout unit that cuts out a background image that enters the field of view based on viewpoint information on the direction, position, and viewing angle of the viewpoint from the wide range of background images. A background depth storage unit 104 stores depth information of a wide range of backgrounds corresponding to the background image stored in 101, and converts the viewpoint information and an object such as the shape and position of a foreground object placed in a three-dimensional space into an image. A foreground object generation unit that generates foreground object information that is information for combining,
Reference numeral 105 denotes an image for comparing the background depth information from 103 and the position of the foreground object from 104, and performing image synthesis of the foreground object on the background image in consideration of the concealment relationship between the background image and the foreground object image from 102. This is the synthesis unit.

Next, the operation of the first embodiment of the present invention will be described with reference to the drawings. FIGS. 2, 3, 4 and 5 are explanatory diagrams of a case where a rectangular background such as a room is represented by a cylindrical panoramic image and is synthesized with a foreground object. FIG. 2 is a schematic diagram showing the relationship between the panoramic image and the background as viewed from above the cylinder. Reference numeral 201 denotes a plan view of a room, which is projected onto a cylinder 203 centering on a viewpoint 202, and is a panoramic image 206 in FIG. This is a two-dimensional array defined by a rotation angle θ around the viewpoint and a height h in the vertical direction.

This panoramic image can be created by a method of rotating the light receiving element of the vertical slit by 360 degrees or by joining two-dimensional images. Further, it can be artificially created by a computer graphics method, for example, a ray tracing method that requires a long processing time. Such an image is stored in the background image storage unit 101 in FIG.

In the background image clipping section, the viewpoint is now at the position 202 in FIG.
Is θ0 and the visual field is between the angle values θs and θe shown in 205, the shaded area 207 on the panoramic image 206 in FIG. 3 is the area corresponding to the background image to be cut out.

For high-speed extraction, the distortion of the projected image of the cylinder and the plane is disregarded approximately and the output of the background image extraction unit is obtained from the area indicated by 207 by primary linear interpolation.

As shown in FIG. 4, the horizontal and vertical axes of the cut-out background image are U and V, respectively, and the width and height of the background image are Uw and Vh.
The value of (1, v1) is obtained as the value of the point (θ1, h1) of the corresponding panoramic image by the following equation.

[0027]

(4) θ1 = u1 / Uw · (θe−θs) + θsh1 = v1 / Vw · Hmax where Hmax is the maximum value of h of the panoramic image.

Since the panoramic image is discretized, the point (θ1, h1) calculated by the above equation may not accurately correspond to the discretized point. In that case, the value is calculated by linear approximation of four neighboring points.

More precisely, projection conversion between a cylinder and a plane may be performed. In this case, the point (θ1, h1) of the panoramic image corresponding to the point (u1, v1) in the composite image is obtained by the following equation, where F is the viewing angle α in the horizontal direction and F is the focal length.

[0030]

Α = (θe−θs) F = 2 · Uw / tan (α / 2) θ1 = arctan (u1 / F) + θ0 h1 = v1 · cos (θ1) Next, background depth information will be described. View point 2 in FIG.
A straight line is extended in the direction of the rotation angle θ from the point at the height h in the vertical direction from 02, and the distance from the point at which it hits the background object 201 is defined as r (θ, h). ,
h) is the background depth. The background depth information is a two-dimensional array of θ and h, similar to the panoramic image, and its element is a depth value r. FIG. 5 shows a change in the depth value r on a straight line where h is 0 in FIG. The background depth information is created by a three-dimensional measuring device such as a range finder or by a human inputting a CAD tool or the like while referring to a background image.

The background depth cutout unit cuts out depth values from θs to θe based on the viewpoint information. The depth value corresponding to the point (u1, v1) in the image is a value r (θ1, 対 応) corresponding to θ1, h1 calculated by (Equation 4) or (Equation 5).
h1).

In the foreground object generation unit, the surface shape of the generated object is represented by a polygon which is a plane approximation. FIG.
a indicates a sphere represented by a triangular polygon. One polygon 301 is composed of a list of vertices constituting the polygon 301 and vertices 302, 303, and 30 of the list.
4 and three polygon attribute information required for rendering, such as polygon color and reflection coefficient. The three-dimensional position of the vertex 304 is represented by an X, Y, Z rectangular coordinate system. As shown in FIG. 6B, the origin is set to the same viewpoint as the cylindrical coordinate system of the panoramic image, and the Z axis is the axis at which the horizontal rotation angle θ of the panoramic image is 0 degrees. The Y axis is in the upward direction of the rotation axis of the cylinder, and the X axis is in a right-handed coordinate system orthogonal to the Z axis and the Y axis. The three-dimensional positions of the vertices, the list of vertices forming the polygon, and the attribute information of the polygons are the foreground object information output from the foreground object generation unit for all the polygons forming the object.

The foreground object is not limited to one having a three-dimensional size. For example, when a two-dimensional image of an object is used, if the image is handled as attribute information pasted on a plane polygon, the same can be handled when the foreground is a two-dimensional image.

Next, a method of synthesizing the foreground and the background from the foreground object information, the background image and the background depth information in the background foreground synthesizing unit will be described with reference to the drawings. FIG. 7 is a flowchart of background foreground synthesis. For the hidden surface processing, a Z-buffer method often used in computer graphics is used. (1) First, a background image output from the background image cutout unit is copied to an output image. This defines values for the entire output image. (2) Each element has the same size as the output image and each element is 2
The background depth information from the background depth information cutout unit is copied into a Z buffer that is a dimensional array. The greater the depth value, the farther it is from the viewpoint. (3) Perspective projection transformation of the coordinate values of each vertex of the polygon constituting the object. Let the coordinates of the vertices be (x0, y0, z0). First, the direction θ0 and the Z-axis of the viewpoint shown in FIG. 2 are rotated around the Y-axis by the following equation to match the direction of the line of sight. If the coordinate values after conversion to this viewpoint coordinate system are (xp, yp, zp)

[0035]

Xp = x0 * cosθ-z0 * sinθ yp = y0zp = x0 * sinθ + z0 * cosθ Next, in the UV coordinate system as shown in FIG. 4, the width Uw and the height Vh
Change of projection on output image

In other words, The point on the image where the vertices are projected (u
0, v0) is obtained by the following equation using F in (Equation 5). However, it is assumed that the vertical and horizontal aspect ratio of the projection conversion is 1.

[0037]

U0 = Fxp / zp v0 = Fyp / zp (4) Each polygon of a foreground object from the foreground object generation unit is sequentially processed. (5) Divide the polygon into scan lines, and perform pixel processing on each scan line. The polygon is divided into scan lines parallel to the U-axis on the output image, and the coordinate values in the visual field coordinate system of the points at both ends of each scan line are calculated, and the internal points (u1, v1) on the scan line are calculated. The coordinate values (xp1, yp1, zp1) in the visual field coordinate system are calculated from the internal parts of the points at both ends. (6) The depth value r of the point (u1, v1) on the scan line in the cylindrical coordinate system is calculated by the following equation.

[0038]

[Mathematical formula-see original document] r = sqrt (xp xp + zp zp) where sqrt () represents a square root. (7) The depth value corresponding to (u1, v1) in the Z buffer is compared with r in (6). If r is smaller, the points of the foreground object are ahead of the background, and the processing of (8) and (9) is performed. If it is large, it remains as the background image.

Thus, a foreground object hidden in the background is not displayed, and a foreground object partially hidden can be expressed. (8) The surface color determined from the polygon attribute information is written to the point (u1, v1) of the output image. (9) The depth value r calculated in (7) is written to the point (u1, v1) in the Z buffer. (10) For all pixels on the scan line (6)
Process from to (9). (11) Process from (5) to (10) for all scan lines of the polygon. (12) From (4) to (1) for all polygons of the object
Process up to 1).

Next, a second embodiment of the present invention will be described with reference to the drawings. FIG. 8 is a schematic configuration diagram of a second embodiment of the present invention. Reference numeral 501 denotes a moving foreground object generation unit that generates a moving object that moves in a three-dimensional space based on the movement control information.
The movement control information includes information that is instructed interactively by a human using a mouse or the like and information that generates a trajectory from internal calculations. For example, as the former example, there is a case in which an object imitating a human hand is moved in a room, and as the latter example, a ball or the like jumps in a room. In this case, if only the initial movement direction is determined, the trajectory until it hits a wall or the like can be calculated dynamically.

This moving foreground object generation unit stores information such as the moving direction and speed of the object, as well as the three-dimensional shape of the object described in the first embodiment.

In the background foreground synthesis unit 502, the processes (8) and (9) of the Z buffer process described in FIG. 7 in the first embodiment are changed to contact determination processes as shown in FIG. . (13) Notify the moving foreground object generation unit that the background has contacted the moving foreground object as movement restriction information. afterwards,
The combining process is interrupted, and the image is combined again with new foreground object information from the moving foreground object.

When the movement restriction information is input, the moving foreground object generation unit inverts the currently held moving direction and generates information of the moving foreground object changed to the position moved by the speed.

As described above, when an image in which an object has moved farther than the background is to be synthesized, it is possible to reproduce a reflection motion that bounces off a wall, instead of being embedded in a wall or the like.

Further, in (13), the combination is not immediately stopped, but the number of the contacted polygon and the coordinates of the point are added to the movement restriction information, and the process proceeds to the next polygon.
The moving foreground object generation unit can know how many polygons have touched. By deforming the contacted polygon based on this information, it is possible to combine not only the reflection but also the shape crushed by the contact.

Alternatively, by adding information for switching between reflection processing and penetration processing for each object and polygon to the foreground object information, if reflection processing and hidden surface processing are switched, an object that rebounds and an object that penetrates a wall can be obtained. Can be combined at the same time.

The output of the composite image is performed when all the processes have been completed, so that the output image can be prevented from flickering due to re-composition.

Next, a third embodiment of the present invention will be described. FIG. 10 is a schematic configuration diagram of a third embodiment of the present invention. A background characteristic storage unit 701 corresponds to the background image in the background image storage unit 101, and as information for determining characteristics when an object interacts with the background, such as collision, the inclination of the background surface and the background attracts a substance. Coefficient of restitution,
The characteristics of the background, such as the impact coefficient that determines the impact strength of the collision based on the hardness of the background material and the temperature of the background material, are stored.

The background characteristic information is a two-dimensional array of the rotation angle θ about the horizontal and the height h in the vertical direction, similarly to the background image, and each element of the array is composed of the characteristic of the corresponding background. For example, considering the inclination of the background surface and the coefficient of restitution as characteristics of the background, the inclination of the background surface is represented by a normal vector (Nx, Ny, Nz).
Can be represented by The coordinate system of the normal vector is an XYZ orthogonal coordinate system similar to the coordinate system of the vertices of the object.

Assuming that the coefficient of restitution of the background is ν1, this is the ratio of the speed of the object before and after the collision. For example, if ν1 is 0, the surface has the property of adsorption, and the object is in a state of sticking to the background. Usually, it is a numerical value between 0 and 1, but it becomes 1 or more in the case of repulsion due to magnetic force or the like.

The collision-based object generation unit 702 adds a three-dimensional movement direction vector (Mx, My, Mz) of the object to the shape and position of the object, and rejects the object in the same manner as the background characteristics as necessary. Object information such as a coefficient ν2 is input to the background object synthesis unit 703. The background object synthesizing unit checks the contact between the background and the object, as in the second embodiment.

In FIG. 9 (13) when contact occurs, the direction vector (Lx, Ly, Lz) of the reflection is calculated from the direction vector of the moving object and the normal vector of the background as follows. It is determined to be symmetric with respect to the normal line of the background inclination.

[0053]

Lx = Mx-NxLy = My-NyLz = Mz-Nz Also, the moving speed of the object before reflection and the coefficient of restitution of the background ν
1. The moving speed of the object after reflection is calculated from the coefficient of restitution ν2 of the object. For example, when the speed before reflection is V1 and the speed after reflection is V2

[0054]

V2 = ν1 × ν2 × V1

The direction and moving speed of these objects after reflection are added to the object change information. The object generation unit obtains the average of the direction and speed of the reflection over the entire object, and sets the average as the direction and speed of the next movement amount. Alternatively, in the object change information, the point where the difference between the background depth value and the depth value r of the object point is the largest is set as the first contact point, and the moving direction and speed of that point are set as the new moving direction and speed. To generate an object.

The object change information is added to the impact coefficient of the background at the point where the object collides. If the impact coefficient exceeds an impact deformation threshold value stored as a characteristic of the object, the impact coefficient is calculated. By generating an object subjected to a deformation process such as reducing the shape of the object in the moving direction in accordance with, it is possible to express an image when a soft object collides with a hard background.

Alternatively, the background temperature is added to the object change information, and the object generation unit generates an explosion, evaporation, or melting object if the temperature is equal to or higher than a certain value depending on the temperature characteristic of the object. The image of the change of the object due to can be synthesized.

A method of calculating the background inclination from the background depth information will be described. A direction vector in a three-dimensional space from a point where the inclination of the background is to be obtained to a nearby point is calculated. The three-dimensional position of a point on the background is represented by three cylindrical coordinate systems of θ and h and a depth value r of the panoramic image. This is converted into the orthogonal coordinate system described with reference to FIG. 6B, a vector is calculated by subtracting the center coordinate value from the coordinates of the neighboring points, and the vector is normalized so that the magnitude becomes 1. In the case where there are four such vectors in the up, down, left, and right neighborhood, four are obtained. The cross product of two adjacent vectors among these vectors is calculated.
In the case of four neighbors, four outer product vectors are obtained. The average of these four cross product vectors and normalized to a size of 1 is used as the background slope of the point to be determined.

Next, a fourth embodiment of the present invention will be described. FIG. 11 is a schematic configuration diagram of a fourth embodiment of the present invention. An edge information storage unit 801 in FIG. 11 stores edge information that indicates whether there is a discontinuous point in the depth of the background, and in some cases, whether or not the discontinuous edge can pass through. The edge information is a two-dimensional array of θ and h. Each element has a value of 0 when there is no edge, 1 when the edge can pass through the edge, and 2 when it cannot. In the case of an edge, it further has a maximum value and a minimum value of the depth of the edge. This value is undefined if there is no edge.

Next, the edge information will be described in detail with reference to the drawings. In FIG. 12A, reference numeral 901 denotes a background image; 902, an object moving to a position (θ1, h1) and its moving direction; and 903, an object at a position (θ2, h1) and its moving direction. I have. Now, it is assumed that the shape of the depth r at the height h1 is as shown by 904 in FIG. 12B. At θ1, the door has a discontinuous edge at the depth, and at θ2, the bookshelf has a discontinuous edge. It is possible to pass through at θ1 and not at θ2. Therefore, in the edge information,
(Θ1, h1) becomes 1 and (θ2, h1) becomes 2, and the maximum value and the minimum value of each edge are stored.

The moving foreground object generation unit inputs the shape, position, and moving direction of the moving object to the background foreground synthesis unit 803 based on the movement control information. In the background foreground synthesis unit, the contact between the background and the foreground moving object is determined. FIG. 13 is a flowchart of the contact determination. (1) The process branches by referring to the flag of the corresponding point in the edge information. (2) If the flag is 0, it is not an edge point, so a contact determination is made as in the second embodiment, and if there is a contact, the contact is notified to the moving foreground object generation unit using the movement control information. (3) When the flag is 1, it is an edge portion that can pass through. As in the first embodiment, hidden surface processing using a Z buffer is performed. (4) When the flag is 2, there is an edge that cannot pass through in the depth direction. In this case, when there is an object between the maximum value and the minimum value of the edge, the movement control information is notified to the moving foreground object generation unit so as to reflect the moving direction in the direction opposite to the depth direction. In accordance with the above processing, the moving foreground object generation unit corrects the moving direction as necessary to generate a moving object.

Although all of the above have been described with reference to the embodiment using a panoramic image, the background image of the present invention is not limited to a panoramic image. For example,
It can be easily applied to the planar image described in the related art by replacing the cylindrical coordinate system with the rectangular coordinate system. Alternatively, a background image projected on a sphere instead of a cylinder may be used. In that case, the cylindrical coordinate system may be replaced with a polar coordinate system.

Since the background image, the background depth information, the background inclination information, and the background edge information are two-dimensional arrays of the same size with respect to the same background point, the elements of the array are expanded. It is also possible to give a plurality of information to the same sequence. In this case, there is an advantage that the array needs to be accessed only once for each pixel.

[0064]

As described above in detail, according to the present invention, in addition to the synthesis of the background by rotating the viewpoint, the hidden surface processing, which is a three-dimensional interaction between the foreground object and the background,
By processing the reflection from the surface, it is possible to perform image synthesis that enhances the sense of reality as a three-dimensional space.

[Brief description of the drawings]

FIG. 1 is a schematic configuration diagram of a first embodiment of the present invention.

FIG. 2 is an explanatory diagram showing a relationship between a background image and a background depth.

FIG. 3 is an explanatory diagram showing a relationship between a background image and a background depth.

FIG. 4 is an explanatory diagram showing a relationship between a background image and a background depth.

FIG. 5 is an explanatory diagram showing a relationship between a background image and a background depth.

FIG. 6 is an explanatory diagram of a shape representation of a foreground object and a coordinate system.

FIG. 7 is a processing flowchart according to the first embodiment of the present invention.

FIG. 8 is a schematic configuration diagram of a second embodiment of the present invention.

FIG. 9 is a processing flowchart according to a second embodiment of the present invention.

FIG. 10 is a schematic configuration diagram of a third embodiment of the present invention.

FIG. 11 is a schematic configuration diagram of a fourth embodiment of the present invention.

FIG. 12 is an explanatory diagram of depth edge information.

FIG. 13 is a processing flowchart according to a fourth embodiment of the present invention.

FIG. 14 is an explanatory view of a conventional example.

FIG. 15 is an explanatory diagram of a conventional example.

[Description of Signs] 101 Background image storage unit 102 Background image cutout unit 103 Waveform depth storage unit 104 Foreground object generation unit 105 Background foreground object synthesis unit 501 Moving foreground object generation unit 502 Background foreground object synthesis unit 701 Background reflection information storage unit 702 Moving foreground object generation unit 703 Background foreground synthesis unit 801 Edge information storage unit 802 Background foreground synthesis unit 803 Moving foreground object generation unit

Continuation of front page (56) References JP-A-4-372079 (JP, A) JP-A-5-183808 (JP, A) JP-A-4-123279 (JP, A) JP-A-5-181940 (JP) , A) JP-A-5-181950 (JP, A) JP-A-6-28451 (JP, A) Masamichi Nakagawa et al., Model-Based Image Synthesis Technology, National Technical Report, Japan, Matsushita Electric Industrial Co., Ltd. , December 18, 1994, Vol. 40, No. 6, 106-113, issued after the priority of the present application Masamichi Nakagawa et al., Generation of stereoscopic images by intelligent image coding, 1990 IEICE Spring Meeting Proceedings of the National Convention (7), Japan, The Institute of Electronics, Information and Communication Engineers, March 5, 1990, PT. 7, 7-39 Junichi Sato et al., Automatic Generation of 3D Shape Model Based on Surface Normal Distribution, Proceedings of the 1992 IEICE Spring Conference, Japan, The Institute of Electronics, Information and Communication Engineers, March 15, 1992 JP, PT. 1,1-302 (58) Field surveyed (Int.Cl. ⁷ , DB name) G06T 17/40 G06T 15/70 G09G 5/36 510 G09G 5/377

Claims (5)

(57) [Claims] 1. A background image storage means for serial <br/>憶images of a wide range of background as a two-dimensional array, in the same two-dimensional array with respect to the image of the background, the element
Background depth storage means for storing depth information in which is a depth value, 3D shape and moving direction of a foreground object moving in a 3D space
A moving foreground object generator that generates foreground object information including speed
A step, the foreground object information, the background image, and the depth information, the image synthesizing device including an image synthesizing unit that synthesizes the background and the foreground object, wherein the image synthesizing unit includes the foreground object information And the depth
Determining contact between the background and the moving foreground object from the information
The contact is output as movement restriction information, and the moving foreground object generating means uses the movement restriction information.
The foreground object information by changing
Equipment. 2. An image of a wide range of backgrounds is rotated by a rotation angle θ around a viewpoint.
Panorama of two-dimensional array defined by height and vertical height h
A panoramic background image storage means for storing as an image, and a two-dimensional array of θ and h like the panoramic image of the background
A background depth storage unit that stores depth information whose element is a depth value r, a background image cutout unit that cuts out a background image included in the field of view from the panorama image based on viewpoint information including a line of sight direction, and viewpoint information including a field of view . Background depth cutout that cuts out depth values based on the viewpoint information
2. The image synthesizing device according to claim 1, further comprising: an output unit . 3. A moving foreground object generating means, comprising:
2. The moving direction of body information is corrected.
An image synthesizing apparatus according to claim 1. 4. The moving foreground object generating means calculates a direction of movement of the moving foreground object due to a contact from the inclination of a background surface obtained from the depth information, and generates new foreground object information. The image synthesizing device according to claim 3, wherein: 5. The background depth storage means stores information on discontinuous portions in a two-dimensional array similar to an image.
The image synthesizing unit also has the edge information , and detects the possibility of passing through the foreground object at the discontinuous point when detecting contact between the foreground object and the background.
The image synthesizing apparatus according to claim 3, wherein the determination is performed using a report .