JPH07320086A - Method and device for image generation - Google Patents

Abstract

PURPOSE:To lower the arithmetic cost and shorten the processing time by generating an image without any detailed three-dimensional shape model. CONSTITUTION:This device consists of a structure information storage part 1 for an object image, a view point position input part 2 for inputting a view point position for a virtual object at the time of viewing, a motion mode selection part 3 regarding prepared objects, a local feature element arrangement and deformation part 4 which alters the information from the structure information storage part 1 on the basis of the view point position input part 2 and motion mode selection part 3, a generation part 5 which performs a generating process so that interpolation between feature elements is performed with a curved line, an image data base 6 which stores a background image, an image composition part 7 which composites the object image generated by the generation part 5 with the background image from the image data base 6, and an image display part 8 which displays the image composed by the image composition part 7.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image generating method and an image generating apparatus for generating a complete image from a small number of partial image data.

[0002]

2. Description of the Related Art Conventionally, as a method for reflecting the three-dimensional shape and distance of an object in an image and generating an image corresponding to a viewpoint position with a small amount of data, for example, image generation disclosed in Japanese Patent Laid-Open No. 12413/1993. Like a device, a point on the surface where the target object can be reproduced is measured in advance by a three-dimensional measuring device, and the three-dimensional position coordinates of such a point are stored, and when a reproduced image is generated, the viewpoint is The position on the screen for the reproduced image is obtained from the positional relationship between the position and the target property, and the luminance level or RGB value is displayed in correspondence with the corresponding point relationship with the images obtained at different viewpoint positions in advance.

Further, in the composite image generating apparatus disclosed in Japanese Patent Publication No. 5-60148, a reference image and a three-dimensional shape model of a target object are stored in a predetermined memory in advance, and a different viewpoint from the one stored at the time of reproduction is stored. It calculates the scaling factor, moving distance on screen, and rotation angle of the target image from the difference between the distance from and the direction, and performs geometric deformation based on them to generate the target image observed from different viewpoints. there were.

[0004]

However, in the above-mentioned conventional example, the image of the specific object in the photographed image is processed to be combined with the image of another scene, or the motion of the specific object while considering the occluded portion, A more detailed three-dimensional shape model of the object is required to perform the transformation. Then, in order to obtain a more detailed three-dimensional shape model regarding the object, detailed measurement is required separately by a shape measuring means or a method of obtaining a shape model from an image with parallax obtained by a twin-lens stereo camera. Therefore, there is a problem that both the calculation cost and the processing time are large.

In view of the above problems of the prior art, the present invention generates an image without a detailed three-dimensional shape model, thereby making it possible to reduce the calculation cost and the processing time, and an image generating apparatus. Is intended to provide.

[0006]

SUMMARY OF THE INVENTION To achieve the above object, the present invention stores pre-defined local feature elements and their spatial arrangement information as image structure information, and based on the structure information. It is characterized by generating an image.

In this image generation method, the local feature element is composed of contour lines that are spatially discrete, and the local feature element is a crossing pattern of edge segments in a plurality of directions and a curvature. It is characterized in that it is composed of a fixed curve or a part thereof and an edge segment, and a part of an image is generated by smoothly connecting the local feature elements by a predetermined method.

The structure information is characterized in that it is a local feature element at each viewpoint position and its spatial arrangement information when it is generated and disappears with a change in the viewpoint position.

Further, the local feature element may be a surface element arranged in a three-dimensional space, and the curved surface data for interpolation which smoothly connects the local feature elements may be generated.

Further, the image generating apparatus suitable for the above method has a structure information storage unit for storing the previously defined local feature elements and their spatial arrangement as the structure information of the target image, and a structure information storage unit for viewing the image. From a viewpoint position input unit for inputting a viewpoint position for a virtual target, a motion mode selection unit for selecting a predetermined motion mode for the target, the viewpoint position input unit and the motion mode selection unit The local feature element placement / deformation unit that changes the information in the structural information storage unit based on the input information of the above and the information from the local feature element placement / deformation unit, and interpolates between the local feature elements. A generation unit that generates a curved line or a curved surface, an image storage unit that stores a background image that is a background of the target image, a background image is captured from the image storage unit, and the generation process is performed by the generation unit. And an image synthesizing portion for synthesizing a target image, an image display unit for displaying the image formed by the image synthesizing unit, and a.

[0011]

According to the present invention configured as described above, an image is generated based on the local feature elements defined in advance and their spatial arrangement information, so that a detailed three-dimensional image such as a target wireframe model can be obtained. Video that generates an image from an arbitrary viewpoint position at a predetermined position by giving an arbitrary size and deformation according to the motion / motion of the target without obtaining a shape model, or inserting a motion / motion of the target It is possible to generate an image.

[0012]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing an example of the configuration of an apparatus for suitably implementing the image generation method of the present invention, and FIG. 2 is shown in FIG.
It is a figure showing an example of a local feature element memorized by structure information storage part shown in.

As shown in FIG. 1, the image generating apparatus of this embodiment has a structure information storage unit 1 for a target image, a viewpoint position input unit 2, a target motion mode selection unit 3, a local feature element placement / deformation unit 4, The inter-feature element interpolation curve generation unit 5, the image database 6, the image synthesis unit 7, and the image display unit 8 are included.

Explaining each of the above-mentioned components in detail, the structure information storage unit 1 is a storage unit for storing structural image information of a target object, living thing, etc. Two-dimensional or three-dimensional local characteristic elements as shown in FIG. 2 and their three-dimensional spatial arrangement information are stored as the minimum necessary information for expressing the accompanying change in the target image.

The viewpoint position input unit 2 is for inputting a position and a direction at which an observer looks at an object, and is a magnetic conversion method (for example, 3SPACE ISOTRAK II manufactured by POLHEMUS) and an ultrasonic method ( For example, Stereo Graphics, Crystal EYES
For example, a method of directly mounting a sensor or the like on the observer's head such as VR System) to measure and input the viewpoint position, or a method of inputting with a mouse or keyboard of a computer terminal may be used.

The target motion mode selection unit 3 determines a predetermined motion / motion relating to the target (for example, when targeting a person, a change in facial expression such as emotions and emotions, and a mouth shape corresponding to a spoken word). It is used to interactively select and generate a moving image that is related to changes in the shape of the face, such as changes in face shape, to walking, running legs, or body movements during sports play, etc. at a certain position in the background image. Be seen.

The local feature element placement / deformation unit 4 receives information from the viewpoint position input unit 2 and the target motion mode selection unit 3, and reproduces an image reflecting the three-dimensional shape and viewpoint position of the target. Changes the relative position of the local feature element on the imaging surface and the deformation of each element.

The inter-feature element interpolation curve generation unit 5 receives the output of the local feature element arrangement / deformation unit 4 and performs a generation process so that the inter-feature elements are correctly and smoothly interpolated by the curvilinear elements.

The image database 6 is an image storage unit that stores a background image that is the background of the target image.

The image synthesizing unit 7 is a processing unit for fetching an image serving as a background other than the specific target from the image database 6 and synthesizing the image with the target image.

The image display unit 8 includes display means for displaying the combined image and its driving means.

Next, the structure information storage unit 1 in this embodiment.
The information storage format will be described by taking a three-dimensional animation image of a face as an example.

In the present embodiment, the face image information is used for the local feature elements shown in FIG. 2 and their three-dimensional spatial arrangement for each part (eg, eyes, nose, mouth, ears, etc.) that constitutes the face. Is held as the structural information of the part image, and further, each part and its (center of gravity) positional relationship are held as the structural information of the face image. For example, the basic structural information of an image of an eye as one of the parts is L-shaped intersection elements at the parts corresponding to the inner and outer corners of the eye.

[0025]

[Equation 1] Is a curve (curvature) element in the arcuate curve part above and below

[0026]

[Equation 2] The local feature elements to which is assigned and their representative point positions (for example, left and right upper and lower end point coordinates) are stored. The image structure information of the contour of the eye is not limited to the above-mentioned configuration, and for example, two curve (curvature) elements corresponding to above and below the eyes.

[0027]

[Equation 3] May be represented by the position of the representative point.

Further, the structural information necessary for expressing the shape of the pupil may be expressed by a circle Cr having a predetermined radius and its center position, or two curvature elements.

[0029]

[Equation 4] And their representative point positions may be represented. In this case, the position of the T-shaped intersection that occurs at the intersections with the curvature elements corresponding to above and below

[0030]

[Equation 5] May be included in the structural information.

As the basic structure information of such a face image, an image viewed from the front and the front in a predetermined size is stored as canonical image information by default.

Further, the viewpoint position is, for example, 4 from the front direction.
Similarly, as the structural information of the image when changed in the horizontal plane in the directions of 5 °, 90 °, 135 °, and 180 °, the local feature elements and their spatial arrangement information as well as the basic structural information from the corresponding original images. Remember.

The characteristic element arrangement / deformation unit 4 receives the information from the viewpoint position input unit 2 or the motion mode selection unit 3 to generate an image from an arbitrary viewpoint position, and obtains a discrete viewpoint direction obtained in advance. Predict the apparent shape and spatial arrangement data of each feature element in the image of the viewpoint direction from the different structure information data,
To generate.

Therefore, the method will be described in detail below.

Drawing based on line drawing such as 3D animation
When expressing an image, the virtual viewpoint changes
Then, the transformation, deletion, and generation of the contour data occur. Example
Then, of these, as for extinction and generation,
Angle data of viewpoint direction θ _iAnd the structural information
Data

[0036]

[Equation 6] Ask for. still,

[0037]

[Equation 7] Is the jth position

[0038]

[Equation 8] Represents a local feature element existing in the. N (i) indicates the total number of characteristic elements of the target image corresponding to the viewpoint direction θ _i .

Next, structural information data in the viewpoint direction θ (θ _i-1 <θ <θ _i ).

[0040]

[Equation 9] To generate. here,

[0041]

[Equation 10] , F are the corresponding feature elements in the viewpoint direction θ from f _k ⁱ⁻¹ , f _k ⁱ

[0042]

[Equation 11] Position of

[0043]

[Equation 12] And a shape (for example, a crossing angle of L-shaped intersection, a direction, a curvature of a curvature element, a direction, etc.), and the position and shape data of two feature elements f _k ^i-1 and f _k ⁱ are represented by θ ( θ
Interpolation is performed from the value of _i-1 <θ <θ _i ). For example, for linear interpolation,

[0044]

[Equation 13] still,

[0045]

[Equation 14] Represents the shape parameter of the local feature element, and represents the intersection angle in the case of the L-shaped intersection and the curvature in the case of the curvature element.

In the present embodiment, the linear method as described above is used as the interpolation method, but the present invention is not limited to this.

In addition,

[0048]

[Equation 15] Is defined for characteristic elements of the same type (for example, types of L-shaped intersections, curvature elements, etc.) that are common to both images of the viewpoint directions θ _i−1 and θ _i , but θ _i−1 ≦ θ < The spatial position of a feature element that exists at θ _i and disappears at θ _i <θ ≤ θ _{i + 1}

[0049]

[Equation 16] Is obtained by the same interpolation as described above, and its shape is invariable, or separately, by generating structural data based on the image from one viewpoint direction in the range of (θ, θ _i ), the position and shape can be calculated. Interpolated data may be obtained.

After the predetermined viewpoint position, the shape of the characteristic element in the target motion mode, and the arrangement data have been obtained as described above, the interpolating curve generating section 5 first sets the face image parts (for example, eyes and nose). , Mouth, etc.) to generate a curve that smoothly connects the feature elements.

Next, the generation method will be described taking the generation of an eye image, which is one of the parts, as an example.

FIG. 3 shows two L-shaped intersections of eyes.

[0053]

[Equation 17] 4 curvature elements

[0054]

[Equation 18] An example of a method for smoothly connecting these local feature elements when described as structural information based on the spatial arrangement of is shown.

In FIG. 3, the size parameter σ of each feature element
Is set in advance, the sum of the lengths of the two line segments forming the L-shaped intersection is about the same as σ,
For curvature elements, the arc length is comparable to σ. in the case of,
The inter-feature element interpolation curve generation unit 5 defines the end point of one line segment L ₁₁ that constitutes the L-shaped intersection L _1.

[0056]

[Formula 19] Connect one end with a straight line (indicated by a dotted line in the figure). Similarly, other line segments L ₁₂ and C ₁ constituting the L _1,

[0057]

[Equation 20] The eye contour is formed by connecting and with a straight line.

Further, in FIG. 3, smooth connection between the characteristic elements is performed by generating a Spline curve. For example, when connecting by a cubic Spline curve element,
Y = ax ³ + bx ² + cx in a preset coordinate system
The coefficient values of + d are selected so that their slopes match at each end point of the feature element to be combined. The drawing range can be determined by obtaining the coordinates of each end point of the feature element.

Furthermore, the method of is based on a contour model called Snakes (M.Kass, A. Witkin, D. Terzopoulos: "S
nakes: Active contour models '' International Journal
of Computer Vision, Vol.1, pp.321-331) is applied, and the evaluation function is applied to the contour line v (s) = (x (s), y (s)).

[0060]

[Equation 21] To obtain a contour line v (s) that minimizes
_{i (s)} (s) is a control point (line) of Snakes, and corresponds to a line or a curve element of a local feature element. Subscript i (s)
Corresponds to the number when the characteristic element is numbered by an appropriate method.

Although the method of generating a curve for interpolation between local feature elements has been described above, the present invention is not particularly limited to this.

As described above, the line image of each part is generated by connecting the characteristic elements for each part of the entire image of the target.

When the target motion mode is set by the selection unit 3, the local feature elements are arranged and arranged in time series based on a model prepared in advance according to each motion and motion.
A transformation is generated by the local feature element placement / transformation unit 4,
Each time, the interpolation curve generator 5 performs shape interpolation processing to generate a target moving image. It goes without saying that the moving image reflects the viewpoint position of the observer.

Further, in the present embodiment, the target image and the background image prepared in advance in the image database 6 are selected and the image synthesizing unit 7 performs the synthesizing process. At that time, the target image may be texture-applied to add a shadow, a color, a pattern, or the like. FIG. 4 shows structure information and a restored image in the case of a face image.

The time-series data of the shape and arrangement of the local feature elements determined by the type of motion / motion of the object may be generated from the actual moving image based on a predetermined local feature element extraction method, or on a computer. It may be generated as a motion / motion model. FIG. 5 shows an example of a storage format of such time series data.

In FIG. 5, reference numeral j is an image frame number, reference numeral N is a characteristic element of the target image in the frame, reference numeral f is a characteristic element type, reference numeral.

[0067]

[Equation 22] Represents the position of the representative point, and x and y represent the horizontal and vertical pixel positions in the frame.

In this way, the local feature element arranging / deforming unit 4 shown in FIG. 1 reads in the respective types and representative point positions of the feature elements as control parameters, and further, according to the viewpoint position of each feature element. Make appropriate changes to the layout and shape.

Note that the data shown in FIG. 5 shows an example of line-of-sight direction data regarding a certain part in the face image contained in the structure information storage unit 1 shown in FIG. If necessary, the value of the parameter that determines the shape of the characteristic element may be stored in a predetermined format.

(Other Embodiments) FIG. 6 is a block diagram showing another example of the configuration of an apparatus for suitably implementing the image generating method of the present invention. In the following description based on this figure, the same components as those in the above embodiment are designated by the same reference numerals, and the description thereof will be omitted.

The present embodiment is characterized in that a three-dimensional surface element is used as a local characteristic element, and a surface for smoothly interpolating the characteristic elements is generated by the inter-characteristic element curved surface generation unit 15. FIG. 7 shows an example of local feature elements used in the image generating apparatus shown in FIG.

In this figure, SL _i has two plane elements L
It is an L-shaped surface element formed by intersecting in a V shape, and is discretely set according to the intersection angle and its direction (the direction that divides the intersection angle into two equal parts and is orthogonal to the intersection line segment).

SC _i is Gaussian curvature, mean curvature (or
Two principal curvatures), a center vector of the center point, and a curved surface element specified by a size parameter, and SL _i
Similarly, is preset by a discrete set of specified parameters. SC ₁ , SC ₂ , and SC ₃ represent different surface elements with different combinations of signs of the two principal curvatures.

In order to generate the image of the target object, the inter-feature element interpolating curved surface producing section 15 produces curve data for smoothly filling the space between the feature elements as x, y, z coordinates, and then, in the viewpoint direction and position. Based on the hidden surface processing, a shadow, a color, a pattern, etc. reflecting the ray direction and the reflection characteristics of the surface are added and displayed on the image display unit 8.

[0075]

As described above, according to the present invention, the image of the specific object is generated from different arbitrary viewpoint positions by expressing the spatial arrangement of the local feature elements defined in advance as the structural information of the image. It is possible to generate an image according to the movement of a target without a detailed three-dimensional shape model such as a wire frame model.

Therefore, the number of data required to generate a three-dimensional image can be greatly reduced compared to the conventional case.

Further, since the generation and disappearance of the shielding contour due to the change of the viewpoint position can be stored in advance, the calculation is performed as compared with the method in which the geometrical calculation is performed from the three-dimensional shape model of the object and the viewpoint position. Both cost and processing time can be reduced.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration example of an apparatus that preferably implements an image generation method of the present invention.

FIG. 2 is a diagram showing an example of local feature elements stored in a structure information storage unit shown in FIG.

FIG. 3 is a diagram for explaining an example of a method for smoothly connecting local feature elements when an eye is described as structural information by spatial arrangement of two L-shaped intersections and four types of curvature elements. .

FIG. 4 is a diagram showing structure information and a restored image in the case of a face image.

FIG. 5 is a diagram showing an example of a storage format of time-series data of shapes and arrangements of local feature elements.

FIG. 6 is a block diagram showing another configuration example of an apparatus that preferably implements the image generation method of the present invention.

7 is a diagram showing an example of local feature elements used in the image generating apparatus shown in FIG.

[Explanation of symbols]

 1 target image structure information storage unit 2 viewpoint position input unit 3 target motion mode selection unit 4 local feature element placement / deformation unit 5 inter-feature element interpolation curve generation unit 6 image database 7 image synthesis unit 8 image display unit 15 feature element Curved surface generator for interpolating

Claims (6)

[Claims] 1. An image generation method characterized in that a local feature element defined in advance and spatial arrangement information thereof are stored as image structure information, and an image is generated based on the structure information. 2. The local feature element comprises a contour line that is spatially discrete and present.
The image generation method described in. 3. The local feature element comprises an intersection pattern of edge segments in a plurality of directions, a curve having a constant curvature or a part thereof, and an edge segment, and the local feature elements are smoothly connected by a predetermined method. The image generation method according to claim 1, wherein a part of the image is generated thereby. 4. The structure information is local feature elements and spatial arrangement information at each viewpoint position when the structure information is generated and disappears with a change in the viewpoint position. 1. The image generation method described in 1. 5. The local feature element is a surface element arranged in a three-dimensional space, and curved surface data for interpolation that smoothly connects the local feature elements is generated. The image generation method described in. 6. A structure information storage unit for storing predefined local feature elements and their spatial arrangement as structure information of a target image, and for inputting a viewpoint position with respect to a virtual target when viewing the image. Point of view position input unit, a motion mode selection unit for selecting a predetermined motion mode for the object, based on the input information from the viewpoint position input unit and the motion mode selection unit, the structural information storage A local feature element arrangement / deformation unit that changes information of a part; a generation unit that receives information from the local feature element arrangement / deformation unit and generates a curve or a curved surface that interpolates between the local feature elements; An image storage unit that stores a background image that is the background of the target image, and an image composition unit that captures the background image from the image storage unit and combines the background image with the target image that has been generated and processed by the generation unit. And an image display unit that displays the image formed by the image composition unit.