JP2000105841A - Method and device for displaying three-dimensional shape model - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system capable of composing computer graphics and actual shot video with good operability. SOLUTION: An image composing device which composes an image of computer graphics and actually shot video is equipped with a means which extracts a specific area in the actually shot video, a means 2 which adds information on a three-dimensional shape to the extracted area, and a means 3 which models the information regarding the extracted area by computer graphics according to the information of the extracted area and the information of the three- dimensional shape.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a method and an apparatus for synthesizing computer graphics and real images.

[0002]

2. Description of the Related Art In recent years, the synthesis of computer graphics (hereinafter referred to as CG) and live-action video has been actively used in the production of movies and commercial films. For example, by synthesizing a fictitious building made of CG with a real photographed image of a human being taken in advance, it is possible to create the effect of a human walking in the building. The video created by combining the CG with the live-action video is real and has a strong impact on the viewer, and is especially indispensable for landscape simulation.

In general, in CG, the shape of an object to be drawn is
It is defined using a simple geometric shape (shape primitive) such as a plane or a quadratic surface, and the surface is colored and any image data is pasted. However, in this method, when a natural object such as a tree or a river is drawn, its shape always looks fixed. So, beforehand, I shot a tree swaying in the wind or a river with flowing water,
A more natural animation can be created by synthesizing the live-action video with the scene created in. In a conventional method, an image is selected from a photographed video image and attached to a simple primitive such as a flat plate to perform a synthesis process with a CG. This synthesis process is repeated for each frame to produce an animation of a continuous synthesized image.

As a well-known document relating to a composite image of a still image, a method for constructing a simple 2.5-dimensional scene model for landscape simulation (July 1992: "Image recognition /
Understanding Symbolium (MIRU '92). Also, Japanese Patent Application Laid-Open No.
-162173 and JP-A-3-244005.

[0005]

In this type of technology, there are the following problems to be solved. (1) To make it easy to convert a live-action video to a CG model having a three-dimensional shape. (2) To allow the intervention of an operator during this conversion. Enable composition (4) Operability when extracting required parts from live-action video,
(5) Enhance operability of rotation, scaling and movement of CG model (6) To facilitate synchronizing the synthesis of CG and live-action video

[Summary of the Invention] The method proposed in the present invention,
In the device (system), using a technique of image processing, a computer performs interactive processing with a user, 1) segmenting video information into object units, 2) generating a moving image object to which 3D geometric information is added, 3 3) Simultaneous display of the CG model and the moving image object is performed. The 1) eliminates the need to measure and record the camera's positional information when shooting a video, and the 3) allows the viewpoint to be changed when displaying the image.

FIG. 1 schematically shows the configuration of a system according to the present invention. The present system includes three processing units: a specific object region extraction unit, a three-dimensional geometric information extraction unit, and a moving image object and CG simultaneous drawing unit. Data called a moving image object to be combined with CG is created by the specific object region extraction unit and the three-dimensional geometric information extraction unit, and stored on the hard disk. Using this data, video objects and
The CG simultaneous drawing unit generates a composite image in non-real time.

Specific Object Region Extraction Unit The specific object region extraction unit performs a process of cutting out a specific object region from video information input by a capturing tool. FIG. 2 shows the flow of this processing. Here, a continuous image sequence is received as input data, and an image sequence of a rectangular area containing a specific object as an output and an alpha map sequence storing an alpha value of the same size are created. When the region of the object is cut by the binary mask, unnatural aliasing occurs at the boundary. To prevent this, use alpha values to define regions. After the area of the specific object is determined for a certain n-th image by dialog processing with the user, the processing of the (n + 1) to (n + m) -th images is semi-automatically performed by a computer using the processing result of the previous frame.

The extraction unit for three-dimensional geometric information The three-dimensional geometric information extraction unit uses a two-dimensional image sequence and an alpha map sequence of a rectangular region containing a specific object created by the above-described specific object region extraction unit. 3D geometric information is extracted from the video information. The extraction of the three-dimensional geometric information is performed by a user transforming, rotating, or transforming a plurality of simple shape primitives (such as a rectangular parallelepiped) into an object on a two-dimensional image.
It is realized by performing the operation of moving and performing fitting. In this system, not only viewpoint information but also shape information of an object and a texture image attached to each surface are extracted. In this extraction unit, in order to create a data structure called a moving image object, three-dimensional geometric information is given to the object in the video, the video information attached to each surface of the CG modeled object is extracted, and viewed from the front. Normalize and store it.

Structure of Moving Image Object The data generated through the processing of the specific object area extraction unit and the three-dimensional geometric information extraction unit has a structure called a moving image object. FIG. 3 schematically shows the structure of the moving image object.
The video object is a newly created data structure for fusing CG and video.
A pointer to video information (still image or moving image) attached to each surface is stored.

Synthesized image generator The synthesized image generator simultaneously draws the generated moving image object data and CG data. In this case, specifying the time T _i and the time interval Δt of the CG scene to be rendered as meta-information. The CG data includes position information at each time in addition to the shape data of each object. Moreover, the time T _i, the image to be pasted on each side from the video data in the video object is selected. After the time T _i is the surface attributes of the in the object is determined to generate a composite scene at time T _i.

Hereinafter, the present invention will be described specifically. The invention of the present application is roughly divided into two. (1) Concerning the overall configuration (2) Concerning the part for extracting the area of the specific object and the part for adding three-dimensional shape information

(2) relates to 2-1 the configuration of a specific object region extraction unit and a three-dimensional shape information addition unit. 2-2 relates to the configuration of the extraction unit. 2-3 relates to the display of a three-dimensional shape model. Divided into two. Hereinafter, (1) is referred to as a first group of inventions, and 2-1, 2-2, and 2-3 are referred to as second, third, and fourth group inventions, respectively.

Although the subject of the present application is the fourth group of inventions, the first to third group inventions related thereto will also be described. The invention of the first group will be described in detail. [Invention of First Group] (Summary) The invention of the first group relates to the overall configuration of the synthesis of CG and real shot video.

[0015]

2. Description of the Related Art The synthesis of a CG and a live-action video is performed by assuming the synthesized video,
Only the part to be synthesized was cut out from the actual image and superimposed on the CG. In the case of shooting using such a method, a large-scale environment is required, such as a need for studio shooting with a blue background or a need to measure the position of a camera during shooting.

The method supported by a computer is also the MIRU described above.
Proposed in '92. This extracts viewpoint information from a live-action video, approximates the object in the video to a plane,
This is a method of combining with CG. However, since the model does not have complete three-dimensional information, the viewpoint cannot be changed at the time of synthesis. Japanese Patent Application Laid-Open No. 3-138784 discloses that in order to treat an object in a still image as three-dimensional, the object in the still image is reconstructed based on a three-dimensional model.
A method of mapping and displaying a partial image corresponding to a three-dimensional object as a surface texture of a three-dimensional object model has been proposed. This method also proposes synthesizing a surface texture from a plurality of input images for one three-dimensional portion. However, in the case of a video (moving image), the surface texture may change every moment. When a plurality of textures are combined, a texture smoothed in a time-series direction may be obtained, which is inappropriate.

[0017]

SUMMARY OF THE INVENTION The present invention has been made under such a technical background, and proposes an image synthesizing method capable of easily converting a real image into a CG model having a three-dimensional shape. The first purpose is to do so. It is a second object of the present invention to provide a video synthesizing method in which the CG model is generated for each frame and, as a result, a moving image can be obtained.

[0018]

SUMMARY OF THE INVENTION An image synthesizing method according to the present invention is an image synthesizing method for synthesizing computer graphics and a real shot video, wherein a step of extracting a specific area in the real shot video is performed. The main feature is to include a process of adding information of a three-dimensional shape and a process of computerizing information relating to the extraction region into a computer graphics model based on the information of the extraction region and the information of the three-dimensional shape.

Further, the method further comprises a step of synthesizing the computer graphics model information and the other computer graphics models so that the information on the extraction region and other computer graphics models are mixedly displayed. Then, the same processing is executed over a plurality of frames to create a moving image.

An image synthesizing apparatus according to the present invention is an image synthesizing apparatus for synthesizing computer graphics and a real photographed image, means for extracting a specific area in the real photographed image,
The main feature is that it comprises means for adding information of a three-dimensional shape to an extracted area, and means for computer graphics modeling of information relating to the extracted area based on the information of the extracted area and the information of the three-dimensional shape. I do.

Further, there is provided means for synthesizing the computer graphics modeled information relating to the extraction area and another computer graphics model so as to display them together.

An area corresponding to a specific object is cut out from the actual photographed video, and three-dimensional shape information is added to this. As a result, a CG model having the surface attribute of the specific object in the actual video is generated. Use it alone or mix it with other CG models. Further, by performing the same processing over a plurality of frames, image synthesis with a moving image can be performed.

[0023]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the drawings showing the embodiments. FIG. 4 is a block diagram of an apparatus for implementing the method of the present invention, and FIG. 5 is a flowchart of the processing. In FIG. 4, reference numeral 9 denotes a video supply device such as a TV camera, a video tape, or a video disc. The real image information stored in the image storage unit 5 is supplied to the specific object region extracting unit 1, where a specific object region in the real image is extracted. The operator designates the extraction area using the pointing device 12 such as a mouse. The specific contents are detailed in the description of the inventions of the second to fourth groups. FIG. 6 shows a case where a quadrangular prism and a triangular pyramid are displayed as live-action images.
This shows a state in which a prism is designated as an extraction area (shown by a thick line). The video information of the area designated in this way is sent to the three-dimensional shape information adding unit 2, where the three-dimensional shape information is added, and stored in the shape / surface attribute information storage unit 6.

The three-dimensional shape information adding unit 2 adds three-dimensional shape information to the information provided from the specific object region extracting unit 1 and stores the information in the shape / surface attribute information storage unit 6. Although the specific configuration of the three-dimensional shape information adding unit 2 is detailed in the second group of the invention, an example will be described here.

FIG. 7 is a flowchart. First, as shown in FIG. 8, an area or an object designated and extracted is displayed on the screen of the image display device 10, and the operator inputs the focal length f of the image (S1). For this input, a character / numerical input device 11 such as a keyboard is used. Next, a ridge line is drawn on the screen using the pointing device 12, and a value of the depth is input (S2). In FIG. 8, the drawn ridgeline is indicated by a thick line, and the value of the designated depth is indicated by Z. This drawing and depth designation can be canceled and corrected. Since the surface can be basically specified by a triangle (three ridges), the present invention also draws the ridges so as to be divided into triangles as shown in FIG. 9 and also assists the division into triangles. Draws a line (diagonal line of a rectangle), and all points in the extraction area are surrounded by three ridge lines without crossing two different ridge lines (including auxiliary lines). State (S3).

Next, the three-dimensional coordinates of the end point are calculated (S4).
X = (x / f) × Z Y = (y / f) × Z based on the focal length f, the depth Z, and the coordinates (x, y) of the end point on the image input in S1 and S2. . Note that the three-dimensional coordinates of a point on the ridge line and a point in a region surrounded by the ridge line can be calculated by the following equation.

Point on Edge Line The three-dimensional coordinates of the edge point of the edge line and the coordinates on the image are respectively
(X _i , y _i ), (X _i , Y _i , Z _i ) (i = 1, 2)
Then, the three-dimensional coordinates (X, y) of the point (x, y) on the ridge line
Y, Z) is obtained by X = (1-t) X 1 + tX 2 Y = (1-t) Y 1 + tY 2 Z = (1-t) Z 1 + tZ 2. Here, t is (x−x) when x ₁ ≠ x _2.
x ₁ ) / (x ₂ −x ₁ ), and when x ₁ = x ₂ , (y−y)
₁ ) / (y ₂ −y ₁ ).

A point in the area surrounded by the ridgeline Each point in this area is surrounded by three ridgelines,
Their intersection is guaranteed to be the endpoint of the three ridges. The plane of the three intersections of coordinates thus can be determined by three intersections of the coordinate _{(X i, Y i, Z} i) (i = 1,2,3) ( trivial). Assuming that the equation of this plane is aX + bY + cZ-1 = 0, the three-dimensional coordinates (X, Y, Z) of the coordinates (x, y) on the image of the point in the area are: X = x / (ax + by + cf) Y = y / (ax + by + cf) Z = f / (ax + by + cf)

From the ridge lines obtained in this manner, the three-dimensional coordinates and connection relations of the end points are used as shape information, the coordinates of the end points on the specific object area, and the image data of the specific object area image are used as surface attribute information.・ Surface attribute information storage unit 6
(S5). Table 1 shows the storage contents of the shape / surface attribute information storage unit 6. The above processing of S1 to S5 is repeated for all frames (S6).

[0030]

[Table 1]

Next, the contents of the shape / surface attribute information storage unit 6 are converted into CG models by the video CG model generation unit 3. CG of shape information
With respect to modeling, a CG model can be generated as it is by regarding the end points as vertices, the ridge lines as sides, and the enclosed portion as a surface from the connection relation of the ridge lines and the three-dimensional coordinates.

On the other hand, regarding the surface attribute information, as the surface attribute information of the portion regarded as a surface, the image information corresponding to the position is used as the texture of the CG model to be generated. At that time, the image information is normalized as an image viewed from a normal direction in a three-dimensional space. The rotation matrix R at the time of normalization is given by the following equation.

[0033]

(Equation 1)

Here, the rotation angle ψ and the rotation angle κ are based on a, b, and c when the plane equation of this area is expressed as aX + bY + cZ−1 = 0.

[0035]

(Equation 2)

Is as follows. a, b, and c can be obtained from the three-dimensional coordinates (X _i , Y _i , Z _i ) of three vertices (i = 1, 2, 3). Such a CG modeling process is applied to all the frames, a CG model for the actual video is obtained as a CG model sequence for each frame, and this is stored in the video CG model storage unit 7b. The CG model creation unit 13 creates a normal CG model that is not created from the above-described actual shot video, and the created CG model is stored in the CG model storage unit 7a.

The composite information storage unit 8 stores information (CG model arrangement information) for synthesizing the CG model and the video CG model created from the actually shot video in the composite image generation unit 4. The unit 4 synthesizes both CG models based on this, and causes the image display device 10 to display them,
Alternatively, it is recorded on a recording medium (not shown). The composite image generation unit 4 and the composite information storage unit 8 are described in detail in the fifth group of the invention.

[0038]

As described above, in the case of the present invention, a real image is converted into a CG model having a three-dimensional shape.
The same handling as a model can be performed, and processing such as synthesis can be easily performed. The CG model of the actual image can be formed by a simple operation of extracting a region and adding three-dimensional shape information thereto. At this time, since manual intervention by the operator is possible, fine adjustments or intentional changes can be made, and the degree of freedom is increased. In addition, by processing a plurality of frames in the same manner, application to moving images is also possible.

[Invention of Second Group] (Summary) The present invention relates to the configuration of the specific object region extracting unit 1 and the three-dimensional shape information adding unit 2 shown in FIG.

[0040]

2. Description of the Related Art A method for completely extracting the shape of a three-dimensional object in an image has not yet been established. The prior art is proposed assuming the reflection characteristics of the surface of the object, a method of calculating the inclination of the surface of the object from the observed color values, and storing a model of the object to be observed in an image in advance. There is a method of collating the appearance of an object observed in a model and an image, and the like. These have been developed with the development of image understanding research.

However, none of these methods can be applied unless the conditions are met. For example, the former method cannot be applied to an object whose reflection characteristics cannot be assumed, and the latter method cannot be applied to an object of a model that is not stored. Therefore, in the present invention, when obtaining the three-dimensional shape of an object included in a video, a human specifies an approximate shape of the object, and while displaying a model of the shape on an image, a method of superimposing is designated by a human. Another object of the present invention is to provide a method and apparatus for extracting a three-dimensional shape, in which automatic adjustment by a computer using an image processing technique is alternately and interactively performed.

[0042]

The method of the second group of the invention comprises:
In a method for extracting a three-dimensional shape of an object included in a real video, a process of preparing data of a plurality of geometric shapes in advance, and a process of extracting a region corresponding to the object from the real video, A step of selectively displaying one of the shapes according to the data on a screen displaying an area,
Position of the shape to match the region and shape,
Adjusting the orientation and size.

In a second group of the invention, there is provided an apparatus for extracting a three-dimensional shape of an object included in a real video, wherein a plurality of data of a plurality of geometric shapes are prepared in advance; Means for extracting a region corresponding to the object;
A means for selecting and displaying one of the shapes on the screen displaying the extracted area based on the data; and adjusting means for adjusting the position, orientation, and size of the shape so that the area and the shape match. It is characterized by.

The adjusting means includes automatic adjusting means for adjusting the position, orientation and size based on the shape and hue value of the object.
Further, there is provided a means for mapping image information of an area extracted from the actual photographed video to the adjusted shape.

A shape close to the region of the extracted object is selected from the prepared shapes and displayed. Then, the three-dimensional shape of the target object can be extracted by adjusting the region to match the shape. This result is similar to the result obtained by the above-described three-dimensional shape information adding unit 2.

[0046]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The second group of the invention will be described below in detail with reference to the drawings showing the embodiments. FIG. 10 is a block diagram of the three-dimensional shape extraction device. In the figure, reference numeral 21 denotes an object area extracting unit, which extracts an area of a required object from a photographed video and causes the image display device 27 to display the extracted area. This is detailed in the third group of inventions. The actual video and the extracted video are stored in the image storage unit 25. The basic shape selection unit 22 stores a large number of basic shape patterns as shown in FIG. 12, and the operator selects this and displays it on the image display device 27. The basic shape superimposing unit 23 compares the pattern of the basic shape selected by the operator and the image of the extracted object in FIG.
Are displayed in a superimposed manner as shown in FIG. 1 and both are matched by an operation of an operator and automatic adjustment by a computer as described later. The result of superimposition so as to match is stored in the superimposition information storage unit 26. The superimposition result display unit 24 maps the image information of the surface of the extracted object onto the adjusted basic shape.

Next, a three-dimensional shape extraction method will be described with reference to FIG. The real image taken out from the image storage unit 25 is taken out and displayed on the image display device 27, and the object region extracting unit
At 21, required objects are extracted (S 21). FIG. 14 is an explanatory diagram of this operation. The operator draws closed curves each exemplifying the object region and the background region on the screen using the drawing device. The object area extraction unit 21 expands the closed curve of the object area and contracts the closed curve of the background area. The expansion and contraction are recognized only for the parts having similar hues. Then, both closed curves come into contact at the boundary, whereby the boundary is specified and the region of the required object is extracted. As shown in FIG. 14B, as a result of expansion and contraction, a boundary may be recognized as being thick due to the presence of a shadow or the like. In this case, the inner line of the thick boundary is set as the boundary of the object.

Next, a pattern of the basic shape (shape primitive) is displayed by performing a predetermined operation, and a pattern similar to the shape of the extracted object is selected (S22). As a result, superimposed display is performed as shown in FIG. 13 (S23). Generally, the extracted object shape does not match the selected basic shape. Therefore, until the two coincide (S24), the position adjustment (S27), the orientation adjustment (S28), and the size adjustment of the object shape and the basic shape are performed.
(S29) and partial deformation of the shape (S30). The agreement and disagreement are determined by the operator.

FIG. 15 is a flowchart of a process relating to position movement, and FIG. 16 is an explanatory diagram thereof. The principle of this movement is to match the center of gravity of the object region with the shape primitive wireframe. That is, calculation of the center of gravity G _RO of the object area (S31)
, And the center of gravity G _RP wireframe shape primitives
Is calculated (S33). Then, the display position of the wire frame of the shape primitive is moved so as to match these.
(S32).

The display position of the shape primitive wire frame is moved so that the center of gravity R _{RO of the} object region matches the center of gravity R _{RP of the} region surrounded by the wire frame display of the shape primitive (FIG. 16). Incidentally, the center of gravity G _R region R can be obtained by _{G R = (m 10 / m} 00, m 01 / m 00) T. Here, m ₀₀ m ₀₀ = ∫fdR indicates the number of points constituting the region R. This value represents the area. m ₁₀ m ₁₀ = ∫xdR Indicates the sum of the x-coordinates of the points constituting the region R. m ₀₁ m ₀₁ = ∫ydR Indicates the sum of the y-coordinates of the points constituting the region R. In accordance with this, the wire frame display position of the shape primitive is moved by (G _RO -G _RP ).

FIG. 17 is a flowchart showing a process of rotating a shape primitive for adjusting the direction, and FIG. 18 is an explanatory diagram thereof. The principle of the orientation adjustment by rotation lies in the parallelization of the long axes of the object region and the shape primitive wireframe. The major axis direction θ _R of the region _R can be obtained as a principal axis of inertia around the center of gravity of the region _R. That is, tan ² θ _R + [｛m ₂₀ −m ₁₀ / m ₀₀ ) − (m ₀₂ −m ₀₁ /
m ₀₀ )｝ / (m ₁₁ −m ₁₀ m ₀₁ / m ₀₀ )] tan θ _R −1 =
0 may be obtained as a solution. Here, m ₁₁ m ₁₁ = ∫xy R Indicates the total sum of the products of the x coordinate and the y coordinate of each point constituting the region R. m ₂₀ m ₂₀ = ∫x ² dR Indicates the sum of squares of the x-coordinate of each point constituting the region R. m ₀₂ m ₀₂ = ∫y ² dR Indicates the sum of squares of the y coordinate of each point constituting the region R. In accordance with this, the wireframe display position of the shape primitive is rotated by (θ _RO −θ _RP ).

As shown in the flowchart of FIG. 17, the moment amounts of the extracted object region and the shape primitive wire frame are calculated (S41, S44). The moment calculation of the amount of _{^{m ij = ∫x i y j dR}} (ij) = {(0,0), (1,0), (0,1), (1,1), (2,0), (0,2)｝.

On the other hand, as described above, the long axis of the object area
And the long axis and weight of the shape primitive wireframe
Heart G_RPIs obtained as described above (S42, S45, S46). Soshi
G _RPΘ around_RO-R_RPOnly shape primitives
The wireframe display position is rotated (S43).

[0054] Adjustment of the size is performed by enlarging or reducing the wireframe shape primitives around the center of gravity G _RP regions R _P as display area therebetween, as shown in FIG. 20 coincide. That is, the area S _{R of the} region R can be obtained as S _R = m ₀₀ . In accordance with this, the shape primitive wire frame may be multiplied by S _RO / S _RP . Where S _RO is the area of the object region S _RP is the area of the shape primitive wireframe.

In the flowchart of FIG. 19, the moment amount of the region of the extracted object and the moment amount of the wire frame display region of the shape primitive are calculated (S51, S54).
. Then, using these, the areas S _RO and S _RP of both are calculated (S52, S55). Further, the center of gravity G _RP of the shape primitive wire frame is calculated (S56). Then, the shape primitive wire frame is multiplied by S _RO / S _RP (S53). Area can be determined as a moment amount m _00.

Returning to FIG. 11, deformation of the shape will be described. If the shape of the object area is partially different from the basic shape, the basic shape is partially deformed by a command input by the operator. When the object area and the basic shape match as described above, this is stored in the overlay information storage unit 26 (S25).
. Then, as shown in FIG. 21, the image information of the extracted object region is mapped on the wire frame of the shape primitive (S26). In other words, a required portion of the actual video is cut out and attached to the shape primitive wire frame.

[0057]

According to the invention of the second group as described above, since the three-dimensional shape can be extracted by the operator in an interactive manner, there is no restriction on application conditions, and known information (reflection information, etc.) on the object is required. Extraction is possible without performing.
In addition, since the operation of matching the basic shape with the object area is automatically performed by the computer, the burden on the operator is light.
In addition, since the mapping is performed, the suitability of the extracted three-dimensional shape information can be intuitively determined.

[Invention of Third Group] (Summary) The invention of the third group is a specific object region extracting unit or FIG.
This relates to ten object region extraction units.

[0059]

2. Description of the Related Art An image is electrically synthesized as shown in FIG. For example, an image of a person is photographed by the image input unit A as a blue background, and a landscape image is photographed by the image input unit B. Then, a blue component is detected from the image of the image input unit A, the blue component is inverted and amplified, and an appropriate mixing ratio control is performed.
And a signal from the multiplexing unit, and synthesizes the signal with a mixing amplifier and outputs the synthesized signal to an image output unit. As a result, the background from the image input unit A disappears, and a person image with the image input unit B as the background is synthesized.

[0060]

In such a conventional system, a blue background is required and there is a burden on facilities. In addition, only those intended for image synthesis can be used from the beginning, and lack versatility. Further, it is difficult to set parameters for synthesis in the mixing amplifier, and the operation is complicated. The present invention has been made to solve such a problem,
It is an object of the present invention to provide an image synthesizing apparatus which does not require any special photographing equipment, has high versatility, and is easy to operate, and particularly, a key image generating apparatus to be synthesized.

[0061]

According to a first method of a third group of the invention, there is provided a method of extracting a specific area from an image, wherein a plurality of pixels in the area to be extracted are designated.
A step of obtaining a predetermined feature amount of the designated pixel, a step of obtaining a maximum value and a minimum value of the obtained feature amount, and a step of obtaining a feature amount for pixels inside and outside the region to be extracted. Selecting a pixel between the maximum value and the minimum value, wherein a region formed by the pixel is set as an extraction region.

A second method is a method of extracting a specific area from a video, in which a plurality of pixels in an area to be extracted are specified, a step of obtaining a predetermined feature amount in the specified pixel, The process of calculating the difference between adjacent pixels for the obtained feature amount of the specified pixel, the process of obtaining the maximum value of the calculated difference, and the process of calculating the difference between the feature amounts of adjacent pixels starting from the specified pixel. Connecting pixels near four or eight pixels smaller than the maximum value, wherein a region formed by the connected pixels is set as an extraction region.

A third method is to assign different values to the pixels in the extraction region and the pixels in the non-extraction region, and to assign a value intermediate between the values to a pixel located at the outer edge of the boundary of the extraction region. And a step of generating an image based on the assigned values.

In a fourth method, a different intermediate value is assigned to each of a plurality of pixels adjacent in the direction away from the extraction area.

A fifth method is a method of extracting a specific area from a video, in which a plurality of pixels in an area to be extracted are specified, a step of obtaining a predetermined feature amount in the specified pixel, Calculating the maximum value and the minimum value of the determined feature amount; determining the feature amount for pixels inside and outside the region to be extracted; and determining whether the feature amount is in the range between the maximum value and the minimum value. Determining, whether to assign a constant K to the pixels within the range, and calculating the difference between the feature value of the pixel outside the range and the maximum or minimum value,
Assigning a value obtained by subtracting a value determined in relation to the difference from the constant K to the pixels outside the range, and generating an image based on the assigned values.

A sixth method is a method of extracting a specific region from a video, in which a plurality of pixels in a region to be extracted are specified, and a plurality of predetermined feature values in the specified pixel are obtained. Obtaining the maximum value and the minimum value of the obtained characteristic amount for each of the characteristic amounts, obtaining the characteristic amount for the pixels inside and outside the region to be extracted, and calculating the characteristic amount between the maximum value and the minimum value. Determining whether the pixel is within the range, providing a constant K to pixels within the range, and calculating a difference between the feature value of the pixel outside the range and the maximum value or the minimum value. And assigning a value obtained by subtracting a value determined in relation to the difference between the respective characteristic amounts from the constant K to pixels outside the range, and generating an image based on these assigned values. And

A seventh method is a method of extracting a specific area from a video, in which a plurality of pixels in an area to be extracted are specified, a step of obtaining a predetermined feature amount in the specified pixel, The process of obtaining the average value and variance of the obtained feature amount, and the feature amount of pixels inside and outside the region to be extracted is determined, and it is determined whether the feature amount is in a range determined by the average value and variance. A step of assigning a constant K to pixels within the range, a step of calculating a deviation between the feature amount of the pixels outside the range and the average value, and a step of calculating the deviation from the constant K for the pixels outside the range. And giving a value obtained by subtracting a value determined in relation to the image data, and generating an image based on the given value.

An eighth method is a method for extracting a specific region from a video, in which a plurality of pixels in a region to be extracted are specified and a plurality of predetermined feature values in the specified pixel are obtained. And a step of obtaining an average value and a variance of the obtained characteristic amount for each of the characteristic amounts, and obtaining a characteristic amount for pixels inside and outside the region to be extracted, so that the characteristic amount falls within a range determined by the average value and the variance. A step of determining whether there is a pixel, a step of assigning a constant K to pixels within the range, a step of calculating a deviation between a feature value of the pixel outside the range and the average value, and a step of Assigning a value obtained by subtracting a value determined in relation to the deviation of each feature amount from the constant K, and generating an image based on these assigned values.

A ninth method is a process of designating a pixel included in any one of a plurality of regions extracted from an image, and a process of performing 4-link or 8-link labeling with the designated pixel as a starting point. And a step of changing an unlabeled area into a non-extracted area.

A tenth method is a method of extracting a specific region from a plurality of frames of video, in which a plurality of pixels in a region to be extracted in one frame are specified, and the specified pixel is used as a starting point. A step of performing 4-connected or 8-connected labeling, a step of changing an unlabeled area to a non-extracted area, and a step of calculating a geometric feature of the extracted area, and labeling in the next frame. Calculating the geometric feature for each of the regions with different labels, and leaving the region having the geometric feature close to the geometric feature of the extraction region of the previous frame as the extraction region. Changing other areas to non-extraction areas.

An eleventh method is a method for extracting a specific area from a video of a plurality of frames, in which a plurality of pixels in an area to be extracted in one frame are specified, and the specified pixel is used as a starting point. A step of labeling four or eight links, a step of changing an unlabeled area into a non-extracted area, and a step of calculating an optical feature of the extracted area, and a step of labeling in the next frame And a process of calculating an optical feature amount for each of the regions with different labels, and a region having an optical feature amount close to the optical feature amount of the extraction region of the previous frame is left as an extraction region, and other regions To a non-extraction area.

The first apparatus is an apparatus for extracting a specific area from an image, a means for designating a plurality of pixels in an area to be extracted, a means for obtaining a predetermined feature amount of the specified pixel, Means for obtaining the maximum value and the minimum value of the obtained feature amount; means for storing the maximum value and the minimum value; and obtaining the feature amount for pixels inside and outside the region to be extracted. Means for selecting a pixel between a maximum value and a minimum value, wherein a region formed by the pixel is set as an extraction region.

The second device is a device for extracting a specific region from an image, a device for designating a plurality of pixels in a region to be extracted, a device for obtaining a predetermined feature amount of the designated pixel, Means for calculating the difference between adjacent pixels for the calculated feature amount of the specified pixel, means for calculating the maximum value of the calculated difference, means for storing the maximum value,
Means for linking pixels in the vicinity of 4 or 8 in which the difference in the feature value between adjacent pixels is smaller than the maximum value with the designated pixel as a starting point, and a region formed by the connected pixels is defined as an extraction region. It is characterized by what it does.

The third device includes means for assigning different values to the pixels in the extraction area and pixels in the non-extraction area, and a pixel located at the outer edge of the boundary of the extraction area, , And generating an image based on the assigned values.

The fourth device is adapted to give a different intermediate value to each of a plurality of pixels adjacent in the direction away from the extraction area.

A fifth device is a device for extracting a specific region from a video, which is a device for designating a plurality of pixels in a region to be extracted, a device for obtaining a predetermined characteristic amount of the designated pixel, Means for obtaining the maximum value and the minimum value of the obtained feature amount; means for storing the maximum value and the minimum value; and obtaining the feature amount for pixels inside and outside the region to be extracted. Means for determining whether or not the pixel is in a range between the minimum value and a minimum value.
Means for calculating the difference between the feature value of the pixel outside the range and the maximum value or the minimum value, and subtracting a value determined in relation to the difference from a constant K for the pixel outside the range. Means for assigning the assigned values to generate an image based on the assigned values.

The sixth device is a device for extracting a specific area from a video, wherein the means for specifying a plurality of pixels in the area to be extracted and the means for obtaining a plurality of predetermined feature values in the specified pixel. Means for obtaining the maximum value and the minimum value of the obtained characteristic amounts for each of the characteristic amounts; means for storing these maximum values and the minimum values; and obtaining the characteristic amounts for the pixels inside and outside the region to be extracted. Means for determining whether or not the feature value is in a range between the maximum value and the minimum value; means for assigning a constant K to pixels within the range; Means for calculating a difference from a maximum value or a minimum value, and means for assigning a value obtained by subtracting a value determined in relation to the difference between the respective characteristic amounts from a constant K to a pixel outside the range, To generate an image with the value given by And wherein the Rukoto.

A seventh device is a device for extracting a specific region from a video, a device for designating a plurality of pixels in a region to be extracted, a device for obtaining a predetermined characteristic amount of the designated pixel, Means for obtaining the average value and variance of the obtained feature amount; means for storing the average value and variance; and obtaining the feature amount for pixels inside and outside the region to be extracted.
Means for determining whether or not the feature value is within a range determined by the average value and the variance; means for assigning a constant K to pixels within the range; and a feature value of pixels outside the range and the average value. Means for calculating a deviation, and means for applying a value obtained by subtracting a value determined in relation to the deviation from a constant K to pixels outside the range, so as to generate an image based on the value provided. There is a feature.

An eighth device is a device for extracting a specific area from a video, wherein the means for specifying a plurality of pixels in the area to be extracted and the means for obtaining a plurality of predetermined feature values in the specified pixel. Means for obtaining the average value and variance of the obtained feature amount for each of the feature amounts; means for storing the average value and variance; and obtaining the feature amount for pixels inside and outside the region to be extracted. Means for determining whether the amount is within a range determined by the average value and the variance; means for assigning a constant K to pixels within the range; and deviation between the feature amount of pixels outside the range and the average value. And a means for assigning a value obtained by subtracting a value determined in relation to the deviation of each feature value from a constant K to a pixel outside the range, and generating an image using these assigned values. Characterized by what they do .

The ninth apparatus is a means for designating a pixel included in any one of a plurality of areas extracted from an image, and a means for labeling four or eight links with the specified pixel as a starting point. And means for changing an unlabeled area into a non-extracted area.

A tenth apparatus is an apparatus for extracting a specific area from a plurality of frames of video, wherein a means for specifying a plurality of pixels in an area to be extracted in one frame, Means for performing 4-connected or 8-connected labeling, means for changing an unlabeled area to a non-extracted area, and means for calculating a geometric feature value of an extracted area, and labeling in the next frame. Means, means for calculating a geometric feature amount for each of the regions provided with different labels, and leaving an area having a geometric feature amount close to the geometric feature amount of the extraction area of the previous frame as the extraction area. Means for changing another area to a non-extraction area.

An eleventh apparatus is an apparatus for extracting a specific area from a plurality of frames of video, wherein a means for specifying a plurality of pixels in an area to be extracted in one frame, Means for performing four-link or eight-link labeling, means for changing an unlabeled area into a non-extracted area, means for calculating an optical characteristic amount of the extracted area, and means for labeling in the next frame And means for calculating an optical characteristic amount for each of the regions with different labels, and a region having an optical characteristic amount close to the optical characteristic amount of the extraction region of the previous frame is left as an extraction region, and other regions To a non-extraction area.

In the first method and apparatus, a plurality of pixels in an image portion to be extracted are designated (tracing with a light pen or a cursor operated by a mouse). The feature amount (one or more of R, G, B, hue, saturation, lightness, brightness, etc.) in the traced pixel group is obtained, and the maximum value and the minimum value are selected and stored.

Next, it is checked whether or not each feature amount is within the range between the maximum value and the minimum value for each pixel of the entire image. Since the pixels within the range have the same feature amount as the image portion desired to be extracted, it is determined that the image portion belongs, and a value larger than 0 is assigned, and the pixel outside the range is a non-extracted portion. Is given as 0. Thus, a desired image portion can be extracted by extracting a non-zero portion.

In the second method and apparatus, the difference between the characteristic value of the traced pixel and the characteristic value of the adjacent pixel is obtained, and the maximum value is stored. Pixels in the vicinity of 4 or 8 having a difference equal to or less than the maximum value are traced and have a threshold value close to that of the region desired to be extracted. Gives 0. Thereby, a non-zero portion can be extracted.

The third method and apparatus give an intermediate value between 1 and 0 to the pixel at the boundary between them when 1 is added to the extraction area and 0 is added to the non-extraction area. This makes the boundaries milder,
When the extracted images are combined, they blend well into the background.

In the fourth method and apparatus, the boundary is further mildened by setting a plurality of intermediate values.

The fifth method and apparatus are for adaptively controlling the mildness of the boundary, and determine the intermediate value of 1 to 0 by determining the feature value of the non-extracted area and the maximum value (or minimum value) of the feature value. Is determined according to the difference between Thereby, the boundary of the extracted image blends well into the background.

In the sixth method and apparatus, the feature amount is not one kind but two or more kinds, and the above difference is obtained for a plurality of feature amounts, and an intermediate value is determined based on, for example, a weighted average thereof. Since a plurality of feature values are used, a more natural boundary can be obtained.

The seventh and eighth methods and apparatuses are different in that the fifth and sixth methods use the maximum value and the minimum value, whereas the variance is used.

In the ninth method and apparatus, an excessively extracted portion is set as a non-extraction area. That is, different labels are assigned to a plurality of similarly extracted regions by labeling. this house,
Only the area including the traced pixel is left and the others are erased.

The tenth method and apparatus correspond to moving images. Labeling is performed in the same manner as in the ninth method and apparatus, and the non-extraction area is deleted. Similar labeling is performed in the next frame, but regions between frames are identified based on the similarity of the geometric feature values. Therefore, only the extraction region remains, and the others disappear. By performing this over a plurality of frames, the extraction processing for a moving image can be automatically performed.

The eleventh method and apparatus use optical features instead of the above-mentioned geometric features, and have the same effect.

[0094]

FIG. 23 is a block diagram of a first area extracting apparatus according to a third group of the present invention. Three image input units 31,3
The NTSC-RGB converters 31a, 32a, and 33a convert the NTSC signal into an analog RGB signal, and convert the analog RGB signal into a digital RGB signal.
A / D converters 31b, 32b, and 33b are provided. These image input units 3
Inputs from 1, 32 and 33 are applied to image memories 37, 38, 39 and 40 comprising a dual-port RAM, and image data read from these is applied to an image output unit 34 and output therefrom. The image output unit 34 converts a digital RGB signal from the image memory 37 or the like into an analog RGB signal, a D / A converter 34b, and converts the converted analog RGB signal into an NT RGB signal.
Monitors the output of RGB-NTSC converter 34a that converts to SC signal
(Not shown).

Reference numeral 35 denotes a coordinate input unit, which includes a light pen and its coordinate recognition means, and is used to trace a part of the image displayed on the monitor. The coordinate information input by the coordinate input unit 35 is input to the processing unit 41. 400 is a semiconductor memory, a memory used for calculation
42, a feature amount upper limit register 43 and a lower limit register 44 described later. Reference numeral 36 denotes a large-capacity recording unit such as a hard disk or a magneto-optical disk, which records images of a plurality of frames.

The processing section 41 composed of a microprocessor or the like performs the following processing for area extraction. Fig. 24
Is a flowchart showing the procedure of this processing, and FIG. 25 is an explanatory diagram thereof. The area to be extracted as shown in Fig. 25 (a)
(Shown in white) is traced with the pen of the coordinate input unit 35. During this time, the feature amount (one or more types) is calculated for a plurality of pixels of the pen locus. Then, the maximum value or the minimum value of the feature value is stored in the feature value upper limit register 43 and the lower limit register 44, respectively. This is performed by updating the stored contents of the register each time the pen moves. After the tracing, the maximum and minimum values of the feature values in all the trajectories are obtained. The feature amounts include R, G, B, hue, saturation, lightness, luminance, and the like.

Next, the pixel dot feature amount of the entire screen
(One or a plurality of types), and a value larger than 0 (for example, 255) is assigned to a pixel within the range from the maximum value to the minimum value, and 0 is assigned to a pixel outside the range. As a result, a key image, that is, an image including the extraction region is obtained. FIG.
(b) illustrates this. If the extraction cannot be performed as desired, a retry may be performed by variously changing the selection or combination of the feature amounts. By repeating the above process for a plurality of frames, a moving image can be processed.

FIG. 26 is a block diagram of the second region extracting apparatus. What differs from the first region extraction device is the semiconductor memory.
400 is provided with a feature amount threshold register 45 in place of the feature amount upper limit register 43 and the lower limit register 44. Since other configurations are the same, the same reference numerals are given and the description is omitted.

FIG. 27 is a flowchart of the processing by the processing section 41, and FIG. 28 is an explanatory diagram thereof. As shown in FIG. 28 (a), the feature value of the pixel of the locus traced by the pen is obtained in the same manner as in the first device, but the feature between the adjacent pixels among the traced pixels in the second device. The difference between the quantities is calculated, and the maximum value is stored in the threshold register 45. Then, for each of the traced pixels, it is checked whether or not the adjacent pixels (near four or eight) are equal to or smaller than a threshold value, and the following adjacent pixels are successively connected (FIG. 28 (b)). A value greater than 0 is assigned to the areas connected in this manner.

FIG. 29 is a block diagram of the third to sixth region extracting devices. The difference from the first apparatus is the operation content of the processing unit 41, which performs a mixture ratio calculation 41a and a total mixture ratio calculation 41b described below. FIG. 30 is a flowchart showing the processing procedure of the processing unit 41. As shown in FIG. 31 (a), for the pixel of the locus traced by the pen, the feature value is calculated in the same manner as in the first device, and the maximum value and the minimum value are respectively set to the feature value upper limit register 43,
Put into lower limit register 44.

Next, a feature amount is calculated for all the pixels of the image, and it is checked whether or not this is within the range of the maximum value and the minimum value stored in the upper and lower limit value registers 43 and 44, respectively.
If it is within the range, a non-zero value K is assigned. If the difference is outside the range, the difference between the calculated feature value and the maximum value (when the feature value is large) or the difference between the calculated feature value and the minimum value (when the feature value is small) is calculated. According to K
A value (mixing ratio) in a range from 0 to 0 is obtained for each feature amount. Then, an overall mixture ratio is obtained by weighted averaging the mixture ratio for each feature amount. Then, a value corresponding to the total mixture ratio is given to the corresponding pixel. Then, as shown in FIG. 31 (b), an extracted image with a gradation added to the boundary is obtained. By repeating this for a plurality of frames, a moving image can be handled.

FIG. 32 is a block diagram of the third, fourth, seventh and eighth region extracting devices. 29 is different from FIG. 29 in that the feature amount upper limit register 43 and the lower limit register 44 are replaced with a feature amount average value register.
46, a feature value variance value register 47 is provided. FIG. 33 is a flowchart showing a processing procedure in this case.
As shown in (a), the feature amount of the pixel of the locus traced by the pen is calculated, the average value and the variance value are calculated, and these are stored in the feature amount average value register 46 and the feature amount variance value register 47. deep.

The apparatus checks whether or not the feature amounts of all the pixels in the image are within a predetermined deviation (for example, the average value ± variance value). If the feature amount is within the range, K is added. If the value is out of the range, the mixture ratio is calculated for each feature value according to the deviation from the average value, a weighted average of the calculated values is obtained as an overall mixture ratio, and K is assigned according to the calculated average.

FIG. 34 (b) shows the result, and an extracted image having a gradation at the boundary is obtained. FIG. 35 shows the third,
FIG. 14 is a block diagram of another embodiment of the region extracting device of No. 4; This device is different from the other devices in the processing of the processing unit 41. This processing will be described with reference to FIGS. In this embodiment, the outline (between pixels) of an extraction region obtained by the first region extraction device or the like and a non-extraction region (given a value of 0)
A value obtained by subtracting a constant K from the value X of the inner peripheral contour point (pixel) inside the image is given to the outer peripheral contour point (pixel) outside the contour. This processing may be performed only on one pixel in the centrifugal direction, but by performing the processing on a plurality of pixels as shown in FIG. 38, a smoother edge can be obtained.

FIG. 36 is a flow chart showing the procedure of this processing. The processing is repeated so as to trace the contour from the upper left corner on the screen. By performing this processing for a plurality of frames, it is possible to deal with moving images.

FIG. 39 is a block diagram of a ninth region extracting apparatus. This apparatus deletes noise, that is, a part that appears as an extraction area when it is not originally desired to extract, by performing a labeling processing (48) described later in the processing unit 41. This process is performed, for example, on a key image (FIG. 41 (a)) obtained by the first region extraction device or the like. This image includes a noise region (non-zero region) having the same characteristic amount other than the portion where the center is desired to be extracted.

FIG. 40 is a flowchart of this process.
In this processing, four-link or eight-link labeling is performed with the pixel of the locus traced by the pen as a starting point. Since the non-zero region of noise is discrete, labeling does not reach that region. Then, the unlabeled area is erased. Then, a desired extraction area is obtained as shown in FIG. 41 (b).

FIG. 42 is a block diagram of a tenth region extracting apparatus. This device performs the same processing as the ninth region extraction device for one.
The subsequent frames are for moving images that can be subjected to noise elimination by simple processing only by performing this operation once. To make this possible, the process of calculating the geometric features (e.g., area, center position) of the traced area and the area of the other frame associated with it (49) The processing unit 41 performs the association (50) of associating in the frame. FIG. 43 is a flowchart of this process, and FIG. 44 is an explanatory diagram thereof. Ninth
The same processing as in the region extraction device of
The noise is eliminated as shown in (a). Then, for the remaining non-zero area, that is, the extracted area, the geometric feature amount is calculated.

Next, in the second frame, a geometric feature value is calculated for a non-zero area (including a noise area). Then, the one having the geometric feature amount closest to the feature amount of the non-zero region of the first frame is selected, and the others are deleted (the non-extraction region). Thereafter, the same process is repeated for the two preceding and succeeding frames, so that the noise region automatically disappears.

FIG. 45 is a block diagram of an eleventh region extracting apparatus. This device uses optical (texture) features, whereas the tenth device uses geometric features to identify regions in a frame. Therefore, the processing unit 41
Of these, pixel value configuration calculation (5
Do 1). FIG. 46 is a flowchart of noise elimination, and FIG. 47 is an explanatory diagram thereof. In each case, the geometric feature is merely replaced with the optical feature, and the description is omitted.

[0111]

According to the third group of the invention as described above, the photographing equipment for the blue background is unnecessary. In addition, extraction can be performed even from a video that is not particularly conscious of image synthesis. The user only has to trace the necessary parts, and the operation is simple.

[Invention of the Fourth Group] (Summary) The invention of the fourth group is a display method and apparatus for easily processing or changing the display mode of a CG model (including a three-dimensional shape model extracted from a video). To provide.

[0113]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional character display method for rotating, enlarging, reducing and translating a three-dimensional shape model two-dimensionally displayed on a display by an interactive method and sequentially displaying the results. . With the speeding up of the computer, it is possible to rotate, scale and translate the 3D shape model in real time and display it, so that humans can interactively operate the 3D shape model and redisplay the results. Functions are required. Therefore, an operation method for rotating, enlarging, reducing, and translating the three-dimensional shape model without hindering human thinking is required.

In order to convert a three-dimensional shape model in a three-dimensional space, a total of six degrees of freedom, ie, three degrees of freedom of rotation and three degrees of translation are required. When the three-dimensional shape model is displayed two-dimensionally on the display, the movement in the depth direction with respect to the display among the above degrees of freedom can be expressed by scaling. Therefore, in this case, conversion is performed in three degrees of freedom, one degree of enlargement / reduction, and two degrees of parallel movement, for a total of six degrees of freedom. In the conventional three-dimensional model operation, this operation is assigned 6 degrees of freedom × positive / negative to each of the 12 keys of the keyboard. Further, in a three-dimensional model operation using a pointing device such as a mouse, the mode is switched in order to make a pointing device having only two degrees of freedom correspond to conversion of six degrees of freedom. As a fusion type of the two, there is also a method of operating two degrees of freedom with a pointing device and operating the remaining four degrees of freedom with a keyboard.

[0115]

In the operation method using the keyboard, since two keys in the positive and negative directions are assigned to each axis, only the axial direction conversion can be performed. For example, if vertical and horizontal axes are prepared for parallel movement on a plane, a two-step operation of performing vertical movement and then horizontal movement (or horizontal movement and then vertical movement) is required for oblique parallel movement. I need. Furthermore, in the case of rotation, there is a problem that it is very difficult to decompose the assumed transformation into an axial vector.

In the operation method using the pointing device, the conversion can be performed obliquely with respect to the axis. However, there are problems that the mode switching is troublesome and that the three degrees of freedom of rotation cannot be operated well. In the operation method using both the keyboard and the pointing device,
It does not make up for the shortcomings of each, simply adding to the difficulty of operation due to input on three different devices. Further, there is a problem in that it is difficult to know where the image is to be rotated unless the image is actually rotated. The present invention can directly perform conversion in any direction, eliminates troublesome operations such as mode switching, and realizes an easy-to-understand operation screen,
It is an object of the present invention to provide a display method and an apparatus capable of performing high-speed and free operation.

[0117]

According to a third aspect of the present invention, there is provided a method for displaying a three-dimensional shape model on a two-dimensional plane, wherein a part or all of the three-dimensional shape model is included therein. It is characterized in that polygons are displayed together. Here, the polygon includes a sphere. The polygon is translucent, and its color is a color that is easily seen in comparison with the color of the three-dimensional shape model and the background color. Further, the display mode such as movement, enlargement / reduction, rotation, etc. is changed based on the relative positional relationship between the designated point and the polygon by the pointing device.

A three-dimensional character display device according to the present invention is a device for displaying a three-dimensional model on a two-dimensional plane, comprising: means for calculating a polygon containing a part or the whole of the three-dimensional model. A pointing device; means for determining a relative positional relationship between a point designated by the pointing device and the polygon; and means for changing a display mode of the three-dimensional shape model according to a result of the determination. I do.

FIG. 48 shows a display example of polygons. 3
The three-dimensional shape model has a step shape, and a sphere (showing parallel lines and meridians) enclosing the model is displayed as a guide polygon for guiding recognition or operation. With such a display, the center of enlargement, reduction, or rotation can be recognized at a glance. As shown in FIG. 49, rotation is instructed when a point specified by a pointing device such as a mouse is inside the polygon, translation is performed when the point is outside the polygon, and enlargement / reduction is instructed when the point is a periphery. Then, the amount is specified in the next operation.

[0120]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a fourth group of the invention will be described in detail with reference to the drawings showing the embodiments. FIG. 50 is a block diagram of the three-dimensional shape model display device of the present invention. In the figure
A display device 60 includes a pointing device 61 such as a mouse. The shape of the three-dimensional shape model is stored in the shape storage unit 63, the position is stored in the position storage unit 70, and the background image is stored in the background image storage unit 62.

The guide polygon generating section 64 includes a color analyzing section 64a, a color selecting section 64b, and a shape determining section 64c, and determines the size and color of the guide polygon required for the subsequent operation. The color analysis unit 64a reads the color information of the background image from the background image storage unit 62,
Further, the color information of the three-dimensional shape model is fetched from the shape storage unit 63 and analyzed, and the color selection unit 64b selects a color that does not hinder the display of the background and the three-dimensional shape model and is easily visible.
The shape determining unit 64c determines the shape and size of the guide polygon to be displayed.

FIG. 51 is a flowchart of color selection, and FIG. 52 is a flowchart of shape and dimension determination. First, information on the background image and the three-dimensional shape model is fetched from the background image storage unit 62 and the shape storage unit 63, respectively, which hue is used for what percentage of the background image (S75), and Then, it is checked which hue is used and what percentage is used (S71).
Then, the used hue of the three-dimensional shape is appropriately weighted (S72). The above is the function of the color analysis unit 64a described above, and the following is the function of the color selection unit 64b. That is, based on the results of the above analysis, the amount of hue in the vicinity is examined from among the candidate display colors prepared in advance (S73). Then, a candidate display color having the smallest hue is selected as a guide polygon color (S74).

Next, the determination of the shape and position will be described.
First, data is taken in from the shape storage unit 63 and the position storage unit 70 to calculate the center of gravity of the three-dimensional shape model (S61). The center of gravity is set as the center of the guide polygon (S62). Next, the distance from this center to each vertex of the three-dimensional shape model is obtained.
(S63). Then, let the longest distance be the radius of the guide polygon (S
64) Then, the information of the guide polygon is stored in the guide polygon storage unit 65. A similar technique may be used when a regular polyhedron other than a sphere is used as a guide polygon.

The guide polygon created as described above is displayed on the display device 60 by the display unit 66. The display unit 66 includes a guide polygon storage unit 65, a background image storage unit 62, a shape storage unit 63, and a superimposition unit 66a for superimposing contents read from the position storage unit 70, and a conversion unit for displaying the superimposition unit 66a on the display device 60. And a display unit 66b.

On the other hand, an input from the pointing device 61 is taken into the interface section 67. Input control unit 67
“a” controls the pointing device 61. When the input is linked with the immediately preceding operation such as dragging the mouse, it is determined that the continuation of the immediately preceding conversion is performed. The operation position determination unit 67b determines whether the input operation start point is outside, inside, or on the boundary of the guide polygon.If the operation is performed outside, the translation is performed, if the operation is performed inside, the rotation is performed, and the operation is performed on the boundary. If so, enlarge or reduce. If the conversion is the continuation of the conversion performed immediately before, the same conversion processing as the previous conversion is selected.

FIG. 53 is a flowchart showing a processing procedure of the operation position discriminating section 67b. Guide polygon generator 64
The radius determined in is set to r (S81), and the distance 1 between the point designated by the pointing device 61, that is, the operation start point and the center point of the guide polygon is obtained (S82). If r = 1 (S83) enlargement / reduction processing (S86), if r> l, rotation processing
(S87) If r <1, a parallel movement process (S85) is performed.

The operation position information or the operation position discrimination information is input to the conversion amount determining unit 68, and the parallel movement amount determining unit 68
a, the translation amount, the enlargement / reduction amount, and the rotation amount are determined by the enlargement / reduction amount determination unit 68b and the rotation amount determination unit 68c, respectively, and these conversion amounts are given to the conversion unit 69, where the conversion according to the conversion amount is performed. Is performed. The translation unit 69a, the enlargement / reduction unit 69b, and the rotation unit 69c perform translation, enlargement / reduction, and rotation, respectively.

Next, these conversions will be described. First, for the parallel movement, the user specifies the parallel movement by a method such as placing the cursor in an area outside the polygon (see FIG. 49) and clicking, and moves (drags) the cursor in a desired direction. Thereby, the three-dimensional shape model and the polygon move in conjunction.
The unit of movement is a pixel. This translation can use various techniques known per se.

Next, enlargement / reduction will be described. FIG. 54 is a diagram for explaining the principle. First, a point P on the periphery of the guide polygon is set.
Click _1, click at the position P ₂ corresponding to the magnification to be went then enlarged or reduced drag. When the center of the guide polygon and 0 becomes enlarged or reduced is that the bar OP ₂ / bar OP _1. For the processing of the enlargement / reduction itself, a technique known per se may be appropriately used.

Next, the rotation will be described. Figure 55 shows the source
FIG. 56 is a flowchart showing a processing procedure for rotation.
It is a chart. In FIG. 55, D represents the display device 60.
2D plane, H is the center of the guide polygon expressed optically
Is a plane parallel to D passing through. Now point P₁Click on
Command to rotate, drag, α_PJust rotated
P_TwoIt is assumed that a click is made at a point (S91). This P₁, P_Twoso
Point R projected on guide polygon₁, R_Two(S9
2) ∠P_TwoO'P₁(O 'is the guide point on the plane D
Rigon's center) = α_P(S93). Then ∠R₁OR_Two
To obtain α _rAnd Then R₁O, R_TwoO (O is
The angle formed by the polygon center on the plane H) is obtained,
₁OR_TwoIs α_r(S94). Next, the reference line on the reference point O
L to bar R ₁O and bar R_TwoDefined as a straight line perpendicular to O
Yes (S95). And α about this axis L_rJust rotate
Yes (S96). Known figures for the processing after the rotation amount determination
The shape rotation method may be used. In the case of such a rotation operation
When using a sphere as a guide polygon, its meridians,
Easy rotation by tracing or referencing parallels
Operation can be performed. The model converted as described above is
Is stored and stored in the position storage unit 70.

FIG. 57 is an overall flowchart of the three-dimensional shape model display device. As described above, first, the guide polygon is determined (S101), then the background, the three-dimensional shape model and the guide polygon are mixed and displayed (S102), and if there is a conversion designation by the operator (S103), the operation area or Determination of movement, enlargement / reduction, and rotation is performed (S104), the amount of conversion is determined (S106), and the conversion is executed (S107).

[0132]

According to the present invention as described above, the origin (center) of enlargement / reduction or rotation can be intuitively recognized. Further, the posture of the three-dimensional shape model can be easily recognized by comparison with the guide polygon. In addition, since the color of the guide polygon is automatically determined, there is no possibility that the three-dimensional shape model becomes difficult to see. In addition, complicated operation of mode switching regarding movement, enlargement / reduction, and rotation does not require a special device. Further, as for the rotation, a rotation amount and a direction of three degrees of freedom can be input only by inputting two degrees of freedom on the two-dimensional display plane of the display device,
Moreover, the operation is easy if it follows the shape of the guide polygon.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram of a system of the present invention.

FIG. 2 is a flowchart of a process performed by a specific object region extraction unit.

FIG. 3 is an explanatory diagram of a structure of a moving image object.

FIG. 4 is a block diagram of an apparatus used for implementing the first group of the invention.

FIG. 5 is a flowchart of a process.

FIG. 6 is an explanatory diagram of area designation of a specific object.

FIG. 7 is a flowchart of a three-dimensional shape information adding process.

FIG. 8 is an explanatory diagram of designation of a depth of a ridge line and an end point.

FIG. 9 is an explanatory diagram of designation of a ridge line.

FIG. 10 is a block diagram of a three-dimensional shape extraction device.

FIG. 11 is a flowchart of a three-dimensional shape extraction method.

FIG. 12 is a three-dimensional view of a basic shape.

FIG. 13 is an explanatory diagram showing an example of superimposed display of a basic shape and an image.

FIG. 14 is an explanatory diagram of object region extraction.

FIG. 15 is a flowchart of a process of moving the position of the basic shape.

FIG. 16 is an explanatory diagram of a process of moving the position of the basic shape.

FIG. 17 is a flowchart of a process of rotating a basic shape.

FIG. 18 is an explanatory diagram of a process of rotating a basic shape.

FIG. 19 is a flowchart of a size changing process.

FIG. 20 is an explanatory diagram of a size changing process.

FIG. 21 is an explanatory diagram of mapping.

FIG. 22 is an explanatory diagram of a conventional image synthesis method.

FIG. 23 is a block diagram of a first region extraction device.

FIG. 24 is a flowchart of region extraction.

FIG. 25 is an explanatory diagram of region extraction.

FIG. 26 is a block diagram of a second region extraction device.

FIG. 27 is a flowchart of region extraction.

FIG. 28 is an explanatory diagram of region extraction.

FIG. 29 is a block diagram of third to sixth region extracting devices.

FIG. 30 is a flowchart of region extraction.

FIG. 31 is an explanatory diagram of region extraction.

FIG. 32 is a block diagram of a third, fourth, seventh, and eighth region extraction devices.

FIG. 33 is a flowchart of region extraction.

FIG. 34 is an explanatory diagram of region extraction.

FIG. 35 is a block diagram of a third and fourth region extracting devices.

FIG. 36 is a flowchart of edge processing.

FIG. 37 is an explanatory diagram of edge processing.

FIG. 38 is an explanatory diagram of edge processing.

FIG. 39 is a block diagram of a tenth region extraction device.

FIG. 40 is a flowchart of noise cancellation.

FIG. 41 is an explanatory diagram of noise cancellation.

FIG. 42 is a block diagram of a tenth region extraction device.

FIG. 43 is a flowchart of noise cancellation.

FIG. 44 is an explanatory diagram of noise cancellation.

FIG. 45 is a block diagram of an eleventh region extracting apparatus.

FIG. 46 is a flowchart of noise cancellation.

FIG. 47 is an explanatory diagram of noise cancellation.

FIG. 48 is a screen diagram showing a display example of polygons.

FIG. 49 is an explanatory diagram of an operation.

FIG. 50 is a block diagram of a three-dimensional shape model display device.

FIG. 51 is a flowchart of guide polygon color selection.

FIG. 52 is a flowchart of determining the shape of a guide polygon.

FIG. 53 is a dimensional flowchart of operation position determination.

FIG. 54 is a diagram illustrating the principle of enlargement / reduction.

FIG. 55 is a view for explaining the principle of rotation.

FIG. 56 is a flowchart of rotation.

FIG. 57 is an overall block diagram of a display device for a three-dimensional shape model.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Specific object area extraction part 2 Three-dimensional shape information addition part 3 Video CG model generation part 4 Synthetic image generation part 5 Image storage part 6 Shape / surface attribute information storage part 7a, 7b CG model storage part 8 Synthesis information storage part 10 Image Display device 12 Pointing device 13 CG model creation unit

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Kazuki Matsui 1015 Uedanaka, Nakahara-ku, Kawasaki City, Kanagawa Prefecture Inside Fujitsu Limited (72) Inventor Shuichi Shiiya 1015 Kamidadanaka, Nakahara-ku, Kawasaki City, Kanagawa Prefecture Fujitsu Limited

Claims (4)

[Claims] 1. A method for displaying a three-dimensional shape model on a two-dimensional plane, wherein a polygon including a part or the whole of the three-dimensional shape model is displayed together, and a center of the polygon is obtained. A method for displaying a three-dimensional shape model, wherein the three-dimensional shape model is enlarged / reduced with reference to a reference point. 2. The polygon according to claim 1, wherein the polygon is semi-transparent, color information of a background color on a two-dimensional plane is extracted, and a color that does not include the extracted color information element most is determined as the color of the polygon. 3D shape model display method. 3. A method for displaying a three-dimensional shape model on a two-dimensional plane, wherein a polygon including a part or the entirety of the three-dimensional shape model is also displayed, and a point designated by a pointing device and the polygon are displayed. A method for displaying a three-dimensional shape model, wherein the three-dimensional shape model is enlarged or reduced in accordance with a relative positional relationship. 4. An apparatus for displaying a three-dimensional shape model on a two-dimensional plane, comprising: means (64) for calculating a polygon including a part or all of the three-dimensional shape model therein; and calculating a center of the polygon. Means (64), a pointing device (61), means (68) for determining a relative positional relationship between a point designated by the pointing device (51) and a center point of the polygon, and the polygon according to the determination result. Means (69) for enlarging / reducing the three-dimensional model using the center of the three-dimensional model as a reference point.