JP2000057377A - Image processor, image processing method and medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily perform three-dimensional editing, etc., to a two-dimensional image. SOLUTION: In the case that the two-dimensional image (Fig. (A)) on which a three-dimensional object to simulate a house is displayed is stored in a noticing buffer, for example, when four vertexes of a grid wall are specified as feature points together with shape information that the shape of the wall is rectangular by a user, a developed image (Fig. (B)) formed by developing the grid wall on a two-dimensional plane is generated and stored in a paste buffer based on the shape information and the feature points. A two-dimensional image (Fig. (C)) of a cylinder is stored in the noticing buffer and when a specified point on the side of the cylinder is specified as the feature point together with the shape information that the two-dimensional image is the side of the cylinder by the user, the image (Fig. (B)) stored in the paste buffer is pasted on the side of the cylinder in the noticing buffer (Fig. (D)), based on the shape information and the feature point.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, an image processing method, and a medium.
For 3D objects displayed in 3D images,
The present invention relates to an image processing apparatus, an image processing method, and a medium that enable dimensional editing and the like to be performed.

[0002]

2. Description of the Related Art Hitherto, various methods have been proposed for performing various processes on a two-dimensional image and extracting information necessary for performing the process from the two-dimensional image. Documents describing such a method include, for example, Ja
mes D. Foley, Andries van Dam, Steven K. Feiner, John
F. Hughes, "Computer Graphics, principles and prac
tice ", ADDISON-WESLEY PUBLISHING COMPANY, 1996 (hereinafter referred to as Reference 1), Paul E. Debevec, Camillo J. Tayl
or, Jitendra Malik, "Modeling and Rendering Archit
ecture from Photographs: A hybrid geometry-and ima
ge-based approach ", proceedings of SIGGRAPH 96, p
p.11-20 (hereinafter referred to as reference 2), Oliver Faugeras, "T
hree-dimensional computer version ", The MIT press
(Hereinafter referred to as Reference 3), Kenneth P. Fishkin, Brian
A. Barsky, "A Family of New Algorithms for Soft Fil
ling ", proceedings of SIGGRAPH 84, pp235-244 (hereinafter referred to as reference 4), Pat Hanrahan and Paul Haeberl
i, "Direct WYSIWYG Painting and Texuturing on 3D Sh
apes ", proceedings of SIGGRAPH 90, pp.215-233 (hereinafter referred to as Reference 5), Youichi Horry, Ken-ichi Anjyo,
Kiyoshi Arai, "Tour Into the Picture: Using a Spi
dery Mesh Interface to Make Animation from a Singl
e Image ", proceedings of SIGGRAPH 97, pp.255-232
(Hereinafter referred to as Reference 6), Michael Gleicher, "Image S
napping ", proceedings of SIGGRAPH 95, pp.183-190
(Hereinafter referred to as Reference 7).

[0003] In Document 1, a designer or the like causes a computer to perform the same work as drawing a picture on paper using a paintbrush or an airbrush (a technique of drawing a paint into powder and flying it on paper). An image processing technique called 2D (two-dimensional) paint is described.

However, in the conventional 2D paint,
Even when a three-dimensional object is displayed in an image, the image itself is handled in a two-dimensional plane, and therefore, the orientation of the three-dimensional object displayed in such a two-dimensional image in a three-dimensional space is not considered. If a character is drawn or a figure is added, the image becomes unnatural.

That is, for example, as shown in FIG.
When a three-dimensional object imitating a house is displayed in a two-dimensional image and characters are drawn on the wall of the house without considering its orientation, as shown in FIG. , The letters do not look like they are drawn on the wall. In addition, even if a rectangle is drawn to add a rectangular parallelepiped room to the wall of a house, if the rectangle is drawn without considering its orientation, as shown in FIG. Becomes unnatural. Further, for example, as shown in FIG. 2A, when a cylinder is displayed in a two-dimensional image, even if a character is drawn on the side surface, the orientation of the side surface is ignored. As shown in B), the characters do not become drawn on the side of the cylinder.

Therefore, in order to prevent the image from being unnatural when performing 2D painting, characters and figures are deformed according to the orientation of the three-dimensional object displayed in the two-dimensional image. You need to do that drawing,
Performing such an operation for drawing requires some skill.

[0007] Therefore, the user can input the tilt angle and the like of the three-dimensional object displayed on the two-dimensional image using a numeric keypad or a GUI (Graph).
There is a method in which a character or a figure to be newly drawn is deformed based on the input by having the input be performed using a hic user interface) or the like. However,
In this method, it is necessary for the user to adjust the tilt angle of the three-dimensional object input to the computer while viewing the drawing result so that the result does not become unnatural. Requires skill.

As described above, when a user draws a character or a figure without considering the orientation of a three-dimensional object displayed on a two-dimensional image, the computer makes the drawing result natural. That is, the fact that it is not possible to obtain a drawing result as if the photographing was performed in a state where the character or figure is present is due to the 3D image displayed on the 2D image.
Information on where the two-dimensional object is located in the three-dimensional space and from which position the two-dimensional image is obtained by photographing (the two-dimensional image is a picture of a painting In such a case, the position at which the landscape or the like shown in the painting is observed) is missing.

Therefore, there is a method of obtaining the position of the three-dimensional object displayed thereon in the three-dimensional space and the photographing position from the two-dimensional image by using an image recognition (Computer Vision) technique.

That is, Document 2 discloses a method in which a three-dimensional shape such as a rectangular parallelepiped is made to correspond to a building photographed in a plurality of photographs by using a GUI, and the size and the photographing position of the building are obtained. ing. Reference 3 discloses various other methods for obtaining the position of a three-dimensional object and its shooting position.

However, the method of calculating the position of a three-dimensional object and its photographing position using such an image recognition technique utilizes the principle of triangulation. Therefore, a plurality of images obtained by photographing the same three-dimensional object from a plurality of photographing positions are required. However, when performing 2D painting, it is not always possible to prepare such a plurality of images.
Furthermore, when the two-dimensional image is a picture of a painting, it is general that the plurality of images do not exist as described above.

[0012] Even if a plurality of images of the same three-dimensional object can be prepared from a plurality of photographing positions, a computer can obtain three-dimensional images from the plurality of images.
To calculate the position of the three-dimensional object and its shooting position,
It is necessary to specify the corresponding position of the same three-dimensional object displayed in a plurality of images (for example, a vertex of a roof of a building which is a three-dimensional object), and such specification is performed for each of the plurality of images. Is troublesome and time-consuming. Furthermore, based on the position in the three-dimensional space and the photographing position of the three-dimensional object displayed in the two-dimensional image,
In the case of performing three-dimensional natural drawing, it is necessary to process three-dimensional data such as three-dimensional coordinates when a three-dimensional object is viewed from a shooting position, even though a two-dimensional image is handled. Instead, the amount of calculation for this is enormous.

Next, Document 4 discloses a method of processing colors when performing 2D painting.

That is, in 2D paint, for example, a user selects a color to be used for drawing using a GUI and performs drawing with that color. However, the color and brightness of a three-dimensional object displayed in a two-dimensional image are: It changes according to the positional relationship between the direction of the surface and the light source. Therefore, for example, the entire surface of a three-dimensional object is represented by the same color (the same RGB (Red, Green, Blue)).
Value), the drawing result will be unnatural. Therefore, in order to make the drawing result natural, it is necessary to change the color to be painted stepwise in consideration of the orientation of the surface to be painted and the positional relationship with the light source. In particular, when the surface to be colored is a curved surface, it is necessary to continuously change the pixel value in order to make the drawing result natural, and therefore, for example, the color to be painted is selected every several pixels. It has to be done again, and the amount of work is enormous.

Therefore, in Reference 4, the color of a pixel is
A method called tint fill is described, which is a mixture of colors, and when two colors are known, one of them can be changed to another color. However, the tint fill can effectively cope with, for example, a portion where foreground and background are mixed, but it is difficult to cope with shading of an object and reflection of illumination by the object.

Next, in Document 5, a character or a handwritten figure input by operating a tablet or a mouse is deformed along the surface (surface) of a three-dimensional object displayed in a two-dimensional image. A method of attaching is disclosed.

[0017] However, the method disclosed in Reference 5 discloses three-dimensional information on a three-dimensional object displayed in a two-dimensional image (for example, the position, photographing position, size, and inclination of the three-dimensional object in a three-dimensional space). Therefore, it is necessary to accurately recognize the three-dimensional information using the image recognition technology as described above. However, when the three-dimensional information is recognized using the image recognition technology, As described above, a plurality of images are required, and the amount of calculation increases.

The method of Reference 5 can use a three-dimensional model of a three-dimensional object generated inside a computer. However, in this case, it is necessary to handle three-dimensional data. The amount of calculation is enormous.

Here, as a method of generating a three-dimensional model, for example, an outline of the shape of a three-dimensional object displayed thereon is extracted from a two-dimensional image using an image recognition technique, and the shape of the three-dimensional object is extracted. There is a method of attaching a photographed image to the surface (performing texture mapping), and the method of generating a two-dimensional image viewed from a free viewpoint (photographed from a free photographing position) using this method is described above. Reference 2 has been disclosed. However, this method requires a plurality of images, as in the case described above, in order to extract the outline of the shape of the three-dimensional object displayed thereon from the two-dimensional image using image recognition technology. Become.

Next, Document 6 discloses a method of generating an image viewed from a viewpoint other than the shooting position from one two-dimensional image. However, this method only generates an image viewed from another viewpoint other than the shooting position, and furthermore, the object displayed in the two-dimensional image and the objects around the two-dimensional image have a very simple plate-like shape. Since it is assumed that this is the case, it is not so much a problem if the image is only approximately generated from another viewpoint. However, when a character or a figure is added to the two-dimensional image, the drawing is not performed. It is difficult to make the results natural. Specifically, for example, when a cylinder is displayed on a two-dimensional image, it is difficult to obtain a natural image in which characters and figures are added to the side surfaces of the cylinder.

Further, when processing is performed on a two-dimensional image, the user is required to specify a predetermined feature point such as a vertex of a three-dimensional object in the two-dimensional image. Often. This designation is performed by operating a mouse, a pen, or the like.
It is difficult to accurately specify vertices of a three-dimensional object. Therefore, in Document 7, a method is disclosed in which an edge (a portion where brightness or color greatly changes) in a two-dimensional image is extracted, and the cursor is moved over the edge when the cursor is located near the edge. Have been. However, in the method of Reference 7, since all edges in a two-dimensional image are detected, the cursor may be moved to an unnecessary position in response to the noise or texture of the two-dimensional image.

[0022]

As described above, many image processing techniques have been conventionally proposed. However, three-dimensional addition, deletion, movement, and the like of a character or a figure with respect to a two-dimensional image have been proposed. A method particularly suitable for performing editing or processing such as deformation, change of color or pattern, or change of light source has not yet been proposed.

That is, in the conventional method, to perform three-dimensional editing and processing on a two-dimensional image, a certain amount of operation skill and time is required, or the same object is photographed from different directions. Requires multiple images. In addition, the amount of calculation may be enormous, and unnecessary cursor movement may occur. Further, when a three-dimensional object or the like in a two-dimensional image is deleted, it is necessary to reproduce the background in the area after the deletion. However, in the methods described in the above-mentioned documents 1 to 7, such a method is used. It is difficult to reproduce a perfect background.

The present invention has been made in view of such a situation, and is intended to easily perform three-dimensional editing or the like on a two-dimensional image.

[0025]

According to the present invention, there is provided an image processing apparatus comprising: first shape information relating to a shape of a surface constituting a first three-dimensional object displayed in a first two-dimensional image; First operation means operated when designating a first feature point that is a point on a first two-dimensional image that constitutes a surface;
Based on the first shape information and the first feature point, the surface constituting the first three-dimensional object in the first two-dimensional image is defined as 2
First calculating means for calculating a first conversion formula for converting into a developed image which is an image developed on a two-dimensional plane; and a first calculating means for calculating a first conversion formula in the first two-dimensional image based on the first conversion formula. First converting means for converting the surface constituting the three-dimensional object into a developed image, storage means for storing the developed image, second shape information relating to the shape of the surface constituting the second three-dimensional object, Second operation means operated when designating a second feature point which is a point on a second two-dimensional image forming the surface;
A second image for converting the image stored in the storage unit into a projection image which is an image obtained by projecting a surface constituting a second three-dimensional object, based on the second shape information and the second feature point. A second calculating means for calculating a conversion formula; a second converting means for converting an image stored in the storage means into a projection image based on the second conversion formula; Pasting means for pasting to a portion specified by the second feature point in the two-dimensional image.

According to the image processing method of the present invention, a surface constituting a first three-dimensional object in a first two-dimensional image is converted into a two-dimensional plane based on first shape information and a first feature point. A first calculating step of calculating a first conversion formula for converting the image into a developed image which is a developed image, and a first three-dimensional image in the first two-dimensional image based on the first conversion formula. A first conversion step of converting a surface constituting the three-dimensional object into a developed image;
Based on the storage step of storing the developed image in the storage unit and the second shape information and the second feature point, the image stored in the storage unit is projected onto a surface forming a second three-dimensional object. A second calculation step of calculating a second conversion equation for converting the image into a projection image, and a second step of converting the image stored in the storage unit into a projection image based on the second conversion equation. And a pasting step of pasting the projection image to a portion specified by the second feature point in the second two-dimensional image.

The computer program which the medium of the present invention causes a computer to execute includes first shape information and first shape information.
Based on the feature points of the first three-dimensional image in the first two-dimensional image
A first calculating step of calculating a first conversion formula for converting a surface forming the two-dimensional object into a developed image that is a developed image on a two-dimensional plane; and a first calculating step based on the first conversion formula.
The surface constituting the first three-dimensional object in the two-dimensional image of
A first converting step of converting the image into a developed image, a storing step of storing the developed image in the storage means, and an image stored in the storage means based on the second shape information and the second feature point. A second calculation step of calculating a second conversion formula for converting a surface constituting the two three-dimensional objects into a projection image, and storing the second conversion formula in a storage unit based on the second conversion formula. And a pasting step of pasting the projected image to a portion specified by the second feature point in the second two-dimensional image.

[0028]

FIG. 3 shows an example of the configuration of an embodiment of an image processing apparatus to which the present invention is applied. This image processing apparatus is configured based on a computer,
Three-dimensional editing or the like can be easily performed on a two-dimensional image.

That is, the arithmetic processing circuit 1 includes, for example, a CPU
(Central Processing Unit) and the like (OS) stored (loaded) in the program memory 2.
By executing an application program also stored in the program memory 2 under the control of the
Various processes described below are performed on the two-dimensional image. The program memory 2 is, for example,
It is composed of a RAM (Random Access Memory) or the like, and temporarily stores the OS and application programs stored (recorded) in the external storage device 7. The data memory 3 is, for example, a RAM or the like, and temporarily stores data necessary for the processing of the arithmetic processing circuit 1. The frame memory 4 is, for example, a RAM or the like, and stores image data to be displayed on the image display device 5. The image display device 5 is, for example, a CR
It comprises a T (Cathode Ray Tube), a liquid crystal display, or the like, and displays image data stored in the frame memory 4. The input device 6 includes, for example, a mouse, a combination of a pen and a tablet, and a keyboard, and is used to input necessary commands and data.
The operation is performed when the image display device 5 is instructed to a predetermined position on the screen. External storage device 7
Is, for example, a hard disk or floppy disk, C
It comprises a D-ROM (Compact Disc ROM), a magneto-optical disk, etc., and stores an OS and application programs. Further, the external storage device 7 stores data necessary for the operation of the arithmetic processing circuit 1 and two-dimensional images (digital data) to be processed. The external interface 8 functions as an interface for taking in data supplied from the outside, such as a two-dimensional image captured by the camera 9 or the like, or data transmitted via a communication line (not shown). It has been done.

The arithmetic processing unit 1, the program memory 2, the data memory 3, the frame memory 4, the input device 6,
The external storage device 7 and the external interface 8 mutually
They are connected via a bus, and programs and data are exchanged between the blocks via the bus.

In the image processing apparatus configured as described above, when the power is turned on, the OS stored in the external storage device 7 is read out, expanded in the program memory 2, and
This is executed by the arithmetic processing circuit 1. Then, when the input device 6 is operated so as to execute the application program, the arithmetic processing circuit 1 operates under the control of the OS.
The application program is read from the external storage device 7, expanded in the program memory 2, and executed.
Thus, for example, various processes are performed on the two-dimensional image captured by the camera 9 and stored in the external storage device 7 via the external interface 8.

FIG. 4 shows an example of a functional configuration of the image processing apparatus of FIG. 3, which is realized by the arithmetic processing circuit 1 executing an application program.

The input event processing unit 11 is supplied with a GUI event from the OS. The input event processing unit 11 analyzes the GUI event,
According to the analysis result, the copy operation processing unit 12, the paste operation processing unit 13, the paste buffer selection processing unit 1
4, erase operation processing unit 15, mat operation processing unit 16,
The object attribute operation processing unit 17 and the paint processing unit 18 are activated to perform processing.

Here, the GUI event includes, for example, a movement of a mouse cursor, a click of a mouse (pressing or pushing a mouse button), a menu selection on the GUI, and an operation of a button. The GUI event is defined in a window system such as X Window, for example.

The copy operation processing section 12 performs a copy process of storing (copying) an image in a paste buffer described later. Paste operation processing unit 13
Is configured to perform a paste process of pasting (pasting) an image stored in a paste buffer to a two-dimensional image stored in an image buffer described later. The paste buffer selection processing unit 14 performs a paste buffer selection process of selecting an image buffer to be used as a paste buffer and storing the buffer ID (Identification) in the buffer ID storage unit 21.

Here, in the data memory 3, an area for storing pixel values of an image in units of one screen, for example, is secured as an image buffer. Here, a plurality of image buffers are secured in the data memory 3, and each image buffer has
Buffer ID so that each can be identified
Is attached. Among the plurality of image buffers, the buffer whose buffer ID is stored in the buffer ID storage unit 21 is particularly called a paste buffer. The image buffer can store pixel values (for example, RGB values, etc.) of a two-dimensional image, and can also store a matte of an object displayed in the two-dimensional image. ing. The mat will be described later.

The erase operation processing section 15 performs an erase process for deleting a part of the image stored in the paste buffer. The mat operation processing unit 16 performs a mat process for generating a mat for the three-dimensional object displayed on the two-dimensional image stored in the image buffer. The object attribute operation processing unit 17 is a so-called material property (such as a material property) such as the color and texture of the three-dimensional object displayed in the two-dimensional image stored in the image buffer.
rty) and a light source change process for changing the light source (changing the lighting condition of the three-dimensional object in the two-dimensional image stored in the image buffer). The paint processing unit 18 performs a paint process for applying 2D paint to the image stored in the paste buffer.

The conversion designation processing unit 19 is configured to operate the input device 6 in accordance with the instructions of the copy operation processing unit 12 and the paste operation processing unit 13 and to specify the two-dimensional image on the two-dimensional image stored in the image buffer. The position of the feature point is corrected, and the corrected feature point is used as the copy operation processing unit 12,
The processing is returned to the paste operation processing unit 13. The image conversion processing unit 20 converts an image stored in an image buffer or a paste buffer in accordance with instructions from the copy operation processing unit 12 and the paste operation processing unit 13. The buffer ID storage unit 21 stores the buffer ID of the image buffer selected as the paste buffer by the paste buffer selection processing unit 14. The buffer ID stored in the buffer ID storage unit 21 is supplied to the erase operation processing unit 15, the paint processing unit 18, and the image conversion processing unit 20, whereby the erase operation processing unit 15 , Paint processing unit 18 and image conversion processing unit 20
Recognizes an image buffer serving as a paste buffer.

The display processing unit 22 performs processing necessary for displaying the two-dimensional image stored in the image buffer on the image display device 5. The display processing unit 2
2 is passed to the display processing of the OS, whereby the image data is
The data is written in the frame memory 4 and displayed on the image display device 5.

The blocks in FIG. 4 except for the data memory 3 and the buffer ID storage unit 21 are realized by the arithmetic processing circuit 1 executing the application program, and correspond to the respective blocks. The program is, for example, modularized.

According to the image processing apparatus configured as described above, for example, a two-dimensional image displaying a three-dimensional object imitating a house as shown in FIG. 5A is stored in the image buffer. In this case, even if the characters are drawn without being conscious of the direction of the wall of the house, as shown in FIG.
It is made possible to obtain a two-dimensional image as if a character is drawn along a wall. In addition, even if a rectangle is drawn without considering the direction of the wall of the house, FIG. 5 (C)
As shown in FIG. 1, a two-dimensional image in which a rectangular parallelepiped room is added to the wall of a house can be obtained. Further, for example, as shown in FIG. 6A, when a two-dimensional image in which a cylinder is displayed is stored in an image buffer, the direction of the side surface is not particularly conscious, and
Even if a character is drawn, a two-dimensional image can be obtained as if the character is drawn along the side surface, as shown in FIG. 6 (B).

That is, for example, a three-dimensional object on which a character or the like is not drawn is actually captured by the video camera 9 without actually capturing the three-dimensional object on which the character or the like is drawn. A three-dimensional operation is performed on the three-dimensional image, thereby making it possible to easily obtain a two-dimensional image as if the three-dimensional object on which the character or the like is drawn is actually photographed. ing.

Next, referring to the flowchart of FIG.
The processing of the image processing apparatus in FIG. 4 will be described.

When the application program is executed in the arithmetic processing circuit 1 (FIG. 3), the image display device 5
For example, a main window as shown in FIG. 8 is displayed. On the right side of the main window, a command button (Command) 31 and a shape button (Shape) 32 are displayed, and on the left side thereof, a window in which a two-dimensional image stored in the image buffer is displayed (hereinafter referred to as a buffer window as appropriate). ) Is displayed.

Here, the command button 31 is operated when inputting a command. In the embodiment of FIG. 8, the "Paste Sel.""Copy" button operated when performing processing, "Paste" button operated when performing paste processing, "Erase" button operated when performing erase processing, operated when performing matte processing "Matte" button, "Material" button operated when performing object attribute processing, "Light" button operated when performing light source change processing, "Paint" button operated when performing paint processing Have been.

The shape button 32 is operated to input the shape (shape information) of the surface of the three-dimensional object to be processed. In the embodiment shown in FIG. "Rectangle" button, "Cylinder" button, "Spher"
An "e" button and a "Cone" button are provided.

Further, in the embodiment shown in FIG. 8, three buffer windows are displayed. Each of the three buffer windows is provided with image buffers # 1, # 2, and # 2 among a plurality of image buffers.
The storage content of # 3 is displayed (the suffix i of the image buffer #i represents, for example, a buffer ID). When a plurality of buffer windows are displayed, for example, of the plurality of buffer windows,
The storage contents of the buffer of interest, which will be described later, are displayed at the foreground (activated).

The operation of the main window is basically unified with a user interface for selecting an image buffer to be processed and then selecting a process. Therefore, the user first operates the main window. One of the buffer windows displayed in
By clicking by operating the input device 6,
Select the image buffer to be processed.

That is, when the user clicks on one of the buffer windows, in step S1, the input event processing unit 11 selects an image buffer whose contents are displayed in the buffer window as a buffer of interest.

Thereafter, the user clicks one of the command buttons 31. In this case, step S
In step 2, the input event processing unit 11 recognizes which of the command buttons 31 has been clicked, and proceeds to step S3 to instruct what kind of processing (operation) based on the recognition result. Judge whether it is. If it is determined in step S3 that the paste buffer selection processing has been instructed, that is, if the “Paste Sel.” Button in the command button 31 has been operated (clicked), the process proceeds to step S4, and the input event The processing unit 11 activates the paste buffer selection processing unit 14 to perform the paste buffer selection processing. That is, in this case, the paste buffer selection processing unit 14 overwrites the buffer ID of the buffer of interest in the buffer ID storage unit 21, whereby the image buffer that is currently the buffer of interest is used as the paste buffer and the process proceeds to step S 1. Return.

If it is determined in step S3 that one of the copy processing and the paste processing is instructed, that is, if the "Copy" button or the "Paste" button of the command buttons 31 is operated. If so, the process waits for the shape button 32 to be operated, and then proceeds to step S5. That is, the copy process is a process of generating a developed image in which a surface constituting the three-dimensional object displayed in the two-dimensional image stored in the buffer of interest is developed on a two-dimensional plane and storing the developed image in a paste buffer. The process generates a projection image by pasting the image on the two-dimensional plane stored in the paste buffer onto the surface constituting the three-dimensional object and projecting the projection image on the screen, and stores the projection image in the buffer stored in the buffer of interest. It is a process of pasting on a two-dimensional image,
In any case, since the surface constituting the three-dimensional object is to be processed, when the user operates the “Copy” button or “Paste” button, the three-dimensional object to be subsequently subjected to copy processing or paste processing is processed. Specify (input) the shape information on the shape of the surface by operating the shape button 32
I do.

In step S5, based on the operation of the shape button 32, the shape (shape information) of the surface of the three-dimensional object to be copied or pasted is recognized. Then, after waiting for the input of the feature point, the process proceeds to step S6. That is, in the copy processing, it is necessary to specify a part (surface) for generating a developed image by designating some points constituting the surface of the three-dimensional object displayed in the two-dimensional image stored in the buffer of interest. There is. In the paste processing, a projection image of the image stored in the paste buffer is attached to, for example, a surface constituting a three-dimensional object displayed in the two-dimensional image stored in the buffer of interest. It is necessary to identify the surface by specifying some points to do. So, the user says "Cop
By operating the "y" button or the "Paste" button, and subsequently by specifying the shape of the surface of the three-dimensional object to be subjected to the copy processing or the paste processing by operating the shape button 32, the buffer of interest is further added. By operating the input device 6, some points constituting the surface of the three-dimensional object to be processed in the two-dimensional image stored in are designated as feature points.

In step S6, the coordinates of the thus specified feature point in the two-dimensional image are recognized.

In the above case, the feature point is specified after the shape of the three-dimensional object is specified. However, the feature point is specified first, and then the shape of the three-dimensional object is specified. It is also possible to do so. The processing in steps S5 and S6 is performed by the copy operation processing unit 12 when the copy processing is instructed, and is performed by the paste operation processing unit 13 when the paste processing is instructed.

In step S 6, the copy operation processing section 12 or the paste operation processing section 13 that has recognized the feature point designated by the user supplies the feature point to the conversion designation processing section 19. Upon receiving the characteristic point, the conversion designation processing unit 19 corrects the position in step S7, and supplies the corrected characteristic point to the characteristic point of the copy operation processing unit 12 or the paste operation processing unit 13. Supply those who have been.

Thereafter, in step S8, the input event processing section 11 determines which of the "Copy" button and the "Paste" button has been operated. If it is determined in step S8 that the “Copy” button has been operated, the copy operation processing unit 12 performs a copy process. That is, in this case, steps S8 to S9
In the copy operation processing unit 12, based on the shape information and the feature points, a conversion formula () for converting a surface constituting the three-dimensional object in the two-dimensional image stored in the buffer of interest into a developed image ( Hereinafter, an inverse conversion formula is calculated as appropriate) and supplied to the image conversion processing unit 20. In step S10, the image conversion unit 20 determines a surface (a surface specified by a feature point) that forms a three-dimensional object in the two-dimensional image stored in the buffer of interest based on the inverse conversion formula.
It is converted to a developed image. Further, in step S11, the image conversion unit 20 converts the developed image into a buffer ID
The image is copied (overwritten) to the image buffer in which the buffer ID is stored in the storage unit 21, that is, the paste buffer, and the process returns to step S1.

According to the above-described copy processing, for example, when a two-dimensional image displaying a three-dimensional object V imitating a house as shown in FIG. 9A is stored in the buffer of interest. In, when a rectangle that is the shape of a wall surface S with a vertical line is specified as shape information, and four vertices P ₁ , P ₂ , P ₃ , and P ₄ of the wall surface S are specified as feature points, As shown in FIG. 9B, the three-dimensional object V
Is developed on a two-dimensional plane, and a two-dimensional image (developed image) of the wall surface S having a vertical line is generated and stored in the paste buffer.

Therefore, when a two-dimensional image on which a three-dimensional object V is displayed as shown in FIG. 9A is stored in an image buffer, it is desired to obtain a developed image of the wall S having the vertical line. Sometimes, the user first sets the image buffer in which the two-dimensional image in which the three-dimensional object V is displayed is stored as the target buffer, and secondly, instructs the copy process to “Co
py ”button, and thirdly, a“ Rectangle ”button for specifying the shape of the wall surface S. Fourth, as shown in FIG. The four vertices P ₁ , P ₂ , P ₃ , and P ₄ of the quadrangle constituting the wall S may be designated as feature points.

On the other hand, in step S8, "Paste"
When it is determined that the button has been operated, the paste operation processing unit 13 performs a paste process. That is, in this case, the process proceeds from step S8 to S12, where the paste operation processing unit 13 converts the image on the two-dimensional plane (developed image) stored in the paste buffer into a projected image based on the shape information and the feature points. A conversion formula (hereinafter, appropriately referred to as a forward conversion formula) is calculated and supplied to the image conversion processing unit 20. In step S13, the image conversion processing unit 20 converts the image stored in the paste buffer into a projection image based on the forward conversion formula. Further, in step S14, the image conversion unit 20 pastes the projection image on the surface of the three-dimensional object displayed on the two-dimensional image stored in the buffer of interest (the surface specified by the feature point), and in step S1. Return to

According to the above paste processing, for example, the copy processing described with reference to FIG.
Is stored in the paste buffer, and a two-dimensional image in which a three-dimensional object V imitating a house is displayed as shown in FIG. When stored in the buffer, a rectangle that is the shape of a plain wall surface S ′ is specified as shape information, and the four vertices P of the wall surface S ′ are designated as feature points.
_{1 ', P 2', P} 3 ', P 4' when is designated, Figure 1
0 (C), when the image stored in the paste buffer is pasted on the wall surface S ′ of the three-dimensional object V and the three-dimensional object V is projected on the screen, A projection image of the portion S ′ is generated and attached to the wall surface S ′ specified by the feature points P ₁ ′ to P ₄ ′ of the three-dimensional object V displayed in the two-dimensional image stored in the buffer of interest. .

Therefore, the image as shown in FIG. 10A is stored in the paste buffer and the image shown in FIG.
When a two-dimensional image displaying a three-dimensional object V imitating a house as shown in FIG. 0 (B) is stored in the image buffer, the image stored in the paste buffer is replaced with the three-dimensional object V. When a user wants to obtain a two-dimensional image pasted on the wall surface S ′, the user first sets the image buffer storing the two-dimensional image in which the three-dimensional object V is displayed as a target buffer, and secondly, "Paste" for paste processing
Third, the “Rectangle” button for designating the shape of the wall surface S ′ is operated. Fourth, FIG.
As shown in (4), the four vertices P ₁ ′, P ₂ ′, P ₃ ′, and P ₄ ′ of the square constituting the wall surface S ′ in the two-dimensional image may be designated as feature points.

As is clear from the above, the copy processing described in FIG. 9 is performed using the image buffer storing the two-dimensional image in which the three-dimensional object V is displayed as the target buffer, and further described in FIG. By performing such a paste process, the texture of the wall surface S can be copied to another wall surface S ′. That is, the user does not need to be particularly aware of the directions of the wall surfaces S and S ′, and only needs to perform the operation for performing the copy processing and the paste processing as described above.
The texture of the wall S in the dimensional image can be copied to another wall S ′.

Further, for example, in the paste buffer,
The characters “ABC” and “DEF” can be converted to the conventional 2D
Using paint or the like, draw in the same manner as drawing a character on a plane, and paste it by the paste processing described above in FIG.
By pasting on the wall of the house shown in FIG.
A natural two-dimensional image as shown in (B) can be obtained. That is, there is no need to draw characters along the wall surface.

Further, copy processing is performed, and some
Save the image in the paste buffer, or whatever
Paste the image buffer with the textures
The surface S selected as a buffer and shown in FIG.₁Configure
Vertex p₁, P_Two, P_Three, P_Four, Surface S_TwoVertices that make up
p_Three, P_Four, P_Five, P₆, Surface S_ThreeVertex p that constitutes_Two, P_Three,
p ₆, P₇And paste as
By performing the processing three times, as shown in FIG.
A two-dimensional image with a rectangular parallelepiped room added to the house wall
Can be easily created (in this case, the surface S₁No
To S_ThreeSpecify a rectangle as the shape information for
).

Here, as can be seen from FIG. 5, the three-dimensional object to be subjected to the paste processing may be displayed in the two-dimensional image stored in the buffer of interest, or may be displayed. You don't have to. That is, FIG.
Indicates that a three-dimensional object (a three-dimensional object to which a character is to be pasted) to be subjected to paste processing is an original two-dimensional image (FIG. 5).
FIG. 5C shows a case where the image is displayed on the two-dimensional image shown in FIG. 5A, and FIG. 5C shows a three-dimensional object (surfaces S ₁ and S in FIG. ₂ and S ₃ ) are not displayed in the original two-dimensional image. When the three-dimensional object to be subjected to the paste processing is not displayed in the original two-dimensional image, the feature point designated by the user is corrected as a point on the three-dimensional object, that is, as shown in FIG. Step S7
Is skipped. This means that in step S7,
As will be described later, the feature point specified by the user is corrected so as to be located on the contour line of the three-dimensional object, but if the three-dimensional object is not displayed in the original two-dimensional image, the contour is corrected. This is because there is no line.

A developed image obtained by copying from a two-dimensional image can be pasted on a two-dimensional image other than the original two-dimensional image. Need not have the same shape as the surface of the three-dimensional object from which the developed image is generated.
That is, for example, a three-dimensional object (a three-dimensional object in the shape of a house) displayed in a two-dimensional image as shown in FIG.
11B, a copy process is performed on the lattice pattern wall surface, and the developed image is stored in a paste buffer as shown in FIG. 11B, and then displayed on a two-dimensional image as shown in FIG. 11C. It is possible to perform a paste process on the side surface of the formed cylinder, and to paste a developed image of the lattice pattern along the side surface of the cylinder as shown in FIG.
That is, the texture of a certain three-dimensional object can be pasted as the texture of another three-dimensional object.

Further, in the paste buffer, any image other than the image obtained by the copy process can be stored. Therefore, according to the paste process, any image can be displayed as a two-dimensional image. It can be attached to a three-dimensional object (however, the three-dimensional object does not have to be displayed in the two-dimensional image as described above).

According to the copy processing and the paste processing, various three-dimensional operations can be performed on a two-dimensional image in addition to the above-described operations. That is, for example,
If the surface of the three-dimensional object displayed in the two-dimensional image has irregularities, the developed image of the surface is stored in the paste buffer by copy processing, and the irregularities are deleted from the paste buffer. If the image after deletion is pasted on the surface of the original three-dimensional object by the paste processing, a two-dimensional image in which the three-dimensional object with the unevenness removed is displayed. Also, for example, when performing a copy process and a paste process on the same three-dimensional object,
By changing the feature points specified when performing the paste processing to the feature points specified when performing the copy processing, movement, enlargement, reduction, and other non-linear deformation of a three-dimensional object (for example, special Deformation as if shooting with a lens having excellent optical characteristics).

Returning to FIG. 7, if it is determined in step S3 that an instruction to perform mat processing has been given, that is, if the "Matte" button of the command buttons 31 has been operated, the flow advances to step S15 to enter The event processing unit 11 activates the mat operation processing unit 16 to perform the mat processing. That is, in this case, the mat operation processing unit 16
Generates a mat of the three-dimensional object displayed on the two-dimensional image stored in the buffer of interest, stores the mat in the buffer of interest, and returns to step S1. The details of the mat processing will be described later.

If it is determined in step S3 that an instruction to perform the object attribute processing or the light source change processing has been issued, that is, if the “Materia
If the “l” button or the “Light” button has been operated, the process proceeds to step S16, where the input event processing unit 11 activates the object attribute operation processing unit 17, and executes the object attribute process or the light source change process (object attribute / light source change process). ), And returns to step S1. The details of the object attribute / light source change processing will be described later.

On the other hand, if it is determined in step S3 that the erasing process has been instructed, that is, if the "Erase" button of the command buttons 31 has been operated, the process proceeds to step S17, where the input event processing unit 11
Starts the erasing operation processing section 15, causes the erasing process to be performed, and returns to step S1. The details of the erasing process will be described later.

If it is determined in step S3 that the user has instructed to perform the paint process, that is, if the "Paint" button of the command buttons 31 has been operated, the flow advances to step S18 to execute the input event processing unit. 11
Starts the paint processing unit 18 to perform the paint processing, and returns to step S1. That is, in the paint processing, for example, the user operates the input device 6 using a paint tool that performs the same 2D paint as in the related art.
Part of the image stored in the paste buffer is deleted,
Alternatively, another image (including characters and graphics) is added.

For example, as described above, when the surface of the three-dimensional object displayed in the two-dimensional image has irregularities, the developed image of the surface is stored in the paste buffer by the copy process, and is stored in the paste buffer. Then, the three-dimensional object from which the unevenness has been removed is displayed by removing the uneven portion by the paint process and pasting the image after the deletion on the surface of the original three-dimensional object by the paste process. A two-dimensional image can be obtained. In addition, for example, a copy process is performed on a two-dimensional image in which a three-dimensional object as shown in FIG. 5A is displayed, and a developed image of the wall surface is stored in a paste buffer, and the paste buffer stores the developed image. By painting the stored image,
By drawing the letters "ABC" and "DEF" and pasting them on the original wall by paste processing, a natural two-dimensional image as shown in FIG. 5B can be obtained. it can. That is, three-dimensional drawing can be easily performed by 2D paint.

Next, for example, when performing the copy processing or the paste processing described in FIG. 9 or FIG.
(A) or as shown in FIG. 10 (B), the user positions the vertex P ₁ to P ₄ or vertices P ₁ 'to P _4', precisely, be designated as a feature point is skilled in the operation Even difficult. That is, for example, when performing the copy processing as shown in FIG. 9, the feature points P _{1 to} P ₄ designated by the user are shifted from the positions of the vertices to be designated as shown in FIG. Is common. Furthermore, even if the user is quite skilled in operation, if the two-dimensional image is blurred or the two-dimensional image is blurred, the position of the vertex to be originally specified is accurately specified as a feature point. It is difficult to do.

Therefore, in step S7 in FIG. 7, an automatic correction process for correcting a feature point specified by the user to a position to be originally specified is performed.

That is, the copy operation processing section 12 or the paste operation processing section 13 activates the conversion specification processing section 19 in step S7 and supplies the shape information and the feature points specified by the user operating the input device 6. I do.

Upon receiving the shape information and the feature points, the conversion designation processing unit 19 performs an automatic correction process for correcting the feature points.

Here, the points to be designated as the feature points are basically points on the contour line (boundary line) of the three-dimensional surface to be copied or pasted as described later. ing. Therefore, the conversion designation processing unit 19 extracts the contour of (the surface of) the three-dimensional shape to be subjected to the copy processing or the paste processing, and corrects the feature points so as to be positioned on the contour. I have.

FIG. 13 is a flowchart for explaining the automatic correction process performed by the conversion designation processing unit 19. Here, for simplicity of explanation, for example, FIG.
As shown in FIG. 2, the surface of the three-dimensional object displayed in the two-dimensional image to be copied or pasted is
It is a rectangle, and therefore, as shown in the figure, a case where four vertices of the rectangle should be designated as the feature points will be described as an example.

First, in step S21, the conversion designation processing section 19 selects a pair of two adjacent points from among the feature points input (designated) by the user. That is, as described above, the points to be designated as the feature points are basically points on the contour line of the surface of the three-dimensional object displayed in the two-dimensional image to be subjected to the copy processing or the paste processing, Further, as described later, which position on the contour is to be designated is determined in advance by the shape of the three-dimensional object. Therefore, among the feature points input by the user, two feature points adjacent on the contour line can be recognized based on the shape information. One of the set of points is selected in step S21.

Thereafter, the process proceeds to step 22, where a region to be subjected to edge detection (hereinafter, appropriately referred to as an edge candidate region) is detected in step S23 described later. That is, in step S22, for example, the 2 selected in step S21
A line segment connecting two feature points (hereinafter, appropriately referred to as a selected feature point) is obtained, and pixels within a predetermined distance (for example, a distance of 5 pixels) from pixels on the line segment are detected. . Then, an area composed of such pixels is set as an edge candidate area.

That is, for example, as shown in FIG. 12, when feature points P _{1 to} P ₄ are designated, when feature points P ₁ and P ₂ are selected as selected feature points, A region indicated by hatching in FIG. 14 is detected as an edge candidate region.

In step S22, it is possible to set in advance how far a pixel within a range from the pixels on the line segment connecting the two selected feature points is to be detected. Alternatively, it is also possible to have the user input by operating the input device 6.

After the edge candidate area is detected, step S2
Proceeding to 3, the edge is detected for the edge candidate area.

Here, various methods have been conventionally proposed as a method for detecting an edge, and it is possible to use any one of them.
This is performed by the following method.

That is, in this case, pixels in the edge candidate area are filtered by an edge detection filter called a Sobel operator,
A gradient (gradient vector) at each pixel is determined. A method of detecting a pixel having a large gradient as an edge is conventionally known, but here, further, a unit vector in a direction orthogonal to a line connecting two selected feature points, that is, a normal vector, An inner product of each pixel in the edge candidate area with each gradient is calculated, and a pixel whose inner product is equal to or larger than a predetermined threshold is detected as a pixel forming an edge (hereinafter, appropriately referred to as an edge pixel).

Here, as shown in FIG. 12, the feature points to be subjected to copy processing or paste processing are as follows.
A point near, if not coincident with, a vertex of a quadrangle which is a surface forming a three-dimensional object is specified. Therefore, the line segment connecting the two selected feature points almost coincides with the rectangular outline, which is a surface constituting the three-dimensional object to be subjected to the copy processing or the paste processing. As a result, the inner product of the normal vector and the gradient represents a component of the gradient in the direction of the normal vector, that is, a component of the gradient in a direction substantially orthogonal to the contour. Therefore, the edge is detected based on the magnitude of the inner product. This makes it possible to perform edge detection in which the influence of noise and the like is reduced as compared with the case where edges are detected based on the magnitude of the gradient itself. That is, it is possible to perform edge detection that is not easily affected by noise on a two-dimensional image or texture such as a stripe pattern.

After detecting the edge pixel, the process proceeds to step S24, and a straight line passing through the edge pixel is obtained by, for example, the least square method. That is, in step S24, a straight line that minimizes the sum of the squares of the distances from the edge pixels detected in step S23 (hereinafter, appropriately referred to as an edge straight line) is obtained.

Then, the process proceeds to a step S25, where it is determined whether or not an edge straight line has been obtained for all pairs of two adjacent feature points among the feature points input by the user. If it is determined that the edge straight line has not been obtained for all the cases, the process proceeds to step S
21, two adjacent feature points for which no edge straight line has been determined are selected as selected feature points, and the same processing is repeated thereafter.

If it is determined in step S25 that an edge straight line has been obtained for all two sets of adjacent feature points, that is, in FIG. 12, the set of feature points P ₁ and P ₂ , P ₂ P ₃ set, the set of P ₃ and P _4, P ₄ and P ₁
For each set of, if the edge straight line is found,
Proceeding to step S26, the intersections of the four edge straight lines determined for the four sets are determined. That is, an intersection (hereinafter, appropriately referred to as a first intersection) between an edge straight line obtained for the set of feature points P ₄ and P ₁ and an edge straight line obtained for the set of feature points P ₁ and P ₂ , a feature point P ₁ and set edge obtained for the P ₂ straight and the feature point P ₂
And the intersection between the obtained edge straight for the set of P ₃ (hereinafter referred to as a second intersection point), the edge line and the feature point obtained for the set of feature points P ₂ and P ₃ of P ₃ and P ₄ Intersecting points (hereinafter appropriately referred to as third intersecting points) with the edge straight line obtained for the set, and the edge straight lines and the feature points P ₄ and P ₁ obtained for the set of feature points P ₃ and P ₄
(Hereinafter, appropriately referred to as a fourth intersection) with the edge straight line determined for the set of. Further, in step S26, the nearest feature point at each intersection, is corrected to the position of the intersection, i.e., the feature point P ₁ to P ₄ are respectively corrected to a position of the first to fourth intersection, returns .

According to the automatic correction process described above, the feature points P _{1 to} P ₄ designated by the user are moved to the positions of the vertices of the quadrangle that should be designated as feature points. Thus, the position to be specified as a feature point does not have to be specified accurately. That is, the user only needs to specify a point close to the position to be originally specified as the feature point.

In the embodiment of FIG. 7, the automatic correction processing of the feature points is always performed.
It is possible to perform it only when there is an instruction from the user. Further, the feature points after the correction by the automatic correction process can be further corrected by the user.

Next, the feature point automatic correction processing can be performed, for example, as follows, using the edge pixel detection method described with reference to FIG.

That is, it is assumed that a two-dimensional image as shown in FIG. 15 is stored in the buffer of interest. In FIG. 15, the hatched portions are
It represents a three-dimensional object displayed on a two-dimensional image (since this is a three-dimensional object in a three-dimensional space projected onto a screen, hereinafter, appropriately referred to as a projected image).

In FIG. 15, for the two-dimensional image, the point P _A
And P _B are designated as feature points of the projected image of the three-dimensional object. Note that the feature points P _A and P _B are adjacent on the contour line of the projected image of the three-dimensional object.

[0096] Now, the two-dimensional image stored in the target buffer, as shown in FIG. 15, x _p axis or y _p axis, two-dimensional coordinate system with the horizontal axis or the vertical axis, respectively (hereinafter, the screen Define a coordinate system)
The shape of the three-dimensional object displayed in the dimension image can be recognized from the shape information, further, knowing the shape of the three-dimensional object, of the projection image, the outline between the characteristic points P _A and P _B contour line (the projected image is basically because constitutes a closed loop is present contour two between the characteristic points P _a and P _B, wherein, the two contour (FIG. 15 In FIG. 15, a thin line and a thick line indicate that no other feature point exists on the line (shown by a thick line in FIG. 15). In general, x _p is expressed using one or more parameters. and y _p can be defined by the equation f = 0 by the function as a variable f (function f has every shape of the three-dimensional object, different types of (implementation as referred by programming)).

That is, the parameters are now represented by C ₁ , C ₂ ,.
..., expressed as C N _(N is an integer of 1 or more), the contour line between the characteristic points P _A and P _B can be defined by the following equation.

F (x _p , y _p , C ₁ , C ₂ ,..., C _N ) = 0 (1) On the other hand, in a pixel forming a true contour,
The component of the pixel value in the direction orthogonal to the contour line changes rapidly. Therefore, of the differential value of the pixel value of the pixel on the screen coordinate system represented by the equation (1), the parameter C ₁ that maximizes the component of the pixel in the direction of the normal vector (the direction orthogonal to the contour) at that pixel. , C ₂ ,..., C _N , it can be said that the line defined by equation (1) expressed using the parameters represents the true contour.

Therefore, a line corresponding to the contour between the feature points P _A and P _B defined by the equation (1) (as described above, the feature point PA and the feature point P _A corresponding to the contour of the projection image). there two lines between the P _B, but of its two lines, for one) other feature points are not present, the energy E _f, for example, is defined by the following equation.

[0100]

(Equation 1)(2) Here, in Expression (2), Σ is represented by Expression (1).
Take summaries for all the pixels on the line
Represent. ▽ represents a nabula operator, and B (x _p, Y_p)
Is the position of the screen coordinate system (x_p, Y_p)
Represents a pixel value. Further, the vector n_f(X_p, Y_p)
Position (x_p, Y_pThe normal vector (Fig.
15) (Vector in the direction orthogonal to the line represented by equation (1)
(In this case, outside or inside the area surrounded by the outline)
Of the sides, for example, the vector in the outward direction)
Position (x_p, Y_p) Represents a unit vector). Ma
| X | represents the norm of the vector x.
Represents Further, K is a pixel on the line represented by equation (1).
Represents the total number of

▽ B (x _p , y _p ) in the equation (2) represents the differential value of the pixel value of the pixel on the line represented by the equation (1), that is, the gradient. It can be obtained by using That is, ▽ B
(X _p , y _p ) can be obtained according to the following equation.

[0102] _{▽ B (x p, y p} ) = (△ x B (x p, y p), △ y B (x p, y p)) △ x B (x p, y p) = B (x _{p + 1, y p -1)} + 2B (x p + 1, y p) + B (x p + 1, y p +1) - (B (x p -1, y p -1) + 2B (x p -1, y p _{) + B (x p -1,} y p +1)) △ y B (x p, y p) = B (x p -1, y p +1) + 2B (x p, y p +1) + B (x p +1, _{y p +1) - (B (} x p -1, y p -1) + 2B (x p, y p -1) + B (x p + 1, y p -1)) ··· (3) In addition, the energy E parameters to maximize _f C _{n (n} =
, N) can be obtained, for example, as follows. That is, for example, _{assuming that} the initial value of the parameter C _n is represented by C _n ′, the parameter C _n is represented by C _n ′.
Equation (2) may be calculated by changing the range from −ε _{n to} C _n ′ + ε _n , and the parameter C _{n at} which the value is maximized may be obtained (ε _n is a predetermined minute value). .

Specifically, assuming that the contour between the feature points P _A and P _B is, for example, a line segment L, the equation (2) can be expressed as follows.

[0104]

(Equation 2) (4) Here, as described above, Σ indicates that the summation is performed on the line represented by the expression (1), that is, here, all the pixels on the line segment L are taken. Also, the vector n
_L corresponds to the vector n _f (x _p , y _p ) in equation (2) and represents a normal vector of the line segment L. Further, in equation (4), K represents the total number of pixels on the line segment L.

On the other hand, when the contour between the feature points P _A and P _B is a line segment, the equation (1) can be expressed as follows using, for example, three parameters C ₁ , C ₂ and C _3. Can be expressed as

[0106] _{f (x p, y p,} C 1, C 2, C 3) = C 1 x p + C 2 y p + C 3 = 0 ··· (5) Then, Now, the feature point P _A or P _B By the coordinates of (x ₁ ,
y ₁ ) or (x ₂ , y ₂ ), the expression (5)
The parameters C ₁ , C ₂ , and C ₃ in can be expressed as follows.

C ₁ = y ₁ −y ₂ , C ₂ = −x ₁ + x ₂ , C ₃ = x ₁ y ₂ −x ₂ y ₁ (6) Accordingly, in this case, As the initial values C ₁ ′, C ₂ ′, C ₃ ′ of the parameters C ₁ , C ₂ , C ₃
The energy E _f represented by the equation (4) is calculated while changing each value, and the parameters C ₁ , C ₂ , and C ₃ when the energy E _f is maximum are obtained. Parameter C _1, C _2, C ₃ with formula represented in this case (5) represents the true contour line between the characteristic points P _A and P _B, therefore, the formula (5) in the line represented, if moving the feature points P _a and P _B, the feature point P _a and P _B, is to be moved on the true contour line.

When the contour between the feature points P _A and P _B is a line segment, as is apparent from the equation (6), the parameter C
Changing ₁ , C ₂ and C ₃ means that the characteristic point P _A or P
Each coordinates _B (x _1, y ₁₎ or (x _2, y ₂₎ corresponds to changing, therefore, the coordinates _{x 1, y 1, x 2} , y 2
Is a parameter of the function f.

When the contour line between the feature points P _A and P _B is a line segment, the feature points for all _three parameters C ₁ , C ₂ and C ₃ of the function f in the equation (5) are obtained. P _A and P
_An initial value is given by _B , but if the contour between the feature points P _A and P _B is a curve, the initial value is not always given for all parameters of the function f. In this case, for a parameter for which an initial value is not provided, a preset value may be provided as an initial value, or a value to be used as an initial value may be input by a user.

Further, in the above case, for simplicity of description, the description has been made focusing on only two adjacent feature points. In this case, however, the following problems occur in actual processing. May occur. That is, assuming that three feature points, a feature point # 1, a feature point # 2 adjacent to the feature point # 1, and a feature point # 3 adjacent to the feature point # 1, have been designated on the contour, attention is paid to feature points # 1 and # 2. When the above-described automatic correction processing is performed, and the automatic correction processing is performed by focusing on the feature points # 2 and # 3, the automatic correction processing is performed by focusing on the feature points # 1 and # 2. In some cases, the position of the corrected feature point # 2 and the position of the corrected feature point # 2 obtained by performing the automatic correction process while paying attention to the feature points # 2 and # 3 may be shifted.

Therefore, in the actual processing, the parameters of equation (1) defining the contour between feature points # 1 and # 2 (hereinafter referred to as first parameters as appropriate), and feature points # 2 and # 2 The parameters of equation (1) that define the contour between 3 (hereinafter referred to as second parameters, as appropriate) are:
It is desirable to change the corrected feature point # 2 so that the above-described positional displacement does not occur. In this case, the first and second parameters are no longer independent and the relationship between them is
Since the positional deviation of the feature point # 2 after correction is restricted to the condition that no, for example, first to maximize the energy E _f characteristic points # 1 and # are expressions obtained by paying attention to 2 (2) 1
And the second parameter that maximizes the energy E _f in equation (2) obtained by focusing on feature points # 2 and # 3 satisfy the condition that no displacement of feature point # 2 occurs. May not be in a relationship. Therefore, in this case, for example, the energy E _f characteristic points # 1 and # are expressions obtained by paying attention to 2 (2), feature points # 2 and # 3
, The first and second parameters that maximize the sum with the energy E _f of Equation (2) obtained by paying attention to

Here, for the sake of simplicity, the case where the position of one feature point # 2 is shifted has been described as an example.
Since the occurrence of this shift can be said for all the feature points on the same contour, the parameters of the equation (1) that define the contour between adjacent pairs of feature points on the same contour are all Is desirably determined by applying the above method. That is, they are on the same contour,
It is desirable to find the sum of the energies E _f for the pair of adjacent feature points and find the parameter that maximizes the sum.

Next, the processing when the above-described automatic correction processing is performed in step S7 in FIG. 7 will be further described with reference to the flowchart in FIG.

Here, for simplicity of explanation,
The projected image is a square, the four vertices of the rectangle, and what is designated as a feature point P ₁ to P _4.

In this case, the three parameters C ₁ , C ₂ and C ₃ in the equation (5), that is, the x-coordinates x ₁ and x ₂ and the y-coordinates y ₁ and y ₁ , of the two feature points in the equation (6) While changing y ₂ , the energy E _f expressed by the equation (4) is calculated, and x ₁ , x ₂ , y ₁ and y ₂ that maximize the value are obtained. It will be more appropriate. Therefore, here, a simplified method is used to reduce the amount of calculation.

That is, here, first, step S
At 31, four feature points P₁Or P_FourAny of
One is selected as the feature point of interest, and step S32
To pass through the feature point of interest (the feature point of interest and other
(Made by connecting feature points) Energy of two sides of a square
Lugi E_fAre required respectively. That is, for example,
In FIG. 17, the point P₁
Or P_FourAre designated as feature points, and the feature points P_Four
Is selected as a feature point of interest. In this case,
In step S32, the square P₁P_TwoP_ThreeP_FourOf the four sides of
Attention feature point P _FourSide P passing through_FourP₁And P_ThreeP_FourEnergy
E_fIs required.

[0117] Then, the process proceeds to step S33, among the two sides energy E _f is determined in step S32, the larger its energy E _f is selected as that along the contour of the more projected image, the sides The feature point of interest is moved along a straight line (hereinafter, this straight line is also referred to as an edge straight line) as appropriate. Then, the feature point at the position after the movement, it the energy E _f of the two sides each of the two feature points and the adjacent building is calculated. That is, for example, in FIG. 17, when the feature point P ₄ is set as the feature point of interest, for example, the energy of the side P ₃ P ₄ of the energy E _f of the side P ₄ P ₁ or P ₃ P ₄ Is larger than the energy of the side P ₄ P ₁ , the target feature point P ₄ is moved along the straight line P ₃ P ₄ as shown by the arrow D in the figure. Then, the side P ₃ P ₄ of the energy E _f, and after moving feature point P ₄ with other 1 adjacent thereto to the feature point P ₃ of 1 adjacent thereto as the target feature point P ₄ after the movement is made side P ₄ P ₁ of the energy E _f to make the feature point P ₁ of is determined.

[0118] Then, the process proceeds to step S34, when the sum of the energy E _f of the two sides determined at step S33 it is determined whether the maximum value, is determined not to have reached the maximum value, step Returning to S33, the feature point of interest is moved again along the edge straight line, and the same processing is repeated thereafter.

[0119] On the other hand, in step S34, when the sum of the energy E _f of the two sides obtained in step S33 is determined to be the maximum value, i.e., for example, as described above, feature point P _4, by moving along a straight line P ₃ P _4, the normal vector n ₁₄ sides P ₄ P ₁ is originally the side connecting the two vertices to be designated by the feature point P ₁ and P ₄ , If they become substantially orthogonal, the process proceeds to step S35, and the feature point P designated by the user
All ₁ to P _4, is determined whether or have been processed as the target feature points, still, all feature points P ₁ to P ₄ specified by the user, is determined not processed as feature point In this case, the process returns to step S31, and a feature point that has not yet been set as a feature point of interest is newly selected as a feature point of interest, and the processing of step S32 and subsequent steps is repeated.

If it is determined in step S35 that all the feature points P _{1 to} P ₄ specified by the user have been processed as feature points of interest, the process proceeds to step S3.
6 and among the feature points P _{1 to} P ₄ , the previous step S
It is determined whether or not the movement amount (distance between the feature point before movement and the feature point after movement) moved in the process of 33 is larger than a predetermined threshold value ε. In step S36, among the characteristic points P ₁ to P _4, the last step S33
If it is determined that there is a movement amount that is larger than the predetermined threshold value ε in the processing of step, the process returns to step S31, and thereafter, the same processing is repeated. That is, the feature points P _{1 to} P ₄ are sequentially set as the feature points of interest, and their positions are moved so that the sum of the energies of the two sides created by each feature point of interest becomes maximum.

On the other hand, in step S36, the characteristic point P
Among ₁ to P _4, when the amount of movement that has moved in the process of the previous step S33 has been determined that there is no greater than a predetermined threshold epsilon, i.e., the processing of the previous step S33, the feature point P ₁ or none of the P _4, when hardly moved, and returns.

Next, the copy operation processing unit 12 or the paste operation processing unit 13 shown in FIG. 4 uses the characteristic points corrected as described above and the shape information to form an inverse conversion formula or a forward conversion formula, respectively. The calculation method of the inverse conversion formula and the forward conversion formula will be described.

For example, the camera 9 (FIG. 3)
When a three-dimensional object in a three-dimensional space as shown in FIG. 8A is photographed, the two-dimensional image output by the camera 9 includes FIG.
As shown in FIG. 8B, a projection image (projection image) obtained by projecting the three-dimensional object on a screen is displayed. Therefore, a two-dimensional image displaying such a projected image is stored in the image buffer (excluding the paste buffer).

On the other hand, in the paste buffer, FIG.
As shown in (C), a developed image obtained by developing the surface of a three-dimensional object (FIG. 18A) in a three-dimensional space on a two-dimensional plane is stored. That is, if the plane of the three-dimensional object is a plane, for example, a two-dimensional image of the plane viewed from the front is stored in the paste buffer. If the surface of the three-dimensional object is a side surface of a cylinder, for example, a two-dimensional image obtained by developing the side surface on a two-dimensional plane and making it rectangular is stored in the paste buffer. Further, if the surface of the three-dimensional object is a spherical surface, a two-dimensional image in which the spherical surface is developed on a two-dimensional plane by, for example, an equirectangular projection method or the like used for creating a map is stored. For example, if the surface of the three-dimensional object is a cubic surface represented by two auxiliary variables s and t, such as a Bezier surface, the auxiliary variable s or t is set to the horizontal axis or the vertical axis, respectively. The two-dimensional image obtained by developing the curved surface into a two-dimensional plane is stored in the paste buffer. Furthermore, for example, if the three-dimensional object is a rotating body, the direction of the circumference or the direction of the rotating axis is set to the horizontal axis or the vertical axis, respectively, and the rotating body is developed on a two-dimensional plane to form a two-dimensional image.
Stored in paste buffer.

In the copy process, as shown in FIG. 18 (B), several characteristic points on the (plane) projection image of the three-dimensional object displayed on the two-dimensional image stored in the buffer of interest. 18 (B), a portion indicated by a triangle) and 3
When the shape information of the (dimensional) surface of the three-dimensional object is designated, an inverse transformation equation is calculated. According to the inverse transformation equation, a projection image as shown in FIG. A developed image is generated and stored in the paste buffer.
Also, in the paste processing, as shown in FIG. 18B, in the buffer of interest, some feature points (FIG. 18) on the region where the (plane) projection image of the three-dimensional object is to be pasted
In (B), when a part indicated by a triangle) and shape information (of a surface) of the three-dimensional object are designated, a forward conversion formula is calculated, and the forward conversion formula is calculated according to the forward conversion formula. Image (expanded image) as shown in 18 (C)
Thus, a projection image as shown in FIG. 18B is generated and pasted to the image buffer.

Therefore, when calculating the inverse transform formula and the forward transform formula, the mutual relationship between the three-dimensional object in the three-dimensional space, its projected image, and its developed image becomes a problem.

Therefore, for example, as shown in FIG.
The two-dimensional coordinate system for the developed image and the three-dimensional
3D coordinate system and 2D for projected image
Consider a coordinate system. Note that the two-dimensional coordinates of the developed image
The x, y coordinates in the system are x_e, Y_eAnd about three-dimensional objects
X, y, z coordinates in all three-dimensional coordinate systems are x_o, Y _o,
z_oAnd x, in the two-dimensional coordinate system for the projected image
y coordinate is x_p, Y_pRespectively. Also, the point (x_p,
y_pPixel value of the projected image in ()
Value) to B (x_p, Y_p) And a point (x_e, Y_eDeployment in)
The pixel value of the image (the stored value of the paste buffer) is C
(X_e, Y_e), Respectively. In addition, the developed image
A map to be attached to the surface of a three-dimensional object
A mapping to be converted into a projection image is represented by M.

Here, the two-dimensional coordinate system for the developed image, the three-dimensional coordinate system for the three-dimensional object, and the two-dimensional coordinate system for the projected image can be arbitrarily set. That is, for example, a three-dimensional coordinate system for a three-dimensional object can be formed in a form convenient for expressing the three-dimensional object. Specifically, for example, when the three-dimensional object is a cylinder, the center of one of the two bottom circles is defined as the origin (0, 0, 0), and its rotation axis (two bottom surfaces) (A straight line connecting the centers of the circles) can be made to coincide with the _yo axis. Further, for example, when the surface of the three-dimensional object is a rectangle, for example, the lower left one of the four vertices of the rectangle is defined as the origin (0,
And 0,0), the horizontal axis or the vertical axis of the rectangle, it is possible to so as to coincide with the x _o axis or y _o axis.

The points (x _o , y _o , z _o ) on the three-dimensional object are:
The points (x _e , y _e ) on the developed image correspond one-to-one, and the relationship between them, that is, the mapping ψ (or ψ ⁻¹ ) can be obtained by geometric calculation. Therefore, a point (x _o , y _o , z _o ) on the three-dimensional object and a point (x _p ,
If the relationship with y _p ), that is, the mapping M (or M ⁻¹ ) is known, the point (x _e , y _e ) on the developed image is converted into a point (x _p , y _p ) on the projection image. The mapping Mψ can be determined.

However, when attention is paid only to the three-dimensional object and the projection image, the mapping M is such that the point (x _o , y _o , z _o ) on the three-dimensional object is the point (x _p , y If it is not known which of _p ) is projected, it cannot be determined. Then, a point (x _o , y _o ,
z _o) is the point (x _p on the projection image, whether being projected in any y _p), in order to obtain the projected image is basically as described above, a plurality of two-dimensional images It is necessary to prepare and use the principle of triangulation. Furthermore, in order to represent a three-dimensional model of the three-dimensional object, a point (x _o , y _o , z _o ) on the three-dimensional object in the three-dimensional space when viewed from a viewpoint (the origin of camera coordinates described later) A memory for temporarily storing the three-dimensional data is required, and a calculation for handling such three-dimensional data is also required.

Therefore, here, the mapping ψ is obtained based on the shape information given by the user, the mapping M is found based on the feature points given by the user, and the relationship between the developed image and the projected image is calculated. That is, a mapping Mψ that converts a developed image into a projected image is determined. Note that this mapping Mψ is a forward conversion formula, and this mapping Mψ
The mapping (Mψ) ⁻¹ that performs the inverse transformation of the transformation corresponding to ψ is the inverse transformation equation. Therefore, the inverse transformation equation can be obtained by calculating the inverse matrix of the mapping Mψ using, for example, Gaussian elimination.

As described above, the inverse transform equation (Mψ) ⁻¹ can be easily obtained by calculating the forward transform equation Mψ.
Here, a method of calculating the forward conversion formula M # will be described.

As described above, since the point (x _o , y _o , z _o ) on the three-dimensional object and the point (x _e , y _e ) on the developed image correspond one-to-one, the point (X _o , y _o , z _o )
Using the mapping ψ and the point (x _e , y _e ), it can be expressed as in equation (7).

[0134]

(Equation 3) (7) Now, in a three-dimensional space, when a three-dimensional object is photographed by the camera 9 and its projection image is obtained, the position of the camera 9 (x
_c , y _c , z _c ) can be considered, and the points (x _o , y _o ,
z _o) the camera coordinate (x _c of, y _c, z _c) is a 9 elements representing a rotating movement in three-dimensional space _{r 11, r 1 2, r} 13, r
₂₁ , r ₂₂ , r ₂₃ , r ₃₁ , r ₃₂ , r ₃₃ and 3
The three elements representing the translation in the three-dimensional space are t ₁ ,
t ₂ and t ₃ can be represented by the following equations.

[0135]

(Equation 4) (8) points on the projected image (x _p, y _p) is 3 in the camera coordinate
A point (x _c , y _c , z _c ) on a three-dimensional object is projected on a screen, and this transformation is a perspective transformation (projection transformation) as described in the above-mentioned document 1. From the above, it is assumed that both the denominator and the numerator are linear expressions of any of the x-coordinate, y-coordinate, and z-coordinate (primary and constant terms of x, y, and z). It can be represented by a rational expression. That is, assuming that w is an arbitrary number representing homogeneous coordinates, the point (x _p , y _p ) of the projected image on the screen can be represented by the homogeneous coordinates, for example, as in equation (9). .

[0136]

(Equation 5) (9) Here, assuming that the focal length of the camera 9 is f and the horizontal or vertical length of one pixel of the screen in the camera coordinate system is h or v, equation (9) f _h or f _v in represent respectively the f × h or f × v.

As is apparent from equation (9), the value obtained by dividing the x coordinate wx _p or y coordinate wy _p of homogeneous coordinates by the z coordinate w is the x coordinate x _p or y coordinate y _{p of the} projected image. Becomes

From equations (7) to (9), equation (10) can be derived.

[0139]

(Equation 6) (10) In equation (10), M is ψ (x _e , y _e ), that is, (x _o , y _o , z _o ), as is clear from FIG.
This is a matrix to be converted into (x _p , y _p ), and is represented by the following equation.

[0140]

(Equation 7)... (11) ψ (ψ in equation (10)₁, Ψ_Two, Ψ_Three) Gives the user
Based on the shape information given by
Can be asked. Also, the line in equation (10)
Column M is calculated based on feature points given by the user.
Can be That is, the matrix M is represented by Expression (11).
Thus, in general, the eleven elements m₁₁, M ₁₂, M₁₃,
m₁₄, M_{twenty one}, M_{twenty two}, M_{twenty three}, M_{twenty four}, M₃₁, M₃₂, M₃₃Not
Because it has as a known number, for example, the degree of freedom on the projected image
Are 5 points with 2 points and 1 point with 1 degree of freedom.
If specified as₁₁To m₁₄, M_{twenty one}To m_{twenty four}, M
₃₁To m₃₃Can be requested. Note that on the projected image
If more than the above 6 points are specified,
For example, by using the least square method or the like, 11 elements m
₁₁To m₁₄, M_{twenty one}To m_{twenty four}, M₃₁To m₃₃Ask for.

As described above, here, since the matrix M is determined based on the feature points given by the user, there is no need to handle the three-dimensional data of the three-dimensional object in the camera coordinates. There is no need to perform calculations using three-dimensional data. Furthermore, the same 3
It is not necessary to use a plurality of two-dimensional images obtained by photographing a three-dimensional object from a plurality of positions. That is, the matrix M can be obtained from one two-dimensional image obtained by photographing a three-dimensional object from a certain position. As a result, the mapping Mψ (and (Mψ) ⁻¹ ) can also be obtained without performing calculations using three-dimensional data in camera coordinates or using a plurality of two-dimensional images.

Next, a description will be given of a method of calculating ψ and M when several primitive shapes are used as a three-dimensional object.

In the case where the three-dimensional object is, for example, a rectangular parallelepiped, ψ in equation (7), that is, a point (x _o , y _o , z _o ) in a three-dimensional space on a plane (rectangle) of the rectangular parallelepiped. ) And the point (x _e , y _e ) on the developed image can be expressed, for example, as follows.

[0144]

(Equation 8) (12) Here, as the two-dimensional coordinates (hereinafter, appropriately referred to as screen coordinates) of the projected image, the three-dimensional coordinates (hereinafter, simply referred to as three-dimensional coordinates) of the three-dimensional object are used. coordinates on a plane of z _o = 0,
The one having the same x and y coordinates (x _e = x _o , y _e = y _o ), that is, the xy plane of the three-dimensional coordinate system for the three-dimensional object is used. Also, as a three-dimensional coordinate system,
One in which one vertex of a certain surface of the rectangular parallelepiped coincides with the origin and the surface is included in the xy plane is used.

[0145] In formula (12), since it is z _o = 0, the third column _{m 1 3, m 23, m} 33 of the matrix M in equation (11) it becomes unnecessary. That is, in this case, equation (10) can be expressed as follows.

[0146]

(Equation 9)(13) Therefore, in this case, the element of the matrix M to be obtained is m₁₁,
m₁₂, M₁₄, M_{twenty one}, M _{twenty two}, M_{twenty four}, M₃₁, M₃₂Of eight
Therefore, four points on the projected image corresponding to four points on the developed image
Is specified as a feature point (t of four feature points)_p, Y_pseat
(Providing a total of eight pieces of information for the target))
Element m₁₁, M₁₂, M₁₄, M_{twenty one}, M_{twenty two}, M_{twenty four}, M₃₁, M₃₂To
You can ask.

When the three-dimensional object is a rectangular parallelepiped, the projected image of the plane (rectangle) is, for example, as shown in FIG.
It becomes a square as shown in FIG. So, here,
For the user, for example, the four vertices of the square projection image,
It will be designated as a feature point. Also, here, for example, as shown in FIG. 20A, the vertices of the rectangular projection image are set to the points P ₁ ,
Thereby respectively _{P 2, P 3, P 4} , FIG. 20 (B)
As shown in the figure, the vertex of the developed image stored in the paste buffer in a rectangular shape is shifted clockwise from the upper left vertex to the point E.
_The points P ₁ and E ₁ and the point P ₂ are denoted as ₁ , E ₂ , E ₃ and E ₄ respectively.
And E ₂ , points P ₃ and E ₃ , and points P ₄ and E ₄ , respectively.

Further, two-dimensional coordinates for the developed image
(Hereinafter referred to as paste buffer coordinates as appropriate)
Point E₁, E_Two, E_Three, E_FourOf the coordinates (0,
0), (L _W, 0), (L_W, L_H), (0, L_H)
You. Where L_WOr L_HIs a rectangular exhibition
Indicates the horizontal or vertical length of the open image.
User can set it as much as capacity allows
It is. Also, L_WAnd L_HIs set up for the user in this way.
In addition, the expanded image is the paste buffer, for example,
A size (area) of a predetermined ratio such as 80%
It is also possible to set things.

In this case, screen coordinates of points P ₁ , P ₂ , P ₃ , and P ₄ as feature points designated by the user are
(X _p1 , y _p1 ), (x _p2 , y _p2 ), (x _p3 ,
If y _p3 ) and (x _p4 , y _p4 ), the following equation holds from equation (10).

[0150]

(Equation 10) (14) In Expression (14), w ₁ , w ₂ , w ₃ , and w ₄ are arbitrary numbers representing homogeneous coordinates.

The matrix M in the expression (14) is obtained by the expression (13)
8 elements m in₁₁, M₁₂, M ₁₄, M_{twenty one}, M_{twenty two}, M
_{twenty four}, M₃₁, M₃₂Is an unknown.
From (14), can we make eight equations?
Eight unknowns m₁₁, M₁₂, M₁₄, M_{twenty one}, M_{twenty two},
m_{twenty four}, M₃₁, M₃₂That is, the matrix M can be obtained.
You. Note that the feature points for obtaining the matrix M are basically
The feature points specified by the user are automatically
What is corrected by the correction processing is used.

When the three-dimensional object is a rectangular parallelepiped and a copy process or a paste process is performed on the surface (rectangle), ψ is obtained from Expression (12) and M is obtained from Expression (14). Then, in the case of paste processing, the mapping Mψ is directly used as a forward conversion formula, and in the case of copy processing, the inverse matrix of the mapping Mψ is obtained and used as an inverse transformation formula. Here, not only in the case where the three-dimensional object is a rectangular parallelepiped, the conversion formula used in the paste processing is:
Since the mapping M # is used as it is, it is called a forward conversion formula, and the conversion formula used in the copy processing is called an inverse conversion formula since an inverse mapping of the mapping M # is used.

In the case where the copy processing is performed, the image conversion processing unit 20 stores the buffer of interest in FIG.
Each of the pixels in the quadrangle P ₁ P ₂ P ₃ P ₄ which is the projected image shown in FIG. 20 is in accordance with the inverse transformation formula (Mψ) ⁻¹ , and the rectangle E ₁ E _{2 in the} paste buffer shown in FIG. E ₃ are written into the corresponding location in E _4.

If one pixel of the buffer of interest corresponds to a plurality of pixels of the paste buffer, the pixel value of one pixel of the buffer of interest may be written for all of the plurality of pixels. The interpolation value is obtained by performing interpolation using a plurality of pixels of the buffer of interest,
The interpolation value may be written in the paste buffer. That is, for example, the pixel A having the buffer of interest is
In a case where _two pixels a ₁ and a ₂ adjacent to each other in the paste buffer correspond to each other, the pixel value of the pixel A is written to the pixel a ₁ , and the pixel A and the adjacent buffer of the target buffer are written to the pixel a ₂ . The average value of the pixel values with the pixels can be written.

Conversely, when a plurality of pixels in the buffer of interest correspond to one pixel in the paste buffer, for example, any one pixel value of the plurality of pixels may be written in the paste buffer. Alternatively, a weighted addition value or the like may be obtained by applying a filter to the plurality of pixels, and the weighted addition value may be written in the paste buffer.

Here, if the pixel value is written into the paste buffer according to the inverse transformation formula (Mψ) ⁻¹ , as described above, each pixel of the projected image of the buffer of interest becomes It may be performed by calculating which pixel is converted, or conversely, by calculating which pixel of the projected image is converted into each pixel of the developed image.

On the other hand, in the case of performing the paste processing, the image conversion processing section 20 prepares a rectangle E ₁ E ₂ in the paste buffer, which is a developed image shown in FIG.
Each pixel in E ₃ E ₄ is in accordance with the forward conversion formula M 式,
The square P ₁ P ₂ P shown in FIG.
It is written to the corresponding location in ₃ P _4, thereby, on a two-dimensional image stored in the target buffer, the projected image is pasted.

When one pixel in the paste buffer corresponds to a plurality of pixels in the buffer of interest, the pixel value of one pixel in the paste buffer is written for all of the plurality of pixels, as in the copy process. Alternatively, an interpolation value may be obtained by performing interpolation using a plurality of pixels of the paste buffer, and the interpolation value may be written.

Conversely, a plurality of pixels in the paste buffer
In the case of corresponding to one pixel of the buffer of interest, for example, the pixel value of any one of the plurality of pixels may be written to the buffer of interest or weighted by applying a filter to the plurality of pixels. Find the added value, etc.
The weighted addition value may be written in the buffer of interest.

Here, if the writing of pixel values to the buffer of interest also follows the forward conversion formula Mψ, as described above, each pixel of the developed image of the paste buffer is converted to any pixel of the projection image of the buffer of interest. Alternatively, it may be calculated by calculating which pixel is to be converted into each pixel of the projected image, or conversely, which pixel of the developed image is converted.

Next, in the case where the three-dimensional object is, for example, a cylinder, copy processing for generating a developed image of the side surface and storing the image in the paste buffer, or pasting the image stored in the paste buffer to the side surface of the cylinder When the paste processing is performed, the reverse conversion formula or the forward conversion formula is calculated as follows.

That is, in this case, as shown in FIG.
And three-dimensional coordinates representing the position of the cylinder in three-dimensional space
Then, of the upper or lower bottom (circle),
If the center of the upper bottom is the origin (0,0,0)
Y from the bottom of_oTake the axis,
Point O described later_FiveFrom the direction of _o
Consider the axis (x_oThe axis is y_oZ from axis_oAxial
When turning the right hand screw in the direction, the right hand screw advances
Take in the direction).

Then, for example, the buffer 2 of interest
Dimension image, the cylinder of FIG. 21 (A), the a projection image obtained by photographing from above the front of the point O ₅ is displayed. In this case, the projected image of the cylinder is, for example, as shown in FIG.
(B) is shown with a shadow.

In FIG. 21B, points P ₁ and P ₂ ,
Alternatively, points P ₃ and P ₄ are the intersections between the side surface of the cylinder in the projected image and the bottom surface above or below it (FIG. 21B).
, The end points of the portions indicated by thick lines). That is, the point P ₁ is the intersection of the left side contour line of the side surface of the cylinder in the projection image and the bottom surface above the cylinder, and the point P ₂ is
The intersection point between the right side contour line of the cylinder side in the projected image and the bottom surface above the cylinder, the point P ₃ is the intersection point between the right side contour line of the cylinder side surface in the projection image and the bottom surface below the cylinder,
Point P ₄ is the intersection of the bottom surface of the lower contour and cylinder of the left side surface of the cylinder in the projected image, and represents respectively.

Then, the points O ₁ ,
O ₂ , O ₃ , and O ₄ correspond to points P ₁ , P ₂ ,
P ₃ and P ₄ respectively.

A point O ₅ in FIG. 21A is a point that bisects the arc O ₁ O ₂ (an arc seen as a projected image), and a point P ₅ in FIG. corresponds to the O _5. Further, a point in FIG. 21 (A) O ₆ also, the arc O ₃
This is a point at which O ₄ (an arc seen as a projected image) is bisected, and a point P ₆ in FIG. 21B corresponds to this point O ₆ .

Here, as described above, the _zo axis is the point O ₅
Directions are oriented in the direction of the center of the upper bottom surface from, and because through the center of the upper bottom surface is the origin, the point O ₅ is, y _o
lies in the _zo plane, that is, in the plane of _xo = 0. Therefore, point O
_{6 is} also in the plane of x _o = 0.

In FIG. 21B, the point P ₇ is
Vanishing point of the cylinder, i.e., it represents the intersection of the straight line P ₁ P ₄ and P ₂ P _3, therefore, the point P ₆ is positioned on a straight line connecting the point P ₅ and P _7. That is, the degree of freedom of the point P ₆ is only one of the directions of the straight line P ₅ P _7.

Angle θ (radian) in FIG. 21 (A)
Is the point O ₁ , the center of the upper bottom surface (here the origin of the three-dimensional coordinates), and the angle formed by the point O ₂ (hence the point O ₃ , the center of the lower bottom surface, and the point O ₄ ), and when the radius of the bottom surface of the cylinder is represented by r, if the developed image corresponding to the projected image portion on the side surface of the cylinder is stored in the paste buffer, the radius r
And the angle θ can be expressed as follows (here, for example, the circumferential direction of the bottom surface of the cylinder corresponds to the horizontal direction of the developed image).

R = L _W / θ (15) Here, L _W represents the horizontal width (horizontal length) of the developed image, as described above.

The angle φ is set as follows.

Φ = θ (x _e −L _W / 2) / L _W (16) Note that x _e represents the x coordinate (horizontal direction coordinate) of the paste buffer coordinate system. buffer coordinate system, for example, the upper left corner of the rectangular expanded image as the origin, the downward direction from the right or top left, we shall take x _e axis or y _e axis.

When the angle φ of the cylinder is represented by the equation (16), ψ of the equation (7), ie, 3
The relationship between the point (x _o , y _o , z _o ) on the dimensional space and the point (x _e , y _e ) on the developed image is given by considering the developed view of the cylinder in FIG. Can be expressed as

[0174]

[Equation 11](17) In this case, unlike the case described with reference to FIG.
There are no unnecessary elements, and therefore,
11 elements m₁₁To m₁₄, M_{twenty one}To m_{twenty four}, M₃₁No
To m₃₃Need to ask. This eleven element m₁₁To m
₁₄, M_{twenty one}To m _{twenty four}, M₃₁To m₃₃Is, as mentioned above,
The combination of predetermined five points and one point with one degree of freedom on the projected image
A total of 6 points can be obtained if they are specified as feature points.
You.

Therefore, here, the user is asked, for example,
6 points P shown in 21 (B)₁Or P₆Specify as feature points
do that for me. Here, as shown in FIG.
And the expanded image stored in the paste buffer in a rectangular shape
From the upper left corner clockwise to the point E₁, E_Two, E
_Three, E_FourAnd the line segment E₁E_TwoOr E _Three
E_FourAre divided into two equal points, respectively, as points E_FiveOr E₆age
And point P₁And E₁, Point P_TwoAnd E_Two, Point P_ThreeAnd E_Three, Point P_FourWhen
E_Four, Point P_FiveAnd E_Five, Point P₆And E₆Correspond to each
It shall be.

Further, in the paste buffer coordinate system,
Point E₁, E_Two, E_Three, E_Four, E_Five, E₆Coordinates of
(0,0), (L_W, 0), (L_W, L_H), (0,
L_H), (L _W/ 2, 0), (L_W/ 2, L_H).

By using the above relationship and formulating and solving equations for _eleven elements m _{11 to} m ₁₄ , m _{21 to} m ₂₄ , and m _{31 to} m ₃₃ in accordance with equation (10), these 11
Elements m ₁₁ to m _14, m ₂₁ to m _24, m ₃₁ to m _33, i.e., can be calculated matrix M.

Here, in this case, the feature points P _{1 to} P ₆ used for obtaining the matrix M are corrected by the automatic correction process, but one of the feature points P _{5 and} P ₆ is further corrected. , as corrected by the automatic correction of the other feature point is corrected so as to be located on the straight line connecting the vanishing point P _7. That is, for example, which is one of the feature points of the feature point P ₅ or P _6, for example, the feature point P ₆ is corrected by the automatic correction processing, further corrected by the automatic correction processing other feature points P ₅ and those are corrected so as to be located on the straight line connecting the vanishing point P _7. This is the feature point P ₆
This is because the degree of freedom is one as described above.

It should be noted that the height of the cylinder in FIG.
If it is equal to the vertical length L _H of the developed image of 1 (C),
The coordinates of points O ₁ , O ₂ , O ₃ , O ₄ , O ₅ , and O ₆ are
(-Rsin (θ / 2), 0, -rcos (θ /
2)), (rsin (θ / 2), 0, −rcos (θ /
2)), (rsin (θ / 2), L _H , −rcos (θ
/ 2)), (-rsin (θ / 2), L _H , -rcos
(Θ / 2)), ( 0,0, -r), (0, L H, -r)
Becomes

The matrix M obtained for the cylinder as described above is a function of the angle θ in FIG. 21A, and it is necessary to obtain this angle θ. Furthermore, in FIG. 21B, the feature points P
₅ and connecting the vanishing point P ₇ straight line, although the projected image of the cylinder, such as to be parallel to the y _p axis is displayed, generally, cylindrical projection image, for example, as shown in FIG. 22, Feature point P
₅ and the straight line connecting the vanishing point P ₇ is often has a like inclined with respect to the y _p axis.

Therefore, a method for obtaining θ by setting an angle between a straight line connecting the characteristic point P ₅ and the vanishing point P ₇ to the _xp axis as α will be described.

In this case, considering the x _p axis and the y _p axis which are the coordinate axes of the screen coordinate system and the x _p 'axis and the y _p ' axis which are the coordinate axes rotated by α counterclockwise, The relationship with the projected image of the cylinder coincides with the case shown in FIG. Therefore, so, x _p axis and y _p
X _p 'axis and y _p ' with the axis rotated counterclockwise by α
Considering a coordinate system defined by axes, θ is determined.

In this case, the point (x _p , y _p ) and the point (x _p ′,
y _p ′) can be expressed by the following equation using homogeneous coordinates.

[0184]

(Equation 12) (18) The matrix M 'is defined as follows, for example.

[0185]

(Equation 13) (19) However, M in Expression (19) is represented by Expression (11).

In this case, the following equation is derived from equations (10) and (15) to (19).

[0187] _{x p '= (m 11'} rsinφ + m 12 'y e -m 13' rcosφ + m 14 ') / (m 31' rsinφ + m 32 'ye-m 33' rcosφ + 1) y p '= (m 21' rsinφ + m 22 ' _{y e -m 23 'rcosφ + m} 24') / (m 31 'rsinφ + m 32' y e -m 33 'rcosφ + 1) ··· (20) that is, in the cylinder as a three-dimensional object as shown in FIG. 21 (a), On the arc O ₁ O ₅ O ₂ displayed as a projection image,
Taking the point O, the point O is projected as a point P on the arc P ₁ P ₅ P ₂ on the projection image in FIG. 21 (B). Then, assuming that this point P corresponds to point E in the paste buffer coordinate system as shown in FIG.
Of a reference point E _5, x _e coordinates may be represented by x _e -L _w / 2.

[0188] Furthermore, in the paste buffer coordinate system of FIG. 21 (C), the point E ₅ or point E, 3 of FIG. 21 (A)
Since corresponding to the point O ₅ or O on a cylinder in a dimensional space, of the point E, relative to the point _{E 5, x e -L w /} 2 is a x _e coordinate, FIG. 21 (A) In the cylinder, the point O ₅
Corresponds to the position of a point O along the circumference of the upper surface (upper bottom surface) of the cylinder.

That is, φ defined by the above equation (16)
Represents the point O ₅ , the origin, and the angle formed by the point O, and x _p ′ and y _p ′ represented by the equation (20) represent the coordinates of the point P on the projection image corresponding to the point O.

Now, when the point O moves along the circumference of the upper surface of the cylinder with reference to the point O _5, that is, when the angle φ is changed from 0 radian, the point O The corresponding point P on the projected image is also an arc P ₅ P with respect to the point P _5.
2 above and, although moves on the arc P ₅ P _1, the point P, the vanishing point P _7, and the angle of the point P ₅ is made, as shown in FIG. 23, and [nu.

Here, the coordinates of the point P are represented by (x _p ′, y _p ′) obtained by the equation (20). Thus, the coordinates of the point _{P 7 (x p7 ', y} p7') is expressed as the angle ν can be determined by the following equation (Fig. 23).

[0192] _{^{ν = tan -1 (| y p}} '-y p7' | / | x p '-x p7' |) ··· (21) angle [nu is the point P moves, i.e., the angle φ Is a function of the angle φ because of the change.

In FIG. 23, when the clockwise direction is the positive direction of the angle ν with respect to the line segment P ₅ P ₇ , the angle ν becomes maximum when the point P coincides with the point P _2. , the minimum when the position of control point P _1.

The point P coincides with the point P ₁ or the point P ₂ when the equation | φ | = θ / 2 holds (the point P coincides with the point P ₁ is represented by the equation φ = −θ / 2 holds, and the point P ₂ coincides with the case where the equation φ = θ / 2 holds). Therefore, the angle ν is calculated by the equation | φ | = θ / 2
Takes an extreme value if

Therefore, the value obtained by partially differentiating the angle ν in equation (21) with φ is set to 0 as follows.

[0196]

[Equation 14] (22) As described above, equation (22) is satisfied when | φ | is equal to θ / 2, and θ (= 2φ) at this time is θ to be obtained (in FIG. 21, When P ₁ or P ₂ corresponds to the point O ₁ or O ₂ , respectively, the point O ₁ , the center of the bottom surface above the cylinder, and the angle formed by the point O ₂ ).

The partial derivative of ν in equation (22) with φ (the left side of equation (22)) is a function having two variables θ and φ. The calculation for is complicated. Therefore, here, equation (22)
Is determined by approximation calculation.

That is, FIG. 24 is a flow chart for explaining a process for obtaining θ satisfying the equation (22) by approximation calculation (hereinafter, appropriately referred to as θ calculation process). The θ calculation processing is performed by the copy operation processing unit 12 when performing the copy processing, and is performed by the paste operation processing unit 13 when performing the paste processing, for example.

In the θ calculation processing, first, at step S
At 41, π (radian) is set to θ as an initial value, and the process proceeds to step S42, where a matrix M ′ that satisfies Expression (22) is obtained using the θ and the feature points. Further, in step S43, using the matrix M ′ obtained in step S42, ν represented by Expression (21) is obtained, and the process proceeds to step S44, where φ when ν takes an extreme value, that is, Φ that satisfies Expression (22) is obtained, and the process proceeds to Step S45.

Two φs satisfying the expression (21) can be obtained. Now, if these are φ ₁ and φ ₂ , step S
At 45, | φ ₁ | + | φ ₂ | is newly set to θ, and the flow advances to step S46 to determine whether θ has converged, that is, the absolute value of the difference between the new θ and the previous θ. Is determined to be equal to or less than a predetermined threshold. If it is determined in step S46 that θ has not converged, that is,
If the absolute value of the difference between the new θ and the previous θ is not equal to or smaller than the predetermined threshold, the process returns to step S42, and the processing from step S42 is repeated using the new θ.

When it is determined in step S46 that θ has converged, that is, when the absolute value of the difference between the new θ and the previous θ is equal to or smaller than a predetermined threshold, the new θ
Is determined as θ to be obtained, and the θ calculation processing ends.

[0202] In the above, the columnar projection image (FIG. 21 (B), the Fig. 23), the case where the vanishing point P ₇ is present has been described as an example, if the vanishing point does not exist (In the case where the line segment P ₁ P ₄ and the line segment P ₂ P ₃ are parallel in the projected image), y _p ′ of the equation (20) is calculated by the equation | φ | =
Since the extreme value is obtained when θ / 2 is satisfied, the value obtained by partially differentiating y _p ′ in Expression (20) with φ is set to 0 as in Expression (23), and the above Expression (21) or ( By using the equation (20) or (23) instead of 22) and performing the θ calculation processing of FIG. 24, θ when y _p ′ takes an extreme value can be obtained.

[0203]

(Equation 15) (23) Next, when the three-dimensional object is, for example, a sphere,
When performing a copy process of generating an expanded image of the spherical surface and storing the image in the paste buffer, or performing a paste process of pasting the image stored in the paste buffer onto the spherical surface, the inverse conversion formula or the forward conversion formula is performed as follows. Each is calculated. Here, it is assumed that the developed image of the spherical surface is generated by, for example, the equirectangular projection.

In this case, as shown in FIG.
As the three-dimensional coordinates representing the position of the sphere in dimensional space, the center of the sphere as the origin (0, 0, 0), so as to be perpendicular to each other, consider what took x _o, y _o, a z _o axis (although , X _o
When turning a right-handed screw in the direction of the _zo axis from the _yo axis,
Take the direction in which the right-hand screw advances).

[0205] Here, hereinafter, the appropriate sphere and y _o axis, y _o
Coordinates positive intersection, called the Arctic, sphere and x _o z _o plane (y _o
The line of intersection (circle) with the = 0 plane is called the equator.

Then, for example, the buffer 2 of interest
It is assumed that a projected image obtained by photographing the sphere of FIG. 25A from a predetermined position is displayed on the two-dimensional image, for example, as shown in FIG. 25B. In FIG. 25B, a straight line P ₅ P ₆ connecting points P ₅ and P _6, which will be described later, is x
It is inclined by an angle α with respect to the _p- axis.

In FIG. 25B, the point P₁And P
_ThreeIs the sphere and y_oz_oPlane (x_o= 0 (plane)
Are the two end points of the portion displayed in the projection image.
Point P ₁Is the end point on the North Pole side and point P_ThreeIs x_oz_oPlane
This is the end point on the opposite side of the North Pole when used as a reference. Ma
Point P_TwoAnd P_FourIs the sphere and x_oz_oIs the line of intersection with the plane
The two end points of the portion of the equator displayed in the projected image
And point P_TwoIs x_oIf the coordinates are positive end points, point P_FourIs x_o
The coordinates are the end points on the negative side. Then, in FIG.
Point O₁, O_Two, O_Three, O_FourIs the point in FIG.
P₁, P_Two, P_Three, P_FourRespectively.

Also, the point O in FIG._FiveIs a sphere
The center of the point O₆Goes to the North Pole, point O₇Is an arc along the equator
O_TwoO_Four(Arc visible as projected image)
Each point is represented by a point P in FIG._Five,
P₆, P₇Means that these points O_Five, O ₆, O₇Corresponding to each
are doing.

Now, for example, assuming that a developed image of the entire sphere in FIG. 25A by the equirectangular projection is stored in the paste buffer, the radius R of the sphere can be expressed as follows (however, Here, the equator corresponds to the horizontal direction of the developed image).

R = L_W/ (2π) (24) Further, regarding the spherical surface, ψ in Expression (7), that is,
A point (x _o, Y_o, Z_o) And its equirectangular
Point (x_e, Y_e) And figure
Consider the development of the 25 (A) spherical surface by equirectangular projection
Thus, it can be expressed as follows.

[0211]

(Equation 16) (25) Also in this case, there are no unnecessary elements in the matrix M, and therefore, the eleven elements m _{11 to} m ₁₄ ,
it is necessary to obtain the m ₂₁ or m _24, m ₃₁ to m _33.

Therefore, here, the user is asked to designate, for example, the seven points P _{1 to} P ₇ shown in FIG. 25B as feature points. That is, for example, first, the points P ₂ , P ₄ , P ₅ ,
The P _6, have them designated as the feature point. Note that points P ₂ , P
_4, P _5, P ₆ are all degrees of freedom in terms of 2. Then, so as to be located on a straight line P ₅ P ₆ connecting the points P ₅ and P ₆ ,
Points P ₁ , P ₃ , and P ₇ are designated as feature points. Therefore, the points P ₁ , P ₃ , and P ₇ are points on the straight line P ₅ P ₆ , and thus have one degree of freedom.

As described above, the point P_Two, P_Four, P_Five, P₆Is
Both have two degrees of freedom, and point P₁, P_Three, P₇Is
Since each of them has one degree of freedom, these seven points P₁
Or P ₇If you know, 11 elements m₁₁To m₁₄, M_{twenty one}No
To m_{twenty four}, M₃₁To m₃₃Can be requested.

In this case, a developed image obtained by developing the entire spherical surface of FIG. 25A by the equirectangular projection method is stored in a paste buffer. The developed image is, for example, shown in FIG.
5 (C).

In FIG. 25 (C), the hatched portion shows the developed image of the projected image portion shown in FIG. 25 (B), and the portion without the hatched portion shows the developed image of FIG.
FIG. 3B shows a developed image of a spherical portion that is not displayed in the projection image shown in FIG. Accordingly, the curve L ₁ is 2
It corresponds to the boundary between the spherical surface visible as the projected image shown in FIG. 25B and the invisible spherical surface of the sphere shown in FIG. Further, the line L ₂ which bisects the longitudinal direction of the rectangular expanded image corresponds to the equator.

Points E ₁ , E ₂ , E ₃ , E on the curve L _1
4 are the feature points P ₁ , P ₂ ,
P ₃ and P ₄ respectively. Incidentally, the point E ₁ or point E ₃ is on the curve L _1, y _e-coordinate is the minimum or maximum point, the line segment point E ₂ or E _4, respectively, corresponding to the curve L ₁ and the equator L of the two intersections of the _2, a point towards larger or smaller of x _e coordinate.

In FIG. 25C, the line segment E ₆ is defined by the origin (0, 0) and the point (L) in the paste buffer coordinate system.
_W, 0) and a line segment connecting the corresponds to the point P ₆ in FIG. 25 (B). In FIG. 25C, a point E ₇ bisects the line segment E ₂ E ₄ and corresponds to the point P ₇ in FIG. 25B.

Next, as shown in FIG. 26, the angles of the predetermined portions of the sphere in FIG. 25A are represented by μ and λ ₁ and λ ₂ . That is, of the circumference of a circle is a line of intersection between the sphere x _o z _o plane (equator), and two end points P ₁ and P ₃ of the displayed portion in the projected image, the center O ₅ of the sphere making The angle ∠O ₂ O ₅ O ₄ is represented by μ as shown in FIG. Further, as shown in FIG. 26 (B), of the circumference of a circle is a line of intersection between the sphere and y _o z _o plane, the displayed portion in the projected image, the end point O ₃ of the Arctic opposite , Points O ₇ and O ₅
Represents an angle ∠O ₃ O ₅ O _{7 formed} by λ _1, and an end point O ₁ on the north pole side thereof and an angle ∠O ₁ formed by points O ₇ and O _5.
O ₅ O ₇ is represented as λ ₂ . Here, λ ₁ + λ ₂ is, sphere and y _o z
_o Represents the angle ∠O ₁ O ₅ O ₃ formed by the two end points O ₁ and O ₃ of the portion displayed in the projection image and the center O _{5 of} the sphere in the circumference of the line of intersection (circle) with the plane. .

In this case, the points O ₁ ,
The coordinates of O ₂ , O ₃ , O ₄ , O ₅ , O ₆ , and O ₇ are
(0, Rsinλ ₂ , Rcosλ ₂ ), (Rsin (μ /
2), 0, Rcos (μ / 2)), (0, −Rsinλ)
₁ , Rcosλ ₁ ), (−Rsin (μ / 2), 0, Rc
os (μ / 2)), (0, 0, 0), (0, R, 0),
(0,0, R).

The matrix M obtained for the spherical surface is shown in FIG.
Is a function of the angles μ and λ ₁ and λ ₂ , so it is necessary to determine these angles μ and λ ₁ and λ ₂ . Further, in FIG. 25B, a straight line P ₅ P ₆ connecting the point P ₅ corresponding to the center of the sphere and the point P ₆ corresponding to the north pole and the _xp axis form an angle α. In consideration of this, it is necessary to determine the angle μ and λ ₁ and λ ₂ .

[0221] Therefore, even here, the x _p axis and y _p axis,
Considering the x _p 'axis and the y _p ' axis, which are coordinate axes rotated counterclockwise by α, the angle μ and λ ₁ and λ ₂ are determined.

[0222] In this case, the point (x _p, y _p) and the point _{(x p ', y p'} ) the relationship between, using homogeneous coordinates can be represented by the formula (18).

Now, in FIG. 25A, the sphere
Face and x_o= 0 plane (y_oz_oThe intersection (circle) with the plane
(0, R sin β, R cos β)
And y _o= 0 plane (x_oz_oIntersection with the plane (point on the equator)
Is represented as (Rsinγ, 0, Rcosγ). Also point
(0, Rsinβ, Rcosβ) projected on the screen
X when_p’Coordinates or y_p’Coordinate is x_p1
Or y_p1And the point (Rsinγ, 0, Rc
x when osγ) is projected on the screen_p’Coordinates
Or y_p’Coordinate is x_p2Or y_p2And This
X_p1Or y_p2Is the line defined by equation (19)
Using the elements of column M ', equation (26) or
(27).

[0224] _{x p1 = (m 12 'Rsinβ} + m 13' Rcosβ + m 14 ') / (m 32' Rsinβ + m 33 'Rcosβ + 1) ··· (26) y p2 = (m 21' Rsinγ + m 22 'Rcosγ + m 24') / (m ₃₁ 'Rsinγ + m ₃₃ ' Rcosγ + 1) (27) For the sphere of FIG. 25 (A), x _{p1 in} equation (26) can be obtained when the equation β = −λ ₁ or β = λ ₂ holds, Y _{p2 in the} equation (27) takes an extreme value when the equation γ = μ / 2 or γ = −μ / 2 holds. Therefore, equation (2)
6) x _p1 is β, and y _{p2 in} equation (27) is γ, and the partial differentials are set to 0 as follows.

[0225]

[Equation 17] ... (28)

[0226]

(Equation 18) ... (29) (28) is satisfied when β equals 1-? ₁ or lambda _2, the lambda ₁ and lambda ₂ in this case, a lambda ₁ and lambda ₂ to be obtained. Equation (29) indicates that γ is μ / 2 or −
It is satisfied when it is equal to μ / 2, and μ at this time is μ to be obtained.

The elements of the matrix M ′ are μ, λ ₁ ,
Since it is a function of λ ₂ , the partial differential of x _p1 in equation (28) with β is a function of four variables μ, λ ₁ , λ ₂ , and β.
The partial differential of y _p2 in equation (29) with γ is μ,
It is a function of four variables λ ₁ , λ ₂ and γ. Therefore, equation (2)
Although the calculation for obtaining μ and λ ₁ and λ ₂ that satisfy 8) and (29) is complicated, similar to the above θ calculation processing, predetermined initial values are set for μ, λ ₁ and λ _2. Is set, the elements of the matrix M ′ are determined, and β, γ
, And μ, λ ₁ , λ ₂ can be updated relatively easily by using an approximate calculation of updating μ, λ ₁ , λ ₂ .

Here, a developed image in which a spherical surface is developed by the equirectangular projection method is generated, but the method used to generate a developed image of a spherical surface is not limited to this. That is, the spherical surface is, for example,
The image can be developed into a developed image using the Mercator projection, the Lambert conformal conic projection, or the like.

As for the spherical surface, as shown in FIG. 25 (C), the developed image of the entire surface is stored in the paste buffer.
Since only the shaded portion in FIG. 25 (C) is converted to a projected image and pasted onto the two-dimensional image stored in the buffer of interest, it is displayed on the two-dimensional image. Only the shaded portions in FIG. 25 (C). That is, since the entire image stored in the paste buffer is not displayed as a projected image on a two-dimensional image, when performing 2D painting on a developed image to be pasted on a spherical surface, this point is taken into consideration. , Characters and the like must be drawn. That is, FIG.
If the character is to be pasted on the projected image of the sphere in (B), it is necessary to draw the character within the range of the hatched portion in FIG. 25C and perform paste processing (from the hatched portion). When a character is drawn out of the way, the protruding portion is not pasted on the projected image in FIG. 25B.)

Next, for μ, λ ₁ , and λ ₂ in FIG. 26 when the three-dimensional object is a sphere, as described above, the same processing as the θ calculation processing described with reference to FIG. 24 is performed. However, μ can be obtained from the following relationship after obtaining λ ₁ and λ ₂ .

That is, the point O shown in FIG.₁Or O₇
In addition, as shown in FIG. _Five’And O₇’
I can. Where point O_Five’Is a line segment O₁O_ThreeFor point O_Five
And a line segment O₁O_ThreeAnd the point O
₇’Is a line segment O_TwoO_FourAnd the line segment O_FiveO₇Is the intersection with

Now, if ∠O ₅ 'O ₅ O ₇ ' is represented as λ ₃ ,
This λ ₃ is ∠O ₃ O ₅ O ₇ shown in FIG.
₁ and λ ₂ which is ∠O ₁ O ₅ O ₇ , can be expressed as follows.

Λ ₃ = (λ ₁ + λ ₂ ) / 2−λ ₁ = (λ ₂ −λ ₁ ) / 2 (30) Further, the sphere in FIG. 27 and points O ₁ , O ₆ , and O ₇ ,交面the plane (y _o z _o plane) passing through O _3, such as shown in FIG. 28, the point O
A circle centered at ₅ and having the same radius as the sphere in FIG. 27 is obtained. In FIG. 28, if the length of the line segment O ₃ O ₅ ′ is D ₁ , this D ₁ is obtained by the following equation. be able to.

D ₁ = Rsin ((λ ₁ + λ ₂ ) / 2) (31) Note that R represents the radius of a sphere in a three-dimensional space as shown in Expression (24).

Furthermore, in FIG. 28, the line segment O ₅ 'O ₇ '
Placing of a length of D _2, the D ₂ can be determined by the following equation.

D ₂ = Rcos ((λ ₁ + λ ₂ ) / 2) tan λ ₃ (32) Next, the sphere in FIG. 27 and the plane passing through the points O ₁ , O ₂ , O ₃ , and O ₄ of交面, such as shown in FIG. 29, the point O ₅ 'around the, and the line segment O ₃ O _5' becomes a circle radius, in FIG. 29, the line segment O ₅ 'O ₃ , And the line segment O ₅ ′ O ₂ are all equal to each other because they are the radius of a circle centered at the point O ₅ ′. Therefore, the length of the line segment O ₅ ′ O ₂
Equal to the length D ₁ of the line segment O ₅ 'O ₃ obtained in 1). 29, the length of the line segment O ₂ O ₇ ′ is D
_{If 3} is set, this D ₃ can be obtained by the following equation by the theorem of _three squares.

D ₃ = {(D ₁ ² −D ₂ ² ) = R} (sin ² ((λ ₁ + λ ₂ ) / 2) −cos ² ((λ ₁ + λ ₂ ) / 2) tan ² λ ₃ ) ... (33) Moreover,交面of the sphere of FIG. 27, the point _{O 5, O 2, O 7} , O 4 and through the plane (x _o z _o plane) are shown in FIG. 26 (a) If a, as shown in the same Figure 30, centered on the point O _5, and although the radius is a circle of R, from FIG. 30, the sine angle mu / 2, obtained by calculating the D ₃ / R be able to. Therefore, the following expression holds between the angle μ and λ ₁ and λ ₂ .

Sin (μ / 2) = D ₃ / R = √ (sin ² ((λ ₁ + λ ₂ ) / 2) −cos ² ((λ ₁ + λ ₂ ) / 2) tan ² λ ₃ ) = √ ( sin ² ((λ ₁ + λ ₂ ) / 2) −cos ² ((λ ₁ + λ ₂ ) / 2) tan ² ((λ ₂ −λ ₁ ) / 2)) (34) From the above, λ ₁ , Λ ₂ are obtained by performing the same processing as the θ calculation processing described with reference to FIG.
The obtained λ ₁ and λ ₂ can be obtained by substituting into the equation (34).

According to the θ calculation processing, until θ converges,
Although it is necessary to repeat the processing of steps S42 to S45, when μ is determined from λ ₁ and λ ₂ according to the relationship of equation (34), it is not necessary to perform such repetitive processing. Μ can be obtained at a correspondingly high speed with a small amount of calculation.

Next, in the case where the three-dimensional object is, for example, a cone, copy processing for generating a developed image of the side surface and storing it in the paste buffer, or pasting the image stored in the paste buffer to the side surface of the cone When the paste processing is performed, the reverse conversion formula or the forward conversion formula is calculated as follows.

That is, in this case, for example, as shown in FIG. 31A, the three-dimensional coordinates representing the position of the cone in the three-dimensional space are represented by the vertices (when the side surface of the cone is expanded into a fan shape, point) O ₁ corresponding to the center of the sector as the origin (0,0,0), from the vertex O ₁ in the direction of the center O ₅ of the bottom take y _o axis, further, in the direction of point O ₄ to be described later Consider the case where the z _o axis is taken in the direction of the center O ₅ of the bottom surface (the x _o axis is taken in the direction in which the right screw advances when the right hand screw is turned in the direction from the _yo axis to the z _o axis). ).

Then, for example, the buffer 2 of interest
It is assumed that a projection image obtained by photographing the cone of FIG. 31A from slightly above is displayed on the two-dimensional image. In this case, the projected image of the cone is shown, for example, with a shadow in FIG.

In FIG. 31B, the point P ₁ or the point P ₅
Respectively correspond to the O ₁ or point O ₅ points in FIG. 31 (A). The points P ₂ and P ₃ are a projection image of the side surface of the cone (projection image when only the side surface is projected) and a projection image of the bottom surface of the cone (projection image when only the bottom surface is projected). and a line of intersection of the end points of the (end point of the circumference of the projected image of the bottom surface), FIG. 31 (B), the right or left end point of the intersection line, are respectively, a point P ₂ or P _3. This point P ₂ or point P ₃ is a point O ₂ or O ₃ in FIG. 31 (A), respectively correspond.

In FIG. 31A, the point O ₄ or O ₆ is a point that divides the point O ₂ and O ₃ into two equal parts along the circumference of the bottom surface on the near side or the back side, respectively. And FIG.
Point P ₄ or P ₆ in (B), respectively correspond.

Here, as described above, z_oThe axis is point O_Four
To the center O of the bottom_FiveAnd point O_FiveIs
y_oA point on the axis. Furthermore, from the above,
O_Four, O_Five, O₆Exist in a straight line, and therefore these
Point O_FourOr O₆Is all y_oz_oPlane (x_o= 0 flat
Plane). As a result, in FIG.
O _Four, O_Five, O₆Point P corresponding to_Four, P_Five, P₆Moichi
Located on a straight line, point P_FourOr P₆Example of two points of
For example, point P_FourAnd P₆Is specified, the remaining one point
Point P_FiveIs a straight line P_FourP₆Because it will be located above,
The degree of freedom is 1.

The angle θ in FIG. 31 (A) is ∠O ₂ O ₅
O _3, and the radius of the bottom surface of the cone is represented by r. If a developed image corresponding to the portion of the projected image on the side surface of the cone is stored in the paste buffer, the radius r and the angle θ Can be expressed by Expression (15), similarly to the case of the cylinder described with reference to FIG. 21 (however, here, for example, the circumferential direction of the bottom surface of the cone corresponds to the horizontal direction of the developed image) Shall be).

When the angle φ is expressed by Expression (16), similarly to the case of the cylinder described with reference to FIG. 21, ψ in Expression (7) for the side surface of the cone, that is, the three-dimensional shape of the side surface of the cone The relationship between the point (x _o , y _o , z _o ) on the space and the point (x _e , y _e ) on the developed image can be obtained by considering the developed view of the cone in FIG. It can be expressed as follows.

[0248]

[Equation 19] (35) In this case, the eleven elements m of the matrix M described in equation (11)
_It is necessary to obtain _{11 to} m ₁₄ , m _{21 to} m ₂₄ , m _{31 to} m ₃₃ , and these 11 elements m _{11 to} m ₁₄ , m _{21 to} m ₂₄ ,
As described above, m _{31 to} m ₃₃ can be obtained if a total of six points, five points with two degrees of freedom and one point with one degree of freedom, on the projected image are designated as feature points. .

Therefore, here, for example, the user
6 points P shown in 31 (B)₁Or P₆Specify as feature points
do that for me. That is, assuming that five degrees of freedom are two, for example,
Point P ₁Or P_FourAnd P₆As a feature point
Further, as one point having one degree of freedom, for example, the point P_Five
To be specified. Where point P_FourAnd P₆Is specified
The point P_FiveIs, as described above, a straight line P_FourP₆Up
, The degree of freedom is one.

Here, as shown in FIG. 31 (C), the origin (0, 0) in the paste buffer coordinate system of the developed image stored in the paste buffer in a rectangular shape.
A segment connecting the point (L _W , 0) to the point (L _W , 0) is denoted by E _1, and the coordinates (L _W , 0)
L _H ), (0, L _H ), and (L _W / 2, L _H ) are referred to as points E ₂ , E ₃ , and E ₄ , respectively, as points P ₁ and E ₁ and point P ₂ . E ₂ , points P ₃ and E ₃ , and points P ₄ and E ₄ are associated with each other.

By using the above relationship and formulating and solving equations for _eleven elements m _{11 to} m ₁₄ , m _{21 to} m ₂₄ , and m _{31 to} m ₃₃ in accordance with equation (10), these eleven elements are obtained.
Elements m ₁₁ to m _14, m ₂₁ to m _24, m ₃₁ to m _33, i.e., can be calculated matrix M.

Here, in the embodiment of FIG.
As the developed image of the side surface of the cone, a rectangular image is used. As the developed image of the side surface of the cone, for example, a fan-shaped image obtained by simply expanding the side surface of the cone shown in FIG. It is also possible to use. However, when a fan-shaped image is used as the developed image of the side surface of the cone, when a character or the like is pasted on the projected image of the side surface of the cone in a direction parallel to the bottom surface, the character is converted to a fan-shaped expanded image. Must be drawn along the circumference. On the other hand, as shown in FIG. 31C, when a rectangular image is used as the developed image of the side surface of the cone, characters are drawn on the developed image in parallel with the _xe axis. By performing the paste processing, characters can be pasted on the projected image of the side surface of the cone in a direction parallel to the bottom surface thereof.

It should be noted that the height of the cone in FIG.
Vertical length L of the developed image of 1 (C)_HIs equal to
Point O₁, O_Two, O_Three, O_Four, O_Five, O₆The coordinates of
(0,0,0), (rsin (θ / 2), L_H, -Rc
os (θ / 2)), (−rsin (θ / 2), L_H, −
rcos (θ / 2)), (0, L_H, -R), (0,
L _H, 0), (0, L_H, R).

The matrix M obtained for the cone as described above is a function of the angle θ, as in the case of the cylinder described with reference to FIG. 21, and it is necessary to obtain this angle θ. Further, in FIG.
In the two-dimensional image, a projection image of a cone is displayed such that a straight line connecting the feature points P ₄ and P ₅ is parallel to the _yp axis. as shown in 32, is a straight line connecting the feature point P ₄ and P _5, often has a like inclined with respect to the y _p axis.

Then, a straight line connecting the characteristic points P ₄ and P ₅ is
As the angle between x _p axis alpha, a method for obtaining the θ will be described.

Now, in the above equation (35), y _o =
If y _e = L _H , equation (35) is equal to equation (17) representing ψ for a cylinder. Therefore, in this case, it is x _p, y _p of the conical, x _p for cylindrical, because equals y _p, the angle formed by the straight line and the x _p axis connecting the and P ₅ feature point P ₄ with α If so, x the _p-axis and y _p axis, in the coordinate system defined by the x _p 'axis and y _p' axis is a coordinate axis that is rotated by α in the counterclockwise direction, the circle of the circle which is the base of the cone Point P in the projected image of point O on the circumference
(FIG. 31) can be represented as in the case of a cylinder.

That is, with respect to the cone, the x _p ′ coordinate and the y _p ′ coordinate of the point P on the projected image corresponding to the point O on the circumference of the bottom surface are expressed by the following equation (20). Further, the angle formed by the point P on the projection image corresponding to the point O and the points P ₁ and P ₄ is denoted by ν as shown in FIG.
_Assuming that the coordinates of ₁ are (x _p1 ′, y _p1 ′), the angle ν is represented by the following equation.

[0258] _{^{ν = tan -1 (| y p}} '-y p1' | / | x p '-x p1' |) is ... (36) equation (36) [nu, as in a cylinder, Point O
Forms an angle φ with the points O ₅ and O ₄ (= ∠O ₄
O ₅ O) (FIG. 31A), and the equation | φ
It takes an extreme value when | = θ / 2 holds.

Therefore, the cone θ in FIG.
Can be obtained in the same manner as in the case of a cylinder by using equation (36) instead of equation (21).

In the projected image of the cone shown in FIG. 31B or FIG. 33, if the vertex P ₁ is inside the projected image, the entire side is visible.
Θ in that case is 2π (360 degrees).

In the above, the calculation method of 場合 and M in the case where some primitive shapes are used as the three-dimensional object has been described. , And M can be calculated for such a three-dimensional object as long as it can be represented by a function.

The points to be designated as feature points are not limited to those described above. That is, for example, in FIG. 20, the vertices of the rectangle are designated as the feature points of the rectangular projection image which is the surface of the rectangular parallelepiped. It is also possible to specify the midpoint of each side of the above. As for the ball, but so as to specify a feature point P ₁ to P ₇ point corresponding to O ₁ to O ₇ points shown in FIG. 25 (A), the point O ₅ and point in FIG. 25 (A) A point which is not the point O ₇ among the intersections of the straight line passing through the O ₇ and the sphere is defined as a point O _8. For example, the points projected on the respective two-dimensional images of the points O _{1 to} O ₆ and O ₈ are defined as It may be specified as a feature point.

Further, regarding the cone, for example, FIG.
Points can be specified as feature points.
is there. That is, as shown in FIG.
In space, the vertex O of the cone₁And the center of the bottom
O for midpoint of line segment₆₀And Furthermore, the circumference of the bottom and x_o
y_oIntersection with plane is O₃₀And O₅₀And the bottom
The circumference of the surface and y_oz_oIntersection with plane is O₂₀And O₄₀Toss
You. Where point O₂₀, O ₃₀, O₄₀, O₅₀, O₆₀Coordinates
Are (r, L_H, 0), (0, L_H, -R),
(-R, L_H, 0), (0, L_H, R), (0, L_H/
2,0). In this case, the point O₁, O₂₀, O₃₀,
O₄₀, O₅₀, O₆₀The point on the projection image corresponding to
As shown in FIG.₁, P₂₀, P₃₀, P
₄₀, P₅₀, P₆₀Then, these points are used as feature points
Can be specified.

As is clear from the above,
As the feature points, it is possible to specify both points displayed as projected images (visible points) and points not displayed (invisible points).

However, it is necessary to set in advance the correspondence between the point designated as the feature point of the projected image and the position of the developed image stored in the paste buffer.

Next, in the paste processing, basically, as described above, the image stored in the paste buffer is specified by the feature point specified by the user in the two-dimensional image stored in the buffer of interest. Is attached to the surface of the three-dimensional object. That is, the pixel value of the image stored in the paste buffer is overwritten in the buffer of interest.
However, for example, when a new three-dimensional object is added to the two-dimensional image stored in the buffer of interest by the paste processing, the pixel value of the image stored in the paste buffer is simply overwritten on the buffer of interest. Then, a new three-dimensional object can be added in front of the three-dimensional object already displayed in the two-dimensional image stored in the buffer of interest, but a new three-dimensional object is added in the back side. I can't.

Therefore, in this embodiment, it is possible to add a new three-dimensional object to the back side of the three-dimensional object already displayed in the two-dimensional image stored in the buffer of interest by using the mat. It has been made possible.

The mat is a so-called gray scale image representing the shape of the object displayed on the two-dimensional image, and its value represents the contribution ratio (called α) of the object to each pixel constituting the two-dimensional image. Therefore, it is also called an α image. Here, the mat of the two-dimensional image stored in the buffer of interest is generated by performing the mat processing in step S15 of FIG. As described above, the mat of the two-dimensional image is stored in the image buffer in which the two-dimensional image is stored.

Here, the pixel value of each pixel constituting the mat takes, for example, a value in the range of 0 to 1, the pixel value of the pixel constituting the so-called foreground is 1, and the pixel value of the pixel constituting the so-called background is The value is 0, and the pixel value of the pixel forming the boundary between the foreground and the background is a value corresponding to the ratio of the foreground and the background included in the pixel (the value becomes closer to 1 as the ratio of the foreground increases). , The value becomes closer to 0 as the background ratio increases).

Therefore, for example, when a two-dimensional image displaying a three-dimensional object shaped like a house as shown in FIG. 35A is stored in the buffer of interest, When mat processing is performed with the foreground as the foreground and the rest as the background, the mat operation processing unit 16
Then, a mat as shown in FIG. 35B is generated and stored in the buffer of interest. In FIG. 35 (B), the pixel value of the shaded portion is 1, and the pixel value of the unshaded portion is 0.

Here, as a method of generating a mat, for example, Eric N. Mortensen, William A. Barrett, "Intellig
ent Scissors for Image Composition ", Proceedings o
f SIGGRAPH 95, pp. 191-198, and Japanese Patent Application Laid-Open Nos. 8-331455 and 10-14 previously proposed by the present applicant.
No. 3,654, for example.

When the mat is stored in the buffer of interest together with the two-dimensional image, whether the mat is valid or invalid can be set by operating the input device 6. I have.

For example, in the buffer of interest, FIG.
A two-dimensional image or mat as shown in FIG. 35A or FIG. 35B is stored, and paste processing for pasting, for example, a cylinder as a new three-dimensional object to the two-dimensional image in the buffer of interest is performed. When instructed, when the mat is invalid, the projected image of the cylinder is 2
The 3D image is simply overwritten, so that, for example, FIG.
As shown in FIG. 5 (C), a two-dimensional image in which the cylinder is in front of the house is generated.

On the other hand, when the mat is valid, the pixel values of the pixels constituting the two-dimensional image stored in the buffer of interest are A, the pixel values of the pixels constituting the projected image of the cylinder are B, The pixel value of the pixel constituting the mat is α
And the two-dimensional image obtained by the paste processing is represented by C, respectively.
α) Generated based on B. Therefore, in this case, for example, as shown in FIG. 35D, a two-dimensional image in which the cylinder is located on the back side of the house is generated.

As described above, by performing the paste processing with the mat enabled, the part hidden behind the three-dimensional object already displayed in the two-dimensional image is made invisible, that is, so-called occlusion. Therefore, in this case, it can be said that the mat functions as a flag as to whether to permit writing of the projection image.

[0276] The mat can be created by the user.

Next, the object attribute / light source change processing in step S16 in FIG. 7 will be described with reference to the flowchart in FIG. Here, it is assumed that the mat of the two-dimensional image stored in the buffer of interest has already been generated and stored in the buffer of interest.

In the object attribute / light source change processing, first, in step S51, the color in the region of the object represented by the mat stored in the buffer of interest and displayed on the two-dimensional image also stored in the buffer of interest. A distribution is detected. That is, in step S51, the region of the object displayed in the two-dimensional image stored in the buffer of interest is
The distribution (color distribution) in the RGB space of the RGB values of the pixels constituting the area recognized from the mat stored in the buffer of interest is detected.

After the detection of the color distribution, the process proceeds to step S52, and among the pixels constituting the object area, the pixel having the highest color saturation (hereinafter, the maximum saturation pixel as appropriate) is detected in step S51. It is detected based on the obtained color distribution.

That is, FIG. 37 shows an example of the color distribution of the object in the RGB space (portion surrounded by a solid line in FIG. 37). FIG. 37 shows that the maximum values of R, G, and B are 1
Shows the normalized color distribution. Therefore, R in FIG.
In the GB space, points where (R, G, B) are (0, 0, 0) represent black, and points (1, 1, 1) represent white.

In the RGB space, the saturation is represented by a straight line connecting a black point and a white point, that is, a point (0, 0, 0) in FIG.
The distance increases from the line connecting (1, 1, 1) to (1, 1, 1). Therefore, in step S52, in the color distribution detected in step S51, points (0, 0, 0) and (1,
A pixel having a color located at a point farthest from the straight line connecting (1, 1) is detected as the maximum chroma pixel. FIG.
In the embodiment, the pixel of the color indicated by C in FIG.
It is detected as the maximum chroma pixel.

Here, the color (RGB value) of the maximum saturation pixel C
Is considered to be the original color of the object displayed in the two-dimensional image stored in the buffer of interest.

Thereafter, the process proceeds to step S53, in which the color distribution detected in step S51 is converted, and the pixel values (RGB values) corresponding to the converted color distribution form the two-dimensional image stored in the buffer of interest. Then, the data is written to the corresponding pixel and the process returns. That is, in step 53, the color C of the maximum chroma pixel is converted into a predetermined color D,
According to a predetermined rule, the colors of other pixels constituting the object area are also converted.

More specifically, when an object attribute process for changing a material attribute such as a color or a texture of the object is instructed, the color C of the maximum chroma pixel is converted into a predetermined color D in step S53. Similarly to the conversion, the colors of the other pixels constituting the object area are also converted linearly.

That is, in FIG. 37, when the color C of the maximum chroma pixel is changed to the color D, the color C and the point (1,1,1) representing white in the color distribution of the object. The color between
It is linearly converted to a color between the color D and the point (1,1,1) representing white. Furthermore, in the color distribution of the object,
The color between the color C and the point (0,0,0) representing black is linearly converted to a color between the color D and the point (0,0,0) representing black. You. As a result, the color distribution of the object is
In FIG. 37, the one shown by a solid line is changed to the one shown by a dotted line. Here, since the color is linearly converted, the lightest color Q in the original color distribution is converted to the lightest color T in the converted color distribution. Similarly, the darkest color P in the original color distribution is converted to the darkest color S in the converted color distribution.

The color between the color C and the point (1,1,1) representing white is the specular reflection component of the object due to the specular reflection of the light from the light source. Is a diffuse reflection component of the object due to the diffuse reflection of light from the light source, and as described above, the color distribution of the object is represented by the diffuse reflection component and the diffuse reflection component. By dividing it into specular reflection components and converting each to linear,
The color of the projected image of the object displayed on the two-dimensional image stored in the buffer of interest can be changed while retaining the original shadow. That is, in the above case, the color of the object displayed in the two-dimensional image can be changed from color C to color D.

It should be noted that such a color change can be performed by using a dichromatic reflection model.
on model).

In the above case, each of the diffuse reflection component and the specular reflection component of the color distribution of the object is linearly converted. However, in step S53, the diffuse reflection component and the specular reflection component are nonlinearly converted. It is also possible to convert. That is, for example, in FIG. 37, most of the colors between the colors C and Q, which are the specular reflection components,
When the color is converted to the vicinity of the color D and the color Q and the color in the vicinity thereof are converted to the vicinity of the color T, the object can be regarded as having a strong specular reflection in a very small part. In this case, for example, as shown in FIG. 38 (A), an object which has a large amount of diffuse reflection and is entirely bright is changed to an object which partially has strong specular reflection as shown in FIG. 38 (B). can do.

As described above, by non-linearly converting the diffuse reflection component and the specular reflection component, it is possible to narrow or widen the specular reflection area of the object and change the intensity of the specular reflection. Thereby, the material appearance of the object can be changed.

It is to be noted that strong specular reflection partially occurs as described above due to the property of an object having a small surface roughness. Therefore, when the color distribution is changed in such a manner, the surface of the object is changed. Can be changed to a smooth surface.

Next, when a light source change process for changing the light source is instructed, in step S53 of FIG. 36, in FIG. 37, the color between the colors C and Q, which are the specular reflection components, is
The color C is converted so as to be distributed along the color of the new light source. That is, in the color distribution of the object, the brightest color is considered to be the color of the light source, and therefore, in FIG.
By converting the color C so as to be distributed along the color of the new light source, it is possible to obtain an effect as if the light source was changed to a new light source.

Next, the erasing process performed in step S17 of FIG. 7 will be described.

As described above, since the erase process deletes a part of the two-dimensional image stored in the image buffer, basically, the two-dimensional image to be erased is stored. Using the image buffer obtained as a target buffer, copy processing is performed, the developed image is stored in the paste buffer, a part of the developed image stored in the paste buffer is deleted, paste processing is performed, and the projection corresponding to the developed image is performed. This can be done by pasting the image at the original position of the buffer of interest.

However, for example, in the case where a cylinder embedded in a wall on which a lattice pattern is drawn is deleted from a two-dimensional image as shown in FIG. As shown in FIG. 39 (B), if the developed image of the wall is stored in the paste buffer, and then the column portion is simply deleted and the paste process is performed, the portion where the column was present has no pattern. Thus, an unnatural two-dimensional image is obtained.

Therefore, in the erase process, a part of the developed image stored in the paste buffer is deleted, and the background of the deleted part is reproduced.

That is, for example, in the case where a cylinder embedded in a wall on which a lattice pattern is drawn is deleted from a two-dimensional image as shown in FIG. As shown in FIG. 39 (B), when the developed image of the wall is stored in the paste buffer, in the erase process, the columnar portion is deleted, and the background of the deleted portion is drawn. . As a result, the developed image of the paste buffer has a grid-like pattern drawn as a whole, for example, as shown in FIG.

Therefore, by performing paste processing using such a developed image, as shown in FIG. 39 (D), the cylinder is deleted, and a grid-like pattern is formed in the portion where the cylinder was located. A reproduced, natural two-dimensional image can be obtained.

As described above, as a method of deleting a part of an image and reproducing a background in the part where the image has been deleted, for example, Japanese Patent Application Laid-Open No.
No. 28529 (corresponding to EP Publication No. 0772157) and JP-A-10-105700 (US Pat.
2, 853) can be employed.

Next, in the above-described case, in the copy processing, a developed image of only one surface of the three-dimensional object displayed on the two-dimensional image stored in the buffer of interest is generated and stored in the paste buffer. However, in the copy processing, it is also possible to generate a developed image for two or more adjacent surfaces of the three-dimensional object displayed in the two-dimensional image stored in the buffer of interest and store the developed image in the paste buffer. It is.

For example, as shown in FIG. 40A, when a two-dimensional image displaying a three-dimensional object in the shape of a house is stored in the buffer of interest, a grid-like pattern When adding continuous characters and graphics to a drawn wall and a non-patterned wall adjacent to the drawn wall, copy processing is performed on the wall with the lattice pattern drawn, and the developed image is pasted into the paste buffer. , Draw characters and figures, paste it, paste the walls of the lattice pattern on which the characters and figures were drawn, and then perform the copy processing on the plain walls, By storing the developed image in the paste buffer, drawing characters and figures, and performing the paste process, pasting a blank wall with characters and figures drawn on it is troublesome, and continuous characters and figures Is difficult to draw.

Therefore, in the copy processing, as shown in FIG. 40 (B), developed images of both the wall on which the lattice pattern is drawn and the non-patterned wall adjacent thereto are generated and stored in the paste buffer. Can be done. In this case, continuous characters and figures can be easily drawn on the wall on which the lattice-like pattern is drawn and on the non-patterned wall adjacent thereto.

[0302] Similarly, in the paste processing, the developed image stored in the paste buffer can be pasted on two or more adjacent surfaces of the three-dimensional object.

In the above-described automatic correction processing, the contour of the object displayed in the two-dimensional image is detected based on the component of the differential value of the pixel value in the direction of the normal vector, and the feature point designated by the user is determined. Although the correction is made so that it is located on the outline, the method of detecting the outline of the object based on the normal vector direction component of the differential value of the pixel value is widely used in addition to the automatic correction process. Can be applied to the processing that requires the detection of the contour line.

Therefore, a contour extracting device for detecting (extracting) the contour of an object based on the normal vector direction component of the differential value of the pixel value will be described.

FIG. 41 shows a configuration example of an embodiment of such a contour extraction device.

The parameter calculating section 41 is supplied with an image on which an object from which a contour line is to be extracted and shape information representing the shape of the object are supplied. A function represented using one or more parameters is read from the function storage unit 42. That is, the function storage unit 42 stores a function represented by using one or more parameters for defining an outline for each shape of the object, and the parameter calculation unit 41 stores the function storage unit 42 Is read out of the functions stored in the subroutine which represents the contour of the object having the shape corresponding to the shape information.

Then, assuming that the function read from the function storage unit 42 is f, the parameter calculation unit 41 calculates the energy E _f represented by the equation (2) for the function f, and calculates the energy E _f Find the parameter that maximizes Note that the image supplied to the parameter calculation unit 41 is
For example, when the image is a two-dimensional image, for example, a pixel value or the like is used as △ B (x _p , y _p ) in Expression (2). When the image supplied to the parameter calculation unit 41 is, for example, a three-dimensional image (an image drawn using three-dimensional data), △ B in Expression (2)
As (x _p , y _p ), volume data or the like used in volume rendering is used.

When the parameter calculation unit 41 obtains the parameter of the function f, it supplies the parameter to the contour extraction unit 43 together with the function f. The contour extraction unit 43 calculates the function f from the parameter calculation unit 41 by setting the parameter from the parameter calculation unit 41 in the same way, thereby obtaining a point satisfying the equation f = 0, and finding that point of the object. It is output as a point constituting the contour.

Here, the contour extraction device shown in FIG. 41 is practically inconceivable. However, even if it is a contour of an object displayed in an image other than two-dimensional or three-dimensional, it is possible to express the object. A scalar quantity is defined and its contour is 1
Expression f = by function f expressed using the above parameters
Anything that can be defined as 0 can be detected. In this case, △ B in equation (2)
As (x _p , y _p ), a differential value of a scalar quantity for representing an object is used.

As described above, based on the shape information and feature points specified by the user, 3
Calculate an inverse transformation formula for transforming the surface constituting the three-dimensional object into a developed image, and convert the surface constituting the three-dimensional object in the two-dimensional image into a developed image based on the inverse transformation formula. On the other hand, based on the shape information and the feature points specified by the user, a forward conversion formula for converting the developed image into a projected image is calculated, and based on the forward conversion formula, the developed image is converted into a projected image. Then, the projected image is pasted on the portion specified by the feature point in the two-dimensional image, so that the two-dimensional image can be easily subjected to three-dimensional editing and processing.

Therefore, in the image processing apparatus shown in FIG. 3, for example, when there is an image of a certain object photographed by the camera 9, the position and shape of the object in the image can be obtained without photographing the object again. , Texture, etc. can be changed. Further, this image processing apparatus can be used, for example, when a designer performs processing on an image after capturing the image. Further, this image processing apparatus can be used for creating data of a virtual reality space, which has been actively performed recently, and for other image editing in general. Also,
In this image processing apparatus, in addition to an image of a real object, an image obtained by photographing a painting, an image drawn using computer graphics technology, and the like can be processed. is there.

[0312] Copy processing is performed on the surface of the three-dimensional object displayed on the two-dimensional image stored in the buffer of interest, and the resultant is stored in the paste buffer, and the stored contents of the paste buffer are processed. When the paste processing is later applied to the surface of the original three-dimensional object by the paste processing, in the paste processing, the shape information and the feature points used in performing the copy processing are used as they are. This is because, in such a case, the shape information and the feature point used in the copy process and the paste process should be the same, so that the shape information and the feature shape input in the copy process are the same. This is because it is redundant for the user to input information and feature points again during the paste processing. In addition, when the user inputs a feature point when performing the copy process and when performing the paste process, even if the automatic correction process is performed, the feature point used for the copy process and the feature used for the paste process are used. It is conceivable that the points do not match exactly. In this case, the area on the two-dimensional image where the developed image is generated by the copy processing and the area on the two-dimensional image where the projected image is pasted by the paste processing are considered. Do not match, and the two-dimensional image after the paste processing may be unnatural.

Next, with reference to FIG. 42, a description will be given of a medium used to install a program for executing the above-described series of processing in a computer and to make the computer executable.

As shown in FIG. 42A, the program is stored in a hard disk 102 (for example, corresponding to the external storage device 7 in the arithmetic processing circuit 1 in FIG. 3) as a recording medium built in the computer 101 in advance. It can be provided to the user in an installed state.

Alternatively, the program is executed as shown in FIG.
As shown in (B), the floppy disk 111 and the CD-R
OM (Compact Disc Read Only Memory) 112, MO (Magnet
o optical) disc 113, DVD (Digital Versatile Di)
sc) 114, magnetic disk 115, semiconductor memory 116
And the like, can be temporarily or permanently stored in a recording medium such as a storage medium and provided as package software.

Further, as shown in FIG. 42C, the program is transmitted from the download site 121 to the computer 1 via an artificial satellite 122 for digital satellite broadcasting.
23, wireless LAN, LAN (Local Area Network),
Via a network 131 such as the Internet,
The data is transferred to the computer 123 by wire, and
In 3, it may be stored in a built-in hard disk or the like.

[0317] In the present specification, the medium means a broad concept including all these media.

In this specification, the steps of describing a program provided by a medium need not necessarily be processed in chronological order in the order described in the flowchart, and may be performed in parallel or individually. (For example, parallel processing or object processing)
Is also included.

In the present embodiment, the image processing apparatus shown in FIG. 4 is realized by causing the arithmetic processing circuit 1 shown in FIG. 3 to execute an application program stored in the external storage device 7. This image processing apparatus can be realized by hardware corresponding to each block shown in FIG.

[0320]

As described above, according to the image processing apparatus, the image processing method, and the medium of the present invention, the first two-dimensional image based on the first shape information and the first feature point is obtained. A first conversion formula for converting a surface constituting the first three-dimensional object into a developed image which is an image obtained by developing the surface is calculated, and a first conversion formula is calculated based on the first conversion formula. The surface constituting the first three-dimensional object in the three-dimensional image is converted into a developed image and stored in the storage unit. Further, based on the second shape information and the second feature point, a second image for converting the image stored in the storage unit into a projection image which is an image obtained by projecting a plane constituting a second three-dimensional object. 2 is calculated, and the image stored in the storage unit is converted into a projection image based on the second conversion formula. Then, the projection image is pasted on a portion specified by the second feature point in the second two-dimensional image. Therefore, it is possible to easily perform three-dimensional editing or the like on a two-dimensional image.

[Brief description of the drawings]

FIG. 1 is a diagram showing a drawing result by a conventional paint tool.

FIG. 2 is a diagram showing a drawing result by a conventional paint tool.

FIG. 3 is a block diagram illustrating a configuration example of hardware of an image processing apparatus according to an embodiment of the present invention;

FIG. 4 is a block diagram illustrating a functional configuration example of the image processing apparatus in FIG. 3;

FIG. 5 is a diagram showing a processing result of a two-dimensional image by the image processing apparatus of FIG. 4;

FIG. 6 is a diagram illustrating a processing result of a two-dimensional image by the image processing apparatus in FIG. 4;

FIG. 7 is a flowchart illustrating a process of the image processing apparatus of FIG. 4;

FIG. 8 is a diagram showing a main window displayed when the image processing apparatus of FIG. 4 is started.

FIG. 9 is a diagram illustrating a copy process.

FIG. 10 is a diagram for explaining a paste process.

FIG. 11 is a diagram illustrating copying of a texture by a combination of a copy process and a paste process.

FIG. 12 is a diagram showing feature points designated by a user.

FIG. 13 is a flowchart illustrating details of a process in step S7 of FIG. 7;

FIG. 14 is a diagram for explaining the process of step S22 in FIG.

FIG. 15 is a diagram for describing a method of extracting a contour of a three-dimensional object displayed on a two-dimensional image.

FIG. 16 is a flowchart illustrating details of a process in step S7 of FIG. 7;

FIG. 17 is a diagram for explaining a feature point correction method.

FIG. 18 is a diagram for explaining a method of calculating an inverse conversion formula and a forward conversion formula.

FIG. 19 is a diagram illustrating a relationship among a three-dimensional object, a developed image, and a projected image in a three-dimensional space.

FIG. 20 is a diagram for explaining a method of calculating an inverse conversion formula and a forward conversion formula when the three-dimensional object is a rectangular parallelepiped.

FIG. 21 is a diagram for explaining a method of calculating an inverse conversion formula and a forward conversion formula when the three-dimensional object is a cylinder.

FIG. 22 is a diagram for explaining a method of calculating an inverse conversion formula and a forward conversion formula when the three-dimensional object is a cylinder.

FIG. 23 is a diagram for explaining an angle ν.

FIG. 24 is a flowchart illustrating a method of calculating θ in FIG. 21.

FIG. 25 is a diagram for explaining a method of calculating an inverse conversion formula and a forward conversion formula when the three-dimensional object is a sphere.

FIG. 26 is a diagram for explaining a method of calculating an inverse conversion formula and a forward conversion formula when the three-dimensional object is a sphere.

FIG. 27 is a diagram for explaining a method of calculating μ from λ ₁ and λ _{2 in} FIG. 26;

FIG. 28 is a diagram for explaining a method of calculating μ from λ ₁ and λ _{2 in} FIG. 26;

FIG. 29 is a diagram for explaining a method of calculating μ from λ ₁ and λ _{2 in} FIG. 26;

30 is a diagram for explaining a method of calculating μ from λ ₁ and λ _{2 in} FIG. 26;

FIG. 31 is a diagram for explaining a method of calculating an inverse conversion formula and a forward conversion formula when a three-dimensional object is a cone.

FIG. 32 is a diagram for explaining a method of calculating an inverse conversion formula and a forward conversion formula when a three-dimensional object is a cone.

FIG. 33 is a diagram for explaining an angle ν.

FIG. 34 is a diagram illustrating an example of a feature point specified for a cone;

FIG. 35 is a diagram for explaining a mat process.

FIG. 36 is a flowchart illustrating details of a process in step S16 of FIG. 7;

FIG. 37 is a view for explaining the processing in step S53 of FIG. 36;

FIG. 38 is a diagram illustrating a conversion result obtained by nonlinearly converting the color distribution of an object.

FIG. 39 is a diagram for explaining an erasing process;

FIG. 40 is a diagram for describing copy processing.

FIG. 41 is a block diagram illustrating a configuration example of an embodiment of a contour extraction device to which the present invention has been applied.

FIG. 42 is a diagram for explaining a medium to which the present invention is applied.

[Explanation of symbols]

1 arithmetic processing circuit, 2 program memory, 3 data memory, 4 frame memory, 5 image display device, 6 input device, 7 external storage device, 8 external interface, 9 camera, 11 input event processing unit, 12 copy operation processing unit, 13 paste operation processing unit, 14 paste buffer selection processing unit,
15 erase operation processing unit, 16 mat operation processing unit, 17 object attribute operation processing unit, 18 paint processing unit, 19 conversion designation processing unit, 20 image conversion processing unit, 21 buffer ID storage unit, 22 display processing unit, 31 command button , 32 shape buttons, 4
1 Parameter calculation unit, 42 function storage unit, 43 contour extraction unit, 101 computer, 102 hard disk, 103 semiconductor memory, 111 floppy disk, 112 CD-ROM, 113 MO disk,
114 DVD, 115 Magnetic disk, 116
Semiconductor memory, 121 download site, 12
2 satellites, 123 computers, 131 networks

Claims (42)

[Claims] An image processing apparatus for editing a two-dimensional image, comprising: first shape information relating to a shape of a surface forming a first three-dimensional object displayed in the first two-dimensional image; A first operation means operated when designating a first feature point which is a point on the first two-dimensional image, wherein the first shape information and the first feature point A first conversion formula for converting a surface constituting the first three-dimensional object in the first two-dimensional image into a developed image which is an image developed on a two-dimensional plane based on the first conversion formula; First conversion means for converting a surface constituting the first three-dimensional object in the first two-dimensional image into the developed image based on the first conversion formula; Means, a storage means for storing the developed image, and a shape of a surface constituting a second three-dimensional object. A second shape information, the second is operated to specify a second characteristic point is a point on the second two-dimensional image to be constituting the surface
A projection image, which is an image obtained by projecting a plane constituting the second three-dimensional object from an image stored in the storage unit based on the second shape information and the second feature point. A second calculating unit that calculates a second conversion formula for converting the image into a second image, and a second conversion unit that converts the image stored in the storage unit into the projection image based on the second conversion expression. Means for attaching the projection image to a portion of the second two-dimensional image specified by the second feature point. 2. The image processing apparatus according to claim 1, further comprising processing means for processing an image stored in said storage means. 3. The image processing apparatus according to claim 2, wherein the processing unit deletes a part of the image stored in the storage unit or adds another image. 4. The image processing apparatus according to claim 1, further comprising: a mat generating unit configured to generate a mat representing an area of a predetermined object displayed in the second two-dimensional image. 5. The image processing apparatus according to claim 4, wherein the pasting unit pastes the projection image on the second two-dimensional image according to the mat. 6. The image processing apparatus according to claim 4, further comprising: a color distribution detecting unit that detects a color distribution in an area of the object represented by the mat; and a color distribution converting unit that converts the color distribution. Prepare. 7. The image processing apparatus according to claim 6, further comprising: a maximum saturation detection unit that detects a maximum saturation pixel that is a pixel having the highest saturation in the color distribution, wherein the color distribution conversion unit Transforms the color distribution linearly so that the color of the maximum chroma pixel becomes a predetermined color. 8. The image processing apparatus according to claim 6, wherein said color distribution conversion means non-linearly converts said color distribution. 9. The image processing apparatus according to claim 6, wherein the color distribution conversion unit converts the color distribution into a diffuse reflection component due to diffuse reflection of light from a light source and a specular reflection due to specular reflection in the object. And a diffuse reflection component or a specular reflection component is converted. 10. The image processing apparatus according to claim 1, wherein the first two-dimensional image and the second two-dimensional image are the same two-dimensional image. 11. The image processing apparatus according to claim 1, wherein a function for mapping the developed image onto a surface constituting the first three-dimensional object in a three-dimensional space is represented by ψ, and a function of the first three When a function for projecting a two-dimensional object onto a screen and forming the first two-dimensional image is M, the first calculating means calculates an inverse function of a function Mψ as the first conversion formula. I do. 12. The image processing apparatus according to claim 1, wherein a function for mapping the image stored in the storage unit onto a surface constituting the second three-dimensional object in a three-dimensional space is:
When each of the functions of projecting the second three-dimensional object on the screen to form a two-dimensional image is represented by M, the second calculating means calculates a function Mψ as the second conversion equation. 13. The image processing apparatus according to claim 1, wherein the first or second calculation unit is a two-dimensional coordinate system defined locally as a two-dimensional coordinate system of an image stored in the storage unit. The first or second conversion formula is calculated using a coordinate system. 14. The image processing apparatus according to claim 1, further comprising a correction unit configured to correct the first or second feature point designated by operating the first or second operation unit. Prepare. 15. An image processing method for editing a two-dimensional image, comprising: first shape information relating to a shape of a surface constituting a first three-dimensional object displayed in the first two-dimensional image; The first shape information and the first feature point are changed by a first operation unit operated when designating a first feature point which is a point on the first two-dimensional image. When specified, a surface constituting the first three-dimensional object in the first two-dimensional image is developed into a two-dimensional plane based on the first shape information and the first feature point A first calculation step of calculating a first conversion formula for converting the image into a developed image, which is a converted image, and the first conversion formula in the first two-dimensional image based on the first conversion formula. A first conversion step of converting a surface constituting a three-dimensional object into the developed image; A storage step of storing the developed image in storage means for storing the image; second shape information relating to a shape of a surface forming the second three-dimensional object; and a second shape information on the second two-dimensional image forming the surface. A second point operated when designating a second feature point which is a point
The second shape information and the second
When the feature point is designated, the image stored in the storage means is projected on the surface constituting the second three-dimensional object based on the second shape information and the second feature point. A second calculation step of calculating a second conversion formula for converting the image into a projection image, and converting the image stored in the storage unit into the projection image based on the second conversion formula. And a pasting step of pasting the projection image to a portion specified by the second feature point in the second two-dimensional image. 16. The image processing method according to claim 15, further comprising a processing step of processing an image stored in the storage unit. 17. The image processing method according to claim 16, wherein in the processing step, a part of an image stored in the storage unit is deleted or another image is added. 18. The image processing method according to claim 15, further comprising a mat generating step of generating a mat representing an area of a predetermined object displayed in the second two-dimensional image. 19. The image processing method according to claim 18, wherein, in the pasting step, the projection image is pasted on the second two-dimensional image according to the mat. 20. The image processing method according to claim 18, further comprising: a color distribution detecting step of detecting a color distribution in an area of the object represented by the mat; and a color distribution converting step of converting the color distribution. Prepare. 21. The image processing method according to claim 20, further comprising a maximum chroma detection step of detecting a maximum chroma pixel that is a pixel having the highest chroma in the color distribution, wherein the color distribution conversion step is performed. , The color distribution is linearly converted so that the color of the maximum chroma pixel becomes a predetermined color. 22. The image processing method according to claim 20, wherein, in the color distribution conversion step, the color distribution is nonlinearly converted. 23. The image processing method according to claim 20, wherein, in the color distribution conversion step, the color distribution is converted into a diffuse reflection component due to diffuse reflection of light from a light source and a specular reflection due to specular reflection in the object. And a diffuse reflection component or a specular reflection component is converted. 24. The image processing method according to claim 15, wherein the first two-dimensional image and the second two-dimensional image are the same two-dimensional image. 25. The image processing method according to claim 15, wherein a function for mapping the developed image onto a surface constituting the first three-dimensional object in a three-dimensional space is represented by ψ, and the first three When a function for projecting a two-dimensional object onto a screen and forming the first two-dimensional image is M, in the first calculation step, an inverse function of the function Mψ is calculated as the first conversion formula. I do. 26. The image processing method according to claim 15, wherein 関 数 represents a function for mapping the image stored in the storage means onto a surface constituting the second three-dimensional object in a three-dimensional space.
When the function of projecting the second three-dimensional object on the screen to form a two-dimensional image is represented by M, in the second calculation step, a function Mψ is calculated as the second conversion equation. 27. The image processing method according to claim 15, wherein in the first or second calculation step, a two-dimensional coordinate system defined locally as a two-dimensional coordinate system of an image stored in the storage unit. Using the coordinate system, the first or second
Are calculated respectively. 28. The image processing method according to claim 15, further comprising a correction step of correcting the first or second feature point designated by operating the first or second operation means. Prepare. 29. A medium for causing a computer to execute a computer program for performing image processing for editing a two-dimensional image, comprising a first three-dimensional object displayed in the first two-dimensional image. First shape information relating to the shape of the surface to be formed and first operation means operated when designating a first feature point, which is a point on the first two-dimensional image, constituting the surface. , When the first shape information and the first feature point are designated, the first shape information and the first feature point in the first two-dimensional image based on the first feature point are designated. A first calculation step of calculating a first conversion formula for converting a surface forming the three-dimensional object into a developed image that is a developed image on a two-dimensional plane; and, based on the first conversion formula, The first three-dimensional object in the first two-dimensional image A first conversion step of converting a surface to be configured into the developed image; a storing step of storing the developed image in a storage unit that stores an image; and a second step related to a shape of a surface configuring a second three-dimensional object. 2 which is operated when designating the second shape information and the second feature point which is a point on the second two-dimensional image constituting the surface.
When the second shape information and the second feature point are designated by the operation means, the image stored in the storage means is determined based on the second shape information and the second feature point. A second calculating step of calculating a second conversion formula for converting a surface constituting the second three-dimensional object into a projection image, which is a projected image, based on the second conversion formula, A second conversion step of converting the image stored in the storage means into the projection image, and affixing the projection image to a portion of the second two-dimensional image specified by the second feature point. And causing the computer to execute a computer program including the attaching step. 30. The medium according to claim 29, wherein the computer program further comprises a processing step of processing an image stored in the storage unit. 31. The medium according to claim 30, wherein in the processing step, a part of an image stored in the storage unit is deleted or another image is added. 32. The medium according to claim 29, wherein the computer program further comprises a mat generating step of generating a mat representing an area of the predetermined object displayed in the second two-dimensional image. 33. The medium according to claim 32, wherein in the attaching step, the projected image is attached to the second two-dimensional image according to the mat. 34. The medium according to claim 32, wherein the computer program comprises: a color distribution detecting step of detecting a color distribution in an area of the object represented by the mat; and a color distribution converting step of converting the color distribution. And further provided. 35. The medium according to claim 34, wherein the computer program comprises:
The method further includes a maximum saturation detection step of detecting a maximum saturation pixel that is a pixel having the highest saturation, wherein the color of the maximum saturation pixel is a predetermined color, and Transform the distribution linearly. 36. The medium according to claim 34, wherein in the color distribution conversion step, the color distribution is nonlinearly converted. 37. The medium according to claim 34, wherein, in the color distribution conversion step, the color distribution is changed to a diffuse reflection component due to diffuse reflection of light from a light source and a specular reflection component due to specular reflection in the object. And the respective diffuse reflection components or specular reflection components are converted. 38. The medium according to claim 29, wherein the first two-dimensional image and the second two-dimensional image are the same two-dimensional image. 39. The medium according to claim 29, wherein 関 数 represents a function for mapping the developed image onto a plane constituting the first three-dimensional object in a three-dimensional space, and the first three-dimensional object Is projected onto a screen, and the function that sets the first two-dimensional image is M. In the first calculation step, an inverse function of the function M 関 数 is calculated as the first conversion equation. 40. The medium according to claim 29, wherein a function for mapping the image stored in the storage means onto a surface of the second three-dimensional object in a three-dimensional space is ψ,
When the function of projecting the second three-dimensional object on the screen to form a two-dimensional image is represented by M, in the second calculation step, a function Mψ is calculated as the second conversion equation. 41. The medium according to claim 29, wherein in the first or second calculation step, a two-dimensional coordinate system locally defined as a two-dimensional coordinate system of an image stored in the storage unit. By using the first or second
Are calculated respectively. 42. The medium according to claim 29, wherein the computer program corrects the first or second feature point designated by operating the first or second operating means. The method further includes a step.