JP3723875B2 - Model image replacement method on a two-dimensional image, model image replacement apparatus therefor, and computer-readable program recording medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to image processing in which a part of an image on a two-dimensional image such as a photographic image is replaced with another model image by an information processing device. After the model image is arranged based on the size data after corresponding to the X, Y, Z orthogonal coordinate system, the model image is converted into the coordinate system of the two-dimensional image, thereby converting the model image into the original two-dimensional image. It can be taken in the same direction.
[0002]
In general, a part of a two-dimensional image showing the current state is replaced with another by image processing, for example, a sink in an image obtained by copying the current kitchen from an oblique direction, and another sink shown in an image form from the front When the model image is replaced with the model image, it is desirable that the orientation of the model image can be automatically adjusted to that of the original image, and the present invention meets such a demand.
[0003]
In the present invention, a partial image in an arbitrary two-dimensional image is a replacement target. In the following description, however, for convenience of explanation, a kitchen sink, an outer wall portion of a house, or the like is used as an example of a replacement target.
[0004]
[Prior art]
Conventionally, when the image of a kitchen sink after being reformed is used, the sink part of the photographic image showing the current kitchen is replaced with a model image of another sink using a paint tool. .
[0005]
At this time, as a matter of course, if the orientation of the model image of the sink does not match that of the photographic image, the entire image of the kitchen after replacing the sink portion becomes unnatural.
[0006]
[Problems to be solved by the invention]
With the conventional method of replacing a part of a two-dimensional image with a model tool using a paint tool, it is difficult to create a model image in accordance with the orientation of the two-dimensional image, which can be said to be a background. There was a problem that skill and careful work were required.
[0007]
Therefore, in the present invention, the feature part to be replaced on the two-dimensional image (for example, each straight line part extending in three directions from the vertex of the rectangular parallelepiped) is made to correspond to the coordinate system of the actual three-dimensional space, and then into the coordinate system. Image processing for arranging the model image in the same orientation by arranging the actual shape of the model image and then converting the arranged three-dimensional image into the coordinate system of the two-dimensional image The purpose is to automate.
[0008]
[Means for Solving the Problems]
The present invention solves this problem as follows.
(1) In an image processing system in which a partial image on a two-dimensional image is replaced with another model image by an information processing device.
Means for obtaining four vertices for specifying three axes of a rectangular parallelepiped that is the two-dimensional partial image to be replaced;
Means for converting the four vertices into three-dimensional coordinates using a focal length corresponding value based on a focal length at the time of photographing the two-dimensional image;
Means for obtaining the three-dimensional coordinates of the remaining vertices in the rectangular parallelepiped from the transformed three-dimensional coordinates;
Means for inversely transforming the three-dimensional coordinates having the four vertices and the remaining vertices in the rectangular parallelepiped into two-dimensional coordinates using the focal length correspondence values;
Means for obtaining virtual display data corresponding to the partial image from the inversely transformed two-dimensional coordinates;
Means for displaying the obtained virtual display data;
Means for obtaining a true value corresponding to the focal length in accordance with the change of the focal length corresponding value so that the displayed virtual display data matches the partial image;
Means for inversely transforming the three-dimensional coordinates of the model image into two-dimensional coordinates using the obtained true value corresponding to the focal length;
Means for replacing the virtual display data with a model image having two-dimensionally transformed coordinates;
Is provided.
(2) In (1) above,
Means for associating the virtual display data with the model image based on one side of the virtual display data and its actual size value is further provided.
(3) In an image processing method for replacing a part of a two-dimensional image with another model image by an information processing device,
Find four vertices that specify the three axes of the rectangular parallelepiped that is the two-dimensional partial image to be replaced,
Using the focal length corresponding value based on the focal length at the time of shooting the two-dimensional image, the four vertices are converted into three-dimensional coordinates,
Find the three-dimensional coordinates of the remaining vertices in the rectangular parallelepiped from the transformed three-dimensional coordinates,
Using the focal length corresponding value, the three-dimensional coordinates having the four vertices and the remaining vertices in the rectangular parallelepiped are inversely transformed into two-dimensional coordinates,
Obtain virtual display data corresponding to the partial image from the inversely transformed two-dimensional coordinates,
Display the obtained virtual display data,
Along with the change in the focal length correspondence value so that the displayed virtual display data matches the partial image, the focal length correspondence true value is obtained,
Using the obtained focal length corresponding true value, the three-dimensional coordinates of the model image are inversely converted into two-dimensional coordinates,
The virtual display data is replaced with a model image having two-dimensional coordinates obtained by inverse transformation.
[0009]
The focal length correspondence value used in this specification is
This is a concept corresponding to the value of [focal length of photographing camera × width of image (number of pixels)] / width of film.
[0010]
According to the present invention, the model image is arranged based on the size data after the replacement target on the two-dimensional image is made to correspond to, for example, the X, Y, Z orthogonal coordinate system in the actual three-dimensional space. By converting this into a coordinate system of a two-dimensional image, the model image can be automatically taken into the original two-dimensional image in the same direction.
[0011]
The present invention is also directed to a computer-readable program recording medium storing a program for causing a computer to realize this function.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described with reference to FIGS.
[0013]
1 to 7 are explanatory diagrams showing changes in the display screen at the time of image replacement. The contents of the display screen are shown in a simplified form, omitting a part of an actual photographic image for the convenience of drawing creation.
[0014]
FIG. 1 shows a state in which the vertices A, B, C, and D for specifying three axes orthogonal to each other in the real space of the sink to be replaced are indicated and the three-axis portions are highlighted. The “yellow axis” in the figure corresponds to the line segment AD.
[0015]
FIG. 2 shows a result of coordinate conversion processing (see FIG. 12) described later.
(1) First, a rectangular parallelepiped corresponding to the three axes in FIG. 1 is realized by using the two-dimensional coordinate data on the display screen of each vertex of A, B, C, and D and the reference value fs corresponding to the focal length. Set to the orthogonal X, Y, Z coordinate system of the three-dimensional space of
(2) Next, the rectangular parallelepiped was inversely converted on the display screen with the reference value fs corresponding to the focal length, and this was displayed as a two-dimensional image with a dotted line.
It is explanatory drawing which shows a state.
[0016]
The reference value fs corresponding to the focal length is set in advance or input as necessary. In the following description, the rectangular parallelepiped set in the three-dimensional space (1) is called “real rectangular parallelepiped”, and the rectangular solid displayed in the dotted line (2) is called “virtual rectangular parallelepiped”.
[0017]
In FIG. 2, for example, the lower side from the vertex D of the virtual rectangular parallelepiped does not coincide with the original line segment DE (see FIG. 3) corresponding to the sink on the display screen. The reference value fs corresponding to the focal length used in the coordinate conversion process. This is because the true value ft corresponding to the focal length when the photographic image of FIG. Note that the AB, AC, and AD line segments in FIG. 1 coincide with the virtual rectangular parallelepiped in FIG. 2 regardless of the deviation.
[0018]
Usually, the image itself on the display screen is one of photographic images created in various situations, and the focal length corresponding true value ft for the display image is often not recognized by the user.
[0019]
The user can eliminate the inconsistency state such as the lower side of the virtual rectangular parallelepiped by operating the orientation adjusting unit 1 in FIG.
[0020]
As will be described later, the focal length corresponding value changes from the reference value fs according to the degree of operation of the orientation adjusting unit 1, and the above-described (1) using the new focal length corresponding value for the virtual rectangular parallelepiped at that time. The processes of ▼ and (2) are repeatedly executed.
[0021]
FIG. 3 is an explanatory diagram showing a state in which a virtual rectangular parallelepiped (= a sink portion in FIG. 1) 2 with respect to the true value ft corresponding to the focal length obtained by this adjustment operation is extracted.
[0022]
The actual size value of the line segment DE of the virtual rectangular parallelepiped 2 is input on the display screen of FIG.
[0023]
As a result, the actual length of each side of the shape of the solid cuboid (corresponding to the true value ft corresponding to the focal length) that has already been set by the coordinate system conversion process based on the operation of the orientation adjusting unit 1 in FIG. 2 is also specified. Is done. The length of each side is obtained from the ratio of the length of each side based on the position data of each vertex of the actual rectangular parallelepiped and the actual size value of the line segment DE.
[0024]
FIG. 4 is an explanatory diagram showing a state in which a model image prepared in advance is selected by a mouse click operation, and the right side surface is selected as a reference plane for model image replacement.
[0025]
In addition, as data for replacing the model image (a sink) on the display screen,
Actual size data in the X, Y, and Z directions (vertical, horizontal, and height) Image data for each of the right side, left side, front, back, top, and bottom is held in the storage device.
[0026]
FIG. 5 is an explanatory diagram showing a state in which the sink portion is replaced with the model image selected in FIG.
[0027]
In this replacement process, as described later,
(1) In the three-dimensional coordinate system space, the shape of the model image is arranged in such a way that the right side of the model image matches that of the actual rectangular parallelepiped, with reference to the vertex F of the actual rectangular parallelepiped corresponding to the virtual rectangular parallelepiped 2 in FIG. ,
{Circle around (2)} The three-dimensional coordinate data of each vertex of the actual rectangular parallelepiped after being arranged is inversely converted into two-dimensional coordinate data with the above-mentioned true value ft corresponding to the focal length. Replace the virtual rectangular parallelepiped 2 in FIG.
{Circle around (3)} A total of three image data on the left side, front, and top of the sink base after replacement are displayed on the screen. In the three-surface image display, the well-known CG for processing the retained image data of these three surfaces in accordance with the orientation of the display screen of FIG. 5 and each shape of the three surfaces on the display screen, respectively. Use the technique.
[0028]
FIG. 6 is an explanatory diagram showing a state in which a portion that is actually visible in the foreground, that is, a preferential display range (a portion of the refrigerator) is designated. The priority display range on the display screen is filled.
[0029]
FIG. 7 is an explanatory diagram showing a state in which priority display processing is executed. A known CG method is used for this priority display processing.
[0030]
FIG. 8 is an explanatory diagram showing another display screen example as an image replacement target.
[0031]
This display screen is used for reforming the outer wall surface of the house. As in the case of FIG. 1, first, four points J, K, L, and M for specifying the three orthogonal axes of the replacement target portion are displayed. Is specified.
[0032]
Next, a true virtual rectangle corresponding to the virtual rectangular parallelepiped in FIG. 2 is set and the focal length corresponding true value ft is obtained, and as in the case of the sink reform,
・ Input of actual size value of wall surface not to be replaced ・ Selection of model image unit, designation of reference point ・ Repeated arrangement of model image unit in 3D coordinate system space ・ 2 of actual model image square after placement using ft Image data display (processing of retained image data) in the virtual model image square area after replacement / replacement of the virtual model image square after inverse transformation / inverse transformation to the dimensional coordinate system and the true virtual square, etc. These operations and processes (see FIG. 12) are executed.
[0033]
FIG. 9 is an explanatory diagram showing a hardware configuration example.
11 is a digital camera or an image data input device comprising a camera and a scanner,
Reference numeral 12 denotes a mouse or keyboard for designating each vertex of A, B, C, and D in FIG. 1, operating the orientation adjusting unit 1 in FIG. 2, and designating the priority display range in FIG. -Image replacement instruction device comprising
13 is an input / output interface;
14 is an image processing apparatus, 14a is a coordinate conversion unit that converts each vertex coordinate data between a two-dimensional coordinate system and a three-dimensional coordinate system, for example, when setting the virtual cuboid and the real cuboid, 14b Is a model image arrangement control unit that arranges the selected model image in the real space area to be replaced based on the actual size data, and 14c is an image display or diagram of the three surfaces of the sink (model image) in FIG. An image display control unit for performing priority display of 7;
15 is a memory for temporarily storing data used for image processing and image processing results;
Reference numeral 16 denotes current image data from the image data input device (display screen data in FIG. 1), model image data in FIG. 4 (the above-described actual size data and image data), and the like. Storing external storage device,
17 is a display device, 17a is a frame memory, 17b is a display screen,
Respectively.
[0035]
FIG. 10 and FIG. 11 are explanatory diagrams showing image replacement processing procedures, the contents of which are as follows.
(S1) The current image data taken out from the external storage device 16 is held in the frame memory 17a and displayed on the display screen 17b.
(S2) A two-dimensional coordinate value in the display screen coordinate system of each vertex A, B, C, D (see FIG. 1) to be replaced designated on the display screen 17b is obtained.
(S3) fs is set as a reference value of the focal length corresponding value f used in the coordinate conversion.
(S4) By the arithmetic processing using the focal length correspondence value f (see FIG. 12), the two-dimensional coordinate values of the vertices A, B, C, and D are converted into the three-dimensional coordinate values in the real space.
(S5) The length and direction of each line segment AB, AC, AD in the real space are obtained using the actual size values in FIG. 3 and the three-dimensional coordinate values of the vertices A, B, C, D. This direction is set, for example, in the form of unit vectors of line segments orthogonal to each other.
(S6) The remaining four vertices E, F, G of the real rectangular parallelepiped are obtained by simple geometrical calculation using the three-dimensional coordinate values of the vertices A, B, C, D and the orientations obtained in step (S5). , H (see FIG. 3) are obtained. The simple geometric calculation is, for example, to obtain an intersection coordinate between a straight line parallel to the line segment AB and passing through the vertex C and a straight line parallel to the line segment AC and passing through the vertex B, and this is the three-dimensional coordinate value of the vertex H. Become.
(S7) By the arithmetic processing using the focal length correspondence value f (see FIG. 12), the three-dimensional coordinate value of each vertex of the real rectangular parallelepiped is inversely converted to the two-dimensional coordinate value on the display screen.
(S8) The virtual rectangular parallelepiped display data specified by the two-dimensional coordinate value is added to the frame memory 17a, and the image of FIG. 2 is displayed on the display screen 17b.
(S9) After this display, it is determined whether or not the orientation adjustment unit 1 has been operated. If “YES”, the process proceeds to the next step, and if “NO”, the process proceeds to step (S11).
(S10) The focal length correspondence value f is changed in a manner corresponding to this operation, and the process returns to step (S4).
(S11) Based on the actual size data of the line segment DE, the actual size data of the model image, the reference plane, and the like, the actual cuboid of the model image is made to correspond to the actual cuboid to be replaced. In this handling process, for example, the right side surface of the replacement target real rectangular parallelepiped is matched with the vertex F as a reference. With this process, display target surfaces that must actually be displayed on the screen, for example, the upper surface, the front surface, and the left surface surface, are specified among the surfaces of the model image.
(S12) By the calculation process using the focal length corresponding true value ft (see FIG. 12), the three-dimensional coordinate values of the eight vertices of the real rectangular parallelepiped of the model image after the corresponding process in step (S11) are displayed on the display screen. Inversely converted to the two-dimensional coordinate value of. The focal length corresponding true value ft is the focal length corresponding value f at the stage where “NO” is determined in the step (S9).
(S13) The model image display data in which the image data is incorporated in each display target surface of the virtual rectangular parallelepiped of the model image specified by the two-dimensional coordinate value is input to the frame memory 17a. 5 is displayed on the display screen 17b.
(S14) It is determined whether or not there is an instruction for priority display processing. If “YES”, the process proceeds to the next step, and if “NO”, the series of processing ends.
(S15) The image of FIG. 7 is displayed on the display screen 17b by the priority display process, and the series of processes ends.
[0036]
FIG. 12 is an explanatory diagram showing a calculation formula for converting two-dimensional coordinate values of four points designated on the display screen into three-dimensional coordinate values. The two-dimensional coordinate value is represented by lowercase letters x and y, the three-dimensional coordinate value is represented by uppercase letters X, Y, and Z. The center of the display screen is the origin of the three-dimensional coordinates, and any of the four points The Z coordinate value of one point (here, Z _A of point _A ) is set.
[0037]
This formula is roughly
Formula (1) for associating (x, y) with (X, Y, Z)
An equation that associates an unknown number of Z _B and Z _C (2)
-Equation (3) for associating unknown numbers of Z _B and Z _D
Equation (4) for associating unknown numbers of Z _C and Z _D
It is made up of. It should be noted that the equations (2) to (4) are satisfied because the line segments AB and AC, the line segments AB and AD, and the line segments AC and AD set in a three-dimensional space intersect at substantially right angles. This is because the inner product between the line segment vectors is “0”.
[0038]
When the two-dimensional coordinate value of point A is converted into a three-dimensional coordinate value, the set value of Z _A and equation (1) may be used, and the two-dimensional coordinate values of the three points B, C, and D are obtained. When converting to a three-dimensional coordinate value, the equation (1) may be used after solving the equations (2) to (4) to obtain the Z coordinate value of each point.
[0039]
When inversely converting the three-dimensional coordinate values of the four points A, B, C, and D into the two-dimensional coordinate values, it is only necessary to use Expression (1).
[0040]
FIG. 13 is an explanatory diagram showing an outline of a computer system that reads and executes a program from a computer-readable recording medium, 20 is a computer system, and 21 is a main unit incorporating a CPU, a disk drive device, and the like. , 22 is a display for displaying an image in accordance with an instruction from the main unit 21, 23 is a display screen, 24 is a keyboard for inputting various information to the computer system 4, and 25 is an arbitrary on the display screen 23. A mouse for designating the position of the computer, 26 is an external database (line-destination memory such as DASD), 27 is a modem for accessing the external database 26, 28 is a CD-ROM, floppy disk, etc. Each of these portable recording media is shown.
[0041]
As a recording medium for storing the program,
-Database 26 on the program provider side (destination memory)
・ Portable recording medium 28
Any memory or the like on the main unit 21 side may be used, and the program is loaded into the main unit 21 and executed on the main memory.
[0042]
【The invention's effect】
In this way, according to the present invention, the feature portion (for example, each straight line portion extending in three directions from the vertex of the rectangular parallelepiped) on the two-dimensional image is made to correspond to the coordinate system of the actual three-dimensional space, and The actual shape of the model image is placed inside, and then the coordinate data of the three-dimensional image after the placement is converted into the coordinate system of the two-dimensional image and displayed on the screen. It is possible to automate image processing for arranging images in the same orientation.
[Brief description of the drawings]
FIG. 1 is an explanatory diagram showing a display screen change (three-axis designation) at the time of image replacement according to the present invention.
FIG. 2 is an explanatory diagram showing a display screen change (dotted line display of a virtual cuboid to be replaced) at the time of image replacement according to the present invention.
FIG. 3 is an explanatory diagram showing a display screen change (extraction display of a true virtual rectangular parallelepiped) at the time of image replacement according to the present invention.
FIG. 4 is an explanatory diagram showing a display screen change (display and selection of a model image) at the time of image replacement according to the present invention.
FIG. 5 is an explanatory diagram showing a display screen change (after model image replacement processing) during image replacement according to the present invention;
FIG. 6 is an explanatory diagram showing display screen change (priority display range designation) at the time of image replacement according to the present invention.
FIG. 7 is an explanatory diagram showing a display screen change (after priority display) at the time of image replacement according to the present invention.
FIG. 8 is an explanatory diagram showing another display screen example of an image replacement target according to the present invention.
FIG. 9 is an explanatory diagram showing a hardware configuration example according to the present invention.
FIG. 10 is an explanatory diagram showing an image replacement processing procedure (part 1) according to the present invention;
FIG. 11 is an explanatory diagram showing an image replacement processing procedure (part 2) according to the present invention;
FIG. 12 is an explanatory diagram illustrating a calculation formula for converting four-dimensional two-dimensional coordinate values designated on the display screen into three-dimensional coordinate values according to the present invention.
FIG. 13 is an explanatory diagram showing an outline of a computer system that reads and executes a program from a computer-readable recording medium according to the present invention.
[Explanation of symbols]
1: Orientation adjustment unit (adjustment of focal length corresponding value f)
2: Virtual cuboid for the focal length corresponding true value ft (= sink part in FIG. 1)
11: Image data input device (digital camera, etc.)
12: Image replacement instruction device 13: Input / output interface 14: Image processing device 14a: Coordinate conversion unit 14b: Model image arrangement control unit 14c: Image display control unit 15: Memory 16: External storage device
17: Display device 17a: Frame memory 17b: Display screen 20: Computer system 21: Main unit 22 incorporating CPU, disk drive device, etc. 22: Display 23: Display screen 24: Keyboard 25: Mouse 26: External database (Destination memory such as DASD)
27: Modem 28: Portable recording medium

Claims (4)

In an image processing system in which a part of an image on a two-dimensional image is replaced with another model image by an information processing device,
Means for obtaining four vertices for specifying three axes of a rectangular parallelepiped that is the two-dimensional partial image to be replaced;
Means for converting the four vertices into three-dimensional coordinates using a focal length corresponding value based on a focal length at the time of photographing the two-dimensional image;
Means for obtaining the three-dimensional coordinates of the remaining vertices in the rectangular parallelepiped from the transformed three-dimensional coordinates;
Means for inversely transforming the three-dimensional coordinates having the four vertices and the remaining vertices in the rectangular parallelepiped into two-dimensional coordinates using the focal length correspondence values;
Means for obtaining virtual display data corresponding to the partial image from the inversely transformed two-dimensional coordinates;
Means for displaying the obtained virtual display data;
Means for obtaining a true value corresponding to the focal length in accordance with the change of the focal length corresponding value so that the displayed virtual display data matches the partial image;
Means for inversely transforming the three-dimensional coordinates of the model image into two-dimensional coordinates using the obtained true value corresponding to the focal length;
Means for replacing the virtual display data with a model image having two-dimensionally transformed coordinates,
An apparatus for replacing a model image on a two-dimensional image. Based on one side of the virtual display data and the actual size value thereof, the apparatus further comprises means for associating the virtual display data with the model image,
The model image replacement apparatus for a two-dimensional image according to claim 1. In a recording medium storing a program used for image processing for replacing a partial image on a two-dimensional image with another model image,
The program is
A function for obtaining four vertices in which three axes of a rectangular parallelepiped that is the two-dimensional partial image to be replaced are specified;
A function of converting the four vertices into three-dimensional coordinates using a focal length corresponding value based on a focal length at the time of photographing the two-dimensional image;
A function for obtaining the three-dimensional coordinates of the remaining vertices in the rectangular parallelepiped from the transformed three-dimensional coordinates;
A function of inversely converting the three-dimensional coordinates having the four vertices and the remaining vertices in the rectangular parallelepiped into two-dimensional coordinates using the focal length correspondence values;
A function of obtaining virtual display data corresponding to the partial image from the inversely transformed two-dimensional coordinates;
A function to display the obtained virtual display data;
A function of obtaining a true value corresponding to a focal length in accordance with the change of the focal length corresponding value so that displayed virtual display data matches the partial image;
A function of inversely converting the three-dimensional coordinates of the model image into two-dimensional coordinates using the obtained true value corresponding to the focal length;
A function of replacing the virtual display data with a model image having two-dimensionally transformed coordinates;
Is to make the computer realize,
A computer-readable program recording medium. In an image processing method for replacing a part of a two-dimensional image with another model image by an information processing device,
Find four vertices that specify the three axes of the rectangular parallelepiped that is the two-dimensional partial image to be replaced,
Using the focal length corresponding value based on the focal length at the time of shooting the two-dimensional image, the four vertices are converted into three-dimensional coordinates,
Find the three-dimensional coordinates of the remaining vertices in the rectangular parallelepiped from the transformed three-dimensional coordinates,
Using the focal length corresponding value, the three-dimensional coordinates having the four vertices and the remaining vertices in the rectangular parallelepiped are inversely transformed into two-dimensional coordinates,
Obtain virtual display data corresponding to the partial image from the inversely transformed two-dimensional coordinates,
Display the obtained virtual display data,
Along with the change in the focal length correspondence value so that the displayed virtual display data matches the partial image, the focal length correspondence true value is obtained,
Using the obtained focal length corresponding true value, the three-dimensional coordinates of the model image are inversely converted into two-dimensional coordinates,
Replacing the virtual display data with a model image having inversely transformed two-dimensional coordinates;
A method for replacing a model image on a two-dimensional image.

Images