JP2013008257A - Image composition program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program for creating a correct composite image without using a marker.SOLUTION: A position of a virtual object is limited on one reference plane in three dimensional space, and a photographing direction and a focal distance are obtained from image data of a digital camera. An exact position of the reference plane is determined using data of two points on the plane and distance therebetween instead of markers. As for the position and a direction of the virtual object, input performed when size is decided is used, and the position and the direction of the virtual object can be changed by gesture or the like.

Description

  The present invention relates to a program for synthesizing a photographic image and another image.

  A technique for displaying a composite image obtained by combining a real image (real image) with another image (virtual image) that does not exist in the real world is called augmented reality and is expected to be used. Yes. For example, prior to purchasing furniture or interior, by combining and displaying an image of the furniture or interior to be purchased on an actual indoor picture, it is possible to confirm the indoor situation when the purchase is made.

  In many cases, a composite image is a virtual object (virtual object) arranged in a space (real space) captured by a camera as a real image. In order to determine the position and direction of the virtual object in the real space, a method of performing imaging by including an object called a “marker” and determining and combining the position and direction of the virtual object based on the image of the captured marker Widely known. For example, the technique concerning patent document 1 is disclosed.

  However, the method using a marker has the following problems, making it difficult to use virtual reality. The marker includes a characteristic pattern for facilitating the discovery of the marker from the image photographed by the program, and is not easily obtained for many users. However, there is no known program that determines the position and direction of a virtual object without using a marker, particularly the position of the virtual object in the real space.

  Actually, there is application software that displays a virtual image in which a virtual object is arranged in a real space photographed by a camera. For example, it is shown in Non-Patent Document 1. In these application softwares, although there are certain restrictions on the placement direction of the virtual object, the placement position can be adjusted by the user (by adjusting the size of the virtual object, the virtual object's depth in the real space can be substantially reduced. To adjust). For this reason, for example, even a large piece of furniture (virtual object) that does not enter the room in the real space, a composite image in which it is stored in the real space is created by adjusting the depth (size). . It is not preferable for users who are considering purchasing furniture.

JP 2003-256876 A

Office furniture layout app (https://www.kaunet.com/kaunet/pdf/20110107_iPhone_kaguap.pdf) Tomas Akenine-Moller, Eric Haines: Real-time rendering 2nd edition (pages 50-54)

  The problem to be solved is to provide a program for creating an accurate composite image without using a special marker.

The virtual object is often represented as data of a three-dimensional polygon or a curved surface (hereinafter, such data is referred to as “3DCG” (3-dimensional computer graphics)). In this case, in order to determine the position and direction of the virtual object, the following three things may be performed.
(1) The direction of the reference plane for arranging the virtual object in the real space is determined. The reference plane is a horizontal plane such as a floor or a desk in the real space. If the shooting direction of the camera is known, it can be calculated. (2) The distance from the reference plane to the camera (viewpoint) is determined. (3) The virtual object is arranged at a specific position on the reference plane in the real space (arranged in a direction according to the direction of the reference plane).
The above (1) can be calculated if the shooting direction of the camera is known. (3) can be adjusted after the images are combined, and a correct combined image can be obtained as an arbitrary object. On the other hand, if (2) is not correct, the actual image and the virtual image will not match each other in size. In this sense, (2) is important.

The image composition program of the present invention is:
An image composition program for creating a composite image by combining a real image obtained by photographing a real space with a camera and a virtual image that is an image of a virtual object represented as three-dimensional shape data (3DCG) viewed from one projection direction. Because
The composite image is an image in which the virtual object is placed on a reference plane that is one horizontal plane in the real space,
Virtual object placement means for calculating coordinates in real space for placing the virtual object on the reference plane;
The virtual object placement unit is configured to use a camera (viewpoint) of the reference plane based on a distance (reference distance) between two points in the real space of a reference point that is two points on the reference plane included in the real image. It is characterized by calculating the exact position from.
The real image is a two-dimensional image obtained by projecting the real space based on the shooting state of the camera. In many cases, the virtual image is that of a virtual object represented as 3DCG, and the composite image is a superposition of the virtual image and the real image. Here, in order to create a composite image in which a virtual object is arranged at an arbitrary position in the real space, it is necessary to determine the position and direction of the virtual object.
For this purpose, an accurate position of the reference plane from the camera (viewpoint) (a plane expression representing the reference plane by the camera coordinate system) is obtained.
Since the virtual object is limited to being placed on the reference plane, the position of the virtual object can be limited to movement on the plane, and the direction of the virtual object can also be limited to rotation of one axis on the plane. You can also. Therefore, the determination of the distance between the reference plane and the viewpoint is examined. If the distance between the reference plane and the viewpoint is known, the virtual object can be placed at the correct position in the real space, so that a virtual image with the correct position, orientation, and size can be finally displayed.

The image composition program of the present invention is:
A size instruction means for designating two points in the real image and inputting a distance between the two points;
The virtual object placement means uses the two points as the reference points, and uses the distance input by the size instruction means as the reference distance.
If an object of a known size (or part of a landscape) is photographed on a reference plane in a real image, a reference point is determined based on the image (position, size, direction) of the object, and the known size is used as a reference. It can be a distance. However, even if such an object is photographed, if the size value is not held as data used by the program, it is necessary to input the reference point of the object and its distance to the program. The size instruction means is for making this input. Specifically, for example, two points on the image are specified (clicking with the mouse, tapping the touch screen, etc.), and the distance between the two points is input (an input window is displayed on the keyboard or the like). It is sufficient to let the user perform an operation of inputting.

The image composition program of the present invention is:
The virtual object placement means calculates the position and direction of the virtual object in the real space based on the coordinates of the reference point in the real image, the reference distance, and the shooting direction of the camera when the real image is shot. It is characterized by doing.
In order to determine the position and direction of the virtual object in the real space, the conventional method using the marker displays at least four points on the marker and back-calculates them from the coordinates in the real image of those points whose mutual distance is known. Position, direction and size. For this purpose, at least four points were required as the display on the marker. This is because it is necessary to obtain the three-dimensional position of the virtual object relative to the camera and the focal length of the camera (or information equivalent thereto).
However, (1) the position of the virtual object is limited to one horizontal plane (reference plane) in the three-dimensional space, and (2) if the shooting direction of the camera is known, real images of two reference points on the reference plane The exact position and direction of the reference plane on which the virtual object is placed can be determined only by the coordinates and the reference distance at.
Here, the camera shooting direction can be the one included in the image data of the digital camera (for example, in the Exif format), or a known value of each camera, a gravity sensor provided in the camera, or the like You can also use output from other devices. An accurate position of the reference plane can be determined only by inputting two reference points and a reference distance by the user, or by only recognizing the reference point from the real image by the program and knowing the reference distance.
As for the position and direction of the virtual object placed on the reference plane, it matches the user's wish, but there are two reference points (a first point for specifying the position and a second point for specifying the direction). The position and direction can be determined based on the coordinates of the two reference points. The user can indicate his / her desire in specifying the reference point.
The image synthesizing program of the present invention utilizes the above and determines the position and direction of the virtual image in the real image by letting the user specify only two reference points and a reference distance. The user's effort to specify is small.

The image composition program of the present invention is:
Virtual image moving means for displaying the composite image on a touch panel screen and changing the position and direction of the virtual object in a real space based on a gesture on the touch panel screen is provided.
There may be cases where you want to make a comparative study by moving and rotating a virtual object, such as the arrangement of furniture in a room. By displaying the composite image on the touch panel screen, movement and rotation in accordance with the user's intuition can be easily realized.

The image composition program of the present invention is:
The camera is attached to the smartphone;
The touch panel screen is a screen attached to the smartphone.
Smartphones are widely used as terminal devices equipped with a touch panel screen. The image composition program of the present invention can be widely utilized by utilizing a smartphone.

The image composition program of the present invention is:
The real image is a photograph of a room,
The virtual object is furniture, a home appliance, or an interior ornament.
An application that arranges furniture or interior decorations in the room effectively uses the image composition program of the present invention.

The image composition program of the present invention is:
Runs on the server,
The real image is an image received from a user via the Internet;
3D shape data of the virtual object is stored in the server,
The synthesized image is displayed on the terminal device of the user via the Internet.
According to this configuration, the image composition program of the present invention can be widely used via the Internet.

  The image composition program of the present invention determines the position and direction of a virtual object without using a marker. Since it is sufficient to specify two reference points and input a reference distance, a user operation is easy.

FIG. 1 is a diagram showing the principle of an image composition program. FIG. 2 is a diagram showing an example of the screen of the size input means. FIG. 3 is a diagram showing an example of the flow of the image composition program. FIG. 4 is a diagram illustrating an example of a virtual object arrangement state. FIG. 5 is a diagram illustrating an example of a gesture and its processing. FIG. 6 is a diagram illustrating an example of the configuration of an image composition program.

  Examples of the present invention are shown below.

  This embodiment shows the basic operation of the image composition program of the present invention. In this embodiment, the real image is an indoor image taken by a camera, and the virtual object is furniture.

FIG. 1 is a diagram showing the principle of an image composition program.
An image including the reference surface 1 which is the floor surface is taken by the camera 2. The coordinates where the horizontal direction of the camera is the x axis, the vertical direction is the y axis, and the shooting direction is the z axis are referred to as “camera coordinates”. The direction of gravity indicated by N in the figure can be determined by a device such as a gravity sensor of the camera. The vector N is a unit vector (having a length of 1). Since the reference plane 1 is a floor surface, the direction of gravity is perpendicular to it.
The captured real image is projected onto the image sensor behind the camera 3, but as shown in Non-Patent Document 2, the virtual screen 3 is assumed to be in front of the camera, and the object in the real space May be considered to be projected on the virtual screen 3 (the image projected on the virtual screen 3 is similar to the real image). The virtual screen 3 is a plane separated from the camera by d in the shooting direction of the camera, and is represented by z = d in camera coordinates. Here, the value of d is determined by the focal length of the camera and the like, and can be obtained by the method described in Non-Patent Document 2. Hereinafter, although described based on the virtual screen 3, it is equivalent to a real image.

Two reference points P and Q are set on the virtual screen 3. The reference point is a point at which points P ′ and Q ′ on the reference plane in the real space are photographed. Further, the distance e between P ′ and Q ′ in the real space is acquired.
The setting of the reference point and the acquisition of the distance e can be performed by photographing an object of known size and recognizing the object, but in this embodiment, the user inputs.
A user activates an image composition program using a computer and causes the image composition program to read captured real image data. On the screen on which the image composition program displays a real image, the points P and Q are clicked with the mouse, and the reference distance is input in the window shown in FIG. FIG. 2 is a diagram showing an example of the screen of the size input means. As described above, when two points of an object of known size are designated, the reference distance can be easily input by selection, and in other cases, the actual distance can be input as a numerical value.
FIG. 3 is a diagram showing an example of the flow of the image composition program. The above is No in the branch of “Is there an image of known size” in the figure, and the processing of the size instructing means 5 below it is performed.
The size instructing means 5 inputs the reference distance value e, the reference point P, and the x and y coordinates of Q. The camera coordinates of P and Q have a z coordinate (value is d) in addition to the x coordinate and the y coordinate. Hereinafter, vector values represented by the three-dimensional coordinates are represented by P and Q.

ここで、（Ｐ・Ｎ）及び（Ｑ・Ｎ）は、Ｐ及びＱとＮとの内積である。
Ｐ_０及びＱ_０に基づき、以下の数４の式により、Ｐ’及びＱ’を求める。

ここで、｜Ｐ_０−Ｑ_０｜は、Ｐ_０とＱ_０との距離である。
基準面は、

として表される。基準面の正確な位置が求まる。ここで、｜Ｐ’｜，｜Ｐ_０｜は、ベクトルＰ’，Ｐ_０の大きさである。 The virtual object placement unit 6 obtains P ′ and Q ′ based on P, Q, and N as follows.
Intersections P ₀ and Q _{0 of} a plane connecting the origin of the camera coordinates and the planes P and Q, which are moved by 1 in the N direction from the camera coordinates origin (lens position), and the camera coordinates origin P and Q Is obtained by the following equation (3).

Here, (P · N) and (Q · N) are inner products of P and Q and N.
Based on P ₀ and Q ₀ , P ′ and Q ′ are obtained by the following equation (4).

Here, | P ₀ -Q ₀ | is the distance between P ₀ and Q ₀ .
The reference plane is

Represented as: The exact position of the reference plane is obtained. Here, | P ′ | and | P ₀ | are the magnitudes of the vectors P ′ and P ₀ .

Based on P ′ and Q ′, the virtual object placement means 6 uses the position of P ′ as the origin, the direction from P ′ to Q ′ is the x ′ axis, and the normal of the reference plane that is upward from P ′ in the figure Is a real coordinate with the y′-axis and the z′-axis the direction in the reference plane perpendicular to the x′-axis. Unit vectors (U _x ′, U _y ′) in the real coordinate system representing the x ′ axis and the y ′ axis are as follows in camera coordinates. Since the z ′ axis is a direction perpendicular to the x ′ axis and the y ′ axis, it can be easily calculated.

The virtual object placement unit 6 places a virtual object represented by 3DCG in real coordinates. Here, the virtual object is a bottom surface when the plane y ′ = 0 is placed on the reference plane, the x′-axis direction is the width direction, the y′-axis direction is the height direction, and the z′-axis direction is the depth direction. , 3DCG is configured such that the front side faces the negative side of the z ′ axis. Also, it is specified that one point on the bottom surface of the virtual object is aligned with the origin of the real coordinates. This is because it is placed correctly on the reference plane and the front side is projected onto the virtual screen 3. FIG. 4 is a diagram illustrating an example of a virtual object arrangement state. The virtual object 4 is arranged so that the camera direction is the front, and is correctly placed on the reference plane.
In this embodiment, P ′ is the origin of the actual coordinates, but another position (for example, the midpoint of P ′ and Q ′) may be the origin. Further, the y ′ axis must be a normal line of the reference plane facing upward in the drawing, but the x ′ axis and the z ′ axis may be different from those in the present embodiment. In short, the position and direction (excluding the normal direction of the reference plane) where the virtual object is arranged may be arbitrarily determined by program design.

とするとき、

左辺のベクトルがカメラ座標、右辺のベクトルが現実座標である。
このカメラ座標（ｘ、ｙ、ｚ）が、仮想スクリーン３上の（Ｘ，Ｙ，ｄ）に投影され仮想画像となる。３ＤＣＧのデータを一方向に投影した２次元画像を作成する方法については、既知のものを用いればよい。なお、Ｘ、Ｙは以下の計算で求めることができる。

仮想スクリーン３に投影された画像が、合成画像となる。 The image composition program converts real coordinates into camera coordinates and projects them onto the virtual screen 3.
The conversion from the real coordinates to the camera coordinates is performed by the following calculation.
x'-axis, y'-axis, z'-axis unit vectors (U _x ', U _y ', U _z ') expressed in camera coordinates,

And when

The vector on the left side is the camera coordinates, and the vector on the right side is the real coordinates.
The camera coordinates (x, y, z) are projected onto (X, Y, d) on the virtual screen 3 to become a virtual image. As a method for creating a two-dimensional image obtained by projecting 3DCG data in one direction, a known one may be used. X and Y can be obtained by the following calculation.

The image projected on the virtual screen 3 becomes a composite image.

As shown in the present embodiment, according to the image composition program of the present invention, the user can obtain a composite image only by inputting two reference points and a reference distance. The input of the two reference points may be a mouse click or a screen tap when using a touch panel screen, or the start point and end point of a line segment drawn using a line drawing gesture as reference points. Also good. There is no need to take an image with a marker placed accurately as in the prior art.
Further, unlike the technique of Non-Patent Document 1, a virtual object is synthesized with a real image with a correct size. The user can accurately grasp the case where the virtual object is placed in the real space.

  In this embodiment, a composite image obtained by the image composition program of the present invention is displayed on a touch panel screen, and the composite image is moved based on a gesture on the touch panel screen.

A composite image is displayed on the touch panel screen. The user moves (changes the position) or rotates (changes the direction) the virtual object using the gesture. FIG. 5 is a diagram illustrating an example of a gesture and its processing.
(A) is a gesture for moving, and the gesture 8a points a virtual object with one finger and moves it to a desired position.
(B) is a gesture for performing rotation, and the gesture 8b is for turning two fingers around a virtual object.

ここで、添字Ｏは移動前、添字Ｒは移動後を示す。
回転のジェスチャについては、θだけの回転（ジェスチャ８ｂの始点と終点の角度の差）とすると

ここで、添字Ｏは回転前、添字Ｒは回転後を示す。（ｚ’_Ｃ、ｘ’_Ｃ）は、回転ジェスチャの中心点８ｃに対応する基準面上の点の座標である。 The image composition program that has recognized the gesture transforms the real coordinates where the virtual object is arranged according to the following equation. Note that the virtual object is moved and rotated in the z′x ′ plane, and the y ′ coordinate is the same before and after the gesture.
As for the movement gesture, the movement by (z ′ _m , x ′ _m ) (the difference between the coordinates of the two points on the reference plane corresponding to the start point and the end point of the gesture 8a: the coordinates of the two points on the reference plane are examples) (It can be calculated by Equation 3 and Equation 4 as in 1.)

Here, the subscript O indicates before movement, and the subscript R indicates after movement.
As for the rotation gesture, if the rotation is only θ (the difference between the start point and the end point of the gesture 8b),

Here, the subscript O indicates before rotation, and the subscript R indicates after rotation. (Z ′ _C , x ′ _C ) are the coordinates of a point on the reference plane corresponding to the center point 8c of the rotation gesture.

The image composition program obtains a composite image by projecting a virtual object on a virtual screen based on the values of real coordinates after movement and rotation, as in the first embodiment. Note that the synthesized image may be continuously updated following the gesture without waiting for the gesture to end.
When a touch screen panel is not used, it can be moved by dragging and dropping, and a predetermined handle can be moved to rotate by operating the mouse without using a gesture. Specific user operations for movement and rotation can be arbitrarily designed.

  According to the present embodiment, the user can see a composite image of various virtual object arrangements. Considering the consideration before purchasing furniture, you can see the image of the most favorite arrangement in the room.

  In this embodiment, the image composition program of the present invention is operated on a server.

FIG. 6 is a diagram illustrating an example of the configuration of an image composition program. The server 9 and the user terminal device 10 are connected via the Internet. In the server 9, an image composition program 11 operates and virtual object data 12 is stored. In the user terminal device 10, the browser 13 operates, and the real image data 14 is stored or created. The user terminal device 10 does not have a special program or data, and may be a widely used device such as a personal computer or a smartphone with a camera.
The virtual object data 12 is for furniture, home appliances, or interior decorations. The server 9 is operated by a person who sells these items.

A user who uses the user terminal device 10 which is a personal computer starts up the browser 13 and accesses the server via the Internet. The user holds an indoor photograph (real image data 14) taken using a digital camera in the user terminal device 10, and transmits this to the image composition program 11. Here, the real image data 14 is in the Exif format, and includes data on the focal length and the shooting direction.
Note that the user terminal device 10 which is a smartphone can also take a picture and transmit it to the server including data on the focal length and the shooting direction.

The virtual object data 12 includes 3DCGs of a plurality of products, and the image composition program 11 displays these images on the browser 13 on a screen (not shown). The user selects one to be used for the composite image from the displayed images.
The image composition program 11 displays a real image on the browser 13. The user inputs a reference point and a reference distance for the displayed real image.
The image composition program 11 creates a composite image including the selected virtual object in the same manner as in the first embodiment, and displays it on the browser 13.
Thereafter, the virtual object may be moved / rotated in the same manner as in the second embodiment.

  According to this embodiment, a person who sells furniture, home appliances, or interior ornaments can effectively promote the sale.

    It is a program that creates an accurate composite image without using a marker, and can be expected to be used by many companies, particularly furniture dealers.

DESCRIPTION OF SYMBOLS 1 Reference plane 2 Camera 3 Virtual screen 4 Virtual object 5 Size instruction | indication means 6 Virtual object arrangement | positioning means 7 Step 8 Gesture
9 Server 10 User terminal device 11 Image composition program 12 Virtual object data 13 Browser 14 Real image data

Claims (7)

An image composition program for creating a composite image by combining a real image obtained by photographing a real space with a camera and a virtual image that is an image obtained by viewing a virtual object represented as three-dimensional shape data from one projection direction. ,
The composite image is an image in which the virtual object is placed on a reference plane that is one horizontal plane in the real space,
Virtual object placement means for calculating coordinates in real space for placing the virtual object on the reference plane;
The virtual object placement unit calculates an accurate position of the reference plane based on a distance (reference distance) between two points in the real space of a reference point that is two points on the reference plane included in the real image. An image composition program characterized by: A size instruction means for designating two points in the real image and inputting a distance between the two points;
2. The image composition program according to claim 1, wherein the virtual object placement unit uses the two points as the reference point, and uses the distance input by the size instruction unit as the reference distance. 3.   The virtual object placement means calculates the position and direction of the virtual object in the real space based on the coordinates of the reference point in the real image, the reference distance, and the shooting direction of the camera when the real image is shot. The image composition program according to claim 1 or 2, wherein   The virtual image moving means for displaying the composite image on a touch panel screen and changing a position and a direction of the virtual object in a real space based on a gesture on the touch panel screen. The image composition program according to claim 1. The camera is attached to the smartphone;
The image composition program according to claim 4, wherein the touch panel screen is a screen attached to the smartphone. The real image is a photograph of a room,
The image composition program according to claim 1, wherein the virtual object is furniture, a home appliance, or an interior ornament. Runs on the server,
The real image is an image received from a user via the Internet;
3D shape data of the virtual object is stored in the server,
7. The image composition program according to claim 6, wherein the synthesized image is displayed on a user terminal device via the Internet.

Images