JP6319804B2 - Projection image generation apparatus, projection image generation method, and projection image generation program - Google Patents

Description

  The present invention relates to a projection image generation apparatus, a projection image generation method, and a projection image generation program that perform image projection (projection mapping) on an object arranged in a space.

  2. Description of the Related Art Conventionally, there is a projection mapping technique that performs dynamic change of surface color and spatial rendering by generating and projecting an image corresponding to the shape of a fixed three-dimensional object (for example, Non-Patent Document 1).

Toshiyuki Amano, “Material Projection Mapping”, June 2014, Journal of the Virtual Reality Society of Japan, Vol. 19, No. 2, pp.14-17

  Conventional projection mapping uses a fixed three-dimensional object as a projection target, and cannot project an image following a dynamic three-dimensional object whose position and orientation changes in real time.

  The present invention has been made in view of this problem, and provides a projection image generation apparatus, a projection image generation method, and a projection image generation program that project a moving image or a still image corresponding to a movable three-dimensional object in real time. Objective.

The projection image generation apparatus of the present invention is a projection image generation apparatus that generates an image for projection to be projected onto a real space object arranged in a space, from the position information of a position detection element attached to the real space object. A model matrix generation unit that calculates an orthogonal unit vector of a real space coordinate that is a coordinate system, and generates a model matrix indicating a relationship between the position information of the position detection element, the orthogonal unit vector, and the real space coordinate; , Each vertex position information of the virtual space object in the real space coordinates by multiplying the model matrix by each vertex position information of the virtual space object arranged in the virtual space having the coordinate system independent of the change of the position and orientation A vertex position information calculation unit that calculates a projection volume in the space and a virtual space view volume in the virtual space. Wherein the virtual space viewpoint information for between to coincide with each vertex position information, the virtual space object; and a virtual space image generation unit configured to generate a virtual space image to be projected onto the real space object, The virtual space view volume is obtained from two real space coordinates on a straight line connecting the position of the projector that projects two or more point images onto the projector projection screen projected onto the space and the point image. and viewpoint data is a parameter determined from two or more straight lines, and the line-of-sight direction information, and the overhead direction information, and aspect angle information, in Ru der those represented.

The projection image generation method of the present invention is a projection image generation method for generating a projection image to be projected onto a real space object arranged in a space performed by the projection image generation device, and is provided to the real space object. An orthogonal unit vector of real space coordinates, which is a coordinate system of the space, is calculated from position information of the position detection element, and the relationship between the position information of the position detection element, the orthogonal unit vector, and the real space coordinates is shown. A model matrix generating step for generating a model matrix; and multiplying the model matrix by each vertex position information of a virtual space object arranged in a virtual space having a coordinate system independent of a change in position and orientation, A vertex position information calculating step of calculating each vertex position information of the virtual space object in the space; a projection volume in the space; A virtual space for generating a virtual space image for projecting the virtual space object onto the real space object using the virtual space viewpoint information for spatially matching the virtual space view volume between them and each vertex position information There line and an image generation step, the virtual space view volume, the projector projection screen projected by the projector in the space, on a straight line in which the positions of the two or more points the projector projecting the image connecting the said point image This is expressed by viewpoint position information, line-of-sight direction information, overhead direction information, and vertical and horizontal angle-of-view information, which are parameters obtained from two or more straight lines obtained from two real space coordinates .

  A projection image generation program according to the present invention causes a computer to function as the projection image generation apparatus.

  According to the present invention, it is possible to generate image information that projects a moving image or a still image corresponding to a movable three-dimensional object (real space object) in real time.

It is a whole block diagram which shows an example of the projection image effect method containing the projection image generation apparatus 50 in this Embodiment. It is a figure which shows the example of the vertex position data of the real space object on an object coordinate. It is a figure which shows typically the agreement with the projection volume in real space, and the virtual space view volume in virtual space. It is a figure which shows typically an example of the projection image used by virtual space viewpoint construction, and a mode that two straight lines are calculated | required. It is a figure which shows the procedure of the projection image effect method including the operation | movement flow of the projection image generation apparatus. It is a figure which shows an example of the three-point position detection element, orthogonal unit vector, and model matrix which were given to the one vertex of the real space object. It is a figure which shows typically the geometric transformation which the real space arrangement | positioning condition reproduction part 20 performs. It is a figure which shows typically the image description which the virtual space image generation part 30 performs.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Configuration of Projection Image Generation Device]
First, with reference to FIG. 1, the structure of the projection image generation apparatus 50 of this Embodiment is demonstrated. The projection image generation device 50 includes a real space arrangement state reproduction unit 20 and a virtual space image generation unit 30. The real space arrangement state reproduction unit 20 includes a model matrix generation unit 21 and a vertex position information calculation unit 22.

  The real space arrangement state reproduction unit 20 receives the shape information 14 of the virtual space object and the position information of the three position detection elements given to the real space object 13 acquired from the motion capture device 15 as inputs. Outputs the vertex position data of the virtual space object in the coordinate system.

  The virtual space image generation unit 30 is a virtual space for spatially matching the vertex position data of the virtual space object output from the real space arrangement state reproduction unit 20 with the projection volume in the space and the virtual space view volume in the virtual space. With the viewpoint information 17 as an input, a virtual space image for projecting the virtual space object onto the real space object 13 is output. Here, the virtual space object is a three-dimensional object that matches the shape of the real space object 13. The shape of the virtual space object may be similar to that of the real space object 13.

  The virtual space object is arranged on the object coordinates that do not depend on the change in the position and orientation of the virtual space object. Therefore, the notation of the virtual space object is omitted in FIG.

[Configuration for displaying and directing projected images]
Further, in order to display and produce a projected image using the projected image generation device 50, the real space object 13, the motion capture device 15, and the projector 16 are required. The motion capture device 15 is a device that digitally records the movement of an actual person or object.

  The real space object 13 is a substance of an object arranged in the space. The real space object 13 may be moved manually by the operator 11 or may be automatically rotated, for example. The position and orientation of the real space object 13 changes in real time. The viewer 12 is a person who views a projection image projected on the surface of the real space object 13 by the projector 16.

[Information to be input to the projection image generator]
Next, the virtual space object shape information 14 and the virtual space viewpoint information 17, which are information input to the projection image generation apparatus 50 from the outside, will be described.

[Shape information of virtual space object]
The virtual space object shape information 14 is virtual space object shape information that matches the shape of the real space object 13. The shape of the real space object 13 may be measured by a person using a measure or the like, or may be automatically measured using an existing 3D scanner device or the like.

  As shown in FIG. 2, the X, Y, and Z position data in the object coordinate system that does not depend on the change in the position and orientation of the three-dimensional object is used as the shape information of the virtual space object. The example shown in FIG. 2 is shape information when the shape of the real space object 13 is a cube, for example, and one side is 100 mm. In the following, object coordinates are represented by capital letters X, Y, and Z.

  The shape information of the virtual space object is the vertex position of the three-dimensional object. In the case of a cube, it is eight coordinate information of vertex 1 to vertex 8. The coordinate information of the vertex 1 is X = 100, Y = 100, Z = 100. The coordinate information of the vertex 8 located on the Z axis is X = 0, Y = 0, Z = 100. In the case of a sphere, a surface obtained by dividing an arbitrary number of spherical surfaces is prepared, and coordinate information on the vertex position of each surface becomes the shape information of the sphere.

  As described above, the shape information 14 of the virtual space object is coordinate information of the vertex position of the three-dimensional object in the object coordinate system of the virtual space object that coincides with the shape of the real space object 13, and is measured once corresponding to the virtual space object. It is a good thing.

[Virtual space perspective information]
The virtual space viewpoint information 17 is information for spatially matching the projection volume in the space to be projected (real space) and the virtual space view volume in the virtual space. The projection volume and the virtual space view volume will be described with reference to FIG.

  FIG. 3A schematically shows a three-dimensional “projection region” of a space in which the projector 16 projects an image with a quadrangular pyramid having the projector 16 as a vertex. This three-dimensional “projection area” is called a projection volume.

  FIG. 3B schematically shows the “field of view” from the human eye (viewpoint) in the virtual space as a quadrangular pyramid with the viewpoint as the apex. This three-dimensional “field of view” is called a virtual space view volume.

  In order to project an image of the virtual space onto the space to be projected, it is necessary to spatially match the projection volume and the virtual space view volume. In order to make them coincide spatially, information on the “position”, “projection direction”, “posture (overhead direction)” and “projection angle of view” of the projector 16 is used as the “position”, “ Applies to "Gaze direction", "Posture (overhead direction)", and "View angle of view".

  A method for deriving each parameter of the projection volume will be described with reference to FIG. Each parameter is derived in cooperation with the motion capture device 15. First, an image in which a point is drawn in advance at the center position and the center right end is projected from the projector 16, and a position detection element (hereinafter referred to as a marker) is moved so as to overlap the projection point, and two points A1, A2 on the center axis, and The coordinates x, y, z of the two points B1, B2 on the center right axis are acquired. In the following, the real space coordinates are written in lower case letters x, y, z.

  Specifically, the person grasps the marker, and the person appropriately stops the marker twice twice so that it overlaps the point image projected at the center position of the projection screen of the projector. The dimension position is measured by the motion capture device 15. Further, the motion capture device 15 measures the two-dimensional (B1, B2) three-dimensional positions by appropriately stopping the marker twice so as to overlap the point image projected on the right position of the projection screen of the projector 16. .

  Then, the intersection of the straight line A generated from the two points (A1, A2) on the central axis and the straight line B generated from the two points on the central right end axis is calculated as a “position” parameter (viewpoint position information). The The straight line A is applied to the “projection direction” parameter (gaze direction information). The “posture (overhead direction)” parameter (overhead direction information) is calculated by the outer product of the straight line A and the straight line B. For the vertical and horizontal “view angle” parameters (vertical and horizontal view angle information), known information of the projector 16 to be used is used.

  The virtual space viewpoint information 17 may be measured once once the space to be projected and the projector 16 are determined. In addition, although the example which projected the point image on the center position and the center right side was demonstrated, you may make it project a point image on another part.

  For example, if a point image is projected at the center left end position and the center right end position, straight lines A ′ and B are obtained from these two point images, and the straight line A (that is, the center line) is set as the straight line between these two straight lines The same processing as described above is possible. It is also possible to improve accuracy by drawing three or more point images and averaging them.

[Operation of Projected Image Generation Device]
FIG. 5 shows an operation flow of the projection image generation apparatus 50. The operation of the projection image generating apparatus 50 will be described with reference to FIGS.

  In the virtual space object shape construction in step S1, the virtual space object shape information 14 is generated. The generation method is as described above. The shape information 14 of the virtual space object is prepared before the projection image generation device 50 starts operating.

  In the virtual space viewpoint construction in step S2, virtual space viewpoint information 17 is generated. The generation method is as described above. The virtual space viewpoint information 17 is prepared before the projection image generation apparatus 50 starts operating.

  When the projection image generation device 50 starts operating, the motion capture device 15 acquires the three-point reference position information of the real space object 13 (step S3). The three-point reference position information is acquired from the position of the marker by adding three markers in an L shape to the real space object 13 and using the motion capture device 15.

  FIG. 6A shows the real space object 13 and the marker. For example, a marker P2 is arranged at the vertex 1 of the real space object 13 which is a cube, a marker P1 is arranged in the direction of the vertex 4 on the same plane as the marker P2, and a marker P3 is arranged in the direction of the vertex 2.

  The model matrix generation unit 21 of the real space arrangement state reproduction unit 20 calculates three orthogonal unit vectors v1, v2, and v3 that are orthogonal from the position information of the markers P1, P2, and P3 acquired by the motion capture device 15 (FIG. 6). (B)). The orthogonal unit vector v1 can be obtained by v1 = P1-P2, the orthogonal unit vector v2 can be obtained by v2 = P3-P2, and the orthogonal unit vector v3 can be obtained by the outer product v1 × v2 of v1 and v2.

  And the model matrix production | generation part 21 produces | generates the model matrix M (FIG.6 (c)) from the coordinate value of the marker P2 arrange | positioned at one vertex, and orthogonal unit vector v1, v2, v3 (step S40). The model matrix M shows the relationship between one vertex P2 of the real space object 13, the orthogonal unit vectors v1, v2, and v3 and the real space coordinate values. In this example, the vertex P2 is used as the reference position information, but another vertex may be used as the reference position information.

  By multiplying the model matrix M by each vertex position vector {X, Y, Z, 1} obtained from the coordinate information of each vertex (vertex 1 to vertex 8) of the shape information 14 of the virtual space object, the real space is obtained. Each vertex position {x ′, y ′, z ′, 1} of the virtual space object in the coordinate system is calculated (formula (1)).

式（１）から明らかなように、モデル行列Ｍはオブジェクト座標系を実空間座標系に座標変換する行列である。実空間配置状況再現部２０の頂点位置情報算出部２２は、この幾何学変換により、実空間座標系におけるマーカＰ２を原点としたｖ１，ｖ２，ｖ３の軸上に、オブジェクト座標系のＸ，Ｙ，Ｚ座標軸が移動し、結果的に実空間オブジェクト１３と同形状、同位置姿勢の仮想空間オブジェクトの立体形状を実空間内に配置することができる（ステップＳ４１）。ステップＳ４０とＳ４１は、実空間配置状況再現部２０が行う処理である（ステップＳ４）。

As is clear from the equation (1), the model matrix M is a matrix for performing coordinate transformation from the object coordinate system to the real space coordinate system. By this geometric transformation, the vertex position information calculation unit 22 of the real space arrangement state reproduction unit 20 has X, Y in the object coordinate system on the axes of v1, v2, and v3 with the marker P2 in the real space coordinate system as the origin. , The Z coordinate axis is moved, and as a result, the three-dimensional shape of the virtual space object having the same shape and the same position and orientation as the real space object 13 can be arranged in the real space (step S41). Steps S40 and S41 are processes performed by the real space arrangement state reproduction unit 20 (step S4).

  FIG. 7 schematically shows processing performed by the vertex position information calculation unit 22 (step S41). FIG. 7A is the same view as FIG. In FIG. 7A, coordinate information of each vertex (vertex 1 to vertex 8) of the shape information 14 of the virtual space object is obtained. As described above, the shape information 14 of the virtual space object is obtained in advance (step S1).

  By multiplying each vertex position vector {X, Y, Z, 1} obtained from the coordinate information of each vertex of the shape information 14 of the virtual space object and the model matrix M, a virtual space having the same position and orientation as the real space object 13 is obtained. The three-dimensional shape of the object can be arranged in the real space (FIG. 7B).

  The virtual space image generation unit 30 generates a virtual space image for projection by rendering the vertex position data of the virtual space object output from the real space arrangement state reproduction unit 20 using the virtual space viewpoint information 17 (step). S5). A technique for rendering a virtual aerial image is described in, for example, Reference 1 (Computer Graphics, Supervision: Computer Graphics Editorial Board, Publisher: CG-ARTS Association, pp32-40).

  The virtual space image generated by the virtual space image generation unit 30 is projected from the projector 16 onto the real space object 13 (step S6). As a result, it is possible to produce a space effect in which the real space object 13 is superimposed in real time with the three-dimensional image of the virtual space object (step S7). That is, for example, assuming that the cubic real space object 13 is rotating around the y axis (see FIG. 7B), the virtual space image is projected onto a plurality of projection planes (cube surfaces) that appear one after another. be able to. As the virtual space image, for example, a different color corresponding to each projection surface can be considered.

  Note that the virtual space image generated by the virtual space image generation unit 30 may be recorded in a memory. If the change in the position and orientation of the real space object 13 is periodic, the same space effect as described above can be obtained by reading out the virtual space image recorded in synchronization with the period.

  From the step of acquiring the three-point reference position information of the real space object 13 in step 3, the steps of browsing the virtual space object projected from the projector 16 in step S7 are repeated. The repetition speed is, for example, about 60 f / s (frame / second).

  As described above, according to the present embodiment, it is possible to generate image information (virtual space image) that projects a moving image or a still image corresponding to a movable three-dimensional object (real space object 13) in real time. Note that there may be a plurality of real space objects present in the irradiation range of the projector 16. Further, although the shape of the real space object 13 has been described by taking a cube as an example, the shape is not limited to this shape. For example, the shape of the real space object 13 may be a rectangular parallelepiped. Moreover, it may be a sphere as described above. As described above, the present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of the gist thereof.

  When the processing unit in the above apparatus is realized by a computer, the processing content of the function that each processing unit should have is described by a program. Then, by executing this program on a computer, the processing unit in the above apparatus is realized on the computer.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a recording device of the server computer and transferring the program from the server computer to another computer via a network.

11: Operator 12: Viewer 13: Real space object 14: Virtual space object shape information 15: Motion capture device 16: Projector 17: Virtual space viewpoint information 20: Real space arrangement state reproduction unit 21: Model matrix generation unit 22 : Vertex position information calculation unit 30: virtual aerial image generation unit 50: projection image generation device

Claims (5)

In a projection image generation apparatus that generates a projection image to be projected onto a real space object arranged in space,
An orthogonal unit vector of real space coordinates, which is a coordinate system of the space, is calculated from position information of the position detection element assigned to the real space object, and the position information of the position detection element, the orthogonal unit vector, and the real unit are calculated. A model matrix generation unit that generates a model matrix indicating a relationship with spatial coordinates;
By multiplying the model matrix by each vertex position information of the virtual space object arranged in the virtual space having the coordinate system independent of the change of the position and orientation, each vertex position information of the virtual space object in the real space coordinates is obtained. A vertex position information calculation unit to calculate,
Projecting the virtual space object onto the real space object using the virtual space viewpoint information for spatially matching the projected volume in the space and the virtual space view volume in the virtual space and the vertex position information A virtual aerial image generation unit for generating a virtual aerial image ,
The virtual space view volume is
It is obtained from two or more straight lines obtained from two real space coordinates on a straight line connecting the position of the projector that projected two or more point images on the projector projection screen projected onto the space and the point image. parameters and viewpoint data is a, and the line-of-sight direction information, and the overhead direction information, and aspect angle information, in the projection image generation apparatus according to claim der Rukoto those represented. The projection image generating apparatus according to claim 1,
The orthogonal unit vector is a vector calculated from the coordinate information of the three points of the position detection element given in an L shape to one vertex of the real space object,
The model matrix is a projection image that includes the orthogonal unit vector as an element, and that converts each vertex position information in the object coordinates of the virtual space object into each vertex position information in the real space coordinates. Generator. A projection image generation method for generating a projection image to be projected onto a real space object arranged in a space performed by a projection image generation apparatus,
An orthogonal unit vector of real space coordinates, which is a coordinate system of the space, is calculated from position information of the position detection element assigned to the real space object, and the position information of the position detection element, the orthogonal unit vector, and the real unit are calculated. A model matrix generation step for generating a model matrix indicating a relationship with spatial coordinates;
By multiplying the model matrix by each vertex position information of the virtual space object arranged in the virtual space having the coordinate system independent of the change of the position and orientation, each vertex position information of the virtual space object in the real space coordinates is obtained. Calculating vertex position information to be calculated;
Projecting the virtual space object onto the real space object using the virtual space viewpoint information for spatially matching the projected volume in the space and the virtual space view volume in the virtual space and the vertex position information There line and a virtual space image generation step of generating a virtual space image,
The virtual space view volume is
It is obtained from two or more straight lines obtained from two real space coordinates on a straight line connecting the position of the projector that projected two or more point images on the projector projection screen projected onto the space and the point image. A projection image generation method, characterized in that it is represented by viewpoint position information, line-of-sight direction information, overhead direction information, and vertical and horizontal angle-of-view information as parameters . In the projection image generation method according to claim 3 ,
The orthogonal unit vector is a vector calculated from the coordinate information of the three points of the position detection element given in an L shape to one vertex of the real space object,
The model matrix is a projection image that includes the orthogonal unit vector as an element, and that converts each vertex position information in the object coordinates of the virtual space object into each vertex position information in the real space coordinates. Generation method. Projection image generation program for causing a computer to function as the projection image generation apparatus according to claim 1 or 2.

Images