JP2002032784A - Device and method for operating virtual object - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a virtual object operating device and a virtual object operating method which can display a virtual object in a virtual world by three- dimensional computer graphics from an optional viewpoint in a state corresponding to the operation of an object in a real world. SOLUTION: A three-dimensional position measuring instrument 2 measures the three-dimensional position of an object OB, a camera 5 photographs the marker MK of the object OB, an arithmetic processor 6 calculates the attitude of the object OB from the three-dimensional position of the object OB and the image of the marker MK and changes the display state of the virtual object in a virtual space in accordance with the three-dimensional position and attitude of the object OB.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual object operation apparatus for operating a virtual object associated with an object in a virtual world by three-dimensional computer graphics by operating the object in the real world. And a virtual object operation method.

[0002]

2. Description of the Related Art In recent years, various researches have been conducted in a mixed reality research field in which a virtual world presented by three-dimensional computer graphics and the like and a real world where humans actually live are fused. For example, Jun Rekimoto,
“Matrix: Realtime Object Identification and Regis
tration Method for Augmented Reality ”, Proc. of AP
CHI '98, Mark Billinghurst, Hirokazu Kato, “Collab
orative Mixed Reality ”, ISMR '99, pp.261-284, a plate-like marker with a predetermined mark printed is photographed by a camera, and three markers are placed at the marker position in the photographed image.
It is disclosed to project a virtual object created by three-dimensional computer graphics.

[0003]

However, in the method used in the above-mentioned document, only an image taken by a camera is used, and the world coordinate system is not specified. Therefore, when viewing the virtual object projected on the camera coordinate system from another arbitrary viewpoint, the camera coordinate system corresponding to the viewpoint cannot be obtained, and the virtual object viewed from the arbitrary viewpoint is displayed. Can not do it.

[0004] It is an object of the present invention to provide a virtual object operation device capable of displaying a virtual object in a virtual world by three-dimensional computer graphics from an arbitrary viewpoint in accordance with operation of an object in the real world. An object of the present invention is to provide a virtual object operation method.

[0005]

Means for Solving the Problems and Effects of the Invention (1)
1st invention A virtual object operation device according to a 1st invention operates a virtual object associated with an object in a virtual world by three-dimensional computer graphics by operating an object in the real world. A virtual object operation device for measuring a three-dimensional position of an object, a photographing means for photographing the object, and a three-dimensional position and photographing means of the object measured by the position measuring means. A posture calculating means for calculating the posture of the object from the image of the photographed object, and a virtual three-dimensional position of the object measured by the position measuring means and the posture of the object calculated by the posture calculating means. Display means for changing the display state of the virtual object in the space.

In the virtual object operation device of the present invention,
The three-dimensional position of the object is measured, the object is photographed, the posture of the object is calculated from the three-dimensional position of the object and the photographed image of the object, and the three-dimensional position of the object is calculated. The display state of the virtual object in the virtual space changes according to the posture of the object.

As described above, since the three-dimensional position of the object as well as the image of the object is measured, the world coordinate system can be defined, and even when the virtual object is viewed from another viewpoint, A camera coordinate system corresponding to the viewpoint can be obtained. Therefore, it is possible to display a virtual object in a virtual world by three-dimensional computer graphics from an arbitrary viewpoint in accordance with an operation of an object in the real world.

(2) Second invention A virtual object operating device according to a second invention is a virtual object operating device according to the first invention, wherein the position measuring means determines a three-dimensional position of the object. It includes wireless position measuring means for wirelessly measuring.

In this case, since the three-dimensional position of the object can be measured wirelessly, it is not necessary to provide wiring or the like for measuring the three-dimensional position of the object on the object.
The object can be freely operated without being disturbed by wiring or the like.

(3) Third invention A virtual object operation device according to a third invention is the virtual object operation device according to the second invention, wherein the wireless position measuring means is attached to the object. It includes a transmitting unit that wirelessly transmits identification information for identifying an object, and a plurality of receiving units that receive the identification information transmitted from the transmitting unit.

In this case, the identification information for identifying the object is transmitted wirelessly by the transmission means attached to the object, and the transmitted identification information is received by the plurality of reception means. The received identification information is received at a plurality of positions, and the three-dimensional position of the object can be measured based on the received time difference.

Further, since the identification information for identifying the object is transmitted, even if there are a plurality of objects, it is possible to specify which object is being operated. Therefore, a plurality of objects can be operated at the same time, and virtual objects corresponding to each operated object can be operated in the virtual space.

(4) Fourth invention A virtual object operating device according to a fourth invention is the virtual object operating device according to any one of the first to third inventions, wherein the object has an ultraviolet ray. A marker having a predetermined shape made of a fluorescent material that emits light is attached, and the photographing means includes an irradiating means for irradiating the marker with ultraviolet light, and a marker photographing means for photographing the marker irradiated with the ultraviolet light by the irradiating means.

[0014] In this case, the marker attached to the object is irradiated with ultraviolet rays, and the marker emitting light by the ultraviolet rays can be imaged. In this way, the marker is photographed using the light emission of the marker itself due to ultraviolet rays, so even if the marker is attached to the bottom of the object and the object is placed on the glass surface, it is reflected on the glass surface The image of the marker attached to the bottom surface of the object can be captured even in a dark place without being affected by the glass surface.

(5) Fifth Invention A virtual object operating device according to a fifth invention is the virtual object operating device according to the fourth invention, wherein the marker is
The posture calculating means has a mark eccentrically located from the center position, and the posture calculating means calculates the object from the three-dimensional position of the object measured by the position measuring means, the image of the marker photographed by the marker photographing means, and the position of the mark on the marker. Is calculated.

In this case, the mark is provided eccentrically from the center position of the marker, and the posture of the object is calculated from the three-dimensional position of the object, the marker image, and the position of the mark. Even if it has an axis and the posture of the object cannot be completely calculated only by the shape of the marker, the rough inclination of the object can be specified by the mark eccentric from the center position of the marker, The posture can be calculated.

(6) Sixth Invention The virtual object operation method according to the sixth invention is such that, by operating an object in the real world, the object is associated with the object in the virtual world by three-dimensional computer graphics. Measuring a three-dimensional position of the object, photographing the object, and measuring the three-dimensional position of the object and the photographed object. Calculating a posture of the object from the image of the object, and changing a display state of the virtual object in the virtual world according to the measured three-dimensional position of the object and the calculated posture of the object. Including.

In the virtual object operation method according to the present invention, the three-dimensional position of the object is measured, the object is photographed, and the object is obtained from the three-dimensional position of the object and the photographed image of the object. Is calculated, and the display state of the virtual object in the virtual space is changed according to the three-dimensional position of the object and the calculated posture of the object.

As described above, since the three-dimensional position of the object as well as the image of the object is measured, the world coordinate system can be defined, and even when the virtual object is viewed from another viewpoint, A camera coordinate system corresponding to the viewpoint can be obtained. Therefore, it is possible to display a virtual object in a virtual world by three-dimensional computer graphics from an arbitrary viewpoint in accordance with an operation of an object in the real world.

[0020]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a virtual object operation device according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a virtual object operation device according to an embodiment of the present invention.

The virtual object operating device shown in FIG. 1 includes a transmitter 1, a three-dimensional position measuring device 2, an I / F (interface).
Unit 3, black light 4, camera 5, arithmetic processing unit 6,
Input unit 7, ROM (read only memory) 8, RAM
(Random access memory) 9, storage device 10 and display unit 11.

FIG. 2 is a schematic perspective view showing the positional relationship between an object in the real world and a three-dimensional position measuring device, a camera and the like. FIG. 3 shows the positional relationship between the object in the real world and a marker and the like. It is a schematic perspective view shown.

As shown in FIGS. 1 and 2, the table T
A glass plate GP is mounted on B, and a real-world object OB is arranged on the glass plate GP. Object OB
Consists of, for example, a hexahedron of a height OH having a shape like a building block (see FIG. 3), and a user performs various operations by gripping the object OB and moving and rotating the object OB. Can be.

A transmitter 1 is mounted on the upper surface of the object OB. The transmitter 1 wirelessly transmits ID information, which is identification information for specifying the object OB, to the three-dimensional position measuring device 2 by ultrasonic waves.

Above the transmitter 1, a substantially cross-shaped three-dimensional position measuring device 2 having four sensor units 2a (see FIG. 2) composed of ultrasonic sensors is installed. The three-dimensional position measuring device 2 transmits the ID information transmitted from the transmitter 1 to 4
The position of the transmitter 1, that is, the three-dimensional position of the object OB, is measured by the time difference between the reception of the ID information and the reception of the ID information by the sensor units 2 a.

Since the ID information for identifying the object OB is transmitted as described above, even if there are a plurality of objects, it is possible to specify which object is being operated. Can be. Therefore, a plurality of objects can be operated at the same time, and virtual objects corresponding to each operated object can be operated in the virtual space.

A square marker MK having a length of 2 HS is attached to the bottom of the object OB (see FIG. 3), and a camera 5 is attached to the bottom of the table TB. A black light 4 for irradiating the marker MK with ultraviolet light is arranged on both sides of the camera 5.
Are irradiated with ultraviolet rays from below by the black light 4.

The marker MK is composed of a fluorescent plate using a fluorescent paint or a fluorescent resin which emits light by ultraviolet rays, and the marker itself emits light when irradiated with ultraviolet rays. Marker M
A mark MA is provided at a position deviated from the center of K (see FIG. 3), and the mark MA does not emit light due to ultraviolet rays. The camera 5 captures an image of the marker MK that emits light, and captures an image of the marker MK that emits light except for the mark MA.

Here, in the case where illumination by ordinary visible light is applied from above the apparatus, if the marker MK is attached to the bottom surface of the object OB, the marker MK becomes a shadow with respect to the illumination, and the marker MK is shaded from the bottom of the table TB. It becomes difficult to take an image with the camera 5. However, in the present embodiment, the ultraviolet light is generated by the black light 4 which does not emit much light, and the marker M which emits light by the ultraviolet light is used.
Since the image of K is taken, no illumination is reflected on the glass surface, and the image of the marker MK can be taken clearly even when the inside of the table TB is dark.

Further, in this embodiment, since the marker MK is photographed by ultraviolet rays and the three-dimensional position of the object OB is measured by ultrasonic waves as described above, the apparatus is installed in a dark place. Also in this case, the three-dimensional position of the object OB can be measured, and the image of the marker MK can be clearly captured.

Further, since the three-dimensional position of the object OB can be obtained wirelessly, it is not necessary to provide a wiring or the like for measuring the three-dimensional position on the object OB, and the object OB can be obtained in a natural state. Can be operated freely. Therefore, the object O can be used without learning a special operation method and without having to use the wiring or the like to complicate the operation.
B can be easily operated.

The position measuring means for measuring the three-dimensional position of the object OB in the real world is not particularly limited to the above example, but may be any one capable of wirelessly detecting the three-dimensional position of the object OB. For example, another three-dimensional position measuring device may be used.

As shown in FIG. 1, a three-dimensional position measuring device 2
Outputs the measured three-dimensional position of the object OB to the arithmetic processing unit 6 via the I / F unit 3. The camera 5 outputs the captured image of the marker MK to the arithmetic processing device 6 via the I / F unit 3.

The input unit 7 includes a keyboard, a mouse, and the like, and is used by a user to perform a desired input.

Various programs are stored in the ROM 8 in advance, and a virtual object operation processing program for executing a virtual object operation process described later is stored in advance.
The RAM 9 is used as a work area when the arithmetic processing device 6 executes various processes. The storage device 10 includes a hard disk drive and the like,
Stores various programs to be executed, and necessary data and the like.

The arithmetic processing unit 6 comprises a CPU (central processing unit) and the like, and executes a virtual object operation processing program stored in advance in the ROM 8 by using the RAM 9 and the like, whereby the image cutout unit 61 and the object The functions of the object cutout unit 62, the main axis acquisition unit 63, the rotation angle acquisition unit 64, and the virtual object drawing unit 65 are executed.

The image cutout section 61 cuts out an image including the marker MK from the image taken by the camera 5.
The target cutout unit 62 specifies the marker MK from the cutout image of the marker. The main axis acquisition unit 63 calculates the main axis vector of the target object OB using the specified center coordinates of the marker MK, the three-dimensional position of the target object OB, and the like. The rotation angle acquisition unit 64 acquires the calculated rotation angle of the object OB with respect to the principal axis vector of the object OB. The virtual object drawing unit 65 determines the three-dimensional position of the object OB and the object O
A virtual object corresponding to the object OB is obtained by using the main axis vector representing the posture of B and the rotation angle.
Data for drawing in a virtual space by graphics is created and output to the display unit 11.

The display unit 11 comprises a CRT (cathode ray tube), a liquid crystal display panel, etc., displays a virtual world by three-dimensional computer graphics according to the control of the arithmetic processing unit 6, and displays a three-dimensional image of the object OB. The display state of the virtual object in the virtual world is changed according to the position and the posture.

According to the above configuration, when the user actually operates the object OB to change the position and orientation of the object OB three-dimensionally, the virtual object based on the three-dimensional computer graphics displayed on the display unit 11 is displayed. The virtual object is displayed in a state where the three-dimensional position and orientation of the virtual object in the space are changed, and the virtual object can be operated in the virtual space by operating the object OB in the real world. In this way, for example, for city landscape simulation, 3D computer graphics layout work, etc.
The virtual object operation device according to the present embodiment can be applied.

In this embodiment, the transmitters 1 and 3
The two-dimensional position measuring device 2 corresponds to a position measuring unit and a wireless position measuring unit, and the black light 4 and the camera 5 correspond to a photographing unit. The unit 64 corresponds to a posture calculating unit, and the virtual object drawing unit 65 and the display unit 1
1 corresponds to the display means. Further, the transmitter 1 corresponds to a transmitting unit, the three-dimensional position measuring device 2 corresponds to a receiving unit, the black light 4 corresponds to an irradiating unit, and the camera 5 corresponds to a marker photographing unit.

Note that the virtual object operation device of the present embodiment can be realized using dedicated hardware, except for the transmitter 1, the three-dimensional position measuring device 2, the black light 4, and the camera 5. For example, it can be realized by a personal computer, a workstation, or the like, and a virtual object operation processing program.

Next, the virtual object operation processing by the virtual object operation device configured as described above will be described.
FIG. 4 is a flowchart for explaining a virtual object operation process executed by the arithmetic processing unit 6 of the virtual object operation device shown in FIG.

Of the following steps, steps S1 to S4 are processes executed as functions of the image cutout unit 61, and steps S5 and S6 are processes executed as functions of the object cutout unit 62. Step S7 is a process executed as a function of the spindle acquisition unit 63, step S8 is a process executed as a function of the rotation angle acquisition unit 64, and step S9 is a process executed by the virtual object drawing unit 65. This is a process executed as a function of.

First, in step S1, the arithmetic processing unit 6 receives data indicating the three-dimensional position of the object OB measured by the three-dimensional position measuring device 2 via the I / F unit 3, and inputs the data to the object OB. Are measured.

Next, in step S2, the arithmetic processing unit 6 estimates the three-dimensional position of a known point on the marker MK from the measured three-dimensional position of the object OB. Specifically, as shown in FIG. 3, when the shape of the marker MK is a square, the coordinates of the vertices C1, C2, C3, and C4 of the marker MK are the three-dimensional position of the measured object OB, that is, the position of the transmitter 1. Assuming that the coordinates are (X, Y, Z), the coordinates can be obtained by the following equation.

C1 = [X-HS, Y + HS, Z-OH] (1) C2 = [X + HS, Y + HS, Z-OH] (2) C3 = [X-HS, Y-HS, Z-OH] (3) C4 = [X + HS, Y-HS, Z-OH] (4) Here, HS is half the length of one piece of the marker MK, and OH is the height of the object OB. is there.

Next, in step S3, the arithmetic processing unit 6 converts the estimated points into points on the camera image using the projection matrix. Specifically, the marker MK obtained by the above equation
Of the vertices C1, C2, C3, and C4 are converted into coordinates of a camera image taken by the camera 5. The conversion from the three-dimensional coordinates to the coordinates of the camera image is performed using the following equations,
Vertices C1, C2, C3, C of marker MK obtained by equation (4)
4 are obtained on the camera image.

[0048]

(Equation 1)

Here, [x, y] are the coordinates of the camera image, [X, Y, Z] are three-dimensional coordinates, s is a scalar, P is a projection matrix, and is obtained by camera calibration. Internal and external variables of the camera being used. This point is described in detail in Xu Go and Saburo Tsuji, "3D Vision", Kyoritsu Shuppan.

Next, in step S4, the arithmetic processing unit 6 cuts out an image including the marker MK based on the converted points. Specifically, a temporary circumscribed rectangle of the marker MK on the camera image is obtained from the vertex of the marker MK on the camera image obtained in step S3, and the image of this circumscribed rectangle is cut out. Here, at this time, the posture of the object OB in the real world is unknown, and there is a possibility that the entire circumscribed rectangle obtained above does not include the entire marker MK.
A region where the circumscribed rectangle is sufficiently enlarged is cut out from the camera image.

At this time, the length L of the side of the circumscribed rectangle is obtained. The length L of the side of the circumscribed rectangle is, for example, the marker M
It can be obtained by calculating the distance between the coordinates of the vertex C1 of K on the camera image and the coordinates of the vertex C2 on the camera image.

Next, in step S5, the arithmetic processing unit 6 executes a marker area specifying process for specifying a marker area corresponding to the marker MK from the cut-out image. FIG. 5 is a flowchart for explaining the marker area specifying process shown in FIG.

As shown in FIG. 5, in the marker area specifying process, first, in step S11, the arithmetic processing unit 6
Converts the image cut out in step S4 into a binary value.

Next, in step S12, the arithmetic processing unit 6 extracts a region that may correspond to the marker MK from the cut-out image, and obtains an area R of a circumscribed rectangle of the extracted region.

Next, in step S13, the arithmetic processing unit 6 obtains the distance between the center of the circumscribed rectangle of the extracted area and the center of the extracted image.

Next, in step S14, the arithmetic processing unit 6 determines whether or not the area of the circumscribed rectangle obtained in step S12 is within a predetermined range, for example, whether the condition of S1 <R <S2 is satisfied. If the condition of <R <S2 is satisfied, the process proceeds to step S15. If the condition of S1 <R <S2 is not satisfied, the process proceeds to step S12 to continue the process on the next region in the image.

Here, S1 = MS · k1 (k1 <1.
0), S2 = MS · k2 (k2> 1.0), MS is the area of the circumscribed rectangle obtained in step S4, and k1 and k2 are predetermined coefficients that satisfy the above inequality.

If the condition of S1 <R <S2 is satisfied, in step S15, the processing unit 6
The distance between the center of the circumscribed rectangle and the center of the image obtained in step 3 is stored.

Next, in step S16, the arithmetic processing unit 6 determines whether or not the above processing has been performed on all the areas in the image, and when the above processing has been completed on all the areas. The process proceeds to step S17. If the above processing has not been completed for all the areas, the subsequent processing is continued for the next area in the image.
Move to 12.

When the above processing is completed for all the areas in the image, in step S17, the arithmetic processing unit 6 rearranges the distances stored in step S15, and sets the area having the minimum distance as a marker area. Then, the process proceeds to step S6 shown in FIG. If an area that satisfies the above condition is not found, it is determined that the recognition has failed, and the posture of the target object OB by the processing described later is not determined.

Next, returning to FIG. 4 again, in step S6, the arithmetic processing unit 6 executes a marker vertex specifying process for specifying a position on the image of a known point of the marker MK from the specified marker region. FIG. 6 is a flowchart for explaining the marker vertex specifying process shown in FIG.

As shown in FIG. 6, in the marker vertex specifying process, first, in step S21, the arithmetic processing unit 6
Extracts a ridgeline on the outer periphery of the specified marker area by applying a Perimeter operator to the cut image.

For example, if the binary image of the area specified as the marker area is the image shown in FIG.
When the elimeter operator is applied, an image as shown in FIG. 8 is obtained. That is, in the case of the image of FIG. 7, the white area MKI is an area corresponding to the marker MK, and the hatched area MAI inside thereof is an area corresponding to the mark MA. When the Perimeter operator is applied to this image, FIG. , Two ridge lines MKR,
The MAR is extracted. Outer ridge lines MKR among the ridge lines obtained in this way are extracted as outer ridge lines of the marker MK.

Next, in step S22, the arithmetic processing unit 6 performs Hough transform on the extracted ridge line. That is, as shown in FIG. 9, the center of the image of the marker area MKI is defined as the origin, the upward direction of the image is defined as the y-axis, and the right direction of the image is defined as the x-axis.
The coordinates of the outer peripheral pixels in the determined coordinate system are obtained, and the obtained coordinates are Hough-transformed.

Here, in the Hough transform, a straight line in the xy space is expressed by the following equation, and the coordinates of points on the straight line are mapped to the parameter space of ρθ.

Xsin θ + ycos θ = ρ (6) Then, the parameter space is divided into small cells, the number of points passing through each cell is recorded, a cell having a large number of points passing through the cell is found, and the cell corresponding to the cell is found. A straight line on the xy space can be specified by the parameter.

Next, in step S23, the arithmetic processing unit 6 obtains a local maximum point representing a linear component from the result of the Hough transform. In the present embodiment, since the marker MK is a square, the number of straight lines is four. Therefore, when the outer peripheral points are mapped to the ρθ space by the Hough transform, four maximum points (white circles in the figure) as shown in FIG. 10 are obtained.

Next, in step S24, the arithmetic processing unit 6 specifies a parallel straight line from the obtained four maximum points. Specifically, the following four linear equations are obtained from the four maximum points.

Xsinθ1 + ycosθ1 = ρ1 (7) xsinθ2 + ycosθ2 = ρ2 (8) xsinθ3 + ycosθ3 = ρ3 (9) xsinθ4 + ycosθ4 = ρ4 (10) Here, two straight lines of the above four straight lines are almost parallel. Therefore, a straight line parallel to the straight line of Expression 7 is specified by finding the one with the smallest difference from θ1 based on the above θ1.

Next, in step S25, the arithmetic processing unit 6 obtains a vertex of the marker MK from the specified parallel straight line. That is, since the combination of the parallel straight lines is known in step S24, the coordinates of the two vertices are obtained from the equation of the two straight lines that are not parallel to the straight line of Expression 7, and the coordinates of the two vertices are similarly calculated from the straight line parallel to the straight line of Expression 7. The coordinates of the other two vertices are obtained, and the process proceeds to step S7 shown in FIG.

Referring again to FIG. 4, in step S7, the arithmetic processing unit 6 calculates the three-dimensional position of the object OB on the camera coordinates based on the center coordinates of the marker area specified in step S5. A main axis vector calculation process for obtaining a main axis vector of the object OB is executed. FIG. 11 shows FIG.
6 is a flowchart for explaining a spindle vector calculation process shown in FIG.

As shown in FIG. 11, in the main axis vector calculation processing, first, in step S31, the arithmetic processing unit 6 obtains the center coordinates (cx, cy) of the marker area specified in step S5.

Next, in step S32, the arithmetic processing unit 6 obtains the center coordinates (rx, ry) of the image cut out in step S4. Specifically, the center coordinates (rx, ry) of the cut-out image are obtained by averaging the coordinates of the outer peripheral pixels according to the following equation. Here, n is the number of peripheral pixels, and x and y are the coordinates of the peripheral pixels.

[0074]

(Equation 2)

Next, in step S33, the arithmetic processing unit 6 obtains an actual size (scale parameter) corresponding to one pixel from the actual size of the marker MK and the size of the marker MK in the camera image. Specifically, since the length L of the side of the circumscribed rectangle has already been determined in step S4 and the size ML of the marker MK is known, the value of the three-dimensional coordinates (scale The parameter s is obtained by the following equation.

S = ML / L (12) Next, in step S34, the arithmetic processing unit 6 obtains the principal axis vector of the object OB using the obtained scale parameter s. Specifically, the X and Y coordinates of the principal axis vector of the object OB are calculated by the center coordinates (r
x, ry) and the center coordinates (cx, cy) of the marker area are obtained by the following equation.

X = s · (rx−cx) (13) Y = s · (ry−cy) (14) Here, as shown in FIG. 12, the length of the main axis vector MV of the object OB is Since the height is equal to the height OH of the object OB and is known, the Z coordinate of the main axis vector MV is obtained by the following equation.

Z = (OH ² + X ² + Y ² ) ^1/2 (15) As described above, the scale parameter s, the center coordinates (rx, ry) of the cut-out image, and the center coordinates (cx, ry) of the marker area cy) and the height OH of the object OB, the main axis vector of the object OB on three-dimensional coordinates is obtained, and the process proceeds to step S8 shown in FIG.

Referring again to FIG. 4, in step S8, arithmetic processing unit 6 determines the rotation angle of marker MK in a plane parallel to the lens surface of camera 5 from each vertex of marker MK specified in step S6. Execute the rotation angle calculation process. FIG. 13 is a flowchart for explaining the marker rotation angle calculation processing shown in FIG.

As shown in FIG. 13, first, in step S4
In 1, the arithmetic processing device 6 obtains a vertex close to the upper left vertex of the cut-out image from the four vertices of the marker MK obtained in step S6, and clockwise along the outer periphery of the marker area based on the vertex. Reorder vertices. Here, as shown in FIG. 14, each sorted vertex is c1,
Defined as c2, c3, and c4.

Next, in step S42, the arithmetic processing unit 6 obtains a rotation vector of the marker MK. That is, as shown in FIG. 15, a coordinate system is set at the center of the clipped image, and a rotation vector RV (vx, vy) indicating the inclination of the marker MK is obtained.

Here, the center coordinates of the marker area are represented by (c
x, cy), the coordinates of vertex c2 are (x2, y2), vertex c
3 is (x3, y3), the rotation vector RV
(Vx, vy) is obtained by the following equation.

RV (vx, vy) = ((x2 + x3) / 2−cx, (y2 + y3) / 2−cy) (16) Next, in step S43, the arithmetic processing unit 6 sets the rotation angle of the marker MK. θ is obtained by the following equation.

Θ = sin ⁻¹ (vy / (vy ² + vx ² ) ^1/2 ) (17) Next, in step S44, the arithmetic processing unit 6 sets the coordinates of each vertex on the camera image and the marker MK. A projection matrix is obtained from vertex coordinates in the three-dimensional local coordinate system.

Here, since the actual rotation angle of the marker MK cannot be determined only from the rotation angle θ, the marker MK is used.
Consider the direction of the mark MA provided in. That is,
In order to determine the direction of the mark MA, as shown in FIG.
As the marker coordinate system, a three-dimensional local coordinate system is set at the center of the marker MK. At this time, the three-dimensional local coordinates of each vertex C1, C2, C3, C4 of the marker MK are:
It is obtained by the following equation.

C1 = [− HS, HS, 0] ^T (18) C2 = [HS, HS, 0] ^T (19) C3 = [− HS, −HS, 0] ^T (20) C4 = [HS, −HS, 0] ^T (21) A projection matrix that maps the above-described three-dimensional local coordinates to the coordinates of each vertex on the camera image already obtained in step S6,

[0087]

(Equation 3) Then, the following relationship holds, and the simultaneous equations are solved to obtain a projection matrix.

[0088]

(Equation 4)

Next, in step S45, the arithmetic processing unit 6 maps the four patterns shown in FIG. 17 onto the camera image using the obtained projection matrix. Here, FIG.
17A is pattern 1, the pattern shown in FIG. 17B is pattern 2, the pattern shown in FIG. 17C is pattern 3, and the pattern shown in FIG. The pattern shown is pattern 4.

Next, in step S46, the arithmetic processing unit 6 performs matching between each mapped pattern and the marker area, and specifies a matching pattern from each pattern shown in FIG.

Next, in step S47, the arithmetic processing unit 6 obtains the rotation angle of the marker MK based on the matched pattern. That is, the angle corresponding to the matched pattern is added to the above θ to determine the rotation angle of the marker MK. Here, assuming that the pattern 1 shown in FIG. 17 is 0 degrees and the number (1 to 4) of the matched pattern is PN, the rotation angle θm of the marker MK is obtained by the following equation.

Θm = θ + (PN−1) · 90 (24) After the rotation angle θm of the marker MK is obtained by the above processing, the flow shifts to step S9 shown in FIG.

Referring to FIG. 4 again, in step S9, the arithmetic processing unit 6 uses the principal axis vector and the rotation angle representing the three-dimensional position and orientation of the object OB obtained by the above-described processing to obtain a three-dimensional computer. Create data for drawing the virtual object in the virtual space by the graphic according to the three-dimensional position and orientation of the object OB, output the data to the display unit 3, and end the process.

In the present embodiment, the three-dimensional position of the object OB is measured and the object O
The marker MK of B is photographed, and the posture of the object OB is calculated from the three-dimensional position of the object OB and the photographed image of the marker MK. The display state of the object is changed.

As described above, since not only the image of the marker MK of the object OB but also the three-dimensional position of the object OB is measured, the world coordinate system can be defined, and the virtual object can be viewed from another viewpoint. A camera coordinate system corresponding to the viewpoint can be obtained even when viewing from the viewpoint. Therefore, it is possible to display a virtual object in the virtual world by three-dimensional computer graphics from an arbitrary viewpoint in a state according to the operation of the object OB in the real world.

In this embodiment, the square marker MK is used. However, the shape of the marker is not particularly limited to this example, and various changes can be made.
The mark MA is provided eccentrically from the center position of, but if a rough rotation angle can be detected from the shape of the marker itself,
The mark MA may be omitted. When the fluorescent paint is applied directly to the object OB and the object OB is directly photographed by the camera 5, the marker MK may be omitted.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a virtual object operation device according to an embodiment of the present invention.

FIG. 2 is a schematic perspective view showing a positional relationship between an object in the real world, a three-dimensional position measuring device, a camera, and the like.

FIG. 3 is a schematic perspective view showing a positional relationship between an object in the real world, a marker, and the like.

FIG. 4 is a flowchart for explaining a virtual object operation process executed by an arithmetic processing unit of the virtual object operation device shown in FIG. 1;

FIG. 5 is a flowchart for explaining a marker area specifying process shown in FIG. 4;

FIG. 6 is a flowchart for explaining a marker vertex specifying process shown in FIG. 4;

FIG. 7 is a diagram illustrating an example of a binary image of an area specified as a marker area.

FIG. 8 is a diagram showing an image after a Perimeter operator is applied to the binary image shown in FIG. 7;

FIG. 9 is a diagram showing a coordinate system set for an image of a cut-out marker area.

10 is a diagram showing local maximum points after the image shown in FIG. 9 is subjected to Hough transform.

FIG. 11 is a flowchart for explaining a spindle vector calculation process shown in FIG. 4;

FIG. 12 is a diagram showing principal axis vectors of an object.

FIG. 13 is a flowchart illustrating a marker rotation angle calculation process shown in FIG. 4;

FIG. 14 is a diagram showing a state in which vertices of specified markers are rearranged.

FIG. 15 is a diagram showing a coordinate system set at the center of a marker area.

FIG. 16 is a diagram showing a three-dimensional local coordinate system set at the center of a marker.

FIG. 17 is a diagram showing four patterns corresponding to the rotation state of the marker.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 transmitter 2 three-dimensional position measuring device 3 I / F unit 4 black light 5 camera 6 arithmetic processing unit 7 input unit 8 ROM 9 RAM 10 storage device 11 display unit 61 image cutout unit 62 target cutout unit 63 spindle acquisition unit 64 Rotation angle acquisition unit 65 Virtual object drawing unit

Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat II (reference) G06T 7/60 150 G06T 7/60 150P 5L096 F term (reference) 5B050 BA08 BA09 BA13 CA07 DA02 EA04 EA07 EA12 FA02 5B057 BA05 DA07 DB03 DC08 5B068 AA05 BB22 BC03 BD11 BD19 BD25 BE03 BE06 BE15 CC11 CC12 EE06 5B087 AA07 AA09 BC12 BC13 BC26 BC32 DJ03 5E501 AA01 AC16 BA05 CA02 CB09 CB14 FA02 FA27 FB04 5L096 AA09 BA08 CA03 CA18 FA38 FA38 FA24 FA24

Claims (6)

[Claims] 1. A virtual object operation apparatus for operating a virtual object associated with the object in a virtual world by three-dimensional computer graphics by operating the object in the real world, Position measuring means for measuring the three-dimensional position of the object; photographing means for photographing the object; three-dimensional position of the object measured by the position measuring means and the object photographed by the photographing means Attitude calculating means for calculating the attitude of the object from the image of the three-dimensional position of the object measured by the position measuring means and the attitude of the object calculated by the attitude calculating means, Display means for changing a display state of the virtual object in a virtual space. 2. The virtual object operation device according to claim 1, wherein the position measurement unit includes a wireless position measurement unit that wirelessly measures a three-dimensional position of the object. 3. The wireless position measuring means is attached to the object, and wirelessly transmits identification information for identifying the object, and receives identification information transmitted from the transmitting means. 3. The virtual object operation device according to claim 2, comprising a plurality of receiving means. 4. A marker having a predetermined shape made of a fluorescent material that emits light by ultraviolet light is attached to the object, the photographing means is: an irradiating means for irradiating the marker with ultraviolet light; and the ultraviolet light is irradiated by the irradiating means. The virtual object operation device according to any one of claims 1 to 3, further comprising a marker photographing means for photographing the selected marker. 5. The marker has a mark deviated from its center position, and the posture calculating means is a three-dimensional position of the object measured by the position measuring means, and is photographed by the marker photographing means. The virtual object operation device according to claim 4, wherein the posture of the object is calculated from the image of the marker and the position of the mark on the marker. 6. A virtual object operation method for operating a virtual object associated with the object in a virtual world by three-dimensional computer graphics by operating the object in the real world, Measuring the three-dimensional position of the object; photographing the object; calculating the posture of the object from the measured three-dimensional position of the object and the photographed image of the object And changing the display state of the virtual object in the virtual world according to the measured three-dimensional position of the object and the calculated attitude of the object. Object operation method.