JP2007293453A - Method for acquiring color information from image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily acquire a color corresponding to a designated position in a real space. <P>SOLUTION: When a position 505 in a real space is designated by a pen 511 to be operated by a user, a projecting position 506 of the position 505 to an image plane 503 where the real space is photographed is found. Then, the position of the pixel of the position shifted from the pixel of the projection position 506 to a direction shown by the pen 511 by the predetermined number of pixels is acquired as the color of the designated position. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for acquiring the color of an object in the real world from an image.

  The VR (virtual reality) system presents a virtual space to the user as if it were a real space by presenting the computer generated 3D CG (computer graphics) to an HMD (head mounted display) worn by the user. It is a system that makes you feel as if. In the VR system using the HMD, the 3D CG presented to the user changes in accordance with the movement of the user's head, and therefore the user has a space that is actually represented by the 3D CG around him / her. You can feel as if you are.

  In recent years, a technology has also been developed that presents the real world, which could not be viewed in VR, by superimposing and presenting a 3D CG on a real space or its video, and this is MR (Mixed Reality). , Mixed reality) system (see Patent Document 1).

  In the MR system, an HMD is generally used as a display device that presents video to a user. The MR system aligns a real space and a virtual space, usually represented by CG, and causes the user to experience mixed reality by following the user's viewpoint position and orientation and displaying them on the HMD.

  The HMD used in the MR system includes an optical see-through method in which an image such as a three-dimensional CG is superimposed on a transflective display device, and an image of a three-dimensional CG or the like synthesized with a real-space image captured by a video camera. There is a video see-through method. In the video see-through method, the HMD is often provided with a video camera that captures an image of a real space.

  The MR system can be used for medical assistance in which the patient's internal state is drawn with CG and superimposed on the patient and its actual image and presented to the doctor. A work assisting application that displays images in an overlapping manner is considered.

  Now, in such an MR system, the user may be asked to select a color. For example, in a product design system using an MR system, the color of a design is specified. Conventionally, in such a case, a method of selecting a color from a GUI for color selection displayed on a display device different from the HMD has been adopted.

  Further, as seen in Non-Patent Document 1, a plurality of virtual spheres representing virtual colors are placed on a plate that is held by a user and whose position and orientation are measured, and the user holds the other hand in the other hand. A method of selecting a color by selecting a sphere with a pen whose position and orientation are measured has also been performed.

  In Non-Patent Document 2, a real object whose position and orientation are measured is held in a hand, and a plurality of points are set on the real object with a pen whose position and orientation are measured. Thereafter, the camera built in the HMD worn by the user acquires an image inside the triangle group defined from the set points, and acquires the appearance of the actual object.

Japanese Patent No. 3363861 Japanese Patent No. 3631151 Fiorentino, M. et al., “Spacedesign: A Mixed Reality Workspace for Aesthetic Industrial Design”, Proc. IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR) 2002, pp. 86-94. Joohi Lee et al., “Modeling Real Objects Using Video See-Through Augmented Reality”, Proc. Second International Symposium on Mixed Reality (ISMR) 2001, pp. 19-26.

  The method for selecting a color from a GUI displayed on a display device different from the HMD has the following problems. In other words, when the user selects a color, the user has to take troublesome procedures such as removing the installed HMD and operating the GUI using the mouse to select the color. Selection takes time.

  This method can be solved by using the method disclosed in Non-Patent Document 1, but another problem arises that the selected color is difficult to adapt to the real world. That is, in the method of Non-Patent Document 1, the color of the sphere presented on the board held by the user is a color predefined on the computer. Since these colors are defined independently regardless of the real world presented in the HMD, applying the selected color to the 3D CG may not be compatible with the real world around the 3D CG. . Note that this problem also occurs in the method using the GUI.

  Since the method shown in Non-Patent Document 2 is a method of acquiring an image inside a triangle defined from a set point, a color cannot be selected. However, if the color is acquired and used using this method, for example, from the image captured by the camera attached to the HMD worn by the user at the moment when a point on the real object is set with a pen, for example. A configuration for obtaining the color of the set point is assumed. However, in this configuration, the color is acquired using an image in which the set point is always hidden by the pen tip, and the color of the pen tip itself is acquired instead of the set point.

  As another problem, the methods disclosed in Non-Patent Document 1 and Non-Patent Document 2 require measurement of the position and orientation of an object held by a user who does not have a pen, and include the position and orientation of the pen. It was necessary to measure the position and orientation of multiple objects. In addition to this, in the MR system, since it is necessary to detect the position and orientation of the camera that captures the real space or the user viewpoint, the load on the position and orientation is heavy.

  The present invention has been made in view of such a problem, and an object of the present invention is to enable easy acquisition of a color corresponding to a position designated in a real space.

  In order to achieve the above-described object, the color information acquisition method of the present invention detects that a position in the real space is specified by a video acquisition step of acquiring a video of the real space and an instruction means operated by the user. An instruction detection step, a calculation step for obtaining a position in the video corresponding to the position in the specified real space in response to the detection, and a position obtained by the calculation step in the vicinity of the position obtained by the calculation step And a color acquisition step of acquiring, as a color designated by the user, a color of a pixel different from the pixel in the video corresponding to.

  In order to achieve the above object, the color information acquisition apparatus of the present invention specifies a position in the real space included in the video by the video acquisition means for acquiring the video in the real space and the instruction means operated by the user. In response to the detection, a calculation means for obtaining a position in the video corresponding to the position in the specified real space, and in the vicinity of the position obtained by the calculation means, and It has a color acquisition means which acquires the color of the pixel different from the pixel in the image | video corresponding to the position which the calculation means calculated | required as a color designated by the user.

  With such a configuration, according to the present invention, it is possible to easily acquire a color corresponding to a position designated in the real space.

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.
In the following embodiments, the color information acquisition method according to the present invention is a method in which a user wearing an HMD operates a pen as an instruction unit to select a color from an object in the real space, and enters the mixed reality space with the selected color. An example will be described in which the present invention is applied to an MR system that allows a user to experience drawing operations.

  FIG. 5 is a diagram for explaining the concept of the MR system of the present embodiment. Reference numeral 510 denotes a real object teapot, 511 denotes a pen operated by the user, 503 denotes an image plane presented to the user, and 512 denotes the position of the camera center of the imaging unit provided in the HMD, that is, the viewpoint position of the user. ing. Other configurations will be described later.

  Note that here, for ease of explanation and understanding, it is assumed that the center position of the camera matches the viewpoint position of the user, but in reality, the center position of the camera attached to the HMD and the user Since it is different from the viewpoint position, the difference is corrected, and it is handled as being taken from the viewpoint position of the user. Since this correction process is not related to the essence of the present invention and can be realized by using a well-known technique, further explanation is omitted.

FIG. 3 is a block diagram illustrating a functional configuration example of the MR system according to the present embodiment.
Reference numeral 100 denotes a video see-through type HMD worn by a user, and includes an imaging unit 101 and a video display unit 102. In the present embodiment, the imaging unit 101 and the video display unit 102 are built in the HMD 100. However, the imaging unit 101 and the video display unit 102 may be arranged in different places.

The imaging unit 101 is an imaging device represented by a video camera using an imaging element such as a CCD sensor or a CMOS sensor. The captured video is sent to the video composition unit 108.
The video display unit 102 is composed of a small LCD or the like, and displays the synthesized video sent from the video synthesis unit 108.

  The position / orientation sensor 103 measures the position and orientation of an arbitrary object, for example, by photographing a marker included in the real space using an internal image sensor. The marker is, for example, a sticker printed with a pattern having a predetermined size, shape, direction, etc., and detects the position and orientation from the shape and size detected in the captured image. It is possible.

  In the present embodiment, the position / orientation sensor 103 is configured by an optical sensor (imaging device). However, the position / orientation sensor 103 may be configured by any device as long as it can measure the position and orientation of a predetermined object. It may be constituted by. From the position / orientation sensor 103, a position / orientation signal of an arbitrary object with a marker attached in advance is output. In this embodiment, the marker is attached to the HMD 100 and the pen 511, and the position / orientation signal is equally output to the position / orientation measurement unit 104 of the imaging unit and the position / orientation measurement unit 105 of the pen.

  In this embodiment, the output of one position / orientation sensor 103 is used by the position / orientation measurement unit 104 of the imaging unit and the position / orientation measurement unit 105 of the pen. However, another position and orientation sensor may exist for each of the position and orientation measurement unit 104 of the imaging unit and the position and orientation measurement unit 105 of the pen. That is, the configuration is arbitrary as long as the positions and orientations of these two real objects are measured.

  In the present embodiment, the position / orientation sensor 103 outputs the position in the form of x, y, and z coordinates in a world coordinate system that is a predetermined xyz coordinate system. Also, the posture is output as a roll angle that is rotation about the z-axis, a pitch angle that is rotation about the x-axis, and a yaw angle that is rotation about the y-axis, and is rotated in the order of yaw rotation, pitch rotation, and roll rotation. In addition, the posture of the measurement object can be obtained.

  In the following description, the position and orientation information is expressed as alphabetic symbols that mean a matrix with 4 rows and 4 columns, for example, M. The contents obtained by the position and orientation sensor 103 will be described as an example with respect to the contents of this matrix.

When the x, y, and z coordinates obtained by the position and orientation sensor are x, y, and z, respectively, the three-dimensional vector t that represents the position obtained by the position and orientation sensor is
It is represented by

Also, assuming that the roll angle, pitch angle, and yaw angle values obtained by the position and orientation sensor are r, p, and y, respectively, three rows and three columns of rotation transformation matrices Rr, Rp obtained from the roll angle, pitch angle, and yaw angle. , Ry
It is represented by

Therefore, the three-row, three-column rotation transformation matrix R representing the posture is
It is represented by

A four-by-four matrix M expressing both the position and orientation obtained from the position and orientation sensor 103 using the t and R obtained above can be expressed as follows.
It is well known that the position and orientation can be expressed as such a 4-by-4 matrix, regardless of the format.

  The position / orientation measuring unit 104 of the image capturing unit receives the position / orientation signal from the position / orientation sensor 103 and generates position / orientation information of the image capturing unit 101. The generated position / orientation information is sent to the CG rendering unit 107 and the color acquisition unit 123.

  In the present embodiment, the position / orientation information output from the position / orientation measurement unit 104 of the imaging unit is obtained only from the output of the position / orientation sensor 103. However, the configuration may be such that the position and orientation information is corrected using the video captured by the imaging unit 101. Such a method is described in Patent Document 2, for example. Further, the position / orientation information may be obtained by the position / orientation measuring unit 104 of the image capturing unit using only the image captured by the image capturing unit 101.

  The pen position / orientation measurement unit 105 receives a position / orientation signal from the position / orientation sensor 103 and generates position / orientation information of the pen 511. The generated position and orientation information is sent to the pen drawing unit 122 and the color acquisition unit 123.

  The CG scene management unit 106 manages information on a three-dimensional CG model of a virtual object necessary for generating a three-dimensional CG. Information of the CG scene management unit 106 is updated by the pen drawing unit 122. The information of the CG scene management unit 106 is referred to by the CG drawing unit 107.

  The CG rendering unit 107 receives the position / orientation information of the imaging unit 101 from the position / orientation measurement unit 104 of the imaging unit, receives the 3D CG scene information from the CG scene management unit 106, and enters the imaging range of the imaging unit 101. A CG image that fits is generated. The generated CG video is transmitted to the video synthesis unit 108.

  The video composition unit 108 has an image memory, draws a captured video received from the imaging unit 101, and generates a composite video by overwriting the CG video received from the CG rendering unit 107 thereon. The synthesized composite video is sent to the video display unit 102.

  The drawing button 120 is provided on the pen 511, and is arranged at a position where the user can easily press the pen 511, for example, a position where the user can press it with the index finger. The drawing button 120 is normally OFF, and is configured to be ON only when the user presses it. A signal (ON / OFF signal) indicating the ON / OFF state of the drawing button 120 is always transmitted to the pen drawing unit 122.

The color acquisition button 121 is disposed at the tip of a pen 511 that the user holds. The color acquisition button 121 is normally OFF, and is configured to be ON only when the user presses the tip of the pen 511 against a real object. When the color acquisition button is turned on, the MR system of this embodiment detects that a position in the real space has been designated. The ON / OFF signal of the color acquisition button 121 is always transmitted to the color acquisition unit 123.
In the present embodiment, the state of various buttons included in the pen 511 is wirelessly transmitted.

  The pen drawing unit 122 acquires the ON / OFF signal from the drawing button 120 and the position / orientation information of the pen 511 from the pen position / orientation measurement unit 105, and performs three-dimensional CG drawing in the space. The pen drawing unit 122 has a current drawing color buffer of the pen 511 and always holds the drawing color of the pen 511. In the present embodiment, the drawing color of the pen 511 is initially set to white. The pen drawing unit 122 has a position / orientation buffer for recording the previous pen position / orientation information.

  The color acquisition unit 123 receives an ON / OFF signal from the color acquisition button 121, a composite video from the video synthesis unit 108, a position / orientation information of the imaging unit from the position / orientation measurement unit 104 of the imaging unit, and a position of the pen from the position / orientation measurement unit 105 of the pen. The position and orientation information is acquired, and the drawing color of the pen 511 is obtained. The color acquisition unit 123 has the focal length information and principal point position information of the imaging unit 101 in advance.

  Note that at least a part of the configuration of the present embodiment shown in FIG. 3 may be realized as an independent device, or as software that realizes a function by being executed by a CPU of one or a plurality of computers. May be. In the present embodiment, each of the position / orientation measurement unit 104 of the imaging unit, the pen position / orientation measurement unit 105, the CG scene management unit 106, the CG drawing unit 107, the video composition unit 108, the pen drawing unit 122, and the color acquisition unit 123 includes It is assumed that each is realized by software and installed in the same computer.

FIG. 1 is a diagram illustrating a basic configuration of a computer that can function as each unit realized by software.
The CPU 1001 controls the entire computer using programs and data stored in the RAM 1002 and the ROM 1003, and controls the execution of each software to realize the functions of the respective units.

  The RAM 1002 includes an area for temporarily storing programs and data loaded from the external storage device 1007 and the storage medium drive 1008, and a work area required for the CPU 1001 to perform various processes.

  The ROM 1003 generally stores computer programs and setting data. A keyboard 1004 and a mouse 1005 are input devices, and an operator can input various instructions to the CPU 1001 using these devices.

  The display unit 1006 is configured by a CRT, a liquid crystal display, or the like, and corresponds to the video display unit 102 of the HMD 100 in the present embodiment. Alternatively, a separate display device may be provided. In this case, for example, the video displayed on the video display unit 102 can be displayed.

  The external storage device 1007 is a device that functions as a large-capacity information storage device such as a hard disk drive, and stores an OS (operating system), a program executed by the CPU 1001, and the like. In the description of the present embodiment, information that is described as being known is stored here, and loaded into the RAM 1002 as necessary.

  The storage medium drive 1008 reads a program or data stored in a removable storage medium such as a CD-ROM or DVD-ROM in accordance with an instruction from the CPU 1001 and outputs it to the RAM 1002 or the external storage device 1007.

  The I / F 1009 acquires an analog video port for connecting the imaging unit 101 or a digital input / output port such as IEEE1394, a serial port such as RS232C or USB for connecting the position / orientation sensor 103, and a button state of the pen 511. For example, a wireless communication port. The data inputted by each is taken into the RAM 1002 via the I / F 1009.

  The above-described components are connected to each other by a bus 1010.

(Color acquisition processing)
FIG. 2 is a flowchart for explaining the color acquisition operation in the MR system of this embodiment. A program related to the color acquisition operation is stored in, for example, the external storage device 1007. When the user instructs execution of the system using the keyboard 1004 and the mouse 1005, the CPU 1001 reads and executes the program.

  S101 is a real space imaging step. In this step, the imaging unit 101 captures a real space image. The captured video is sent to the video composition unit 108. When S101 ends, S102 and S103 are executed in parallel.

  S102 is a position and orientation measurement step of the imaging unit. In this step, the position / orientation measuring unit 104 of the image capturing unit receives the position / orientation signal from the position / orientation sensor 103 and calculates position / orientation information of the image capturing unit 101. In this embodiment, since the marker of the position / orientation sensor 103 is attached to the HMD 100, the position / orientation sensor 103 outputs a position / orientation signal of the marker attached to the HMD 100. However, since the imaging unit 101 is fixedly attached to the inside of the HMD 100, the positional relationship between the imaging unit 101 and the marker can be measured, and the positional relationship can be obtained by calibration in advance.

  In order to perform this calibration, the coordinate system (imaging unit coordinate system) of the imaging unit 101 needs to be determined in advance. In the present embodiment, the imaging unit coordinate system is determined as indicated by 501 in FIG. That is, the camera center of the imaging unit 101 is the origin, the perpendicular line drawn from the origin to the image plane 503 is the negative direction of the Z axis, the upper direction of the image plane 503 is the positive direction of the Y axis, and the horizontal right direction of the image plane 503 is the X axis. The positive direction is determined.

  In S102, the position and orientation of the imaging unit 101 are obtained from a position and orientation signal obtained by photographing a marker attached to the HMD 100 using a relative positional relationship between the imaging unit 101 and a marker (not shown) calibrated in advance. Calculate information. This position and orientation information, that is, the position and orientation of the imaging unit coordinate system in the world coordinate system is transmitted to the CG drawing unit 107 and the color acquisition unit 123.

  S103 is a pen position / orientation measurement step. In this step, the pen position / orientation measurement unit 105 receives the position / orientation signal from the position / orientation sensor 103 and calculates the position / orientation information of the tip portion of the pen 511. In the present embodiment, a marker (not shown) of the position / orientation sensor 103 is attached to the pen 511. Since the positional relationship between the tip of the pen 511 and the marker is fixed, the positional relationship between the two can be obtained by calibration in advance.

  In order to perform this calibration, a pen coordinate system needs to be determined in advance. In the present embodiment, the pen coordinate system is determined as indicated by 502 in FIG. That is, the pen tip is set as the origin, the pen axis direction is defined as the positive direction of the Z axis, and the pen axis drawing button 120 side is defined as the positive direction of the Y axis.

  In S103, the position and orientation information of the tip of the pen 511 is calculated from the relative positional relationship between the tip of the pen 511 and the marker and the signal indicating the position and orientation of the marker attached to the pen 511. . This position and orientation information, that is, the position and orientation of the pen coordinate system in the world coordinate system is transmitted to the pen drawing unit 122 and the color acquisition unit 123. When S102 and S103 are completed, the process proceeds to S110.

  S110 is a drawing button determination step. Here, the pen drawing unit 122 determines ON / OFF of the drawing button 120. If the drawing button 120 is pressed and in the ON state, the process proceeds to S121, and if it is in the OFF state, the process proceeds to S111.

S121 is a pen drawing step. FIG. 8 is a diagram for explaining the processing of this step.
In the pen drawing step, the previous pen position and orientation 802 held in advance by the pen drawing unit 122 is set as the start point, and the current pen position and orientation 803 sent from the pen position and orientation measurement unit 105 is set as the end point. The CG of the line is drawn in the virtual space. In other words, three-dimensional data for drawing the CG of the three-dimensional line by the CG drawing unit 107 is generated. At this time, the line color is set to the color recorded in the current drawing color buffer of the pen. The generated three-dimensional data representing the three-dimensional line 801 is sent to the CG scene management unit 106 and recorded. When S121 ends, the process proceeds to S111.

  S <b> 111 is a CG drawing step, in which the CG drawing unit 107 uses the position and orientation information of the imaging unit 101 to generate a CG image of the virtual space viewed from the viewpoint of the imaging unit 101. At this time, information on the 3D CG model (including information on the line 801 generated by the pen drawing unit 122) stored in the CG scene management unit 106 is referred to. The generated CG video is sent to the video composition unit 108. When S111 ends, S112 is executed.

  S 112 is a video composition step. First, a captured video transmitted from the imaging unit 101 is written in the image buffer of the video composition unit 108. Next, the CG video transmitted from the CG rendering unit 107 is overwritten in the image buffer. As a result, a composite video is generated in which the CG video in the virtual space is drawn with the captured video as the background. The generated composite video is sent to the video display unit 102 and the color acquisition unit 123. When S112 ends, S113 is executed.

S113 is a video display step, in which the video display unit 102 displays a composite video. When S113 ends, the process proceeds to S120.
S120 is a color acquisition button determination step. In this step, the color acquisition unit 123 determines the ON / OFF state of the color acquisition button 121. If the color acquisition button 121 is pressed and in the ON state, the process proceeds to S122, and if it is in the OFF state, the process proceeds to S130.

  S122 is a color acquisition step. This step is executed by the color acquisition unit 123. Here, details of the processing performed in the color acquisition step will be described with reference to the flowchart of FIG.

When S122 is started, S401 is executed.
In S401, the position and orientation of the tip portion (pen tip) of the pen 511 in the imaging unit coordinate system are calculated. If the position and orientation information of the imaging unit 101 is expressed by C and the position and orientation information of the pen 511 is expressed by P, the position and orientation information M of the pen tip in the imaging unit coordinate system is
Is required. When S401 ends, the process proceeds to S402.

  In S402, the projection position of the pen tip on the image plane 503 is calculated. Here, the projection position is first obtained as the position of the pixel corresponding to the position of the pen tip in the image plane 503 in the coordinate system 501 of the imaging unit.

The color acquisition unit 123 has focal length information of the optical system of the imaging unit 101 in advance. That is, the length f of the line 504 that connects the origin of the coordinate system 501 of the imaging unit and the center of the image plane 503 is known in advance. In FIG. 5, 505 represents the position of the pen tip, and 506 represents the projection position of the pen tip on the image plane 503. Here, it is assumed that the position of the pen tip in the coordinate system 501 of the imaging unit is (P _x , P _y , P _z ). In addition, the length of the component 508 in the X-axis direction of the coordinate system 501 of the imaging unit of the vector extended from the center of the image plane 503 to the projection position 506 of the pen tip on the image plane 503 is w and the component in the Y-axis direction. If the length of 507 is h,
It becomes. When S402 ends, the process proceeds to S403.

In S403, the projection position 506 of the pen tip on the image plane 503 in the coordinate system 501 of the imaging unit obtained so far is converted into a coordinate value in the coordinate system of the image plane 503. Here, the coordinate system of the image plane 503 is an xy coordinate system in which the upper left of the image plane 503 is the origin, the right direction is the x axis, and the lower direction is the y axis, as indicated by 509 in FIG. Now, the projection position of the pen tip in the coordinate system of the image plane 503 is (a, b), the principal point position of the imaging unit 101 in the coordinate system of the image plane 503, that is, the origin coordinate of the coordinate system 501 of the imaging unit ( C _x , C _y )
It becomes. Here, the sign before w and h is different, as shown in FIG. 5, in the XYZ coordinate system 501 of the imaging unit and the xy coordinate system 509 of the image plane, the positive direction of x is the same. This is because the positive direction of y is opposite.

Here, when distortion aberration caused by the lens of the camera of the imaging unit 101, for example, radial distortion, is known in advance, correction may be performed in consideration of the influence. This correction process is a two-dimensional conversion process in the image plane, and can be performed independently of the projective conversion.
In the above calculation, all the units of f and (C _x , C _y ) are assumed to be represented by pixels on the image plane.

  When S403 ends, the process proceeds to S404. With the processing so far, the projection position of the pen tip on the image plane has been obtained. However, as described above, if the color of the pixel at the projection position of the pen tip is acquired, the color of the pen tip itself may be acquired. For this reason, in the present embodiment, by adding the following processing, the pixels in the vicinity of the pixel at the projection position of the pen tip on the image plane 503 and the pixel corresponding to the pen imaged in the image in the real space are The color of the different pixel is acquired as the color at the position specified by the user. Specifically, the color of the pixel at a position shifted by a predetermined amount in the direction indicated by the pen as the instruction means, that is, the direction in which the pen tip is directed, is acquired from the pixel at the projection position of the pen tip.

In S404, the tilt direction of the pen tip on the image plane is calculated. FIG. 6 is a diagram illustrating the image plane 503. In this figure, reference numeral 601 represents a tilt direction of the pen 511 on the image plane, that is, a direction indicated by the pen tip when the pen 511 is projected onto the image plane 503, and a vector T (T _x , _Ty ).

  In order to obtain the vector T, first, a pen extension point existing on a line that is virtually extended from the pen tip to the direction of the pen shown in 513 in FIG. 5 is assumed. Next, the projection position (position in the image plane coordinate system) of the pen extension point 513 on the image plane 503 shown in 602 in FIG. 6 is obtained. Then, the vector T is calculated | required by taking the difference of the projection position of the pen tip calculated | required by S403, and the projection position of a pen extension point.

First, the position of the pen extension point 513 is obtained. When the position of the pen extension point 513 in the coordinate system 501 of the imaging unit is represented by S and the distance from the pen tip to the pen extension point 513 is 1,
S = ML
It becomes. Here, L is a matrix indicating a -1 movement in the Z-axis direction, that is,
It is.

M is the position and orientation information of the pen tip in the imaging unit coordinate system obtained in S122. M includes a three-row, three-column rotation matrix R, from which a roll angle, pitch angle, and yaw angle representing rotation can be obtained. When the roll angle, pitch angle, and yaw angle are represented by r, p, and y, S is obtained as follows.

Here, in the pen coordinate system 502, the roll angle r means rotation around the axis of the pen 511 (rotation around the Z axis). Therefore, when obtaining the coordinates of the pen extension point 513, r = 0 may be set. Then, the above formula is
It becomes.

That is, the three-dimensional position of the pen extension point 513 in the coordinate system 501 of the imaging unit is
It is. This three-dimensional position is represented as S = (S _x , S _y , S _z ).

Next, when the projection position (c, d) of the pen extension point 513 on the image plane 503 is obtained, as in the calculation in S403,
It becomes.

Then the vector T is
It becomes. When S404 ends, the process proceeds to S405.

In step S405, an offset from the coordinates of the pen tip on the image plane is added. That is, the coordinates are moved by a predetermined length from the pen tip projection position (a, b) obtained in S403 in the tilt direction obtained in S404. In this embodiment, it is moved by the length of 5 pixels. Then the required coordinates are
It becomes. When S405 ends, the process proceeds to S406.

  In S406, a color is acquired from the image plane 503. That is, the color of the pixel corresponding to the coordinates obtained in S405 is acquired from the synthesized video sent from the video synthesizing unit 108. The acquired color is stored in the drawing color buffer of the pen drawing unit 122. When S406 ends, the entire process of S122 ends, and the process proceeds to S130.

  S130 is an end determination step. If the user has instructed to end the system using the keyboard 1004 and the mouse 1005, the program ends. If not, the process proceeds to S101.

  As described above, according to the present embodiment, if the user holds the pen 511, presses the pen tip against a real object, and turns on the color acquisition button 121 provided on the pen, the position near the pen tip is determined. The color can be acquired. With the above processing, the user can easily acquire the color in the displayed video without erroneously acquiring the pen tip color. Furthermore, when the user moves the pen 511 to a different position and presses the pen drawing button 120, a CG of a three-dimensional line having the acquired color can be drawn starting from the most recent pen tip position.

  In this embodiment, the HMD 100 including an imaging unit and a display unit is used as a device that displays a composite image and a device that captures an image of a real space. However, other configurations such as using a stationary display and a movable camera may be used. Even in such other configurations, the same effect can be realized by the procedure described in the present embodiment.

  In the present embodiment, in order to facilitate explanation and understanding, the HMD 100 uses a monocular HMD having one imaging unit 101 and one video display unit 102. However, a compound eye type HMD having two imaging units 101 and two video display units 102 may be used. In that case, the processing related to the position and orientation measurement of the imaging unit and the generation of the composite image is performed individually. What is necessary is just to perform a part and imaging with respect to an image.

  Further, in the present embodiment, the example in which the color of the surface of the teapot is acquired as a real object for acquiring the color has been described. However, a color chart for color selection as shown at 701 in FIG. 7 may be used as a real object. By using such a color chart 701, the user can easily select various colors. In this case as well, the same procedure as in this embodiment may be executed.

  Further, in the present embodiment, an example in which colors are acquired from a real space image captured by the imaging unit 101 has been described. However, it is also possible to acquire a color from a CG video with the same configuration and procedure as in this embodiment. In this case, for example, if the user performs the same operation as described above while viewing the composite image in which the CG video is superimposed and displayed on the teapot, the color can be acquired from the pixels of the CG video on the composite image.

  Further, in the present embodiment, the color acquisition is performed when the color acquisition button 121 is pressed against the real object and is turned on. However, the color acquisition button 121 is not essential, and the timing of color acquisition can be specified. Any configuration can be used. For example, as an example using a button other than the color acquisition button, a mechanism for recognizing the locus of the pen tip is provided, and when it is detected that the pen tip has drawn a specific locus such as a circle, the position of the pen tip is detected. The color acquisition process described above may be performed.

  Furthermore, in the present embodiment, the position and orientation of the pen tip are measured by the position and orientation sensor. However, the same effect can be obtained even when only the position is measured. In this case, when calculating the direction in which the tip of the pen is facing on the image plane in the step of calculating the tilt direction of the pen tip in S404, the movement direction of the pen tip may be used instead of the posture of the pen. That is, the previous position of the pen tip is recorded, and the vector obtained by the difference from the current position can be obtained as a vector representing the direction in which the pen tip is facing.

  Furthermore, in the present embodiment, the color acquisition position is specified using a pen whose position and orientation is measured, but the position may be specified using another configuration. That is, any means that can designate one point in the real space and can measure the position or the position and orientation can be used as the position designation means. For example, the position of the user's fingertip may be measured from a parallax image captured by a stereo camera provided in the HMD, and the color acquisition position may be designated by the user's fingertip.

  In addition, in the present embodiment, the acquired color is held in the drawing color buffer of the pen drawing unit, and the user cannot confirm the color until a composite image is finally generated. However, by presenting the acquired color each time, the color acquired by the user can be easily visually recognized. In this case, a part of the display area may be used as a drawing color display area, or a three-dimensional virtual brush having an acquired color may be drawn with CG at the pen tip position and displayed in the composite image.

<Second Embodiment>
In this embodiment, in the color acquisition process (S122) of the first embodiment, the drawing position is acquired in addition to the color.

  In the present embodiment, an acquisition mode switch (not shown) is added to the pen 511. The acquisition mode switch is installed near the rear end of the pen, for example. The acquisition mode switch is a switch that keeps its state when set to either ON or OFF.

  In this embodiment, the pen drawing unit 122 has a drawing model buffer instead of the drawing color buffer. The drawing model buffer is a buffer for holding a virtual three-dimensional model drawn on the pen tip when the drawing button of the pen 511 is pressed. The drawing model buffer is initially blank.

  In the present embodiment, the processing procedure shown in FIG. 2 described in the first embodiment is changed as shown in FIG. Hereinafter, the flow of processing in the present embodiment will be described with reference to FIG. In FIG. 10, the same processes as those in FIG. 2 are denoted by the same reference numerals and redundant description is omitted.

  In the present embodiment, when drawing is performed in S121, instead of a three-dimensional line starting from the most recent pen tip position and ending at the pen tip position when the drawing button is pressed, Data for drawing the three-dimensional model stored in the drawing model buffer is generated at the position. This data is sent to the CG scene management unit 106 and recorded.

  In this embodiment, S200 is executed after S113. S200 is an acquisition mode determination step, in which it is determined whether the acquisition mode determination switch is ON or OFF. If ON, the process proceeds to S120, and if OFF, the process proceeds to S130. Further, the state of the previous acquisition mode determination switch is recorded, and when the acquisition mode is changed from OFF to ON, the process proceeds to S201.

  S201 is a drawing model buffer clear step. In this step, the drawing 3D model buffer of the pen drawing unit 122 is cleared and left blank. When S201 ends, the process proceeds to S130.

  In the present embodiment, the process in the color acquisition step of S122 is changed to the process shown in the flowchart of FIG. Hereinafter, with reference to FIG. 11, the color acquisition processing of the present embodiment will be described. In FIG. 11, the same processes as those in FIG. 4 are denoted by the same reference numerals, and redundant description is omitted.

  When S406 ends, S410 is executed. In S410, it is determined whether or not the drawing model buffer is blank. If it is blank, the process proceeds to S411, and if it is not blank, the process proceeds to S412.

  In S411, a drawing model buffer coordinate system is set. In this step, the pen coordinate system 502 at this time is set as the coordinate system of the drawing model buffer. That is, the 3D model recorded in the drawing model buffer is recorded with the position of the pen at this time as the origin. When S411 ends, the process proceeds to S412.

In step S412, the position and color of the pen tip are recorded in the drawing model buffer. At this time, the position of the pen tip is recorded as the position in the coordinate system of the drawing model buffer obtained in S411. That is, assuming that the position and orientation of the pen tip in the world coordinate system at the time of S411 is P, and the position and orientation of the pen tip at the time of S412 is S, P ⁻¹ · S is recorded in the drawing model buffer as the current position. Is done. When S412 ends, S122 ends.

  Thus, in this embodiment, in the acquisition mode in which the acquisition mode switch is ON, every time the color acquisition button is turned ON, the color of the pixel near the pen tip and the position of the pen tip are displayed in the drawing model buffer. To be recorded.

  Therefore, according to the present embodiment, the drawing model is repeated by repeating the operation in which the user holds the pen, turns on the acquisition mode switch, contacts the pen tip with the surface of the real object, and turns on the color acquisition button 121. In the buffer, position and color information relating to points constituting the surface of the real object are accumulated. As a result, the surface shape and color of the real object can be recorded.

  When the drawing button 120 is turned ON, a CG image of the recorded real object can be displayed on the pen tip by performing CG drawing based on the three-dimensional data recorded in the drawing model buffer. it can.

  FIG. 9 is a diagram schematically illustrating a state in which a CG is drawn using the three-dimensional data recorded according to the present embodiment. In FIG. 9, 510 is an actual teapot, and 901 is a virtual teapot drawn based on the recorded three-dimensional data.

  As described above, according to the present embodiment, by acquiring not only the color but also the position of the pen tip, it is possible to record the shape of the real object in addition to the effects of the first embodiment. It has the effect.

(Other embodiments)
As described above, the above-described embodiment can be realized by software by a computer (or CPU, MPU, etc.) of a system or apparatus.

  Therefore, the computer program itself supplied to the computer in order to implement the above-described embodiment by the computer also realizes the present invention. That is, the computer program itself for realizing the functions of the above-described embodiments is also one aspect of the present invention.

  The computer program for realizing the above-described embodiment may be in any form as long as it can be read by a computer. For example, it can be composed of object code, a program executed by an interpreter, script data supplied to the OS, but is not limited thereto.

  A computer program for realizing the above-described embodiment is supplied to a computer via a storage medium or wired / wireless communication. Examples of the storage medium for supplying the program include a magnetic storage medium such as a flexible disk, a hard disk, and a magnetic tape, an optical / magneto-optical storage medium such as an MO, CD, and DVD, and a nonvolatile semiconductor memory.

  As a computer program supply method using wired / wireless communication, there is a method of using a server on a computer network. In this case, a data file (program file) that can be a computer program forming the present invention is stored in the server. The program file may be an executable format or a source code.

  Then, the program file is supplied by downloading to a client computer that has accessed the server. In this case, the program file can be divided into a plurality of segment files, and the segment files can be distributed and arranged on different servers.

  That is, a server apparatus that provides a client computer with a program file for realizing the above-described embodiment is also one aspect of the present invention.

  In addition, a storage medium in which a computer program for realizing the above-described embodiment is encrypted and stored is distributed to the user, and key information for decrypting is supplied to the user who satisfies a predetermined condition, and the user's computer It is also possible to enable installation on Windows. The key information can be supplied by being downloaded from a homepage via the Internet, for example.

  Further, the computer program for realizing the above-described embodiment may use an OS function already running on the computer.

  Further, a part of the computer program for realizing the above-described embodiment may be configured by firmware such as an expansion board attached to the computer, or may be executed by a CPU provided in the expansion board. Good.

FIG. 2 is a diagram illustrating a basic configuration of a computer capable of functioning as each unit realized by software in an MR system. 6 is a flowchart illustrating a color acquisition operation in the MR system according to the first embodiment. It is a block diagram which shows the function structural example of MR system which concerns on 1st Embodiment. It is a flowchart for demonstrating the color acquisition process in MR system which concerns on 1st Embodiment. It is a figure explaining the concept of MR system concerning a 1st embodiment. It is a figure for demonstrating the process which acquires the direction which a nib shows. It is a figure which shows the example of the color chart as a real object used in order to designate a color. It is a figure for demonstrating the pen drawing process in the MR system which concerns on 1st Embodiment. It is a figure which shows typically the state which drawn CG using the three-dimensional data recorded by pen operation in MR system which concerns on 2nd Embodiment. It is a flowchart explaining the color acquisition operation | movement in MR system which concerns on 2nd Embodiment. It is a flowchart for demonstrating the color acquisition process in MR system which concerns on 2nd Embodiment.

Explanation of symbols

100 HMD
DESCRIPTION OF SYMBOLS 101 Image pick-up part 102 Image | video display part 103 Position / orientation sensor 104 Position / orientation measurement part 105 of image pick-up part Pen position / orientation measurement part 106 CG scene management part 107 CG drawing part 108 Image composition part 120 Drawing button 121 Color acquisition button 122 Pen drawing part 123 Color acquisition unit 501 Imaging unit coordinate system 502 Pen coordinate system 503 Image plane 504 Line connecting the origin of the coordinate system of the imaging unit and the center of the image plane 505 Pen tip position 506 Pen tip projection position 507 on the image plane Vector field coordinate system extended from the center of the plane to the projected position of the pen tip on the image plane Y-axis component 508 Vector field coordinates of the vector extended from the center of the image plane to the projected position of the pen tip on the image plane System X-axis direction component 509 Image plane coordinate system 510 Real teapot 511 Pen 512 held by user Camera center 513 of imaging unit Pen extension point 601 Pen tilt direction 602 on the screen Projection position 701 of pen extension point on the image plane Color chart 801 for color selection CG of generated three-dimensional line
802 Previous pen position and orientation 803 Current pen position and orientation 901 Virtual teapot 1001 CPU
1002 RAM
1003 ROM
1004 Keyboard 1005 Mouse 1006 Display unit 1007 External storage device 1008 Storage medium drive 1009 I / F
1010 bus

Claims (10)

An image acquisition process for acquiring an image of a real space;
An instruction detecting step for detecting that a position in the real space is designated by an instruction means operated by a user;
In response to the detection, calculating a position in the video corresponding to the specified position in the real space;
A color acquisition step of acquiring, as a color designated by the user, a color of a pixel different from the pixel in the video corresponding to the position obtained by the calculation step and in the vicinity of the position obtained by the calculation step. The color information acquisition method characterized by this. The color acquisition step
2. The color information according to claim 1, wherein the color of a pixel at a position shifted by a predetermined number of pixels in the direction indicated by the instruction means from the position obtained by the calculating step is acquired as a color designated by the user. Acquisition method. Furthermore, it has a synthesis step of synthesizing a computer graphics video superimposed on the video of the real space,
3. The color information acquisition method according to claim 1, wherein the calculation step and the color acquisition step are applied to an image obtained by the synthesis step.   4. The color information acquisition method according to claim 1, further comprising a display step of displaying computer graphics drawn in the color acquired in the color acquisition step. 5.   5. The color information acquisition method according to claim 1, wherein the color acquisition step records the specified position in the real space together. The instruction means is a pen-type instruction means,
The color acquisition step acquires the color of a pixel corresponding to a position extended by a predetermined distance in the direction of the tip from the tip of the pen-shaped instruction means as a color designated by the user. The color information acquisition method according to claim 1. Image acquisition means for acquiring images of real space;
Instruction detecting means for detecting that the position in the real space included in the video is designated by the instruction means operated by the user;
In response to the detection, calculating means for obtaining a position in the video corresponding to the designated position in the real space;
Color obtaining means for obtaining, as a color designated by the user, a color of a pixel in the vicinity of the position obtained by the calculating means and different from the pixel in the video corresponding to the position obtained by the calculating means. A color information acquisition apparatus characterized by that.   The color information acquisition apparatus according to claim 7, further comprising a display device that presents the video acquired by the video acquisition unit in front of the user's eyes.   The program for making a computer perform the color information acquisition method of any one of Claims 1 thru | or 6.   A computer-readable storage medium in which the program according to claim 9 is stored.