JP3817505B2 - Information processing method and information processing apparatus - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method for obtaining calibration information for converting a position and orientation of an imaging unit measured by a sensor into a position and orientation in a world coordinate system.
[0002]
[Prior art]
In recent years, research on mixed reality has been actively conducted for the purpose of seamless connection between the real space and the virtual space. An image display device that presents mixed reality superimposes an image of a virtual space (for example, a virtual object or character information drawn by computer graphics) on an image of a real space photographed by an imaging device such as a video camera. Realized by displaying.
[0003]
Applications of such image display devices include virtual reality, such as surgical support that superimposes the state of the body on the patient's body surface, and mixed reality games that fight against virtual enemies floating in real space. Different new fields are expected.
[0004]
A common requirement for these applications is how to accurately align the real space and the virtual space, and many efforts have been made in the past.
[0005]
The problem of alignment in mixed reality results in the problem of obtaining the three-dimensional position and orientation of the imaging device in the world coordinate system (hereinafter simply referred to as the world coordinate system) set in the real space. As a method for solving these problems, a three-dimensional position and orientation sensor such as a magnetic sensor, an optical sensor, or an ultrasonic sensor is generally used.
[0006]
In general, the output value output by the three-dimensional position and orientation sensor is the position and orientation of the measurement point in the sensor coordinate system uniquely defined by the sensor, and not the position and orientation of the imaging device in the world coordinate system. For example, taking the magnetic sensor FASTRAK of Polhemus as an example, the position and orientation of the receiver in the coordinate system defined by the transmitter is obtained as the sensor output. Therefore, the sensor output value cannot be used as it is as the position and orientation of the imaging apparatus in the world coordinate system, and some calibration is required. Specifically, coordinate conversion for converting the position and orientation of the measurement point into the position and orientation of the imaging apparatus and coordinate conversion for converting the position and orientation in the sensor coordinate system into the position and orientation in the world coordinate system are required. In the present specification, information for converting the sensor output value into the position and orientation of the imaging apparatus in the world coordinate system is referred to as calibration information.
[0007]
In order to accurately align the real space and the virtual space, it is necessary to set accurate calibration information by some means. Only when accurate calibration information is given, display of a virtual image accurately aligned with the real space is realized.
[0008]
The calibration information may be stored in any form as long as the information can define the position and orientation of the other coordinate system as viewed from one coordinate system. For example, it may be a 4-by-4 viewing transformation matrix that represents transformation from one coordinate system to the other. Alternatively, the position and orientation may be expressed by a total of six parameters, ie, three parameters describing the position and three parameters expressing the orientation by Euler angles. The posture may be expressed by four parameters, a ternary vector defining the rotation axis and a rotation angle around the axis, or the rotation angle may be expressed by the magnitude of the vector defining the rotation axis. It may be expressed by three parameters.
[0009]
Moreover, you may express with the parameter (For example, the position and attitude | position of the world coordinate system in a sensor coordinate system) showing those inverse transformations. However, in any case, the position and orientation of the object in the three-dimensional space have only a total of 6 degrees of freedom, that is, 3 degrees of freedom in the position and 3 degrees of freedom in the orientation. The number of parameters is a total of 12 parameters including 6 parameters necessary for conversion from the world coordinate system to the sensor coordinate system and 6 parameters necessary for conversion from the position and orientation of the measurement point to the position and orientation of the imaging apparatus.
[0010]
As one of known methods for setting calibration information, a user or an operator interactively changes the 12 parameters (or 12 or more equivalent parameters) through an input means (not shown) to accurately There is a method in which adjustment is performed by trial and error until alignment is achieved.
[0011]
Further, according to the calibration method proposed in Patent Document 1, either the conversion parameter from the position and orientation of the measuring point to the position and orientation of the imaging device or the conversion parameter from the world coordinate system to the sensor coordinate system is somehow used. If it is obtained, the remaining unknown parameters can be easily derived by using the virtual image generated based on the position and orientation information fixed to a certain value as the visual cue.
[0012]
[Patent Document 1]
JP 2002-229730 A
[0013]
[Problems to be solved by the invention]
However, in the former method, it is necessary to adjust twelve unknown parameters at the same time, so that the adjustment takes a considerable time, and accurate calibration information is not always obtained. The latter method also has a problem that it is necessary to perform a trial and error work or a work using some kind of calibration instrument when deriving a known parameter.
[0014]
The present invention has been made in view of the above problems, and calibration information for converting the position and orientation of the imaging unit measured by the sensor into the position and orientation in the world coordinate system can be more simply and specially calibrated. It is an object of the present invention to be able to obtain with high accuracy without using.
[0015]
[Means for Solving the Problems]
In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.
That is, an information processing method for obtaining calibration information for converting the position and orientation of the imaging unit measured by the sensor into a position and orientation in the world coordinate system,
When photographing a plurality of captured images obtained by photographing a real space in which a plurality of markers having known world coordinates are arranged at a plurality of positions and orientations using the imaging unit, and each of the photographed images Get the measured value of the sensor of
Detecting the image coordinates of the marker included in the captured image,
In response to a user instruction, select a captured image to be used when calculating the calibration information from the acquired captured images,
The calibration information is calculated using the measured value of the sensor of the selected captured image, the image coordinates of the detected marker of the selected captured image, and the world coordinates of the detected marker. ,
The image obtained from the image coordinates of the marker detected from the captured image, the identification information of the detected marker, and the world coordinates of the detected marker using the calculated calibration information. Combine and display the coordinates
It is characterized by that.
[0016]
In order to achieve the object of the present invention, for example, an information processing apparatus of the present invention comprises the following arrangement.
That is, an information processing apparatus for obtaining calibration information for converting the position and orientation of the imaging unit measured by the sensor into a position and orientation in the world coordinate system,
When photographing a plurality of captured images obtained by photographing a real space in which a plurality of markers having known world coordinates are arranged at a plurality of positions and orientations using the imaging unit, and each of the photographed images Input means for inputting the measured value of the sensor of
Detecting means for detecting image coordinates of a marker included in the captured image;
In accordance with a user instruction, a selection unit that selects a captured image to be used when calculating the calibration information from the acquired plurality of captured images;
The calibration information is calculated using the measured value of the sensor of the selected captured image, the image coordinates of the detected marker of the selected captured image, and the world coordinates of the detected marker. A calculation means;
The image obtained from the image coordinates of the marker detected from the captured image, the identification information of the detected marker, and the world coordinates of the detected marker using the calculated calibration information. Means for combining and displaying coordinates; and
It is characterized by providing.
[0019]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings.
<First Embodiment>
The sensor calibration apparatus according to the present embodiment obtains parameters for converting the position and orientation of the imaging unit measured by the sensor into a position and orientation in the world coordinate system.
[0020]
Therefore, the sensor calibration device of the present embodiment is
(1) World coordinates of four or more markers that are not on the same straight line,
(2) Marker image coordinates on images taken at multiple positions and orientations,
(3) Sensor measurement value when the image of (2) is taken,
(4) Initial value (rough value) of sensor placement information,
The parameter is obtained using
[0021]
Among these, the world coordinates of the marker (1) are data to be prepared as known information in the calibration preparation stage. As for the markers (landmarks, feature points), it is necessary that at least three or more points with known world coordinates are arranged in the real space that is to be displayed by the image display device. Here, each marker is a marker that can detect the image coordinates of the projected image on the captured image and can identify which marker is, for example, each marker having a different color. And The world coordinate system has a predetermined point as an origin, and further takes X, Y, and Z axes in directions orthogonal to the origin. It is assumed that the coordinates of the three markers are already known in this coordinate system. In other words, the distances from the origin in the X, Y, and Z directions are measured in advance for the above three markers.
[0022]
(2) and (3) are data acquired during calibration. The initial value of the sensor arrangement information in (4) is not necessarily necessary data, but if it is not set, an appropriate solution may not be obtained.
[0023]
Using these data as input, calculate the theoretical value of the image coordinates of each marker based on the world coordinates, sensor measurement values, and sensor placement information of each marker, and calibration information that minimizes the sum of the errors from the actual measurement values. (Parameter) is calculated.
Note that the parameters may be obtained using other methods using other information.
[0024]
FIG. 1 is a block diagram showing a schematic configuration in the first embodiment of the sensor calibration apparatus according to the present embodiment.
In FIG. 1, reference numeral 100 denotes an arithmetic processing unit, which is composed of a computer such as a computer. The arithmetic processing unit 100 includes a CPU 101, a RAM 102, an image generation device 103, a system bus 104, a disk device 105, an input device 106, and an image capture device 107 therein.
[0025]
The CPU 101 controls the calibration process based on the calibration program. The CPU 101 is connected to the system bus 104, and can communicate with the RAM 102, the image generation device 103, the disk device 105, the input device 106, and the image capture device 107.
[0026]
The RAM 102 is realized by a main storage device such as a memory. The RAM 102 temporarily holds the program code of the calibration program, the control information of the program, the world coordinates and image coordinates of the marker, the calibration information calculated by the apparatus, and the like via the system bus 104.
[0027]
The image generation device 103 is realized by a device such as a graphics card. Image information generated by a program executed on the CPU 101 is sent to the display unit 200 via the system bus 4.
The system bus 104 is a communication path to which the devices constituting the arithmetic processing unit 100 are connected and the devices communicate with each other.
[0028]
The disk device 105 is realized by an auxiliary storage device such as a hard disk. The disk device 105 holds the program code of the calibration program, the control information of the program, the world coordinates and image coordinates of the marker, the calibration information calculated by this device, and the like.
[0029]
The input device 106 is realized by various interface devices. A signal from a device connected to the outside of the arithmetic processing unit 100 is input as data, and the data is written to the RAM 102 via the system bus 104.
[0030]
The image capturing device 107 is realized by a device such as a capture card. An image sent from the imaging device 302 is input, and image data is written into the RAM 102 via the system bus 104.
[0031]
A display unit 200 is realized by a display device such as a CRT monitor or a liquid crystal monitor. It is used to display a video signal sent from the image generation apparatus 103 and present the result to the user of the apparatus.
Reference numeral 300 denotes a head-mounted unit, which is a target to be calibrated by this apparatus. The head mounting unit 300 includes a receiver 301 and an imaging device 302.
[0032]
For example, in the case of a magnetic sensor, the receiver 301 is realized by a device that measures the magnetic field generated by the transmitter 600. The value of the magnetic field measured by the receiver 301 is sent to the sensor control device 500 and converted into a parameter representing a three-dimensional position and orientation by the sensor control unit 500.
The imaging device 302 is realized by an imaging device such as a CCD camera. A video signal captured by the imaging device 302 is sent to the image capturing device 107.
[0033]
The operation input unit 400 is realized by an input device for controlling the arithmetic processing unit 100 such as a keyboard and a mouse. The operation input unit 400 sends an operation signal to the input device 106. A user of the apparatus (not shown) gives an instruction to control the apparatus by operating the operation input unit 400.
[0034]
For example, in the case of a magnetic sensor, the sensor control unit 500 controls the receiver 301 and the transmitter 500 and calculates the three-dimensional position and orientation information of the receiver 301 based on information obtained from the receiver 301. The three-dimensional position and orientation information calculated by the sensor control unit 500 is sent to the input device 106.
For example, in the case of a magnetic sensor, the transmitter 600 generates a magnetic field in order for the sensor control unit 500 to calculate the three-dimensional position and orientation of the receiver 301.
[0035]
In this embodiment, although the case where the magnetic sensor was used was demonstrated, the sensor to be used is not restricted to a magnetic sensor. For example, an optical sensor or an ultrasonic sensor may be used, and the type of sensor is not limited as long as it can measure a three-dimensional position and posture. In this case, the receiver 301 is the measurement target of the sensor, and the transmitter 600 is the sensor coordinate system origin when the sensor control unit 500 calculates the three-dimensional position and orientation of the receiver 301.
[0036]
FIG. 2 is a block diagram showing a functional configuration of the calibration apparatus according to the present embodiment. The processing of each unit in FIG. 2 is performed in the arithmetic processing unit 100 in FIG.
The world coordinate holding unit 110 holds coordinate data of each marker in the world coordinate system, and outputs this according to a request from the data management unit 112. The world coordinate holding unit 110 holds the coordinate data of each marker in association with data of information unique to each marker (marker color information and identification information).
[0037]
When receiving a data acquisition request from the instruction unit 115, the data management unit 111 inputs the image coordinates and identification information of the marker from the image coordinate acquisition unit 112, and the world coordinates of the marker corresponding to the identification information from the world coordinate holding unit 110 , And a set of image coordinates, world coordinates, and identification information is added to the data list and held. When only the image coordinates of the marker are input from the image coordinate acquisition unit 112 and no identification information is input, only the image coordinates are added to the data list. When receiving a data deletion request from the instruction unit 115, the data is deleted from the data list. Also, when a data identification request is received from the instruction unit 115, the combination of the world coordinates, image coordinates, and identification information of the marker is changed, and the data in the data list is changed. Further, the generated data list is output to the calibration information calculation unit 113 in accordance with a request from the calibration information calculation unit 113.
[0038]
The image coordinate acquisition unit 112 captures the coordinates and identification information of the marker captured in the image (hereinafter referred to as a real image) captured by the image capturing device 302 in FIG. 1 and acquired by the image capturing device 107 in FIG. And outputs these pieces of information to the data management unit 111 according to a request from the data management unit 111.
[0039]
When the calibration information calculation unit 113 receives a calibration information calculation instruction from the instruction unit 115, the calibration information calculation unit 113 inputs a data list from the data management unit 112, calculates calibration information based on the data list, and stores the calculated calibration information in the calibration information holding unit 114. To send.
[0040]
The calibration information holding unit 114 holds the calibration information calculated by the calibration information calculation unit 113, and outputs or changes the held calibration information according to a request from the instruction unit 115, and saves it in a file. When an output request is received from the instruction unit 115, calibration information is output. When a calibration information change request is received from the instruction unit 115, the value of the held calibration information is changed. When a file save request is received from the instruction unit 115, a file is created in the disk device 105, and calibration information is saved in the file. When a file read request is received from the instruction unit 115, the currently held calibration information is discarded, the file instructed from the disk device 105 is read, and the read value is newly set as the current calibration information. When a reset request is received from the instruction unit 115, the currently held calibration information is discarded, and the default value of the calibration information held when the apparatus is activated is newly set as the current calibration information. In response to a request from the calibration information calculation unit 113, the calibration information held by the calibration information holding unit 114 is sent to the calibration information calculation unit 113. The calibration information sent to the calibration information calculation unit 113 is used as an initial value when the calibration information calculation unit 113 calculates the calibration information.
[0041]
The instruction unit 115 issues a data acquisition request when a data acquisition command is input from a user of the apparatus, a data deletion request when a data deletion command is input, and a data identification request when a data identification command is input. Each is sent to the data management unit 111. When a calibration information calculation command is input, a calibration information calculation request is sent to the calibration information calculation unit 113.
[0042]
In the present embodiment, when the user of the apparatus gives an instruction to the instruction unit 115, the user uses the GUI shown in FIG. 3 includes a main window 1000, an image display area 1010, a calibration information display area 1020, operation buttons 1030, and a file menu 1040. As shown in FIG. 4, the file menu 1040 includes a save submenu 1050 and an end submenu 1060.
[0043]
A real image is displayed in the image display area 1010. Further, the mark indicating the coordinate position of the marker specified by the image coordinate acquisition unit 112 and the identification information are also displayed superimposed on the real image.
[0044]
FIG. 5 shows a state in which the mark indicating the coordinate position of the marker specified by the image coordinate acquisition unit 112 and the identification information are displayed in the image display area 1010 superimposed on the real image. In FIG. 5, an ellipse indicates a marker photographed in the real image. In addition, the image coordinates of the marker specified by the image coordinate acquisition unit 112 are indicated by x marks surrounded by a rectangle. Further, the marker identification information specified by the image coordinate acquisition unit 112 is superimposed and displayed with text representing the marker name.
[0045]
By superimposing the mark on the position of the image coordinate of the marker, the user of the present apparatus can confirm the coordinates specified by the image coordinate acquisition unit 112 at a glance.
Further, by displaying the marker identification information in a superimposed manner, the user of this apparatus can check at a glance which marker the marker specified by the image coordinate acquisition unit 112 is.
[0046]
Here, an example is shown in which x marks surrounded by a rectangle are superimposed in order to indicate the image coordinates of the marker, but what is superimposed is not limited to the x marks surrounded by a rectangle. Various symbols such as ○, ×, □, and arrows, icons, and the like may be used, and any symbol can be used as long as the image coordinates of the marker specified by the image coordinate acquisition unit 112 can be visually recognized.
[0047]
Moreover, although the example which superimposed the text showing a name was shown as marker identification information here, the identification information to superimpose is not restricted only to a text. The identification may be performed using an icon or an image, or may be performed by changing the color of text, icon, image, or the color of the marker itself. Any device can be used as long as the user of the apparatus can confirm the correspondence between the marker photographed in the real image and the actual marker.
[0048]
In FIG. 6, although the image coordinate acquisition unit 112 specifies the image coordinates of the marker, the identification information is not specified, and the data management unit 111 uses the marker world coordinates, the image coordinates, and the identification information. The display example of the image display area 1010 in the case where no set is created is shown. In the data management unit 111, the association of the marker with the world coordinates, the image coordinates, and the identification information and creation of the set is hereinafter referred to as “identifying the marker”.
[0049]
In FIG. 6, the x mark is superimposed on the real image on the image coordinates of the marker. Also, the text “?” Is displayed superimposed on the real image. In the image display area 1010, when the identification of the marker is performed, the x mark surrounded by the rectangle and the identification information of the marker are superimposed and displayed, and when the identification of the marker is not performed, the display is performed. Try to switch methods.
[0050]
There is no need to switch the entire image display area 1010 for the display method when the marker is identified or not, and for each marker photographed in the image display area 1010, it is independent for each marker. It is possible to switch.
[0051]
In FIG. 7, the coordinates in the world coordinate system of the marker held by the world coordinate holding unit 110 are drawn by CG from the viewpoint of the imaging device 302 based on the current calibration information and the position and orientation information of the receiver 301. In addition, a state in which the image is displayed superimposed on the real image in the image display area 1010 is shown.
[0052]
If calibration by this apparatus is performed correctly, the coordinates of the marker specified by the image coordinate acquisition unit 112 and the world coordinates of the marker superimposed on the real image theoretically match. The user of this apparatus can visually check the accuracy of the calibration performed by this apparatus at any time by observing the amount of deviation between the image coordinates of the marker and the world coordinates.
[0053]
In the present embodiment, the world coordinates of the marker are drawn by CG, but the object to be drawn by CG is not limited to the world coordinates of the marker.
For example, in order to confirm how the world coordinate system is observed on the real image, the coordinate axes, planes, world coordinate origin, and the like constituting the world coordinate system may be drawn.
[0054]
In addition, when a part of the calibration apparatus is realized by using an image display device that presents mixed reality, a virtual world or a virtual object used when presenting mixed reality is drawn by CG. You may make it do.
[0055]
When the image coordinate acquisition unit 112 performs a process of extracting a color area in the image in order to specify the coordinate of the marker, the image display area 1010 includes the color area extraction instead of the actual image. You may make it display the image which shows the result of a process. In this way, the user of this apparatus can easily adjust the parameters for the color area extraction process.
[0056]
A user of this apparatus can specify image coordinates in the image display area 1010 using a pointing device such as a mouse. For example, the coordinate of the marker (in the real image) performed by the image coordinate acquisition unit 112 is specified by the user of the apparatus, or a desired marker photographed in the real image is selected, and processing parameters for the marker are selected. Or the identification information can be manually given by the user of the apparatus.
[0057]
In the calibration information display area 1020, the current calibration information held by the calibration information holding unit 114 is displayed.
The operation button 1030 is used by the user of the apparatus to control the behavior and GUI of the apparatus by selecting a button area with a pointing device such as a mouse. In the present embodiment, when the user of the apparatus selects the operation button 1030, a data acquisition dialog, an initial value setting dialog, and a detection parameter setting dialog are newly displayed.
[0058]
The file menu 1040 displays a save submenu 1050 and an end submenu 1060 when the user of the apparatus selects a menu area with a pointing device such as a mouse.
[0059]
The save submenu 1050 gives an instruction to the apparatus when the user of the apparatus selects a submenu area with a pointing device such as a mouse. When the save submenu 1050 is selected, the instruction unit 115 issues a file save command and sends a file save request to the calibration information holding unit 114.
[0060]
An end submenu 1060 gives an instruction to the apparatus when the user of the apparatus selects a submenu area with a pointing device such as a mouse. When the end submenu 1060 is selected, the calibration apparatus is ended. When the calibration information holding unit 114 holds unsaved calibration information, the instruction unit 115 issues a file save command, sends a file save request to the calibration information holding unit 114, and the calibration information holding unit 114 After waiting until the file saving process is completed, the calibration apparatus is terminated.
[0061]
FIG. 8 shows an example of a data acquisition dialog. The data acquisition dialog 1100 in FIG. 8 includes a data list display area 1110, a data acquisition operation button 1120, a display image switching button 1130, a calibration information calculation button 1140, a selection data invalidation / validation button 1150, a selection data deletion button 1160, and a selection. A data display area 1170, a selected data operation area 1180, an error display area 1190, and a data acquisition dialog end button 1195 are configured.
[0062]
The data list display area 1110 displays thumbnails of reduced images of real images corresponding to the data lists acquired so far. When the user of the apparatus selects a thumbnail, data corresponding to the image is selected from the data list and displayed in the selected data display area 1170. In addition, the data corresponding to the image is selected. The data in the selected state is subject to processing when the selection data invalidation / validation button 1150 and the selection data deletion button 1160 are pressed.
[0063]
At this time, a real image corresponding to the data list is displayed in the image display area 1010. In the data list display area 1110, a real image is displayed in a reduced size, and it is difficult for the user of this apparatus to confirm details from the reduced image. Therefore, the real image is displayed in the image display region 1010 without being reduced. . Further, the image coordinates, world coordinates, and identification information of the marker selected from the data list are also displayed by being superimposed on the real image by the method described in the description of the image display area 1010.
[0064]
The data acquisition button 1120 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the data acquisition button 1120 is pressed, the instruction unit 115 issues a data acquisition command and sends a data acquisition request to the data management unit 111. Further, the actual image at that time is added to the data list display area and displayed.
[0065]
A display image switching button 1130 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the display image switching button 1130 is pressed, the mode of the image displayed in the image display area 1010 is switched. The user of this apparatus selects “live video display mode” for displaying the currently acquired real image and “acquired data display mode” for displaying the real image currently selected in the data list display area 1110. Is possible. When the display image switching button 1130 is pressed, the screen is switched to the acquired data display mode when the current mode is the live video display mode, and is switched to the live video display mode when the current mode is the acquired data display mode.
[0066]
The calibration information calculation button 1140 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the calibration information calculation button is pressed, the instruction unit 115 issues a calibration information calculation command and sends a calibration information calculation request to the calibration information calculation unit 113.
[0067]
The selection data invalidation / validation button 1150 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. By pressing the selection data invalidation / validation button 1150, when the data (photographed image) currently selected in the data display area 1110 is valid, the data is invalidated, and the selection data is selected from the target of the calibration information calculation process. exclude. When the currently selected data is invalid, the data is validated.
[0068]
The selection data deletion button 1160 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the selection data deletion button 1160 is pressed, the instruction unit 115 issues a data deletion command and sends a data deletion instruction to the data management unit 111. Further, the thumbnail of the currently selected real image is deleted from the data list display area 1110.
[0069]
In the selected data display area 1170, data selected by the user of the apparatus in the data list display area 1110 is displayed.
The selection data operation area 1180 is used by the user of this apparatus to operate selection data. FIG. 9 shows details of the selection data operation area 1180.
The selection data operation area 1180 includes a marker selection button 1181, a marker addition button 1182, a marker identification information selection menu 1183, and a marker deletion button 1184.
[0070]
The marker selection button 1181 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the marker selection button 1181 is pressed, the operation mode is set to “marker selection mode”. When the marker selection mode is set and the user of this apparatus selects the vicinity of the mark indicating the coordinate position of the marker displayed in the image display area 1010, the marker is selected.
[0071]
The marker addition button 1182 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the marker addition button 1182 is pressed, the operation mode is set to “marker addition mode”. When the user of this apparatus selects an arbitrary part of the image display area 1010 in the state where the marker addition mode is set, a new marker having the image coordinates as the marker coordinates is set and added to the data list. . Further, the marker is selected.
[0072]
The marker identification information selection menu 1183 gives an instruction to the apparatus when the user of the apparatus selects a menu area with a pointing device such as a mouse. When the marker identification information selection menu 1183 is selected, a list of marker identification information accumulated in the data list is displayed. In this embodiment, a character string indicating the name of the marker is used as the marker identification information. The user of this apparatus selects one of the marker identification information displayed in a list by using a pointing device such as a mouse. When the marker identification information is selected, the instruction unit 115 issues a data identification command, and sends the selected identification information and the selected marker information to the data management unit 111 at the same time as the data identification request. .
[0073]
The marker deletion button 1184 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the marker deletion button 1184 is pressed, the marker that is currently selected is deleted from the data list.
[0074]
The error display area 1190 displays the calibration error when the calibration information calculation unit 113 calculates the calibration information. In the present embodiment, two types of calibration errors are displayed: an error for each marker in the data list and an average of errors for all data.
[0075]
The user of this apparatus confirms the calibration error for each marker.If the accuracy of the calibration information calculated by the calibration information calculation unit 113 is not sufficient, or if the solution does not converge in the calculation of the calibration information, It is easy to specify the marker that causes the problem.
[0076]
Further, by confirming the average of errors of all data, the result of validation or invalidation of data used for calculating calibration information can be evaluated using the selection data invalidation / validation button.
[0077]
A data acquisition dialog end button 1195 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the data acquisition dialog end button 1195 is selected, the data acquisition dialog is closed. Even if the data acquisition dialog is closed, the contents of the data list are retained.
[0078]
FIG. 10 shows an example of the initial value setting dialog 1200. The initial value setting dialog includes a parameter type selection tab 1210, a parameter setting area 1220, a standard setting viewpoint button 1230, a transmitter observation button 1240, a reset button 1250, and an initial value setting dialog end button 1260.
[0079]
The parameter type selection tab 1210 is used to select the type of calibration information for which an initial value is set by the user of the apparatus selecting a tab area with a pointing device such as a mouse. There are two types of tab areas, “Local Transform” and “World Transform”. When the “Local Transform” area is selected, the calibration information mode is converted into the position and orientation of the image sensor. The first parameter is set to “imaging device-station calibration information mode”. When the “World Transform” area is selected, the calibration information mode is set to the “world coordinate system-sensor coordinate system calibration information mode”.
[0080]
In the parameter setting area 1220, the user of this apparatus can set parameters for calibration information corresponding to the current calibration information mode by means of buttons, sliders, spin buttons, numerical input from a keyboard, and the like. In the present embodiment, the parameters of the calibration information are set by using the following methods alone or in combination.
[0081]
First, a vector defining a position and a rotation axis and a rotation angle around that axis are specified.
Second, specify the position and Euler angle.
Third, the rotation angles of the receiver 301 around the X, Y, and Z axes as viewed from the imaging device 302 and the rotation angles of the imaging device 302 as viewed from the receiver 301 around the X, Y, and Z axes. specify.
When the parameters of the calibration information are set, the instruction unit 115 issues a calibration information change command and sends a calibration information change request to the calibration information holding unit 114.
[0082]
The standard setting designation button 1230 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the standard setting designation button 1230 is designated, the instruction unit 115 issues a calibration information read command and sends a calibration information read request to the calibration information holding unit 114.
[0083]
The transmitter observation button 1240 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse.
By selecting the transmitter observation button 1240, a rough value of Local Transform can be automatically calculated using the sensor measurement value.
[0084]
First, as shown in FIG. 14A, after adjusting the position and orientation of the camera so that the transmitter of the sensor is imaged at the approximate center of the captured image, the transmitter observation button 1240 is clicked. The approximate value of Local Transform is calculated based on the sensor measurement value at that time.
[0085]
When transmitter observation button 1240 is clicked, instruction unit 115 issues a calibration information calculation command and sends a calibration information calculation request to calibration information calculation unit 113. At this time, the calibration information calculation unit 113 uses only the current sensor measurement value in the data list managed by the data management unit 111 without using the world coordinates, the image coordinates, and the identification information. Calibration information (Local Transform) between the receivers 301 is calculated.
[0086]
Calculate the theoretical value of the transmitter's position and orientation using the calculated approximate value, generate a virtual image of the transmitter according to the theoretical value, and superimpose the virtual image of the transmitter on the image position according to the theoretical value on the captured image And displayed (FIG. 14B).
[0087]
The Local Transform set using the transmitter observation button 1240 has an indefinite rotation angle in the Z-axis direction of the imaging device, so this parameter (Z-axis of Local Transform) is set using the azimuth (z-axis) slider bar (parameter setting area 1220). Roughly adjust the posture component). Then, the virtual image of the transmitter is updated in real time according to the adjustment result (FIG. 14 (c)). It is possible to adjust other parameters.
[0088]
The reset button 1250 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the reset button 1250 is designated, the instruction unit 115 issues a reset command and sends a reset request to the calibration information holding unit 114.
[0089]
An initial value setting dialog end button 1260 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the initial value setting dialog end button 1260 is selected, the initial value setting dialog is closed. Even if the initial value setting dialog is closed, the calibration information is retained.
[0090]
As described above, by selecting the transmitter observation button 1240, a rough value of Local Transform can be calculated automatically using the sensor measurement value without using any special preparation by using the transmitter. . Since the transmitter is a necessary configuration in the present system, the user does not need to perform special processing in order to perform this processing.
[0091]
11 and 12 show examples of the marker detection parameter setting dialog 1300. FIG. The marker detection parameter setting dialog 1300 includes a parameter type selection tab 1310, a marker detection parameter setting area 1320, a marker detection result display button 1330, a marker detection parameter reset button 1340, a marker detection button 1350, a marker identification means switching button 1360, a marker identification parameter setting. The area 1370 includes a marker identification parameter reset button 1380 and a marker detection parameter setting dialog end button 1390.
[0092]
The parameter type selection tab 1310 allows the user of this apparatus to select and display either the marker detection parameter setting screen or the marker identification parameter setting screen by selecting a tab area with a pointing device such as a mouse.
[0093]
The marker detection parameter setting area 1320 is displayed when the marker detection parameter setting screen is selected on the parameter type selection tab 1310. In the marker detection parameter setting area 1320, the user of this apparatus uses a means such as a slider or a numerical value input box to set the color threshold value for detecting the marker and the minimum value of the number of pixels in the color area recognized as the marker. Set. The color threshold is used to consider a pixel as a marker candidate when the value of each color component of the pixel exceeds the threshold.
[0094]
In this embodiment, threshold values are set for R, G, and B indicating the luminance values of the respective colors for red, green, and blue. For example, threshold values are set for Y, Cr, and Cb components of pixels. Also good. The minimum value of the number of pixels in the color area is used to prevent the area from being recognized as a marker when the number of pixels in the color area is smaller than the set minimum value. In the present embodiment, only the minimum value of the number of pixels in the color region is determined, but the maximum value may be determined at the same time so that the number of pixels considered as marker candidates falls within a certain range.
[0095]
The marker detection result display button 1330 is displayed when the marker detection parameter setting screen is selected on the parameter type selection tab 1310. The marker detection result display button 1330 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the marker detection result display button 1330 is designated, the real image display mode color extraction result image display mode is switched. In the real image display mode, a real image is displayed in the image display area 1010. In the color extraction result image display mode, the color extraction result image is displayed in the image display area 1010. In the color extraction result image shown in FIG. 13, only the color region extracted in the live-action image is displayed.
[0096]
The marker detection parameter reset button 1340 is displayed when the marker detection parameter setting screen is selected on the parameter type selection tab 1310. The marker detection parameter reset button 1340 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the marker detection parameter reset button 1340 is designated, the marker detection parameter currently held is discarded, and the default value of the marker detection parameter set when the apparatus is activated is set as the current marker detection parameter.
[0097]
The marker detection stop button 1350 is displayed when the marker detection parameter setting screen is selected on the parameter type selection tab 1310. The marker detection stop button 1350 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the marker detection stop button 1350 is selected, the marker detection process is stopped.
[0098]
The marker identification means switching button 1360 is displayed when the marker identification parameter setting screen is selected on the parameter type selection tab 1310. The marker identification means switching button 1360 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the marker identification means switching button 1360 is selected, the marker identification function using the sensor measurement value is switched between valid / invalid. The marker identification function is based on a current sensor measurement value and calibration information, and the detected image coordinates of the marker and the coordinates of the world coordinates of the marker held by the world coordinate holding unit 110 projected onto the imaging surface of the imaging device 302. This is a function for calculating a distance and automatically identifying a marker when the minimum value of the distance is less than or equal to the maximum allowable value.
[0099]
The marker identification parameter setting area 1370 is displayed when the marker identification parameter setting screen is selected on the parameter type selection tab 1310. In the marker identification parameter setting area 1370, the user of this apparatus sets a maximum allowable distance for identifying a marker using means such as a slider or a numerical value input box.
[0100]
The marker identification parameter reset button 1380 is displayed when the marker identification parameter setting screen is selected on the parameter type selection tab 1310. The marker identification parameter reset button 1380 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the marker identification parameter reset button 1380 is designated, the marker identification parameter currently held is discarded, and the default value of the marker identification parameter set when the present apparatus is activated is set as the current marker identification parameter.
[0101]
The detection parameter setting dialog end button 1390 gives an instruction to the apparatus when the user of the apparatus selects a button area with a pointing device such as a mouse. When the detection parameter setting dialog end button 1390 is selected, the detection parameter setting dialog 1390 is closed. Even when the detection parameter setting dialog is closed, the marker detection parameter and the marker identification parameter are retained.
[0102]
FIG. 15 is a flowchart of processing performed by the sensor calibration apparatus of the present embodiment. Note that the program code for realizing the processing according to the flowchart is stored in a storage device such as the disk device 105 or the RAM 102 in the device of the present embodiment, and is read and executed by the CPU 101.
[0103]
In step S000, the sensor calibration apparatus of the present embodiment is activated.
In step S100, the instruction unit 115 determines whether a data acquisition command has been input from the user of the apparatus. If a data acquisition command has been input, the process proceeds to step S110.
In step S <b> 110, the data management unit 111 inputs the current measurement value of the position and orientation of the receiver 301 from the sensor control unit 500.
[0104]
In step S <b> 120, the data management unit 111 inputs marker identification information and image coordinates on the captured image captured by the imaging device 302 from the image coordinate acquisition unit 112. This input is performed for each marker when a plurality of markers on the captured image are captured.
In step S130, the data management unit 111 adds the input data to the data list for each detected marker.
[0105]
In step S200, the instruction unit 115 determines whether or not a data list editing operation has been input by the user of the apparatus. If the data list editing operation has been input, the process proceeds to step S210.
[0106]
In step S210, the instruction unit 115 issues a command corresponding to the data list editing operation input from the user of the apparatus, and edits the data list. For example, to edit a data list, select an element in the data list, delete a selected element from the data list, add a new marker to the data list, delete an existing marker from the data list, An operation of giving identification information to the marker and identifying it is included.
[0107]
In step S300, the calibration information calculation unit 113 determines whether or not the data list acquired so far has enough information to calculate the calibration information. If the data list does not satisfy the condition, the process returns to step S100 again and waits for the input of a data acquisition command. On the other hand, if the data list satisfies the conditions for calculating calibration information, the process proceeds to step S400. As conditions for calculating calibration information, for example, data on three or more different markers is obtained, data is acquired at a plurality of viewpoint positions, and the total number of data is 6 or more. As a condition. However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.
[0108]
In step S400, it is determined whether a calibration information calculation command has been input from the user of the apparatus. When the calibration information calculation command is input, the process proceeds to step S410. When the calibration information calculation command is not input, the process returns to step S100 and waits for the input of the data acquisition command.
[0109]
In step S410, the calibration information calculation unit 113 performs calibration information calculation processing.
In step S500, the instruction unit 115 determines whether or not an editing operation for calibration information has been input from the user of the apparatus. If an editing operation for calibration information has been input, the process proceeds to step S510.
[0110]
In step S510, the instruction unit 115 issues a command corresponding to the editing operation of the calibration information input from the user of the apparatus, and edits the calibration information. For example, editing calibration information includes operations such as changing calibration information parameters, reading calibration information, and resetting calibration information.
[0111]
In step S600, the instruction unit 115 determines whether or not a calibration information storage operation has been input from the user of the apparatus. If the calibration information storage operation has been input, the process proceeds to step S610.
In step S610, the instruction unit 115 sends a file storage request to the calibration information holding unit 114, and outputs the currently held calibration information to the disk device 105.
[0112]
In step S700, the instruction unit 115 determines whether or not an end operation of the apparatus has been input from the user of the apparatus. When the calibration information end operation is input, the process proceeds to step S800. If not, the process returns to step S100 again.
In step S800, processing for terminating the present apparatus is performed.
[0113]
FIG. 16 is a flowchart of a standard operation for the user of the sensor calibration apparatus of the present embodiment to calibrate the sensor using the apparatus.
In step S1000, the user of this apparatus starts the sensor calibration apparatus of this embodiment.
[0114]
In step S <b> 1100, the user of this apparatus images the marker using the imaging device 302. The user of this apparatus checks whether or not the image coordinate acquisition unit 112 of this apparatus correctly specifies the image coordinate and identification information of the marker. If it can be specified correctly, the process proceeds to step S1200. If it cannot be specified, the process proceeds to step S1110.
[0115]
In step S1110, the user of the present apparatus gives an instruction for adjusting the marker detection parameter and the marker identification parameter to the instruction unit 115 using, for example, the marker detection parameter setting dialog 1300.
In step S1200, the user of the apparatus causes the instruction unit 115 to issue a data acquisition command using, for example, the data acquisition button 1120.
[0116]
In step S1300, the user of this apparatus checks whether each data in the acquired data list is good. If the data is good, the process proceeds to step S1400. If the data is not good, the process proceeds to step S1310 or returns to step S1100 depending on the situation.
[0117]
In step S1310, the user of this apparatus gives an instruction for performing a data list editing operation to the instruction unit 115 using the data acquisition dialog 1100, for example.
In step S1400, the user of the apparatus causes the instruction unit 115 to issue a calibration information calculation command using, for example, the calibration information calculation button 1140.
[0118]
In step S1500, the user of this apparatus checks whether or not the calibration information calculation unit 113 has correctly calculated the calibration information. If it has been calculated correctly, the process proceeds to step S1600. If it has not been calculated, the process returns to step S1100.
In step S1600, the user of this apparatus checks whether the calibration information calculated by the calibration information calculation unit 113 is good. If the calibration information is good, the process proceeds to step S1700. If not, the process proceeds to step S1610 or returns to step S1100 depending on the situation.
[0119]
In step S1610, the user of this apparatus gives an instruction to edit the calibration information to the instruction unit 115 using the initial value setting dialog 1200, for example.
In step S1700, the user of the apparatus causes the instruction unit 115 to issue a file save command using the save submenu 1050, for example.
In step S1800, the user of the apparatus ends the apparatus using, for example, the end submenu 1060.
[0120]
<Second Embodiment>
FIG. 17 is a block diagram showing a schematic configuration in the second embodiment of the sensor calibration apparatus according to the present invention. In the first embodiment, the display unit 200 is provided outside the head mounting unit 300. In the second embodiment, the display unit 200 is included in the head mounting unit 300. Such a configuration is realized by using, as the display unit 200, a display device that can be worn on the head by a user of the device, such as an HMD. In the second embodiment, by using a display device that can be worn on the head, the user of this device performs calibration under the same conditions as the conditions for using an image display device that presents mixed reality. Is possible.
[0121]
<Third Embodiment>
FIG. 18 is a block diagram showing a schematic configuration in the third embodiment of the sensor calibration apparatus according to the present invention. In the third embodiment, the display unit 200 is mounted on the head by the display device provided outside the head-mounted unit 300 in the first embodiment and the user of the device in the second embodiment. Consists of both display devices. For example, the HMD generally has a lower resolution than a normal display device such as a CRT monitor or a liquid crystal monitor. When such a display device is used, it may be difficult for a user of the device to give an instruction to the device using the GUI described in the first embodiment. In such a case, for example, it is conceivable that the apparatus is used by two people.
[0122]
That is, the GUI described in the first embodiment is displayed on the display device (display unit in the first embodiment) provided outside the head-mounted unit 300 viewed by the first user of the apparatus. The display device (display unit in the second embodiment) worn by the second user of the device on the head has a variety of real images and real images excluding the GUI described in the first embodiment. A composite image in which information and CG are superimposed, a marker color extraction image, and the like are displayed. The operation of this apparatus is performed by the first user, and the adjustment of the positions and orientations of the imaging apparatus 302 and the receiver 301 is performed by the second user, so that the calibration work can be performed efficiently. .
[0123]
[Other Embodiments]
An object of the present invention is to supply a storage medium (or recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus. Needless to say, this can also be achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. In addition, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.
[0124]
Furthermore, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.
[0125]
When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowchart described above (shown in FIG. 15).
[0126]
【The invention's effect】
As described above, according to the present invention, the calibration information for converting the position and orientation of the imaging unit measured by the sensor into the position and orientation in the world coordinate system can be used more easily and using a special calibration instrument. And can be obtained with high accuracy.
[0127]
In particular, since the user can select a photographed image to be used when obtaining calibration information from a plurality of obtained photographed images, an inappropriate photographed image can be removed when obtaining calibration information.
[0128]
According to the invention of claim 4, since an error is displayed for each detected marker, an inappropriate photographed image can be appropriately selected.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a schematic configuration of a calibration apparatus according to a first embodiment of the present invention.
FIG. 2 is a block diagram illustrating a functional configuration of a calibration device according to the first embodiment.
FIG. 3 is a diagram illustrating a GUI presented by the calibration apparatus according to the first embodiment.
4 is a diagram illustrating a file menu in the GUI presented by the calibration apparatus according to the first embodiment. FIG.
FIG. 5 is a diagram for explaining a state in which a mark indicating image coordinates of a marker and additional information are superimposed on a real image and displayed on the GUI presented by the calibration apparatus according to the first embodiment.
FIG. 6 shows a GUI displayed by the calibration apparatus according to the first embodiment when a marker is not identified and a mark indicating the image coordinate of the marker and additional information are superimposed on the real image and displayed. FIG.
FIG. 7 is a diagram for explaining a state in which the world coordinates of a marker are drawn in CG and displayed superimposed on a real image on the GUI presented by the calibration apparatus according to the first embodiment.
FIG. 8 is a diagram illustrating a data acquisition dialog in the GUI presented by the calibration device according to the first embodiment.
FIG. 9 is a diagram for explaining a selection data operation area in the dialog shown in FIG.
FIG. 10 is a diagram illustrating an initial value setting dialog in the GUI presented by the calibration apparatus according to the first embodiment.
FIG. 11 is a diagram illustrating a detection parameter setting dialog in the GUI presented by the calibration apparatus according to the first embodiment.
12 is a diagram illustrating a detection parameter setting dialog in the GUI presented by the calibration apparatus according to the first embodiment. FIG.
13 is a diagram for explaining a state in which a color extraction result image is displayed in the image display area of the GUI in FIG. 3 in the calibration apparatus according to the first embodiment.
FIG. 14 is a diagram illustrating a procedure for automatically calculating a rough value of Local Transform using a sensor measurement value in the calibration device according to the first embodiment.
FIG. 15 is a flowchart showing a processing procedure of the sensor calibration apparatus according to the first embodiment.
FIG. 16 is a flowchart showing a procedure for a user of the sensor calibration apparatus to use the sensor calibration apparatus in the first embodiment.
FIG. 17 is a block diagram illustrating a schematic configuration of a sensor calibration apparatus according to a second embodiment.
FIG. 18 is a block diagram illustrating a schematic configuration of a sensor calibration apparatus according to a third embodiment.

Claims (7)

An information processing method for obtaining calibration information for converting a position and orientation of an imaging unit measured by a sensor into a position and orientation in a world coordinate system,
When photographing a plurality of captured images obtained by photographing a real space in which a plurality of markers having known world coordinates are arranged at a plurality of positions and orientations using the imaging unit, and each of the photographed images Get the measured value of the sensor of
Detecting the image coordinates of the marker included in the captured image,
In response to a user instruction, select a captured image to be used when calculating the calibration information from the acquired captured images,
The calibration information is calculated using the measured value of the sensor of the selected captured image, the image coordinates of the detected marker of the selected captured image, and the world coordinates of the detected marker. ,
The image obtained from the image coordinates of the marker detected from the captured image, the identification information of the detected marker, and the world coordinates of the detected marker using the calculated calibration information. An information processing method characterized by combining and displaying coordinates. Detection of the image coordinates of the marker is automatically performed by analyzing the captured image,
Furthermore, based on a user instruction using the displayed image, it is possible to delete image coordinates of the detected marker and add image coordinates of a marker included in the captured image. Item 3. The information processing method according to Item 1.   The information processing method according to claim 1, wherein the displayed captured image is an image selected from the plurality of captured images according to a user instruction.   The information according to any one of claims 1 to 3, further comprising displaying an error in the calibration information for each of the detected markers and an average value of the error for each of the detected markers. Processing method.   5. The information processing method according to claim 1, further comprising adjusting a detection condition used when detecting an image coordinate of a marker included in the captured image based on a user instruction. .   A program for causing a computer to execute the information processing method according to any one of claims 1 to 5. An information processing apparatus for obtaining calibration information for converting a position and orientation of an imaging unit measured by a sensor into a position and orientation in a world coordinate system,
When photographing a plurality of captured images obtained by photographing a real space in which a plurality of markers having known world coordinates are arranged at a plurality of positions and orientations using the imaging unit, and each of the photographed images Input means for inputting the measured value of the sensor of
Detecting means for detecting image coordinates of a marker included in the captured image;
In accordance with a user instruction, a selection unit that selects a captured image to be used when calculating the calibration information from the acquired plurality of captured images;
The calibration information is calculated using the measured value of the sensor of the selected captured image, the image coordinates of the detected marker of the selected captured image, and the world coordinates of the detected marker. A calculation means;
The image obtained from the image coordinates of the marker detected from the captured image, the identification information of the detected marker, and the world coordinates of the detected marker using the calculated calibration information. An information processing apparatus comprising: means for combining and displaying coordinates.

Images