JP2016065830A - Image processor and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for more surely and more easily collecting images taken of an indicator, which are used for determining a positional attitude of the indicator.SOLUTION: The attitude of an imaging device is changed within the possible range that the attitude can take. Images taken every time the imaging device changes the attitude are acquired, and information on indicators in the taken images is registered. If information on the same indicator is registered more than once, the positional attitude of the imaging device when the image of the indicator was taken is acquired. The positional attitudes of one or more indicators are calculated with the information on the indicators registered for the imaging devices and the acquired positional attitudes of the imaging devices.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for obtaining the position and orientation of an index placed in a real space.

  In recent years, research on mixed reality (MR) technology has been active. MR technology is a technology that seamlessly fuses a real space with a virtual space created by a computer.

  An image display device in MR technology is realized by a video see-through method or an optical see-through method. The video see-through method is a method for displaying a composite image in which an image in a virtual space generated according to the position and orientation of an imaging device is superimposed on a real space image captured by an imaging device such as a video camera. is there. An image in the virtual space is composed of a virtual object drawn by computer graphics, character information, and the like. The optical see-through method is a method of displaying an image of a virtual space generated according to the position and orientation of the observer's viewpoint on an optical see-through display mounted on the observer's head.

  MR technology includes various types of support, such as surgical support that superimposes the state of the body on the patient's body surface, architectural simulation that superimposes and displays a virtual building on a vacant land, and assembly support that superimposes and displays work procedures and wiring during assembly work. Application to the field is expected.

  One of the most important issues in MR technology is how to accurately align the real space and the virtual space, and many efforts have been made in the past. In the case of the video see-through method, the alignment problem in MR results in a problem of obtaining the position and orientation of the imaging device in the scene (that is, in the reference coordinate system defined in the scene). Similarly, the optical see-through method results in the problem of obtaining the observer's viewpoint or display position and orientation in the scene.

  The following methods are generally used to solve the former problem. That is, a plurality of indices are arranged or set in the scene, and the reference coordinate system is determined based on the correspondence between the projected position of the index in the image captured by the imaging device and the position of the index that is known information in the reference coordinate system. The position and orientation of the image pickup apparatus are obtained. Further, the following methods are generally used as a method for solving the latter problem. That is, an imaging device is mounted on a measurement object (that is, an observer's head or display), the position and orientation of the imaging device are obtained by the same method as the former, and the position and orientation of the measurement object are obtained based thereon. Is generally done.

  A method for obtaining the position and orientation of the imaging device based on the correspondence between the projected image of the index on the image and the three-dimensional position of the index has been proposed for a long time in the field of photogrammetry (Non-patent Document 1).

  Non-Patent Document 2 proposes the following. That is, the initial position is the position and orientation of the imaging device obtained based on the projected image of the index on the image, and the error between the actual observation position of the projected image of the index on the image and the calculated position of the projected image is The position and orientation of the imaging device are optimized by iterative calculation so as to be minimized. Here, the calculated position of the projected image is the position of the projected image calculated from the three-dimensional position of the index and the position and orientation of the imaging device.

  Conventionally, the position and orientation of an imaging device have been obtained based on an image captured by the imaging device by the above method.

  On the other hand, for example, those disclosed in Patent Documents 1 and 2 and Non-Patent Document 3 have been conventionally performed. In other words, a 6-DOF position / orientation sensor such as a magnetic sensor or an ultrasonic sensor is attached to the imaging device to be measured, and the position and orientation are measured by using the detection of the index by image processing as described above. ing. Since the output value of the sensor can be obtained stably although the accuracy varies depending on the measurement range, the method using both the sensor and the image processing can improve the robustness as compared with the method using only the image processing.

  In the technique disclosed in Patent Document 2, the position and orientation of the imaging device obtained from the 6-DOF position and orientation sensor are set as initial values, and an error between the observation position of the projected image of the index on the image and the calculation position is described above. Is obtained by iterative calculation to obtain the position and orientation of the imaging apparatus.

  In the alignment method using the index described above, the following information needs to be known in order to obtain the position and orientation in the reference coordinate system of the imaging device that is the measurement target. That is, in the case of a point-shaped index (point index) such as a circular region of the same color existing in the space, the position of the center of gravity in the reference coordinate system and the attitude of the index with respect to the reference coordinate system must be known. In the case of a polygonal index such as a triangle or a square, the position of the center of gravity in the reference coordinate system needs to be known. In the case of a polygonal index such as a square index, the index itself is often used as a reference for the coordinate system without providing a separate reference coordinate system. However, in the case of using a plurality of indices, the relationship between the positions and orientations of each other needs to be known, so that the reference coordinate system is still necessary.

  Measurement of the position and orientation of the index can be performed manually using a tape measure or a protractor or a surveying instrument. However, measurement using an image is performed because of problems with accuracy and labor. The position measurement of the point index can be performed by a method called a bundle adjustment method.

  The bundle adjustment method is the following method. That is, a plurality of point index images are captured by the imaging device, and the observation position of the projected image of the point index on the image is obtained. It is repeated so that the error (projection error) between the obtained observation position and the calculated position of the projected image of the point index calculated based on the three-dimensional position of the point index and the position and orientation of the imaging device is minimized. The correction of the position of the point index and the position / orientation of the imaging apparatus is repeatedly performed by calculation. Thereby, the position of the point index is obtained.

  Non-Patent Document 4 discloses a method for measuring the positions and orientations of a large number of square markers arranged in a three-dimensional space. In Non-Patent Document 4, a large number of images of a large number of square markers arranged in a three-dimensional space are captured, and the position and orientation of the imaging device that captures each image by iterative calculation so that the projection error is minimized, The position and orientation of the square index are obtained.

  In addition, an “index calibration tool” that calculates the position of an index using a plurality of images obtained by imaging the index arranged in the real space with a plurality of imaging devices has been proposed (non-patent document). Reference 5). By using such an index calibration tool, the position of the index in the real space can be easily measured.

JP 11-084307 A JP 2000-041173 A

RM Haralick, C. Lee, K. Ottenberg, and M. Nolle: “Review andanalysis of solutions of the three point perspective pose estimation problem”, Int'l. J. Computer Vision, vol.13, no.3, pp. 331-356, 1994. Kato, M. Billinghurst, Asano, Tachibana: “Augmented reality system based on marker tracking and its calibration”, Transactions of the Virtual Reality Society of Japan, vol.4, no.4, pp.607-616, 1999. A. State, G. Hirota, D. T. Chen, W. F. Garrett and M. A. Livingston: “Superior augmented reality registration by integrating landmark tracking and magnetic tracking”, Proc. SIGGRAPH’96, pp.429-438, 1996. G. Baratoff, A. Neubeck and H. Regenbrecht: “Interactive multimarker calibration for augmented reality applications”, Proc. ISMAR2002, pp.107-116, 2002. D. Kotake, S. Uchiyama, and H. Yamamoto: "A marker calibration method utilizing a priori knowledge on marker arrangement," Proc. 3rd IEEE / ACM Int'l Symp. On Mixed and Augmented Reality (ISMAR 2004), pp. 89-98, November 2004.

  In the above-described point index position measurement method using the bundle adjustment method, accurate measurement cannot be performed unless at least two images are captured for one index. In the case of a point index, when only one image is captured, the position in the depth direction of the imaging apparatus becomes indefinite simply by determining a straight line where the point index exists in the three-dimensional space. Therefore, information on whether or not the minimum image necessary for position measurement has been acquired is important. Furthermore, since it is necessary to capture two or more images for all the indices, this is a laborious operation for the user.

  In the conventional system, the user moves the imaging device and images the index. Therefore, the user needs to remember whether or not two or more images have already been captured for each index when imaging the index. When the number of indices increases, it is necessary to remember which index the user has imaged, and it takes time to actually perform imaging.

  The present invention has been made in view of such problems, and provides a technique for collecting a captured image of an index used for obtaining the position and orientation of the index more reliably and simply.

  According to one aspect of the present invention, a control unit that changes the posture of an imaging device within a range of postures that can be taken by an imaging device that captures a real space including one or more indices, and whenever the imaging device changes its posture The first acquisition unit that acquires the captured image of the real space that has been imaged, the registration unit that registers the information related to the index included in the captured image acquired by the first acquisition unit, and the same registration unit If the information is registered twice or more, the second acquisition unit that acquires the position and orientation of the imaging apparatus at the time of imaging the index, and the registration for each captured image acquired by the first acquisition unit And calculating means for calculating the position and orientation of each of the one or more indices using the information registered by the means and the position and orientation acquired by the second acquisition means.

  According to the configuration of the present invention, it is possible to collect a captured image of the index used for obtaining the position and orientation of the index more reliably and easily.

The block diagram which shows the function structural example of a system. The figure explaining the parameter | index 100. FIG. The block diagram which shows the hardware structural example of a computer. 5 is a flowchart of processing performed by the image processing apparatus 1; The figure which shows the structural example of a parameter | index management table. The figure explaining the change of a pan angle. The figure explaining the change of a tilt angle.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
In the present embodiment, an image processing apparatus that calculates the position and orientation of one or more indices arranged (or set) in the real space will be described. First, a functional configuration example of a system including an image processing apparatus according to the present embodiment will be described with reference to the block diagram of FIG. As shown in FIG. 1, the system according to the present embodiment includes an imaging unit 110, an imaging unit 111, and an image processing device 1.

  First, the imaging unit 110 will be described. The imaging unit 110 is an imaging device that captures an image of a real space where a plurality of indices 100 are arranged. The panning angle, tilting angle, and zooming (viewing angle) of the imaging unit 110 are controlled by the image processing apparatus 1, and the imaging unit 110 is based on the panning angle, tilting angle, and viewing angle controlled by the image processing apparatus 1. An image of the space is taken.

  The imaging unit 111 is a functional unit similar to the imaging unit 110, and captures an image of a real space where a plurality of indices 100 are arranged based on the pan angle, tilt angle, and angle of view controlled by the image processing apparatus 1. It is an imaging device.

  Next, the image processing apparatus 1 will be described. As shown in FIG. 1, the image processing apparatus 1 includes an image input unit 120, an index detection and identification unit 130, a data management unit 140, an operation unit 150, a data storage unit 160, a display unit 170, an index position and orientation calculation unit 180, and an imaging unit. A control unit 190.

  The image input unit 120 receives a captured image captured every time the imaging unit 110 and the imaging unit 111 change the pan angle, the tilt angle, and the angle of view, and the captured image is transmitted to the index detection identification unit 130 and the display unit 170. Send it out. As a result, the captured image by the imaging unit 110 and the captured image by the imaging unit 111 are displayed on the display screen of the display unit 170.

  When the index is included in the captured image received from the image input unit 120, the index detection / identification unit 130 identifies the index and sends the identification result to the data management unit 140 at the subsequent stage.

  The data management unit 140 manages identification result data received from the index detection and identification unit 130. When the data management unit 140 detects that the user inputs an “image storage” instruction using the operation unit 150, the data management unit 140 registers the identification result data received from the index detection / identification unit 130 in the data storage unit 160.

  In addition, the data management unit 140 compares the latest identification result data received from the index detection and identification unit 130 with the identification result data already registered in the data storage unit 160, and has been identified multiple times in the past. Information about the index is sent to the display unit 170. This process will be described in detail later.

  When the data management unit 140 receives notification from the imaging unit control unit 190 that imaging by the imaging unit 110 and the imaging unit 111 has been completed, the data management unit 140 receives the position and orientation and identification of the imaging unit 110 and the imaging unit 111 from the data storage unit 160. The resulting data is read out and sent to the index position / orientation calculation unit 180.

  The index position and orientation calculation unit 180 calculates the position and orientation of the index using the data received from the data management unit 140.

  In FIG. 1, the output position of the index position / orientation data obtained by the index position / orientation calculation unit 180 is not particularly illustrated, but the output destination is not the main point here. Not. However, in general, since the method of using the index position / orientation data obtained from the captured image is well known, the description and illustration are omitted here.

  The imaging unit control unit 190 controls the pan angle, tilt angle, and field angle of the imaging unit 110 and the imaging unit 111.

  Next, the index 100 will be described with reference to FIG. In the present embodiment, a square-shaped index (hereinafter referred to as a square index) as illustrated in FIG. As shown in FIG. 2A, the square index includes a boundary area (black frame) and an internal area for indicating the index area. The inner area is divided into 5 parts in the vertical and horizontal directions, and the inner 3 × 3 area represents the direction of the square index and the unique identifier (identification information). The four corners of the 3 × 3 area (area surrounded by a circle in FIG. 2A) are areas (orientation determination areas) representing the direction of the square index. Three of the four corners are painted black and the remaining one is painted white. This represents the direction of the square index. The identifier of the square index is represented by five areas (bit areas) excluding the four corners of the 3 × 3 area (areas indicated by diagonal lines in FIG. 2A). Each of the five areas can represent 5 bits, that is, 32 types of identifiers, assuming that 0 is painted white and 1 is painted black.

Further, the position and orientation of the square index are expressed by the position and orientation with respect to the reference coordinate system, as shown in FIG. The reference coordinate system is, for example, a coordinate system in which one point in the real space is an origin, and three axes orthogonal to each other at the origin are an x-axis, a y-axis, and a z-axis, respectively. In this embodiment, the position of the square index is the position of the center of the square (intersection of diagonal lines), and is represented by a three-dimensional vector t _{wm with} respect to the reference coordinate system. Assume that the posture of the square index is the posture of the coordinate system (index coordinate system) of the square index with respect to the reference coordinate system, as shown in FIG. Since the degree of freedom of the posture is 3, it is represented by a three-dimensional vector ω _wm . The index coordinate system is, for example, with the center of the square index as the origin, the direction along one side of the square index as the x axis, the direction along the side adjacent to the one side as the y axis, and the normal direction of the square index as z It is a coordinate system with axes.

  Next, the operation of the system according to the present embodiment will be described in more detail.

  As described above, the imaging unit 110 and the imaging unit 111 are real-space images in which a plurality of indices 100 (square indices) are arranged based on the pan angle, the tilt angle, and the angle of view controlled by the imaging unit control unit 190. Is taken regardless of the operation by the user. Hereinafter, as an example, the controllable range of the pan angle of the imaging unit 110 (imaging unit 111) is 360 degrees, and the controllable range of the tilt angle is 100 degrees.

  Image capturing by the image capturing unit 110 in such an example will be described. The following description is the same for the imaging unit 111. Here, the maximum value of the horizontal field angle α of the imaging unit 110 is αmax, the minimum value is αmin, the minimum value of the vertical field angle is βmin, and a = (αmax−αmin) / n (n is an integer of 2 or more). .

  First, the image capturing unit control unit 190 performs image capturing for each of pan angles = 0, α / 2, α,..., 360 while the horizontal angle of view α of the image capturing unit 110 is fixed to αmin and the tilt angle is fixed to 0. (FIG. 6). Thereby, the imaging unit 110 can capture an image of one round in the pan direction in a state where the horizontal angle of view α = αmin and the tilt angle = 0. The number of captured images obtained by this imaging is (720 / α). The imaging unit control unit 190 performs such an imaging operation for each of the horizontal angle of view α = αmin + a, αmin + 2a,..., Αmin + (n−1) a. By such an imaging operation, (720 / α) captured image sequences obtained with a horizontal angle of view α = αmin, (720 / α) captured image sequences obtained with a horizontal angle of view α = αmin + a,. (720 + α) captured image sequences obtained at an angle of view α = αmax, (n + 1) sets of captured image sequences are obtained.

  Further, the imaging unit control unit 190 causes the imaging unit 110 to perform an imaging operation for obtaining such (n + 1) sets of captured image sequences for each of the tilt angles = βmin / 2,... 7).

  Through the imaging control as described above, the imaging unit control unit 190 causes “horizontal angles α = αmin, αmin + a,..., Αmin + (n−1) a to be“ pan angle = 0, α / 2, α,. 360 ”can be performed for each of the tilt angles = 0, βmin / 2,..., 100, whereby the imaging unit 110 performs (n + 1) × 200 / βmin imaging. As a result, the same number of captured images are captured. As described above, this also applies to the imaging unit 111. However, the imaging unit 110 and the imaging unit 111 sequentially transmit the captured images captured whenever the posture is changed as described above to the subsequent image input unit 120.

  The image input unit 120 transmits the captured images sequentially transmitted from the imaging unit 110 and the captured images sequentially transmitted from the imaging unit 111 to the display unit 170 and also to the index detection identification unit 130. .

  As described above, when the index 100 is present in the captured image received from the image input unit 120, the index detection / identification unit 130 performs processing for identifying the index 100. In order to detect the index 100 as a square index from the captured image, the index detection identification unit 130 performs binarization and labeling of the connected area on the captured image, and the inner area surrounded by the black frame that is the boundary area of the square index. Extract. Further, the index detection / identification unit 130 performs straight line fitting on the inner area of the square index to obtain the image coordinates of the four vertices of the square of the inner area. Then, the square index on the captured image is converted to an orthographic image by two-dimensional projective transformation, and the identifier of the square index is read and identified by reading the orientation determination area and the bit area. The identification of the index 100 varies depending on what kind of index 100 is used, and the identification method of the index 100 may be appropriately handled according to the index 100.

  When the index 100 is included in the captured image, the index detection / identification unit 130 identifies the identifier of the imaging unit that captured the captured image from which the index 100 is extracted (the identifier of the imaging unit 110 or the imaging unit 111), A set (extracted index set) of the frame number of the captured image, the identifier of the index 100, and the image coordinates of the four corners of the inner area of the index 100 is obtained. The extraction index set is obtained for each index 100 in the captured image. As for the identifier of the imaging unit that captured the captured image, for example, the imaging unit transmits the captured image with its own identifier attached, and the index detection and identification unit 130 acquires the identifier from the captured image. For the frame number, a count value that is counted each time the image input unit 120 inputs a captured image of each frame may be used. Of course, it is not limited to the frame number, and any image may be used as long as it can uniquely identify the captured image of each frame.

  The index detection / identification unit 130 sends the extracted index set for each index 100 included in the captured image to the subsequent data management unit 140.

  When the data management unit 140 detects that the user has input an “image storage” instruction using the operation unit 150, the data management unit 140 stores the extracted index set for each index 100 detected from the captured image by the index detection / identification unit 130. Registered in the unit 160. That is, every time an instruction of “image storage” is input to the data storage unit 160, “the identifier of the imaging unit that captured the captured image from which the index 100 is extracted, the frame number of the captured image, and the identifier of the index 100” , A set of “image coordinates of the four corners of the internal region of the index 100” is registered.

  In the registration process, an index management table as illustrated in FIG. 5 is also created in the data storage unit 160. In the index management table, for each index 100, an identifier (index ID) of the index 100, whether the corresponding extracted index set is obtained by identifying the index 100 from the image, or manually input by the user Whether the index 100 is detected from each of the information (input type), the captured image by the imaging unit 110, and the captured image by the imaging unit 111 (imaging unit A and imaging unit B, respectively) and whether the position and orientation are obtained Register information etc.

  In FIG. 5, the identifier of the index is registered in the “index ID” column. In the column of “input type”, whether the index of the corresponding identifier (the index of the identifier registered in the same row) is obtained from the captured image (automatic detection) or manually registered by the user (user Information indicating whether it is a definition). In the column “imaging unit A”, the number of times that the index of the corresponding identifier has been detected in the past from the captured image by the imaging unit 110 is registered. In the column “imaging unit B”, the number of times the index of the corresponding identifier has been detected in the past from the captured image by the imaging unit 111 is registered. In the “calibration state” column, information indicating whether or not the position and orientation of the index of the corresponding identifier has been obtained is registered.

  For example, in FIG. 5, the extracted index set for the index with index ID (identifier) = 1 is the one manually registered in the data storage unit 160 by the user, and the index is once from the image captured by the imaging unit 110 in the past. The index has been detected and has been detected once from the image captured by the imaging unit 111 in the past, and the index is a reference index whose position and orientation are known.

  That is, when the data management unit 140 registers the extracted index set (target extracted index set) of the index detected from the image captured by the imaging unit 110 in the data storage unit 160, first, the identifier in the target extracted index set It is checked whether an extracted index set including the same identifier is registered in the index management table. If it is not registered as a result of the check, the identifier in the attention extraction index set is registered in the last line of “index ID”, and “automatic detection” is registered in the last line of “input type”. , “1” is registered in the last row of “imaging unit A”, “0” is registered in the last row of “imaging unit B”, and “unestimated” is registered in the last row of “calibration state” To do. Of course, if the attention extraction index set is an extraction index set of indices detected from the captured image by the imaging unit 111, “0” is registered in the last row of “imaging unit A”, and “imaging unit B” is registered. "1" is registered in the last line of "." On the other hand, if it is registered as a result of the check, 1 is added to the numerical value in the column of “imaging part A” in the row in which the identifier in the attention extraction index set is registered. Of course, when the attention extraction index set is an extraction index set of an index detected from a captured image by the imaging unit 111, a column of “imaging unit B” in a row in which an identifier in the attention extraction index set is registered. Add 1 to the numerical value at.

  If both the value in the “imaging unit A” column and the value in the “imaging unit B” column are equal to or greater than a threshold (for example, 1), the position and orientation of the imaging unit 110 and the imaging unit 111 are obtained and the data storage unit 160 In addition to storing, “roughly calculated” is registered in the corresponding “calibration state” column.

  Further, when the data management unit 140 registers the attention extraction index set in the data storage unit 160, when the extraction index set including the same identifier as the identifier in the attention extraction index set is registered in the index management table, The following processing is performed. In other words, the information registered in the index management table shown in FIG. For example, when the identifier in the attention extraction index set is 12, “input type” = “automatic detection”, “imaging unit A” = “3”, “imaging unit B” corresponding to the index ID = 12. “1” and “calibration state” = “roughly calculated” are additionally displayed on the display screen of the display unit 170.

  Note that when the imaging unit control unit 190 detects that the imaging unit 110 and the imaging unit 111 have taken (n + 1) × 200 / βmin times of imaging, the imaging unit control unit 190 notifies the data management unit 140 to that effect. In other words, when the imaging unit 110 and the imaging unit 111 complete imaging of all captured images, the imaging unit control unit 190 notifies the data management unit 140 to that effect.

  Upon receiving such a notification, the data management unit 140 sends the extracted index set corresponding to each identifier registered in the index management table and the positions and orientations of the imaging unit 110 and the imaging unit 111 from the data storage unit 160. It is read out and sent to the index position / orientation calculation unit 180. The index position and orientation calculation unit 180 calculates the position and orientation of each index 100 in the reference coordinate system using the position and orientation of the imaging unit 110 and the imaging unit 111 and each extracted index set. The calculation method will be described later.

  Next, processing performed by the image processing apparatus 1 for calculating the position and orientation of each index 100 arranged in the real space in the reference coordinate system will be described with reference to FIG. 4 showing a flowchart of the processing. .

<Step S4000>
The image capturing unit control unit 190 sets the horizontal angle of view, tilt angle, and pan angle of the image capturing unit 110 and the image capturing unit 111 to perform image capturing. As described above, the initial value of the horizontal angle of view is αmin, the initial value of the tilt angle is 0, and the initial value of the pan angle is 0. Each time the processing in this step is executed, the pan angle, tilt angle, and angle of view are changed as described above, whereby the horizontal angle of view α = αmin, αmin + a,..., Αmin + (n−1) a It is realized that “image capturing for each of pan angles = 0, α / 2, α,..., 360” is performed for each of tilt angles = 0, βmin / 2,.

<Step S4010>
The image input unit 120 sends out images captured by the imaging unit 110 and the imaging unit 111 to the display unit 170 and the index detection / identification unit 130, respectively.

<Step S4020>
The index detection / identification unit 130 determines whether or not the index 100 is present in the captured image received from the image input unit 120. As a result of this determination, if it exists, the process proceeds to step S4030. If it does not exist, the process returns to step S4000.

<Step S4030>
The index detection / identification unit 130 performs a process of identifying the index 100 present in the captured image received from the image input unit 120 to obtain an extracted index set of the index 100, and the extracted index set is a subsequent data management unit. 140. When the data management unit 140 detects that the user inputs an “image storage” instruction using the operation unit 150, the data management unit 140 registers the extracted index set of the index 100 in the data storage unit 160.

<Step S4040>
When the data management unit 140 detects that the user inputs an “image save” instruction using the operation unit 150, the extracted index set including the same identifier as the identifier in the extracted index set of the index 100 is already index management. It is determined whether it is registered in the table. As a result of this determination, if registered, the process proceeds to step S4060, and if not registered, the process proceeds to step S4050.

<Step S4050>
The data management unit 140 registers the identifier in the extracted index set of the index 100 in the last line of “index ID” in the index management table, and registers “automatic detection” in the last line of “input type”. Here, the data management unit 140 refers to the “identifier of the imaging unit that captured the captured image from which the index 100 is extracted” in the extracted index set of the index 100, and captures the captured image from which the index 100 is extracted. It is determined whether the imaging unit is the imaging unit 110 or the imaging unit 111. When the index 100 is detected from the image captured by the imaging unit 110, “1” is registered in the last row of “imaging unit A”, and “1” is registered in the last row of “imaging unit B”. 0 ”is registered. On the other hand, when the index 100 is detected from an image captured by the imaging unit 111, “0” is registered in the last row of “imaging unit A”, and “0” is registered in the last row of “imaging unit B”. 1 ”is registered. Further, “unestimated” is registered in the last line of “calibration state”.

<Step S4060>
The data management unit 140 refers to the “identifier of the imaging unit that captured the captured image from which the index 100 is extracted” in the extracted index set of the index 100, and the imaging unit that captured the captured image from which the index 100 is extracted It is determined which is the imaging unit 110 or the imaging unit 111.

  When the index 100 is detected from an image captured by the imaging unit 110, in the column of “imaging unit A” in the row in which the identifier in the extracted index set of the index 100 is registered in the index management table. Add 1 to the number. On the other hand, when the index 100 is detected from an image captured by the imaging unit 111, in the column of “imaging unit B” in the row in which the identifier in the extracted index set of the index 100 is registered in the index management table. Add 1 to the number.

<Step S4070>
The data management unit 140 has both the value in the column “imaging unit A” and the value in the column “imaging unit B” in the row registered in step S4050 or the value added in step S4060 are both equal to or greater than the threshold. Determine whether or not. As a result of this determination, if both are greater than or equal to the threshold, the process proceeds to step S4080, and if either is less than the threshold, the process proceeds to step S4090.

<Step S4080>
The data management unit 140 acquires initial positions and orientations of the imaging unit 110 and the imaging unit 111. Various methods for acquiring the initial position and orientation of the imaging unit 110 and the imaging unit 111 are conceivable, and the method is not limited to a specific acquisition method.

  For example, it may be obtained by using a known method such as a DLT (Direct Linear Transform) method using a captured image obtained by capturing an index whose position and orientation are known in the reference coordinate system. Further, a six-degree-of-freedom position / orientation sensor such as a magnetic type, an optical type, or an ultrasonic type may be attached to the imaging unit 110 and the imaging unit 111 and may be obtained from a measurement value obtained by the sensor.

  The data management unit 140 saves the initial positions and orientations of the imaging unit 110 and the imaging unit 111 acquired in this way in the data management unit 140, and performs numerical addition in the line registered in step S4050 or in step S4060. Change the “calibration state” in the line to “roughly calculated”.

<Step S4090>
When an extracted index set including the same identifier as the identifier in the extracted index set of the index 100 is already registered in the index management table, the data management unit 140 uses the index shown in FIG. Information registered in the management table is sent to the display unit 170.

<Step S4100>
The imaging unit control unit 190 determines whether (n + 1) × 200 / βmin times of imaging have been performed by the imaging unit 110 and the imaging unit 111. As a result of the determination, if (n + 1) × 200 / βmin times of imaging are performed by the imaging unit 110 and the imaging unit 111, the process proceeds to step S4110. If not, the process returns to step S4000.

  When the imaging unit control unit 190 detects that the imaging unit 110 and the imaging unit 111 have performed (n + 1) × 200 / βmin imaging, the imaging unit control unit 190 notifies the data management unit 140 to that effect.

<Step S4110>
When the data management unit 140 receives the above notification from the imaging unit control unit 190, the “initial position and orientation of the imaging unit 110 and the imaging unit 111 stored in the data storage unit 160 in step S4080” registered in the index management table. , “Extracted index set corresponding to each identifier” is read from the data storage unit 160 and sent to the index position and orientation calculation unit 180.

  The index position and orientation calculation unit 180 calculates the position and orientation of each index 100 in the reference coordinate system using the position and orientation of the imaging unit 110 and the imaging unit 111 received from the data management unit 140 and each extracted index set.

Let N be the number of indices for obtaining the position and orientation, and M be the number of captured images. M may be the maximum value among the image numbers in each extraction index set stored in the data storage unit 160. Further, ai represents a 6-dimensional vector representing the position and orientation of the index i (i = 1, 2,..., N), and 6 represents the position and orientation of the imaging unit that images the image j (j = 1, 2,..., M). The dimension vector is represented as sj. The 6-dimensional vector a representing the position and orientation of the index has a position t _wm = [t _wm ^x t _wm ^y t _wm ^z ] ^{t in} the reference coordinate system of the index, and a posture ω _wm = [ω _wm ^x ω _wm ^{y with} respect to the reference coordinate system. ω _wm ^z ] ^t is a vector composed of each component, and can be expressed as a = [t _wm ^x t _wm ^y t _wm ^z ω _wm ^x ω _wmy ^y ω _wm ^z ] ^t . _Assuming that the position of the imaging unit in the reference coordinate system is t _wc = [t _wc ^x t _wc ^y t _wc ^z ] ^t and the attitude ω _wc = [ω _wc ^x ω _wc ^y ω _wc ^z ] ^t with respect to the reference coordinate system, the imaging unit The six-dimensional vector s indicating the position and orientation of can be expressed as s = [t _wc ^x t _wc ^y t _wc ^z ω _wc ^x ω _wc ^y ω _wc ^z ] ^t .

The coordinates x _m ^k = [x _m ^k y _m ^k z _m ^k ] ^t (k = 1, 2, 3, 4) in the index coordinate system of each vertex of the square index are ^{expressed as} follows: For example, it can be expressed as (Equation 1), and when the length of one side of the square is known, it becomes a known value.

Position and orientation with respect to the reference coordinate system t _wm, camera coordinate x _c in the imaging unit position and orientation is t _wc, omega _wc index coordinate of the index is omega _wm is relative to the reference coordinate system of the vertex is x _m is (Formula 2 ).

However, R (ω _wc ) and R (ω _wm ) are 3 × 3 rotation transformation matrices representing postures represented by ω _wc and ω _wm , respectively. Further, a position u = [projected onto an image by a pinhole camera (imaging unit 110 and imaging unit 111 in the present embodiment) whose focal length is f at a point where the camera coordinates are Xc = [xc yc zc] ^t . u _x u _y ] ^t can be expressed as (Equation 3).

That is, u is a function of the index position and orientation t _wm and ω _wm , and the imaging unit position and orientation t _wc and ω _wc . Therefore, the two-dimensional vector u _{i, j, k} representing the position of the projected image on the image j of the vertex k (k = 1, 2, 3, 4) of the index i is expressed as follows: _{This is} a function of _i and s _j .

An observation position on the actual image of the projected image on the image j vertex k of the index i u _{i, j,} expressed as _{k ^{^,}} u _{i, j,} and _k ^{^,} observation position u _{i, j, k} ^{^} and a _i, u is computed from s _{j i, j,} the error Delta] u _i with _{k, j, k (= u} i, j, k ^ -u i, j, k) is the (formula 5) Thus, a first order approximation can be made.

∂u _{i, j, k} / ∂a _i , ∂u _{i, j, k} / ∂s _j are partial differentials of u _{i, j, k} by the components of a _i and s _j in (Equation 4), respectively. It is a Jacobian matrix in which partial differential coefficients are arranged.

(Equation 5) is set for each vertex of each index detected and identified in each image, and all Δa _i (i = 1, 2,..., N), Δs _j (j = 1, 2,..., M) Can be obtained as simultaneous equations as common unknown variables, and Δa _i (i = 1, 2,..., N) and Δs _j (j = 1, 2,..., M) can be obtained. Initial values are given to a _i and s _j in advance, Δa _i (i = 1, 2,..., N) and Δs _j (j = 1, 2,..., M) are obtained, and a _i = The index position / orientation a _i (i = 1, 2,..., N) is obtained by repeating the operation of correcting a _i + Δa _i and s _j = s _j + Δs _j .

  Then, the index position / orientation calculation unit 180 outputs information (position / orientation information) indicating the obtained position / orientation to an appropriate output destination. Update “calibration state” to “optimum calculated”.

<Step S4120>
The data management unit 140 updates the “calibration state” of the index for which the position / orientation has been obtained to “optimally calculated” among the “calibration states” currently displayed on the display screen of the display unit 170. In the above calculation process, “error” is registered in the “calibration state” for an index for which the position and orientation cannot be obtained due to the result of calculation not being converged or the erroneous recognition of the index.

  As described above, according to the present embodiment, the captured image of the real space including one or more indices captured by the imaging device each time the orientation of the imaging device is changed within the range of the orientation that the imaging device can take. Since it can be acquired, the position and orientation of the index included in the captured image can be calculated. It should be noted that an index whose position and orientation cannot be obtained because the calculation result has not converged can be obtained by re-imaging in the same manner after re-imaging only the index and its surrounding indices.

  The index 100 is not limited to the square index as described above, but may be another type of index, for example, a point shape index. In this case, the index identification method is appropriately changed according to the type of the index. It should be noted that index identification according to the type of each index can be performed using a known technique.

  In the present embodiment, the number of image capturing units that capture the real space is set to 2 (the image capturing unit 110 and the image capturing unit 111). However, the number of imaging units is not limited to 2, and may be 1 or 3 or more.

  In the present embodiment, the data management unit 140 performs processing such as storage in the data storage unit 160 when the user inputs an “image storage” instruction using the operation unit 150. However, this instruction is not limited to being input by the user by operating the operation unit 150, and may be input by the imaging unit control unit 190. Further, the data management unit 140 may operate without inputting such an instruction.

  1 may be housed in one apparatus as shown in FIG. 1, or may be distributed among a plurality of apparatuses. In addition, the information described as being displayed on the display unit 170 in the above description is not limited to displaying all of the information, and a part of the information may be displayed. Information to be displayed may be set in advance, or may be selected by the user by operating the operation unit 150.

  Further, the information to be included in the extracted index set is not limited to the information described above, and if the information necessary for calculating the position and orientation of the corresponding index is included, the extracted index set Any information may be included in.

  That is, this embodiment is only an example of the following structure including the modification. That is, the orientation of the imaging device is changed (controlled) within the range of orientations that can be taken by the imaging device that captures the real space including one or more indices, and the captured real space is captured every time the imaging device changes the orientation. A captured image is acquired. Then, information related to the index included in the acquired captured image is registered, and when information about the same index is registered twice or more, the position and orientation of the imaging device at the time of imaging the index are acquired. Then, the position and orientation of each of the one or more indices are calculated using the index information registered for each captured image and the acquired position and orientation of the imaging apparatus.

[Second Embodiment]
The functional units constituting the image processing apparatus 1 shown in FIG. 1 may be configured by hardware, and the functional units other than the image input unit 120, the data storage unit 160, the operation unit 150, and the display unit 170 are included. You may comprise with a software (computer program). In the latter case, this software is executed on a computer such as a PC (Personal Computer), so that the computer executes each of the processes described as being performed by the image processing apparatus 1. Become. A hardware configuration example of a computer applicable to the image processing apparatus 1 will be described with reference to the block diagram of FIG.

  The CPU 301 executes various processes using computer programs and data stored in the RAM 302 and the ROM 303, thereby controlling the operation of the entire computer and performing each process described above as being performed by the image processing apparatus 1. Run.

  The RAM 302 has an area for storing computer programs and data loaded from the external storage device 306, and captured images received from the imaging unit 110 and the imaging unit 111 via the I / F (interface) 307. Further, the RAM 302 includes a work area that is used when the CPU 301 executes various processes. That is, the RAM 302 can provide various areas as appropriate.

  The ROM 303 stores a boot program, setting data of the computer, and the like.

  The operation unit 304 functions as the operation unit 150 and includes a keyboard and a mouse. An operator of this computer can input various instructions to the CPU 301 by operating the operation unit 304.

  The display unit 305 functions as the display unit 170 and includes a CRT, a liquid crystal screen, or the like. The display unit 305 can display the result of processing by the CPU 301 using an image or text.

  The external storage device 306 is a large capacity information storage device represented by a hard disk drive device, and functions as the data storage unit 160. The external storage device 306 stores an OS (operating system) and computer programs and data for causing the CPU 301 to execute the above-described processes performed by the image processing apparatus 1. This computer program is a computer program for causing the CPU 301 to execute the processes described above as performed by the functional units of the imaging unit control unit 190, the index detection and identification unit 130, the data management unit 140, and the index position and orientation calculation unit 180. It is included. Further, this data includes information handled as known information in the above processing. Computer programs and data stored in the external storage device 306 are appropriately loaded into the RAM 302 under the control of the CPU 301 and are processed by the CPU 301.

  The I / F 307 functions as the image input unit 120 and functions as an interface for connecting the imaging unit 110 and the imaging unit 111 to the computer. Data of images captured by the imaging unit 110 and the imaging unit 111 is transmitted to the RAM 302 and the external storage device 306 via the I / F 307. A bus 308 connects the above-described units.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

190: Imaging unit control unit 120: Image input unit 130: Index detection / identification unit 140: Data management unit 180: Index position / orientation calculation unit

Claims (8)

Control means for changing the posture of the imaging device within a range of postures that can be taken by the imaging device that images the real space including one or more indices;
First acquisition means for acquiring a captured image of the real space imaged each time the imaging device changes posture;
Registration means for registering information on the index included in the captured image acquired by the first acquisition means;
A second acquisition unit configured to acquire the position and orientation of the imaging apparatus at the time of imaging the index when the registration unit registers information about the same index twice or more;
Using the information registered by the registration unit for each captured image acquired by the first acquisition unit and the position and orientation acquired by the second acquisition unit, the position and orientation of the one or more indices are obtained. An image processing apparatus comprising: a calculating means for calculating.   The calculating means is configured to acquire each of the one or more indices after the first acquisition means has acquired the captured image captured by the imaging apparatus in each attitude within the range of attitudes that the imaging apparatus can take. The image processing apparatus according to claim 1, wherein the position and orientation is calculated.   The first acquisition unit acquires a captured image of a real space including one or more indices, which is captured whenever the imaging device changes one or more of a pan angle, a tilt angle, and a field angle. The image processing apparatus according to claim 1 or 2.   The registration means registers a set of an identifier of an index included in the captured image and an image coordinate of the index in the captured image as information related to the index. The image processing apparatus according to any one of the above. Furthermore,
The image processing apparatus according to claim 1, further comprising a display unit that displays the captured image acquired by the first acquisition unit.   The image processing apparatus according to claim 5, wherein the display unit further displays the number of times of registration when the registration unit registers information about the same index more than once. An image processing method performed by an image processing apparatus,
A control step in which the control means of the image processing device changes the posture of the imaging device within a range of postures that can be taken by the imaging device that images a real space including one or more indices;
A first acquisition step in which a first acquisition unit of the image processing apparatus acquires a captured image of the real space captured each time the imaging apparatus changes its orientation;
A registration step in which the registration unit of the image processing apparatus registers information about the index included in the captured image acquired in the first acquisition step;
A second acquisition step of acquiring a position and orientation of the imaging apparatus at the time of imaging the index when the second acquisition unit of the image processing apparatus registers information about the same index twice or more in the registration step; When,
The calculation means of the image processing apparatus uses the information registered in the registration step for each captured image acquired in the first acquisition step and the position and orientation acquired in the second acquisition step. An image processing method comprising: a calculation step of calculating the position and orientation of each of the above indices.   A computer program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 6.

Images