JP2006267879A - Image processing method, image processing apparatus and marker - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique for reading identification information written on an index even in observing the identification information from either of a plurality of prescribed viewing points, for example, in observing the information from an oblique direction, and then, for rightly discriminating the read identification information. <P>SOLUTION: The image in a real cavity is obtained, where a plurality of indexes having a lenticular lens for optically changing the observation image of an identifier are arranged so that the observation images become almost the same, even when the identifier is viewed from a plurality of prescribed viewing points and the projected index image is detected in the image (S201), then, the index is discriminated by reading the identifier from the projected image with the detected index (S204). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing technique related to an image of a real space in which a plurality of indices are arranged.

  In recent years, research on mixed reality (MR) technology has been actively conducted. MR technology is a technology that seamlessly fuses real space and virtual space created by computers. Among MR technologies, Augmented Reality (also called AR, augmented reality, augmented reality) technology that displays virtual space superimposed on real space is attracting particular attention.

  An AR image display device uses a virtual space (virtual object or character information drawn by computer graphics) generated in accordance with the position and orientation of the imaging device in an image of a real space taken by an imaging device such as a video camera. A virtual space generated according to the position and orientation of the observer's viewpoint on a video see-through method that displays a composite image with a superimposed image of the image or the like, or an optical see-through display mounted on the observer's head This is realized by an optical see-through method for displaying the image.

  AR technology is applied to various fields such as surgery support that superimposes the state of the body on the patient's body surface, architectural simulation that superimposes a virtual building on a vacant lot, and assembly support that superimposes and displays work procedures during assembly work. There is expected.

  One of the most important issues in AR technology is how to accurately align the real space and the virtual space, and many efforts have been made. In the case of the video see-through method, the alignment problem in the AR results in a problem of obtaining the position and orientation of the imaging device in the scene (that is, in the world coordinate system). Similarly, the optical see-through method results in the problem of obtaining the observer's viewpoint or display position and orientation in the scene.

  As a method for solving the former problem, the position and orientation of the imaging apparatus in the scene are determined by arranging or setting a plurality of indices in the scene and detecting the coordinates of the projected image of the index in the image captured by the imaging apparatus. It is generally done to ask for. As a method for solving the latter problem, an imaging device is attached to a measurement object (that is, an observer's head or display), and the position and orientation of the imaging device are obtained by the same method as the former, and measurement is performed based on the obtained device. In general, the position and orientation of an object are obtained.

  A conventional example of a position and orientation measurement apparatus that measures the position and orientation of an imaging apparatus by detecting an index from an image captured by the imaging apparatus will be described with reference to FIG. FIG. 1 is a block diagram showing a functional configuration of a conventional position and orientation measurement apparatus.

  As shown in the figure, the conventional position / orientation measurement apparatus 100 includes an index detection unit 110 and a position / orientation calculation unit 120, and an imaging device 130 is connected to the index detection unit 110.

  Further, at a plurality of positions in the real space, a plurality of indices Qk (k = 1, 2,..., K1) whose positions in the world coordinate system are known as indices for photographing by the imaging device 130. Is arranged. In FIG. 1, four indexes Q1, Q2, Q3, and Q4 are arranged, and three of them, Q1, Q3, and Q4, are included in the field of view of the imaging device 130.

  The index Qk is an index that can detect the image coordinates of the projected image on the captured image and can identify which index, for example, circular markers having different colors. Any form may be used.

  An image output from the imaging device 130 is input to the index detection unit 110. The index detection unit 110 detects the image coordinates of the index Qk photographed on the image input from the imaging device 130. For example, when each of the indices Qk is composed of markers having different colors, an area corresponding to each marker color is detected from the input image, and the position of the center of gravity is set as a detected coordinate of the index.

Furthermore, the index detection unit 110 outputs the detected image coordinates u ^Qkn of each index ^Qkn and its identifier kn to the position / orientation calculation unit 120. Here, n (= 1,..., N) is a variable representing the serial number of the detected index, and N represents the total number of detected indices. For example, in the case of FIG. 1, N = 3, and the index detection unit 110 positions the identifiers k1 = 1, k2 = 3, k3 = 4 and the corresponding image coordinates u ^Qk1 , u ^Qk2 , u ^Qk3 . Output to the posture calculation unit 120.

The position and orientation calculation unit 120, based on the correspondence between the image coordinates u ^Qkn of the index Qkn of each detected, and advance the held index Qkn world coordinates as known information, the position and orientation of the imaging device 130 Is calculated. A method for calculating the position and orientation of an imaging device from a set of world coordinates and image coordinates of an index has been proposed for a long time in the field of photogrammetry (for example, see Non-Patent Document 1 and Non-Patent Document 2). The position / orientation calculation unit 120 calculates the position and orientation of the imaging device 130 by the method described in Non-Patent Document 1, for example.

  Conventionally, the position and orientation of the imaging device have been acquired based on the image captured by the imaging device by the above method.

  On the other hand, as disclosed in, for example, Patent Documents 1 and 2 and Non-Patent Document 3, a 6-DOF position / orientation sensor such as a magnetic sensor or an ultrasonic sensor is attached to an imaging apparatus that is a measurement object, and the above-described image The position and orientation are also measured by using in combination with detection of an index by processing. In this method, it is possible to avoid a solution greatly deviating from the true solution by performing processing in a framework of correcting the measurement value obtained by the sensor using image information, and stable operation can be expected.

  By the way, when calculating the position and orientation of the imaging apparatus using the index as described above, it is necessary to correctly identify the index identifier and know exactly which index is the index in the image.

  In the case where circular markers having different colors as described above are used as an index and there are markers with similar colors, they cannot be uniquely identified due to illumination conditions or the like. In addition, even when the index is tracked using the image coordinates of the index in the previous frame or the image coordinates of the index predicted using the measurement values of the 6-DOF position and orientation sensor, the position or orientation of the imaging apparatus varies. Cannot be uniquely identified when the number of indicators is large or the number of indicators is large.

On the other hand, by using a marker on which a two-dimensional pattern is drawn as an index, the number of usable indices is increased, and by reading the two-dimensional pattern of the index projected on the photographed image, only the photographed image is obtained. In other words, the identifier of the index is uniquely identified (see, for example, Non-Patent Document 4 and Non-Patent Document 5). In Non-Patent Document 4, a two-dimensional matrix code is used as a two-dimensional pattern, and identification of an identifier is performed by directly reading bit information from a pattern projected on an image. In Non-Patent Document 5, an index is identified by identifying an arbitrary two-dimensional pattern by pattern matching.
Japanese Patent Laid-Open No. 11-084307 JP 2000-041173 A RM Haralick, C. Lee, K. Ottenberg, and M. Nolle: Review and analysis of solutions of the three point perspective pose estimation problem, International Journal of Computer Vision, vol.13, no.3, pp.331-356, 1994. DG Lowe: Fitting parameterized three-dimensional models to images, IEEE Transactions on PAMI, vol.13, no.5, pp.441-450, 1991. A. State, G. Hirota, DT Chen, WF Garrett and MA Livingston: Superior augmented reality registration by integrating landmark tracking and magnetic tracking, Proc.SIGGRAPH '96, pp.429-438, 1996. Junichi Kakimoto: Augmented Reality Construction Method Using 2D Matrix Code, Interactive System and Software IV, Modern Science, 1996. Kato Hirokazu, M. Billinghurst, Asano Koichi, Tachibana Keihachiro: Augmented Reality System Based on Marker Tracking and Its Calibration, Transactions of the Virtual Reality Society of Japan, vol.4, no.4, pp.607-616, 1999 .

  When an index on which a two-dimensional pattern is drawn as described above is observed from an oblique direction (when the angle formed by the normal vector of the index and the line-of-sight vector is close to 90 °), the index is displayed on the image plane due to the influence of projective transformation. There is a problem that the image is crushed and observed in the depth direction, and the identification cannot be performed without correctly reading the two-dimensional pattern.

  Further, even when a two-dimensional pattern can be read, there is a problem that a correct position and orientation cannot be obtained in the position and orientation calculation because incorrect reading is performed and an index is incorrectly identified.

  Further, there was no reference for determining an index that is actually used for position and orientation calculation from among the observed indices.

  The present invention has been made in view of the above problems, and even if the identification information written on the index is observed from any of a plurality of predetermined viewpoints, for example, when observed from an oblique direction, the identification information is used. It is an object of the present invention to provide a technique for correctly identifying read and read identification information.

  Furthermore, even if the identification information is erroneously identified, the error can be detected.

  Furthermore, an index that is actually used for position / orientation calculation is determined based on the priority from among a large number of observed indices.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

In other words, an index having a configuration in which identification information is written and an observation image of the identification information is optically changed so that the identification information is substantially the same even if the identification information is observed from any of a plurality of predetermined viewpoints. An acquisition step of acquiring an image of a physical space in which a plurality of are arranged,
A detection step of detecting a projected image of the index in the image acquired in the acquisition step;
An identification step of identifying the index by reading the identification information from the projected image of the index detected in the detection step.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

In other words, the first identification information and the second identification information are written, and the first identification information should be substantially the same when viewed from any of a plurality of predetermined viewpoints. Observation of the second identification information in order to vary the image of the second identification information in accordance with a configuration for optically changing the observation image of the second identification information and an observation region where a viewpoint for observing the second identification information exists. An acquisition step of acquiring an image of a real space in which a plurality of indicators having a configuration for optically changing an image is arranged;
A detection step of detecting a projected image of the index in the image acquired in the acquisition step;
An identification step for identifying the index by reading the first identification information from the projected image of the index detected in the detection step;
An estimation step of estimating the position of the viewpoint by reading the second identification information from the projected image of the index detected in the detection step;
Using the position of the index on the image seen from the viewpoint of the position estimated in the estimation process and the position of the index detected in the detection process, a check process for checking the identification process in the identification process for the index; It is characterized by providing.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

In other words, the first identification information and the second identification information are written, and the first identification information should be substantially the same when viewed from any of a plurality of predetermined viewpoints. Observation of the second identification information in order to vary the image of the second identification information in accordance with a configuration for optically changing the observation image of the second identification information and an observation region where a viewpoint for observing the second identification information exists. An acquisition step of acquiring an image of a real space in which a plurality of indicators having a configuration for optically changing an image is arranged;
A detection step of detecting a projected image of the index in the image acquired in the acquisition step;
An identification step for identifying the index by reading the first identification information from the projected image of the index detected in the detection step;
A setting step of reading the second identification information from the projected image of the index detected in the detection step, and setting a priority for the index detected in the detection step based on the read second identification information. It is characterized by that.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

In other words, an index having a configuration in which identification information is written and an observation image of the identification information is optically changed so that the identification information is substantially the same even if the identification information is observed from any of a plurality of predetermined viewpoints. Acquisition means for acquiring an image of a real space in which a plurality of are arranged,
Detecting means for detecting a projected image of the index in the image acquired by the acquiring means;
And identifying means for identifying the index by reading the identification information from the projected image of the index detected by the detecting means.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

In other words, the first identification information and the second identification information are written, and the first identification information should be substantially the same when viewed from any of a plurality of predetermined viewpoints. Observation of the second identification information in order to vary the image of the second identification information in accordance with a configuration for optically changing the observation image of the second identification information and an observation region where a viewpoint for observing the second identification information exists. An acquisition means for acquiring an image of a real space in which a plurality of indicators having a configuration for optically changing an image is arranged;
Detecting means for detecting a projected image of the index in the image acquired by the acquiring means;
Identification means for identifying the index by reading the first identification information from the projected image of the index detected by the detection means;
Estimating means for estimating the position of the viewpoint by reading the second identification information from the projected image of the index detected by the detecting means;
Checking means for checking identification processing by the identification means for the index using the position of the index seen from the viewpoint of the position estimated by the estimation means and the position of the index detected by the detection means; It is characterized by providing.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

In other words, the first identification information and the second identification information are written, and the first identification information should be substantially the same when viewed from any of a plurality of predetermined viewpoints. Observation of the second identification information in order to vary the image of the second identification information in accordance with a configuration for optically changing the observation image of the second identification information and an observation region where a viewpoint for observing the second identification information exists. An acquisition means for acquiring an image of a real space in which a plurality of indicators having a configuration for optically changing an image is arranged;
Detecting means for detecting a projected image of the index in the image acquired by the acquiring means;
Identification means for identifying the index by reading the first identification information from the projected image of the index detected by the detection means;
Setting means for reading the second identification information from the projected image of the index detected by the detection means, and setting a priority for the index detected by the detection means based on the read second identification information. It is characterized by that.

  In order to achieve the object of the present invention, for example, the marker of the present invention comprises the following arrangement.

That is, a marker for obtaining the position of the photographing unit based on the marker position in the photographed image,
An object having an identifier image for identifying the marker;
And a lens disposed on the object.

  According to the configuration of the present invention, even if the identification information written on the index is observed from any of a plurality of predetermined viewpoints, for example, when observed from an oblique direction, the identification information is read and the read identification information is correctly identified. can do.

  Furthermore, even if the identification information is erroneously identified, the error can be detected.

  Furthermore, from among a number of observed indices, an index that is actually used for position and orientation calculation can be determined based on priority.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
FIG. 2 is a block diagram illustrating a functional configuration of a system according to the present embodiment for presenting an observer with an image of a mixed reality space obtained by synthesizing a real space and a virtual space. As shown in the figure, the system according to the present embodiment includes an imaging unit 210 and a mixed reality experience device 2.

  First, the imaging unit 210 will be described. The imaging unit 210 is, for example, a camera, captures a moving image of the real space, and sends the captured image of each frame to the subsequent mixed reality experience device 2. A plurality of indices (markers) described later are arranged in the real space. Therefore, a plurality of indices are shown in the image captured by the image capturing unit 210.

  Next, the mixed reality experience device 2 will be described. The mixed reality experience device 2 includes an image capturing unit 220, an index identification unit 230, a position / orientation calculation unit 240, an image composition unit 260, and a virtual space drawing unit 250.

  The image capturing unit 220 inputs an image of each frame captured by the imaging unit 210 into the apparatus, and outputs the image to the image composition unit 260 and the index identification unit 230, respectively.

  The index identifying unit 230 detects a projected image of the index shown in the image received from the image capturing unit 220. Here, “detecting the projected image of the index” means detecting an image coordinate value of the index in the image and information (identification information) for identifying the index. And if identification information is detected, the process which identifies this parameter | index will be performed. When the index is identified, an identifier for identifying the index and the image coordinates of the index are sent to the position / orientation calculation unit 240.

  The position / orientation calculation unit 240 is a position of each index in the world coordinate system (a coordinate system in which one point in the real space is an origin and three axes orthogonal to each other at the origin are an x axis, a y axis, and a z axis). When the index identifier and the image coordinates of the index are received from the index identification unit 230, the world coordinate system of the index specified by the received identifier is read out from the index identification unit 230. Using the received image coordinates, the position and orientation of the imaging unit 210 are calculated by a known technique.

  The virtual space drawing unit 250 uses the position and orientation of the imaging unit 210 calculated by the position and orientation calculation unit 240 to generate an image of the virtual space that can be seen from the imaging unit 210.

  The image composition unit 260 generates a composite image obtained by superimposing the virtual space image generated by the virtual space drawing unit 250 on the real space image received from the image capturing unit 220, that is, an image of the mixed reality space. The data is output to the display unit 270.

  Thereby, an image of the mixed reality space that can be seen from the imaging unit 210 can be displayed on the display unit 270.

  FIG. 3 is a block diagram showing a hardware configuration of a computer applicable to the mixed reality experience apparatus 2 and its peripheral devices.

  A CPU 301 controls the entire computer using programs and data stored in the RAM 307 and the ROM 306 and executes each process described later performed by the computer. That is, the respective functions (or control functions) of the index identification unit 230, the position / orientation calculation unit 240, the virtual space drawing unit 250, and the image composition unit 260 shown in FIG.

  A graphic accelerator 302 processes part of image processing such as image generation processing in a virtual space and image generation processing in a mixed reality space. The generated mixed reality space image is sent to an HMD (head mounted display) 304. The graphic accelerator 302 operates according to control by the CPU 301.

  304 is an HMD, and as is well known, when it is mounted on the head of an observer, a display device corresponding to the display unit 270 is provided in front of the eyes of the observer.

  Reference numeral 305 denotes an external storage device, which is a large-capacity information storage device typified by a hard disk drive device or the like. Here, an OS (Operating System) or a program or data for causing the CPU 301 to execute each process described later performed by the computer. Are stored in the RAM 307 as needed under the control of the CPU 301.

  Reference numeral 306 denotes a ROM that stores setting data and a boot program of the computer.

  Reference numeral 307 denotes a RAM, an area for temporarily storing programs and data loaded from the external storage device 305, an area for temporarily storing image data received from the camera 311 via the image capturing device 310, Various areas such as an area for temporarily storing the position and orientation data received from the sensor 312 and a work area used when the CPU 301 executes various processes can be provided as appropriate.

  Reference numerals 308 and 309 are a mouse and a keyboard, respectively, and various instructions can be input to the CPU 301 when operated by an operator of the computer.

  Reference numeral 310 denotes an image capturing device, for example, a video capture board, which functions as an I / F for inputting real space image data captured by the camera 311 to the computer. That is, it functions as the image capturing unit 220.

  The camera 311 is attached to the HMD 304 so as to be positioned in the vicinity of the viewpoint position of the observer wearing the HMD 304 on the head. The camera 311 captures a moving image of the real space, and the captured image of each frame is an image capturing device. It outputs to 310. The camera 311 functions as the imaging unit 210.

  A sensor 312 (6-degree-of-freedom sensor, posture sensor, inertial sensor, etc.) is attached to the HMD 304 so as to be positioned in the vicinity of the viewpoint position of the observer wearing the HMD 304 on the head, and measures its own position and posture. To do. The measured result is input to the RAM 307. Note that the method for measuring the position and orientation of the sensor 312 is not particularly limited.

  Next, a plurality of indexes arranged in the real space will be described.

  FIG. 4 is a diagram for explaining indices used in the present embodiment. FIG. 4A is a top view of the index. As shown in the figure, the index includes an identifier area in which an identifier for uniquely identifying the index is described as an image (identifier image) and a boundary surrounding the identifier area. For example, it is obtained by printing each such area on paper. In the present embodiment, a two-dimensional matrix pattern is used as an identifier as shown in the figure. However, the identifier can be used as an identifier, but is not limited thereto.

  FIG. 4B is a cross-sectional view of the index along one side of the index. As shown in FIG. 4B, the identifier region is composed of lenticulars. That is, the lenticular lens is disposed on an object having a two-dimensional matrix pattern written in the identifier area (for example, a planar object having a surface on which the two-dimensional matrix pattern is written).

  Usually, a lenticular realizes a three-dimensional image display by allowing a different image to be observed for each viewpoint, but in this embodiment, even when viewed from different viewpoints, the projected image of the identifier region is almost the same. To be the same.

  FIG. 5A is a diagram showing an index viewed from the front (an index that can be seen by an observer when the line-of-sight vector of the observer viewing the index and the normal vector of the index are substantially parallel). (B) is a diagram showing an index viewed from an oblique direction (an index that can be seen by an observer when the line-of-sight vector of the observer viewing the index and the longitudinal direction of the lenticular lens are substantially orthogonal to each other). As shown in each of the figures, the projected image of the index itself on the image is affected by the perspective transformation, but the identifier region portion is observed as almost the same projected image by the lenticular.

  FIG. 6 is a cross-sectional view of the index along one side of the index. The identifier area is composed of a lenticular lens and an identifier image. Different images are observed in the observation areas v1, v2, and v3, but the same image is observed in the same area. Each observation area is a quadrangle as shown in FIG. As the quadrangle extends in the direction from the lens center to the imaging plane (depth direction), the region where the same image can be observed expands in the depth direction. When the focal length of the lenticular lens is f, the distance between the image and the lens center is a, and the distance between the lens center and the imaging plane is b, the following equation holds.

1 / f = 1 / a + 1 / b
In order to widen the observation area in the depth direction, it is necessary to increase the distance b between the lens center and the imaging plane. Therefore, when the focal length f is fixed, the distance a between the identifier image and the lens center is decreased. Bring the lenticular lens closer to the image. When the distance a between the identifier image and the lens center is fixed, the focal length f of the lenticular lens is increased. On the other hand, since b is also determined by the viewpoint position at which the index is observed, a and f may be adjusted after determining b at the assumed observation position. It is also possible to change the size of the observation area in the depth direction by changing the size L of the identifier area. Although the observation area in the depth direction can be expanded by reducing L, the resolution of the captured image is a trade-off because the resolution of the camera is usually finite.

  In order to widen the observation region in the horizontal direction (direction parallel to the imaging plane), the distance a between the image and the center of the lens must be reduced or the identifier image must be enlarged. However, if the identifier image is simply enlarged, it may appear in the adjacent lens, so the lens must also be enlarged at the same time. This leads to a reduction in the number of lenticular lenses when the size L of the identifier area is fixed. In addition, the observation region in the horizontal direction (direction parallel to the imaging plane) can be expanded by increasing the deviation between the image position on the image plane and the center of the lens.

  The number of viewpoints that can be observed (three in FIG. 6) can be set arbitrarily, but it is a trade-off between the resolution of the captured image and the size L of the identifier area.

  In the present embodiment, the identifier image must be generated so that the projected image of the identifier region is almost the same no matter which of the observation regions v1, v2, and v3 is observed. Since the projected image of the identifier area cannot be the same at all points in the observation area, when one representative point is selected in each observation area and observation is performed at this representative point, the identifier area So that the projected images are the same. In FIG. 6, each point indicated by 601 to 603 is a point on the imaging plane, and the observed index is the best point in each observation region. In the present embodiment, points indicated by 601 to 603 are used as representative points.

  FIG. 7 is a diagram for explaining a method for constructing an identifier image. A camera is placed at points p1, p2, and p3 on the imaging plane in the observation area, and the index is photographed with the index center direction directed. Images taken at points p1, p2, and p3 are image 1, image 2, and image 3, respectively. In FIG. 7, assuming that the direction connecting the point p2 and the index center is perpendicular to the lenticular, the projected image when the index is photographed has the same shape as the original index. Therefore, the identifier image is configured such that the projected image of the identifier area of the index photographed at the points p1 and p3 has substantially the same shape as the projected image of the identifier area of the index in the image 2 photographed at the point p2. Of the images for the observation areas v1, v2, and v3 that must be generated for each lens, the image for the observation area v2 uses the pattern of the identifier area as it is. As the images for the observation areas v1 and v3, intersection points of line segments connecting the image centers of the image 1 and the image 3 and the points p1 and p3 and the lens center are obtained, and the pixels of the same point on the image 2 as the intersection coordinates are used.

  In the present embodiment, the indicators configured as described above are arranged in a plurality of real spaces. In the external storage device 305, a set of an identifier of the index and a position of the index in the world coordinate system (for example, the position of the index center of gravity is used as the index position) is registered in advance for each index. Accordingly, when a certain identifier is designated, the position of this index in the world coordinate system can be obtained by searching the external storage device 305.

  Next, processing performed by the computer will be described. As described above, a plurality of such indicators are arranged in the real space. Since the camera 311 attached to the HMD 304 images such a real space, the captured image is input to the RAM 307 via the image capturing device 310 and temporarily stored therein.

  Next, the CPU 301 functions as the index identification unit 230, detects a projected image that is considered to be an index from the image input to the RAM 307, and identifies which index is the detected projection image. Do. FIG. 8 is a flowchart of such a process, that is, a process for detecting a projected image that seems to be an index and identifying which index the detected projected image is.

  First, a process of detecting a projected image (index area) that seems to be an index from an image is performed (step S201). As such detection processing, for example, each pixel value constituting an image is referred to, a pixel representing a specific color is extracted based on whether or not it is in a specific region of the YUV color space image, and the connected pixels are labeled. By doing so, the boundary region of the index is detected. In the detection process in this step, the image coordinates of the detected index area are also detected. Since the processing in this step is well known, further explanation is omitted.

  In this way, the index region is detected from the image. When detected, the variable n is incremented by one. That is, the number of index areas detected at present is stored in the variable n. This variable n is initialized to 0 at the start of this process.

  Next, when the above detection processing is completed, the value of the variable n is referred to and it is checked whether or not it is 0 (step S202). As a result of the check, when the value of the variable n is 0, that is, when no index area is detected in the image, the present process is terminated, but the value of the variable n is 1 or more. In this case, the process proceeds to step S203, and variable i is initialized to 1 (step S203). The variable i indicates an index for the detected index region.

  And the process which reads the identifier in the parameter | index area | region i is performed (step S204). The identifier is read by, for example, pattern matching. When the identifier pattern is a two-dimensional matrix code, the matrix area is detected, and the identifier is read by reading the pixels directly from the image. In addition, since the process which reads the pattern reflected in the captured image and analyzes the pattern is a well-known thing, description regarding this is abbreviate | omitted.

  Here, although the number of viewpoints of the lenticular and the deviation between the actual viewpoint and the representative points of the observation area (p1, p2, and p3 in FIG. 7) do not necessarily eliminate the influence of the perspective projection, the index size L Assuming that the observation distance (distance from the lens center of the lenticular lens to the image plane) b is sufficiently larger than that of FIG. If the identifier area of the index is not affected by perspective projection, the projected image of the identifier area in the image captured by the camera 311 can be used as it is for reading the identifier.

  Next, it is determined whether or not the value of the variable i is the same as the value of the variable n, that is, whether or not the identifier reading process in step S204 has been performed for all detected index areas (step S205). When the identifier reading process in step S204 is performed for n, that is, for all detected index areas, this process ends. However, in the case of i ≠ n, that is, for all detected index areas, the process proceeds to step S204. If the identifier reading process has not been performed, the process proceeds to step S206, the variable i is incremented by 1, and the processes after step S204 are performed for the next index region that has not yet been subjected to the identifier reading process.

  As described above, by using the lenticular lens, the identifier written on the index can be read regardless of the shooting angle of the camera 311 with respect to the index.

  Next, the CPU 301 functions as the position / orientation calculation unit 240 and calculates the position / orientation of the camera 311. For this purpose, first, the CPU 301 reads the position in the world coordinate system stored in the external storage device 305 corresponding to each identifier read in step S204. Then, the position / orientation of the camera 311 is calculated by a known method using the “coordinate positions on the image plane of the respective index regions and the positions of the respective indices in the world coordinate system” detected in step S201.

  For example, a case where the position and orientation of the camera 311 in the world coordinate system is obtained using the method of Non-Patent Document 2 will be described. In the method of Non-Patent Document 2, the relative position and orientation of a camera and an object photographed by the camera are obtained by model fitting. In other words, the model data of the object to be photographed is held in the computer, the model position on the image plane calculated based on the relative position and orientation, and the position of the object on the actually photographed image The position and orientation of the model are optimized by iterative calculation using the Newton method so that the error between the model and the model is minimized. In the present embodiment, a plurality of indices are combined into one group (model) to obtain a relative position and orientation of the index group with respect to the camera, and the camera based on the position of the known index in the world coordinate system. Find the position and orientation in the world coordinate system. Note that an initial value is required for the relative position and orientation, but here the position and orientation of the camera in the previous frame is used. In this case, the initial value is set by placing the camera at a fixed position in the initial frame.

  When the output of the sensor 312 can be used, the position and orientation of the camera 311 in the world coordinate system is calculated using, for example, the method disclosed in Patent Document 1. Assuming that a 6-DOF sensor is used as the sensor 312, the projection position on the image of the marker whose position in the world coordinate system is known is calculated using the output value of the 6-DOF sensor as the position and orientation of the camera 311 Then, the position and orientation of the camera 311 are obtained by correcting the deviation from the projection position in the actual image by correcting the orientation or position of the camera 311.

  Next, the CPU 301 controls the graphic accelerator 302 so that the position and orientation of the camera 311 in the world coordinate system obtained by the above processing and the internal parameters (focal length, principal point position, etc.) of the camera 311 that have been calibrated in advance. Based on this, an image of the virtual space that can be seen from the camera 311 is generated.

  Next, the CPU 301 functions as the image composition unit 260 and generates a mixed reality space image by superimposing a virtual space image on the real space image previously stored in the RAM 307. Then, the generated image is output to the HMD 304.

  As described above, according to the present embodiment, the index is correctly set even when the index is observed from an oblique direction using the index configured so that the projected images of the identifier areas of the index at a plurality of different observation viewpoints are substantially the same. It becomes possible to identify.

[Second Embodiment]
The index used in the present embodiment is the index used in the first embodiment to which a configuration for preventing the identifier in the index area from being erroneously identified is added.

  FIG. 9 is a diagram for explaining an index used in the present embodiment. FIG. 9A is a top view of the index. As shown in FIG. 9, the index according to the present embodiment is from which observation area the index is observed in addition to the identifier area (for example, as shown in FIG. 6). An area (viewpoint identifier area) in which an identifier (viewpoint identifier) indicating which of the observation area representative points 601 to 603 is observed) is provided in the boundary area. In the present embodiment, a one-dimensional matrix pattern is used as the viewpoint identifier as shown in the figure. However, the viewpoint identifier is not limited to this, and other viewpoints can be considered.

  FIG. 9B is a cross-sectional view of the index along one side of the index. As shown in the figure, the identifier area and the viewpoint identifier area are configured by lenticulars. That is, the lenticular lens is arranged on the two-dimensional matrix pattern written in the identifier area and the one-dimensional matrix pattern written in the viewpoint identifier area. In the present embodiment, even when observed from different viewpoints, the projected image of the identifier area is substantially the same as in the first embodiment, but the viewpoint identifier area is as described above. Since the viewpoint identifier written in the area indicates from which observation area the index is observed, when observing from a different viewpoint, a pattern corresponding to the observation area to which the viewpoint belongs is observed Like that.

  FIG. 10A is a diagram showing an index viewed from the front (an index that can be seen by an observer when the observer's line-of-sight vector viewing the index and the normal vector of the index are substantially parallel). (B) is a diagram showing an index viewed from an oblique direction (an index that can be seen by an observer when the line-of-sight vector of the observer viewing the index and the longitudinal direction of the lenticular lens are substantially orthogonal to each other). As shown in each of the figures, the projected image of the index itself on the image is affected by the perspective transformation, but the identifier region portion is observed as almost the same projected image by the lenticular. However, regarding the viewpoint identifier area portion, when the viewpoint is changed, different projection images are observed accordingly.

  FIG. 11 shows a process of detecting a projected image that seems to be an index, identifying which index the detected projection image is, and checking whether the identified result is incorrect. It is a flowchart. In addition, by performing the process according to the flowchart of FIG. 9 instead of the process according to the flowchart of FIG. 8, the position / orientation calculation process of the camera 311 can be performed based on the detection of the index with higher accuracy. .

  First, a process of detecting a projected image (index area) that seems to be an index from an image is performed (step S301). The processing in this step is performed in the same manner as in step S201. When detected, the variable n is incremented by one. This variable n is initialized to 0 at the start of this process.

  Next, when the above detection processing is completed, the value of the variable n is referred to and it is checked whether or not it is 0 (step S302). As a result of the check, when the value of the variable n is 0, that is, when no index area is detected in the image, the present process is terminated, but the value of the variable n is 1 or more. In this case, the process proceeds to step S303, and flags Fi (i = 1, 2, 3,..., N) used in the subsequent processes are initialized to FALSE (step S303). This flag indicates whether or not the detected index is correctly identified and can be used in position and orientation calculation.

  Next, the variable i is initialized to 1 (step S304). The variable i indicates an index for the detected index region. And the process which reads the identifier in the parameter | index area | region i is performed (step S305). The processing in this step is performed in the same manner as in step S204.

  Next, it is determined whether or not the value of the variable i is the same as the value of the variable n, that is, whether or not the identifier reading process in step S305 has been performed for all detected index areas (step S306). If the identifier reading process in step S305 is performed for n, that is, all detected index areas, the process proceeds to step S308. If i ≠ n, that is, step S305 is performed for all detected index areas. If the identifier reading process is not performed, the process proceeds to step S307, the variable i is incremented by 1, and the process from step S305 onward is performed for the next index area for which the identifier reading process is not yet performed.

  The index detection process can be performed by the above process. Next, it is checked whether or not each detected index is correctly identified and can be used in position and orientation calculation by the processing after step S308.

  First, the variable i used in the subsequent processing is initialized to 1 (step S308). And the process which reads the viewpoint identifier in the parameter | index area | region i is performed (step S309). The processing in this step is performed in the same manner as in step S204. Note that the viewpoint identifier may be read simultaneously with the reading of the identifier.

  Next, it is checked whether or not the index area i has been correctly identified (step S310). This check process is performed by determining whether both the identifier and the viewpoint identifier are read correctly.

  More specifically, first, an approximate position and orientation of the camera 311 is obtained from the read viewpoint identifier. Since the viewpoint identifier indicates from which of the plurality of observation areas the index is being observed, if this viewpoint identifier is read, the observation area where the camera 311 observing the viewpoint identifier is arranged is displayed. Can be identified. Here, it is assumed that the camera 311 is arranged at a representative point in the specified observation area. For example, when the read viewpoint identifier indicates the observation region v1, it is roughly estimated that the camera 311 is arranged at the position indicated by 601.

  Regarding the posture of the camera 311, the direction from the roughly estimated placement position of the camera 311 toward the position in the world coordinate system of the index specified by the identifier in the index area i is defined as the posture of the camera 311.

  Therefore, the viewpoint identifier and the position of the representative point in the observation area indicated by the viewpoint identifier are registered in advance in the external storage device 305 for each viewpoint identifier. Thus, when the viewpoint identifier is determined, the position of the representative point is uniquely determined. Further, the direction to the index can be obtained as the posture of the camera 311 from the obtained position of the representative point.

  Then, based on the roughly determined position and orientation of the camera 311, the position of the known index in the world coordinates, and the internal parameters of the camera 311, the image on the image plane that can be seen from the camera 311 having the roughly estimated position and orientation. Calculate the coordinate position of the index. Therefore, if this coordinate position is compared with the coordinate position of the index region i detected in step S301 and both are located in the image and within a predetermined distance, the index of the index region i is incorrect. It is judged that identification has not been performed.

  In addition, the process for checking whether or not the index region i is correctly identified can be considered other than this, and is not limited to the above process.

  If the index region i is correctly identified as a result of the above check processing, the processing proceeds to step S311 and the flag Fi is set to TRUE (step S311).

  Next, it is determined whether or not the value of the variable i is the same as the value of the variable n, that is, whether or not the processing of steps S309 to S311 has been performed for all detected index regions (step S312). = N, i.e., if the processing of steps S309 to S311 is performed for all the detected index regions, this processing is terminated. If i ≠ n, that is, the steps are performed for all detected index regions. If the process from S309 to S311 has not been performed, the process proceeds to step S313, the variable i is incremented by 1, and the process after step S309 is performed for the next index region that has not yet been subjected to the process from step S309 to step S311. I do.

  Next, the CPU 301 functions as the position / orientation calculation unit 240 and calculates the position / orientation of the camera 311 in the same manner as in the first embodiment. In this case, the only indicator used is that whose flag Fi is TRUE. Is used.

  As described above, according to the present embodiment, it is possible to detect an erroneously identified index by adding a viewpoint identifier region in order to identify where in the space the index was taken.

  In the present embodiment, the check processing as to whether the erroneous identification processing has been performed for the index region may be performed based on the position of the index region to be checked on the image surface as described above, or a plurality of processing may be performed. You may perform based on the position on the image surface of a parameter | index area | region.

[Third Embodiment]
In the present embodiment, the priority of the index is determined using the same index as that used in the second embodiment. That is, if you want to limit the number of indices used for position / orientation calculation due to problems such as computer CPU power, or if you want to perform position / orientation calculation using only reliable indices as much as possible, use reliability. Give priority to the index used for, and use from the highest priority.

  When observing an index using a lenticular, observing from an oblique direction is more likely to be out of the observation region than observing the index from directly above. Therefore, give priority to the observed index by making the priority of the index observed from directly above the highest, and lowering the priority as the line-of-sight direction and the normal direction of the index become closer to vertical. be able to.

  FIG. 12 is a flowchart of processing for detecting a projected image that is considered to be an index, identifying which index the detected projection image is, and setting a priority for the identified index. is there.

  First, similarly to step S201, a process of detecting a projected image (index area) that is considered to be an index from an image is performed (step S401). When detected, the variable n is incremented by one. That is, the number of index areas detected at present is stored in the variable n. This variable n is initialized to 0 at the start of this process.

  Next, when the above detection processing is completed, the value of the variable n is referred to and it is checked whether or not it is 0 (step S402). As a result of the check, when the value of the variable n is 0, that is, when no index area is detected in the image, the present process is terminated, but the value of the variable n is 1 or more. In this case, the process proceeds to step S403, and the variable i is initialized to 1 (step S403). The variable i indicates an index for the detected index region.

  Then, similarly to step S309, a process of reading the viewpoint identifier in the index area i is performed (step S404). Then, the priority for the index area i is set using the read viewpoint identifier (step S405). This priority setting will be described with reference to FIG. FIG. 13 is a diagram for explaining setting of priority. As described above, the highest priority is given to the observation area directly above the index center, and the priority is lowered as the observation area is shifted from the index center.

  As described above, since the viewpoint identifier indicates from which of the plurality of observation areas the index is observed, if the viewpoint identifier is read, the corresponding observation area is close to the index center. You can check if there is. For example, as a viewpoint identifier, a viewpoint identifier that is visible when the index area is viewed from directly above is pattern 1, a viewpoint identifier that is visible when the index area is viewed from obliquely 45 ° from directly above is pattern 2, and the index area is directly from above If the viewpoint identifier that can be seen when viewed from an oblique angle of 60 ° is pattern 3, the higher priority of pattern 1, pattern 2, and pattern 3 is registered in the external storage device 305 in this order.

  Thus, when the pattern 2 is read as the identifier area in the index area i in step S404, the second highest priority is set for the index area i.

  Next, it is determined whether or not the value of the variable i is the same as the value of the variable n, that is, whether or not the processing of steps S404 and S405 has been performed for all detected index areas (step S406). If n is performed, that is, if the processing in steps S404 and S405 is performed for all detected index regions, the process proceeds to step S408. If i ≠ n, that is, step S404 is performed for all detected index regions. , S405 is not performed, the process proceeds to step S407, the variable i is incremented by one, and the process after step S404 is performed for the next index region that has not been subjected to the processes of steps S404, S405.

  Through the above processing, priority can be set for each index region based on the read viewpoint identifier. Next, a process of selecting an index to be actually used based on the priority by the process after step S408 will be described.

  Next, the index areas are sorted in descending order of the set priority (step S408). Next, the variable j used in the subsequent processing is initialized to 1 (step S409). Then, similarly to the above step S204, a process for reading the identifier in the index region j is performed (step S410). That is, the first read out in step S410 is the identifier in the index area with the highest priority. The reading of the identifier may be performed simultaneously with the reading of the viewpoint identifier.

  Next, it is determined whether or not the value of the variable j is the same as the value m determined in advance as the number of indices used for calculating the position and orientation of the camera 311 (step S411), and when j = m. Ends this process, but if j ≠ m, the process proceeds to step S412 to determine whether or not the value of the variable j is equal to the number n of index regions in the image ( If they are the same, the process is terminated. If they are not the same, the process proceeds to step S413 and the value of variable j is incremented by 1 (step S413). Repeat the process.

  In the above processing, the number of indices used for calculating the position and orientation of the camera 311 is determined in advance. However, a threshold is set for the priority, and the index has a priority equal to or greater than (or greater than) the threshold. Only the position / orientation calculation may be used.

  As described above, according to the present embodiment, the priority of the index is set using the viewpoint identifier read from the viewpoint identifier area, and the index used for position and orientation calculation is determined based on the set priority. The position and orientation of the camera 311 can be calculated using only the highly reliable index, and more accurate position and orientation calculation processing can be performed.

[Fourth Embodiment]
In the second and third embodiments, since the identifier area and the viewpoint identifier area are provided separately, the reading of the identifier and the reading of the viewpoint identifier are performed separately. In the present embodiment, these are combined into one, and the identifier and the viewpoint identifier are read together. Hereinafter, one identifier obtained by combining the identifier and the viewpoint identifier is referred to as a unified identifier.

  FIG. 14 is a diagram for explaining indices used in the present embodiment. FIG. 14A is a top view of the index. FIG. 14B is a cross-sectional view of the index along one side of the index. As shown in each figure, among the unified identifiers, substantially the same pattern is observed from any viewpoint, but the observed pattern differs depending on the viewpoint. Therefore, it is necessary to adjust each pattern and the lenticular lens as described above.

  FIG. 15 is a diagram showing a configuration of a unified identifier. Hereinafter, processing for obtaining an identifier and a viewpoint identifier from a unified identifier will be described with reference to FIG. In the figure, the unified identifier consists of 4 × 4 16 bits. In this figure, 12 bits are assigned to the index identifier and 4 bits are assigned to the view identifier, and the index identifier and view identifier are obtained by extracting 12 bits and 4 bits respectively from the read 16-bit unified identifier. .

  The procedure of index identification and erroneous identification index detection in the present embodiment is the same as that in the second and third embodiments except that the index identifier and the viewpoint identifier are obtained by the above-described method.

  As described above, according to the present embodiment, by combining the viewpoint identifier area and the index identifier area into one identifier area, it is possible to simplify the process by combining identifier reading processing into one. Become.

[Fifth Embodiment]
In each of the above embodiments, even when observed from different viewpoints, the projected image of the identifier area is substantially the same, and the viewpoint identifier is observed in a different pattern depending on the observation area to which the observed viewpoint belongs. As described above, in order to optically convert the observation image of each identifier, a lenticular is used. In this embodiment, integral photography is used.

  FIG. 16 is a diagram for explaining an index used in the present embodiment. FIG. 16A is a top view of the index and has the same region configuration as that of the first embodiment. FIG. 16B is a cross-sectional view of the index along one side of the index. As shown in the figure, the identifier region is configured by integral photography. That is, the eyelet lens (integral photography lens) is arranged on the two-dimensional matrix pattern written in the identifier region.

  FIG. 17 is a diagram illustrating a configuration of a eyelet lens. FIG. 17A is a top view of the eyelet lens. As shown in FIG. 17, the eyelet lens is a lens array in which small lenses are arranged vertically and horizontally. Integral photography displays a three-dimensional image by observing the image through the eye lens. In lenticular, the viewpoint change is one-dimensional, but in integral photography, a two-dimensional viewpoint change is possible.

  FIG. 17B is a cross-sectional view taken along the cut surfaces 1 and m of the eyelet lens. As shown in the figure, when the eyelet lens is viewed from one direction, it is equivalent to the cross section of the lenticular lens. By using this integral photography, the projected images of the indicator identifier areas at a plurality of different observation viewpoints are made substantially the same.

  Unlike lenticular lenses, integral photography has a volumetric observation region arranged two-dimensionally as shown in FIG. FIG. 18 is a diagram showing an observation region when integral photography is used.

  When the index is observed at the representative point of each observation area, the projected image of the identifier area is made the same. Specifically, the above-described method described with reference to FIG. 7 is performed for all observation regions arranged two-dimensionally, and a two-dimensional pattern is arranged for each lens of the eyelet lens on the identifier image. .

  Integral photography may also be used for viewpoint identifiers.

[Sixth Embodiment]
In each of the above embodiments, even when observed from different viewpoints, the projected image of the identifier area is substantially the same, and the viewpoint identifier is observed in a different pattern depending on the observation area to which the observed viewpoint belongs. As described above, in order to optically convert the observation image of each identifier, a lenticular is used. In this embodiment, a hologram is used.

  Hologram is a technology that realizes three-dimensional display by recording not only the intensity of light but also the direction in which light has traveled on a recording material by utilizing interference of light. Using this technique, the projected image of the identifier region is the same even when the viewpoint is changed.

  Specifically, as shown in FIG. 19, a front image of the identifier region is printed on a film, the film is irradiated with a laser, projected onto a screen, and recorded on a recording material while moving a slit for each viewpoint position. The hologram of the identifier area is generated by going. Thereby, even if a viewpoint is changed, the projection image of an identifier area | region becomes the same. FIG. 19 is a schematic diagram showing how to create a hologram of an identifier area.

  Note that the hologram may also be used for the viewpoint identifier.

[Other Embodiments]
Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU or Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a block diagram which shows the function structure of the conventional position and orientation measuring apparatus. It is a block diagram which shows the function structure of the system which concerns on the 1st Embodiment of this invention for showing the observer the image of the mixed reality space which synthesize | combined real space and virtual space. It is a block diagram which shows the hardware constitutions of the computer applicable to the mixed reality experience apparatus 2, and its peripheral device. It is a figure explaining the parameter | index used in the 1st Embodiment of this invention. (A) is a figure which shows the parameter | index seen from the front, (b) is a figure which shows the parameter | index seen from diagonally. It is sectional drawing of the parameter | index along the one side of the parameter | index. It is a figure explaining the structure method of an identifier image. It is a flowchart of the process which detects the projection image which seems to be a parameter | index, and identifies which projection image is the detected projection image. It is a figure explaining the parameter | index used by the 2nd Embodiment of this invention. (A) is a figure which shows the parameter | index seen from the front, (b) is a figure which shows the parameter | index seen from diagonally. It is a flowchart of the process which detects the projection image considered as a parameter | index, identifies whether the detected projection image is a projection image of the detected index, and checks whether the identified result is incorrect. 10 is a flowchart of processing for detecting a projected image that is considered to be an index, identifying which index the detected projection image is, and setting a priority for the identified index. It is a figure explaining the setting of a priority. It is a figure explaining the parameter | index used by the 4th Embodiment of this invention. It is a figure which shows the structure of a unified identifier. It is a figure explaining the parameter | index used by the 5th Embodiment of this invention. It is a figure which shows the structure of the eyelid lens. It is a figure which shows the observation area | region at the time of using integral photography. It is a schematic diagram which shows a mode that the hologram of an identifier area | region is created.

Claims (15)

A plurality of indices having a configuration in which the identification information is written and the observation image of the identification information is optically changed so that the identification information is substantially the same even when the identification information is observed from any of a plurality of predetermined viewpoints. An acquisition step of acquiring an image of the arranged physical space;
A detection step of detecting a projected image of the index in the image acquired in the acquisition step;
An image processing method comprising: an identification step of identifying the index by reading identification information from a projected image of the index detected in the detection step. The indicator is equipped with a lenticular lens,
2. The identification information written on the lenticular lens and the index is configured to be substantially the same image when the identification information is observed from any of a plurality of predetermined viewpoints. The image processing method as described. The identification information written on the indicator is written by a hologram,
The image processing method according to claim 1, wherein the hologram is configured to have substantially the same image even when the identification information is observed from any of a plurality of predetermined viewpoints. The indicator is equipped with an integral photography lens,
The identification information described on the integral photography lens and the index is configured to be substantially the same image when the identification information is observed from any of a plurality of predetermined viewpoints. Item 8. The image processing method according to Item 1. The first identification information and the second identification information are recorded, and the first identification information is observed so that the first identification information can be substantially the same even if the first identification information is observed from any of a plurality of predetermined viewpoints. In order to change the image of the second identification information according to the configuration in which the image is optically changed and the observation region where the viewpoint for observing the second identification information is present, the observation image of the second identification information is changed. An acquisition step of acquiring an image of a real space in which a plurality of indicators having a configuration that is optically changed is arranged;
A detection step of detecting a projected image of the index in the image acquired in the acquisition step;
An identification step for identifying the index by reading the first identification information from the projected image of the index detected in the detection step;
An estimation step of estimating the position of the viewpoint by reading the second identification information from the projected image of the index detected in the detection step;
Using the position of the index on the image seen from the viewpoint of the position estimated in the estimation process and the position of the index detected in the detection process, a check process for checking the identification process in the identification process for the index; An image processing method comprising: The first identification information and the second identification information are recorded, and the first identification information is observed so that the first identification information can be substantially the same even if the first identification information is observed from any of a plurality of predetermined viewpoints. In order to change the image of the second identification information according to the configuration in which the image is optically changed and the observation region where the viewpoint for observing the second identification information is present, the observation image of the second identification information is changed. An acquisition step of acquiring an image of a real space in which a plurality of indicators having a configuration that is optically changed is arranged;
A detection step of detecting a projected image of the index in the image acquired in the acquisition step;
An identification step for identifying the index by reading the first identification information from the projected image of the index detected in the detection step;
A setting step of reading the second identification information from the projected image of the index detected in the detection step, and setting a priority for the index detected in the detection step based on the read second identification information. An image processing method. A plurality of indices having a configuration in which the identification information is written and the observation image of the identification information is optically changed so that the identification information is substantially the same even when the identification information is observed from any of a plurality of predetermined viewpoints. An acquisition means for acquiring an image of the arranged real space;
Detecting means for detecting a projected image of the index in the image acquired by the acquiring means;
An image processing apparatus comprising: an identification unit that identifies an index by reading identification information from a projected image of the index detected by the detection unit. The first identification information and the second identification information are recorded, and the first identification information is observed so that the first identification information can be substantially the same even if the first identification information is observed from any of a plurality of predetermined viewpoints. In order to change the image of the second identification information according to the configuration in which the image is optically changed and the observation region where the viewpoint for observing the second identification information is present, the observation image of the second identification information is changed. An acquisition means for acquiring an image of a real space in which a plurality of indices having a configuration that is optically changed is arranged;
Detecting means for detecting a projected image of the index in the image acquired by the acquiring means;
Identification means for identifying the index by reading the first identification information from the projected image of the index detected by the detection means;
Estimating means for estimating the position of the viewpoint by reading the second identification information from the projected image of the index detected by the detecting means;
Checking means for checking identification processing by the identification means for the index using the position of the index seen from the viewpoint of the position estimated by the estimation means and the position of the index detected by the detection means; An image processing apparatus comprising: The first identification information and the second identification information are recorded, and the first identification information is observed so that the first identification information can be substantially the same even if the first identification information is observed from any of a plurality of predetermined viewpoints. In order to change the image of the second identification information according to the configuration in which the image is optically changed and the observation region where the viewpoint for observing the second identification information is present, the observation image of the second identification information is changed. An acquisition means for acquiring an image of a real space in which a plurality of indices having a configuration that is optically changed is arranged;
Detecting means for detecting a projected image of the index in the image acquired by the acquiring means;
Identification means for identifying the index by reading the first identification information from the projected image of the index detected by the detection means;
Setting means for reading the second identification information from the projected image of the index detected by the detection means, and setting a priority for the index detected by the detection means based on the read second identification information. An image processing apparatus.   A program causing a computer to execute the image processing method according to any one of claims 1 to 6.   A computer-readable storage medium storing the program according to claim 10. A marker for determining the position of the imaging unit based on the marker position in the captured image,
An object having an identifier image for identifying the marker;
And a lens disposed on the object.   The identifier image and the lens are configured to optically change the observation image of the identifier image so that the identifier image becomes substantially the same even when viewed from any of a plurality of predetermined viewpoints. The marker according to claim 12, characterized in that:   The marker according to claim 13, wherein the identifier image is a two-dimensional matrix pattern.   The marker according to claim 13, wherein the lens is a lenticular lens.

Images