JP2009020614A - Marker unit to be used for augmented reality system, augmented reality system, marker unit creation support system, and marker unit creation support program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a marker unit to be used for an augmented reality system for surely securing identity without damaging any fine sight. <P>SOLUTION: This marker unit MU is used for the augmented reality system 1 for recognizing the position of an object R in an actual environment, and for displaying virtual information by superimposing it, and arranged in the object R so as to be configured as the index of the position recognition of the object R. The marker unit MU is configured of a plurality of marker elements ME, attached with a color approximated to the color of the object R, configuring identification information according to the combination of the external forms and layout. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a marker unit and a mixed reality system used in a mixed reality system. The present invention also relates to a marker unit creation support system and a creation support program for creating a marker unit.

  In recent years, mixed reality technology that amplifies and expands real environment information by superimposing and displaying virtual information such as CG (computer graphics) images and characters on real environment (real space) images taken by a camera ( Research and development related to MR (Mixed Reality) are actively conducted. As an application of this mixed reality technology, for example, a virtual image corresponding to the real field of view can be superimposed on a see-through type display that can be worn on a human head, thereby providing more information to the wearer in real time. A mixed reality system to provide is considered.

  In such a mixed reality system, when a human moves or moves his / her head, the field of view naturally changes, so the virtual information must also follow and change. For this follow-up, it is important to align the object in the real environment with the virtual information corresponding to the object. As a method for performing this alignment, a plurality of artificially created markers are placed on the object in the real environment, and the position of the object in the video is recognized by detecting the marker placed on the object from the image captured by the camera. A method is known in which the position and orientation of the camera are specified from the position and aligned.

Conventionally, artificially created markers have a circular or square shape that can be easily identified, two high-contrast black and white colors, or a combination of red, blue, yellow, etc. Etc. are used. However, since such a marker emphasizes only the discriminability, there is a problem that it does not harmonize with the object on which the marker is placed and impairs the beauty.
In addition, in order not to impair the aesthetic appearance, it is also proposed that the marker itself is made of a retroreflective material that is difficult to see with the naked eye, and the marker irradiated with infrared light is photographed with an infrared camera (non-patent document). 1). However, in this case, since the use of an infrared camera is a prerequisite, the use environment is limited and it is difficult to spread.
Yusuke Nakazato and two others, “Position / Attitude Estimation Using Invisible Markers and Infrared Cameras for Wearable Augmented Reality”, Transactions of the Virtual Reality Society of Japan, Vol. 10, no. 3,2005

The present invention has been made in view of the above circumstances, and has used a marker unit that does not impair the aesthetics and that can sufficiently ensure discrimination (robustness of discrimination) and the marker unit. The purpose is to provide a mixed reality system.
It is another object of the present invention to provide a marker unit creation support system and a creation support program for creating an appropriate marker unit according to an object.

The marker unit of the present invention is used in a mixed reality system that recognizes the position of an object in a real environment and displays virtual information in a superimposed manner, and is a marker used as an index for recognizing the position of the object by being arranged on the object A unit,
It is characterized by comprising a plurality of marker elements which are provided with a color approximate to the color of the object and which constitute identification information by a combination of its outer shape and arrangement.

  According to this configuration, since the color of the marker unit is a color that approximates the color of the object, the aesthetic appearance in the real environment is not impaired, and the identification information is configured by a combination of the outer shape and arrangement of a plurality of marker elements. Therefore, even if the color of the marker unit approximates the color of the object, it is possible to sufficiently ensure the distinguishability of the marker unit.

The color of the marker element can be set so that the density approximates in the same color system as the object color, or can be set so that the color system approximates in the same density as the object color.
It is preferable that at least a part of the outer shape of the marker element has a shape along the outer shape of the object. This makes it possible to arrange marker elements that are in harmony with the object, so that the aesthetic appearance is not impaired.

The mixed reality system of the present invention using the marker unit stores color information of the object and the marker element arranged on the object, and combination information of the outer shape and arrangement of the marker unit. The combination of the storage means, the marker element detection means for detecting the marker element based on the color information from the photographed image of the object, and the detected outline and arrangement of the plurality of marker elements corresponds to the combination information. And a marker unit detecting means for detecting a marker unit constituted by the marker element by determining whether or not.
With such a configuration, it is possible to appropriately detect a marker element having a color approximate to the color of the object and a marker unit identified by the outer shape and arrangement of the marker element.

  A marker unit creation support system of the present invention for creating the marker unit corresponding to an object includes an imaging device that photographs the object, and a processing device that performs a determination process of a marker unit used for the object. The processing device includes an image input unit that inputs an image of a photographed object, a target region extraction unit that extracts a region of the object in which the marker unit is arranged from the input image, and a color of the extracted object And marker unit determining means for determining a marker unit that can be arranged on the object according to the shape.

  In addition, the marker unit creation support program of the present invention for creating the marker unit corresponding to an object includes an image input means for inputting a captured image of the object, and an object on which the marker unit is arranged from the input image. The computer functions as target area extraction means for extracting an area, and marker unit determination means for determining a marker unit that can be placed on the object according to the color and shape of the extracted object. To do.

  According to such a marker unit creation support system and marker unit creation support program, the marker unit can be appropriately determined according to the object in the real environment, and the user (creator of the marker unit) himself can respond to the object. Since it is not necessary to determine the color, shape, etc. of the marker unit, the burden on the user can be reduced and the mixed reality system can be easily introduced into the real environment.

The marker unit determining unit includes a presenting unit that presents a plurality of marker unit candidates that can be arranged on the object, and a selection receiving unit that receives a user's selection from the plurality of marker unit candidates. preferable.
With this configuration, a marker unit that does not impair the aesthetics can be determined and determined by the user's eyes from among a plurality of marker unit candidates.

  According to the marker unit and the mixed reality system of the present invention, the aesthetic appearance of the real environment is not impaired, and the distinguishability of the marker unit can be sufficiently ensured. In addition, according to the marker unit creation support system and the creation support program of the present invention, it is possible to appropriately determine a marker unit according to the real environment and reduce the burden of creating the marker unit on the user.

  FIG. 1 is a diagram showing an overview of a mixed reality system 1 according to an embodiment of the present invention. The mixed reality system 1 arranges a marker unit MU on an object R (R1, R2) existing in an actual environment, for example, a shelf R1 or a door R2 arranged in a room, and displays an indoor image, for example, a human being The head mounted display (HMD) 10 mounted on the head of the camera is photographed by the camera, the marker unit MU is detected from the photographed image, the existence of the objects R1, R2 is recognized, and the objects R1, R2 are associated (alignment). ) And displaying virtual information such as CG images and characters on the head mounted display 10, thereby amplifying and expanding information that falls within the actual visual field.

  The head mounted display 10 can adopt a video see-through method, for example. A video camera (photographing means) 12 (FIG. 3) having a CCD or CMOS image sensor is provided at a position substantially corresponding to both eyes inside the goggle portion of the head mounted display 10, and the back side of the front portion of the goggle portion. In FIG. 3, displays (display means) 13 (FIG. 3) corresponding to both eyes are arranged on both the left and right sides. Then, the video of the virtual space generated by the processing device 11 is displayed on each display 13 together with information captured by the video camera 12. The head mounted display 10 can adopt an optical see-through method.

The processing device 11 is configured by a computer including a storage device such as a CPU, RAM, ROM, and HDD, an I / O interface, and the like, and is connected to the head mounted display 10 so as to be communicable by wire or wirelessly. The processing device 11 can be attached to the human body as a wearable computer, or can be installed at a location away from the human.
The video camera 12 and the display 13 are not limited to being configured as the head mounted display 10, but may be configured in a form that is not individually worn on the human body. In this case, the display 13 can also be configured as a part of a computer constituting the processing device 11 such as a notebook computer.

[Configuration of marker unit]
The marker unit MU is a unit that forms one piece of identification information (ID) for recognizing the objects R1 and R2, and is configured by a combination of a plurality of marker elements ME. Therefore, if the shape and arrangement of some of the marker elements ME are different, the marker unit MU constitutes different identification information.
The marker element ME is formed of paper, plastic board, wood board, or the like that is attached to the surfaces of the objects R1 and R2. The marker element ME has small irregularities when attached to the objects R1 and R2, and the object R1 and R2 have a feeling of strangeness or foreign matter It is made of a thin material that does not give a feeling.

  Further, the marker element ME is a single solid color, and is colored similar to the color (background color) of the objects R1 and R2 to be pasted. The approximate color is, for example, a color having the same color system (hue, color) and a slightly different density (lightness), a color having the same density and a slightly different color system, or a color having a slightly different color system and density, etc. Say. In any case, the color of the marker element ME and the colors of the objects R1 and R2 are not greatly different. It is set in a color that is in harmony and does not impair the beauty.

  The marker element ME is formed in an L shape, an I shape, a T shape, an isosceles triangle, or the like according to the shapes of the objects R1 and R2 to be arranged. In the example shown in FIG. 1, six marker elements ME are arranged on the shelf R1, and are L-shaped on both sides of the upper portion of the shelf R1, T-shaped on both sides of the upper and lower middle portions, and I-shaped on both sides of the lower portion. Each marker element ME is arranged. In addition, two upper and lower marker units MU are arranged on the door R2, and isosceles triangles are arranged on both upper sides of the upper marker unit MU, and I-shaped marker elements ME are arranged on both lower sides. An I-shaped marker element ME is arranged on both upper sides of the marker unit MU below the door R2, and an L-shaped marker element ME is arranged on both lower sides.

  The outer shape of each marker element ME is selected so as to follow the shapes of the corners and sides of the objects R1 and R2. That is, the marker element ME is formed so that at least a part of the outer shape thereof conforms to the outer shapes of the objects R1 and R2, and has a simple shape that does not make the object R1 and R2 visually feel uncomfortable even when arranged on the surface. ing.

Each marker element ME constituting one marker unit MU has a combination of external shapes so that the rotation systems of the objects R1 and R2 can be identified. For example, as shown in FIG. 2, when marker elements ME1 to ME4 are arranged at the four corners of a plane PL on which an object R is formed to form a rectangular marker unit MU as indicated by a two-dot chain line, the plane The outer shape and arrangement of the marker elements ME1 to ME4 are set so that the marker elements ME1 to ME4 having the same shape do not have the same arrangement when the PL is rotated around the center O. That is, in one marker unit MU, all the marker elements ME1 to ME4 cannot be formed in the same shape, or the marker elements ME1 to ME4 cannot be arranged point-symmetrically, and all the marker elements ME1 to ME4 are different. Even if it is a shape or includes the same shape, it is necessary to arrange them symmetrically.
In FIG. 1, a set of marker elements ME that can be arranged on the object R is shown as a marker set MS.

[Functional Configuration of Processing Device 11]
As shown in FIG. 1, the mixed reality system 1 recognizes the presence of the objects R1 and R2 by detecting the marker elements ME and the marker units MU arranged on the objects R1 and R2, and determines the position of the marker unit MU. The position and orientation of the camera 12 (the position and orientation of the human viewpoint) are detected from the arrangement. Then, in accordance with the detected position and orientation of the camera 12, virtual information such as a CG image is superimposed and displayed on an object in the field of view via a display.

  FIG. 3 is a diagram illustrating a functional configuration of the processing device 11 of the mixed reality system 1, and the processing device 11 has information between the camera 12 and the display 13 of the head mounted display 10 as a functional unit. Communication means 27 for inputting / outputting, information on the color of the marker element ME and the objects R1, R2, storage means 21 for storing information on the outer shape and arrangement of the marker element ME, and image input for inputting an image taken by the camera Means 22, marker element detection means 23 for detecting the marker element ME from the input image, marker unit detection means 24 for detecting the marker unit MU from the detected marker element ME, and position of the camera 12 from the detected marker unit MU And camera position / orientation detection means 25 for obtaining the position and the detected position Depending on the position and orientation of the camera 12, and a virtual information output means 26 for outputting display the virtual information on the display 13.

The communication unit 27 is configured as a function of an I / O interface physically connected to the camera 12 and the display 13, and the storage unit 21 is configured as a function of a storage device such as a hard disk. Each detection means 23, 24, 25 functions by an application program installed in a storage device such as a hard disk.
Details of the functions of the mixed reality system 1 will be described together with the following processing procedure.

[Processing procedure of mixed reality system]
FIG. 4 is a flowchart showing a processing procedure of the mixed reality system. Hereinafter, these processes will be described using the model shown in FIG. 2 as an example.
In the processing by the mixed reality system 1, the processing device 11 includes a marker element ME (ME1 to ME4) constituting the marker unit MU and the marker element ME by the function of the storage unit 21 (FIG. 3). The color information with the object R on which is placed is registered in advance. In the present embodiment, the marker element ME and the object R have the same color system (hue and color) but have slightly different densities (lightness). In the storage unit 21, information on the color systems of the marker element ME and the object R and information on the density difference are registered.
In addition, in the storage unit 21 of the processing device 11, combination information of the outer shape and arrangement of each marker element ME of the marker unit MU constituting the identification information (ID) is registered. In the model shown in FIG. 2, combination information of each outer shape of the four marker elements ME <b> 1 to ME <b> 4 arranged so as to form a rectangle and the mutual arrangement relationship is registered in the storage unit 21.

(1) Marker Element Detection Processing As shown in FIG. 3, when shooting is performed at a predetermined frame rate by the camera 12, the processing device 11 inputs the captured image by the image input means 22 each time. Step S1 in FIG. 4 is a process of the marker element detection unit 23, and the processing device 11 first performs raster scanning on the entire image of the first input frame. And the pixel comprised by the color similar to the color information of the marker elements ME1-ME4 and the object R memorize | stored in the memory | storage means 21 is searched from an image.

  FIG. 5 is a schematic diagram illustrating pixel search of an image. As indicated by the frame A, when the corresponding pixels p exist in the image G adjacent to each other, it is determined whether or not the density difference exceeds a predetermined threshold value. When the threshold value is exceeded, it is determined that the pixel p is adjacent at the boundary between the marker elements ME1 to ME4 and the object R, and the same search is repeated using one pixel as a starting point, as indicated by an arrow. The contour line tracking process of the marker element ME is performed.

Note that which of the adjacent pixels p is to be used as the starting point for contour tracking is that the pixel corresponding to the marker element ME is the starting point based on the density of the marker element ME and the object R registered in advance. Select
During contact with the contour line, if the image border or other contour line is touched, or if it touches the contour line currently being tracked, the contour tracking is terminated. As a result of the contour tracking, only the contour that wants to return to the starting point is set as a candidate for the marker element ME.

When the processing device 11 determines the candidate of the marker element ME included in the image by performing the contour tracking process as described above, the processing device 11 next identifies the shape of the marker element ME.
The outer shape of the marker element ME can be classified from the number of vertices of the contour line and the shape (unevenness) of the contour line at each vertex. For example, an L-shaped marker element has six vertices, of which five have convex vertices and one have concave vertices. The I-shaped marker element has four convex vertices, and the isosceles triangular marker element has three convex vertices. Therefore, the shape of the marker element can be identified by determining the number of vertices and the contour shape at the vertices.

The vertex of the contour line acquired by the contour line tracking process is obtained using a piecewise linear approximation method. Originally, this method is a method of approximating a curve with a straight line. However, as in the present embodiment, for a marker element ME that forms a closed contour line by a combination of straight lines, it is used to obtain the vertex. be able to.
Specifically, for example, as shown in FIG. 6A, an arbitrary point P0 on the contour line L obtained by contour tracking is determined, and the farthest point from the point P0 is defined as the first vertex P1 and the first point P1. The point farthest from one vertex P1 is set as the second vertex P2, respectively. Then, as shown in FIG. 6B, assuming two contour lines L1 and L2 having the first vertex P1 (P11, P12) and the second vertex P2 (P21, P22) as end points, each contour line L1 , L2 respectively apply piecewise linear approximation. As a result, as shown in FIG. 6C, the contour lines L1 and L2 are straightened and the vertices P3 to P6 are obtained.
The above processing is performed on the entire frame image, and all marker elements ME in the image are detected.

  In the present embodiment, the color information used for detecting the marker element ME is not an RGB color space generally used in a computer, but an HSI (hue, saturation, brightness) used in a developing system such as Munsell color system. A color space is used. Therefore, the RGB values of the captured image of the object R are converted into HSI values using a bihexagonal pyramid model or a hexagonal pyramid model.

(2) Marker Unit Detection Processing In step S <b> 2 of FIG. 4, the processing device 11 performs marker unit MU detection processing using the function of the marker unit detection means 24. The processing device 11 selects a plurality of marker elements ME that can constitute the marker unit MU from all the marker elements ME detected from the captured image. For example, in the model shown in FIG. 2, since the marker unit MU is configured by the marker elements ME1 to ME4 arranged so as to form a two-dot chain rectangle, the selected marker element ME is It is determined whether they are arranged on the same plane according to the rectangular arrangement rule. When the selected marker element ME is arranged so as to form the rectangle, the combination of the outer shape and arrangement of the marker element ME and the outer shape of the marker element ME stored in the storage means 21 in advance Corresponding with the combination information of arrangement.

The determination as to whether or not the selected marker element ME is arranged to form a rectangle is performed as follows. As shown in FIG. 7A, first, assuming that the selected four marker elements ME1 to ME4 form a rectangle, the vertexes are obtained. Then, the center of gravity Ec _{1 to} Ec ₄ of each marker element ME is obtained, and the center is set as a rectangular pseudo center of gravity C.
In each of the marker elements ME1 to ME4, as shown in FIG. 7 (b), the vertex having the longest distance from the pseudo center of gravity C is obtained, and each is assumed to be the vertex ν _{1 to} ν ₄ of the marker unit MU. At this time, with respect to the L-shaped marker elements ME1 and ME2, since the points that can be rectangular vertices are uniquely determined, the vertices are set as vertices ν ₁ and ν ₂ of the marker unit MU.

When the two sides of each of the marker elements ME1 to ME4 are inscribed in the squares ν _{1 to} ν ₄ , the combination of these four marker elements ME1 to ME4 satisfies the arrangement rule for forming the marker unit MU. Judge.
Specifically, To determine each marker element ME1~ME4 is inscribed in square ν ₁ ~ν _4, as shown in FIG. 7 (c), the point which is assumed to vertices of the marker unit MU, e.g. A direction vector from the vertex ν ₁ to the adjacent vertex ν ₂ in the marker unit MU is V _12, and a direction vector from the vertex ν ₁₂ to the adjacent vertex ν ₁₂ in the marker element ME 1 is U ₁₂ . A direction vector from the vertex ν ₁ to another vertex ν ₄ adjacent in the marker unit MU is V _14, and a direction vector from the vertex ν ₁₄ adjacent in the marker element ME 1 is U ₁₄ . When the direction components of V ₁₂ and U ₁₂ , and V ₁₄ and U ₁₄ are substantially the same, it is assumed that the two sides of the marker element ME1 are inscribed in the squares ν _{1 to} ν ₄ .

When the four selected marker elements ME1 to ME4 are arranged so as to form a rectangle, the marker unit MU registered in advance in the storage means 21 has the external shape and arrangement relationship of the marker elements ME1 to ME4. It is determined whether or not the marker elements ME1 to ME4 match, and the marker unit MU is detected when they match.
When the marker unit MU is detected from the captured image in this way, the three-dimensional positions of the four vertices ν _{1 to} ν ₄ of the marker unit MU can be taken.

  The processing device 11 determines in step S3 of FIG. 4 whether or not the marker unit MU has been successfully detected. If successful, the processing device 11 proceeds to step S4, and if unsuccessful, the processing returns to step S1.

(3) Camera Position / Orientation Detection The processing device 11 detects the position / orientation of the camera 12 by the function of the camera position / orientation detection means 25 in step S4 of FIG.
In order to detect the position and orientation of the camera 12, the coordinate system handled by the mixed reality system 1 according to the present embodiment includes a world coordinate system ( _Xw , _Yw , _Zw ), a marker coordinate system ( X _m , Y _m , Z _m ), camera coordinate system (X _c , Y _c , Z _c ), and screen coordinate system (x _c , y _c ).

The world coordinate system is a reference coordinate system in the mixed reality space. By determining the world coordinate system, consistency between the real space and the virtual space is achieved. The position and orientation of the world coordinate system in the real space and the position of the virtual object in the world coordinate system are set in advance.
The marker coordinate system is a coordinate system that exists for each marker unit MU, and is a three-dimensional coordinate having the normal direction of the marker unit MU as the Z axis. The positional relationship of the marker coordinate system in the world coordinate system is set in advance.

The camera coordinate system is a three-dimensional coordinate system in which the focus position f of the camera 12 is the origin and the direction perpendicular to the image plane is the Z axis. In order to obtain the position and orientation of the camera 12, it is necessary to obtain a transformation matrix between the world coordinate system and the camera coordinate system.
The screen coordinate system is a two-dimensional coordinate system that represents a positional relationship in a captured image.
A transformation matrix between the world coordinate system and the marker coordinate system is registered in the processing device 11 in advance. In addition, camera internal parameters, which are transformation matrices between the camera coordinate system and the screen coordinate system, are obtained in advance.

  The coordinates of the four vertices constituting the marker unit MU in the marker coordinate system and the screen coordinate system have already been obtained by the marker unit detection process (step S2) described above. Therefore, a conversion matrix between the marker coordinate system and the camera coordinate system can be obtained from these correspondences. Since two sets of two straight lines facing each other are obtained from the coordinates of the four vertices of the marker unit MU in the screen coordinate system, the posture of the camera is estimated using these two sets of parallel straight lines. When the posture of the camera is obtained, the position of the camera is obtained from the correspondence between the four vertices of the marker unit MU in the marker coordinate system and the screen coordinate system. Then, after appropriately correcting the rotation component of the transformation matrix between the marker coordinate system and the camera coordinate system, the transformation matrix between the marker coordinate system and the camera coordinate system, and the conversion between the pre-registered world coordinate system and the marker coordinate system Using the matrix, the position and orientation of the camera 12 in the world coordinate system is obtained.

  In step S5 of FIG. 4, the processing device 11 determines whether or not the position and orientation of the camera 12 has been successfully detected. If successful, the processing proceeds to step S6, and if unsuccessful, the processing returns to step S1.

(4) Marker element detection process in the next frame In step S6 in FIG. 4, a process similar to the above is performed on the image of the next frame captured by the camera 12. At this time, in the process of step S1, the marker element ME is detected by performing raster scanning on the entire frame image. However, in step S6, the marker element ME is detected by limiting the scanning range. Do.
In the process of step S6, since the position and orientation detection of the camera 12 has succeeded in the previous frame, it is considered that the marker element ME and the marker unit MU are also accurately detected. Therefore, by limiting the scanning range based on the already detected position information of the marker element ME, it is possible to accurately detect the marker element ME while improving the processing speed.
Therefore, the marker element detection means 23 of this embodiment scans the entire captured image and detects the marker element ME, and the scanning range based on the position information of the marker element ME that has already been detected. A second detection function for detecting the marker element ME in a limited manner is provided.

  In step S6, the marker element ME detected in the previous frame is scanned only in the circumscribed rectangular region. In this case, it is determined from the shape and inclination of the marker element ME whether or not it is the same as that detected in the previous frame, and when the marker element at the same position is detected, the subsequent scanning is not performed.

When a plurality of marker units MU are installed on the object R, in step S6, marker elements of the marker unit MU that have not been detected in the previous frame (marker elements hidden by other objects in the previous frame) are displayed. A new image may appear in the image.
For example, as shown in FIG. 9, when the marker units MU1 and MU2 are set on the two side surfaces of the shelf R, the image of the previous frame includes all the marker elements ME of one marker unit MU1. When a part of the marker element ME of the other marker unit MU2 is captured (only the marker element ME surrounded by the circle B is captured), in step S6, information on the position and orientation of the existing camera is displayed. The perspective projection conversion is performed based on the combination information of the marker units MU1 and MU2 stored in the storage unit 21, and the position of the marker element ME (marker element ME surrounded by the circle C) that has not been captured in the previous frame is determined. Predict and scan the surrounding area.
Accordingly, the marker element ME can be reliably detected within a limited range without scanning the entire frame image.

(5) Output of virtual information When the position and orientation of the camera are detected through the above-described procedure, the processing device 11 performs the CG image or the like in a state of being aligned with the object R by the function of the virtual information output means 26 (FIG. 3). Virtual information such as characters is superimposed on the display 13.

  As described above in detail, in the mixed reality system 1 according to the present embodiment, the marker unit MU has a color that approximates the color of the object R in the real environment. There is almost no loss of aesthetics. The marker element ME constituting the marker unit MU is formed in a shape (L-shaped, I-shaped, triangular, etc.) along the outer shape (corner or side) of the object R. Is less conspicuous, and the person who sees it is less likely to feel uncomfortable. Furthermore, the marker unit MU can sufficiently ensure the identification of the marker unit by configuring the identification information by combining the outer shape of the plurality of marker elements ME and the mutual arrangement relationship.

As described above, in the present embodiment, the color of the marker element ME constituting the marker unit MU is set to a color approximate to the color of the object so as not to impair the beauty of the real environment, and each marker element ME is It is formed in a shape that matches the shape of the object. Therefore, the color and shape of the marker element ME of the present embodiment depends on the actual environment, and it is difficult to determine what color and shape the necessary aesthetics and distinguishability can be obtained.
Therefore, the inventor of the present application has constructed a marker unit creation support system for creating an optimum marker unit according to an object in the real environment. Hereinafter, this system will be described in detail.

[Configuration of marker unit creation support system]
FIG. 10 is a diagram showing a schematic configuration of the marker unit creation support system. The marker unit creation support system 30 includes a processing device 38, a camera (imaging device) 39, and a printer (output device) 40. The processing device 38 includes a storage device including a CPU, RAM, ROM, HDD, and the like, and a computer including an I / O interface. Further, the processing device 38 is exemplified as a notebook personal computer integrally provided with a display 38a. As the camera 39, a digital camera or a digital video camera is used, and at least a still image can be taken. As the printer 40, a color printer such as an ink jet printer or a laser printer is used. The camera 39 and the printer 40 are connected to the processing device 38 via an I / O interface so that they can communicate with each other in a wired or wireless manner.

FIG. 11 is a diagram illustrating a functional configuration of the processing device 38. The processing device 38 includes a communication unit 41 that communicates with the camera 39 and the printer 40, an image input unit 42 that inputs an image photographed by the camera 39, and an area of the object R where the marker unit MU is arranged from the input image. Target area extracting means 43 to be extracted, marker unit presenting means 44 for presenting a plurality of marker unit MU candidates suitable for the extracted area to the user, selection for accepting the user's selection from the presented marker unit MU candidates A receiving unit 45, an output instruction unit 46 that instructs the printer 40 to output the marker unit MU based on the selection, a determination receiving unit 47 that receives the determination of the marker unit MU, and the determined marker unit MU and object R Information registration means 48 for registering information and the like are provided as its function. The communication unit 41 is configured as a function of an I / O interface physically connected to the camera 39 and the printer 40, and other units function by an application program or the like installed in the storage device of the processing device 38.
Details of these functions will be described together with the following processing procedure.

[Marker unit creation processing procedure]
FIG. 12 is a flowchart illustrating a processing procedure performed by the processing device 38 of the marker unit creation support system 30.
In the marker unit creation support system 30, first, as shown in FIG. 10, an image of the real environment RE where the marker unit is installed is taken by the camera 39. The captured image is input to the processing device 38 by the function of the image input means 42 in step S11 of FIG.

In step S <b> 12, the processing device 38 extracts the region of the object (door in the example of FIG. 10) R on which the marker unit is placed from the input captured image by the function of the target region extraction unit 43. An image of the extracted door is displayed on the display 38a of FIG.
Next, in step S13, the processing device 38 displays marker unit candidates suitable for the object R on the display 38a according to the color and shape of the extracted object R by the function of the marker unit presenting unit 44 ( Present). When there are a plurality of marker unit candidates suitable for the object R, the plurality are displayed on the display 38a.

The user selects a marker unit to be used from a plurality of presented marker unit candidates. In step S <b> 14, the processing device 38 accepts the selection of the marker unit candidate by the user by the function of the selection reception unit 45, and further causes the output instruction unit 46 to print the selected marker unit candidate to the printer 40. Send instructions.
The printer 40 prints on the paper SE a marker unit MU that is actually attached to the object R. The user cuts out the marker unit MU printed on the paper SE and actually pastes it on the object R, thereby confirming whether the marker unit MU is in harmony with the object R or not deteriorating the aesthetics, etc. 38, whether or not to use the marker unit MU (whether or not to determine the marker unit) is input.

In step S15, the processing device 38 determines whether or not the determination of the marker unit MU has been received by the function of the determination receiving unit 47. If the determination is received, the process proceeds to step S16 and the determination is made that the determination is not made. If accepted, the process returns to step S13 and the same operation is repeated.
Here, the marker unit presentation unit 44, the selection receiving unit 45, the output instruction unit 46, and the determination receiving unit 47 constitute a marker unit determining unit that determines a marker unit corresponding to an object.

  In step S16, the processing device 38 determines the color and shape of the marker element ME constituting the marker unit MU determined by the function of the information registration unit 48, the position of the three-dimensional space, the color of the object R (background color), and the like. Information is registered in a database on the processing device 38. This registered information can be used as information registered in the storage unit 21 of the processing device 11 in the mixed reality system 1.

By using such a marker unit creation support system 30, it is not necessary for the user himself to determine the color and shape of the marker unit, so the burden on the user can be reduced and a mixed reality system can be easily constructed. It becomes possible to do.
In addition, by presenting a plurality of marker unit candidates by the marker unit presenting means 44, it is possible to select a more optimal marker unit after confirming that the user does not actually impair the aesthetics. it can.

The present invention is not limited to the above embodiment, and can be modified as appropriate.
For example, the shape and size of the marker element ME constituting the marker unit MU are not limited to the illustrated example of the above-described embodiment, and may be changed as appropriate according to the shape and size of the object to which the marker element ME is to be attached. be able to.
The mixed reality system 1 of the present invention can be used not only indoors but also outdoors. For example, when the mixed reality system 1 of the present invention is implemented outdoors in a city or the like, the marker unit MU may be installed on a building such as a building or a house or an object such as a road.

  In addition, as the color of the marker element ME, it is possible to select a color that approximates the color of the object within a detectable range in consideration of illumination conditions and the like. In the experiment conducted by the inventor of the present application, stable detection is possible if the density difference between the marker element ME and the object R is 24 gradations or more (about 10% or more) of 256 gradations under indoor lighting conditions. I found that I can do it. In this way, by obtaining the density difference that can detect the marker element ME according to the environment (illumination conditions, etc.) in which the present invention is implemented, the color of the marker element ME is changed using this density difference as an approximation limit (threshold). Can be set.

  In the above-described embodiment, the marker element ME is arranged on the object R by pasting the marker element ME created separately from the object R to the object R, but the object R itself is colored corresponding to the marker element ME. The marker element ME can be arranged by applying. In addition, the marker element ME can be constituted by a material constituting the surface of the object R. For example, when the marker element ME is arranged in a building R having a brick structure or a tiled outer wall, the brick or tile can be used as the marker element ME.

  In the marker unit creation support system 30, in the above embodiment, the marker unit MU that is actually pasted on the object R is printed (created) by the printer 40. However, the marker unit MU suitable for the object R is designed and designed. It can also be output by the printer 40. In this case, the user himself / herself can easily create the marker unit MU according to the model or the design drawing.

  The mixed reality system of the present invention can be used in various environments and applications such as entertainment systems such as games, work support systems such as medical sites and factory lines, and guidance systems in sightseeing spots and exhibition halls. It is.

It is a figure which shows the outline | summary of the mixed reality system 1 concerning embodiment of this invention. It is a front view which shows the example of arrangement | positioning of a marker unit. It is a figure which shows the function structure of the processing apparatus of a mixed reality system. It is a flowchart which shows the process sequence of a mixed reality system. FIG. 3 is a schematic diagram illustrating pixels of an image. It is a figure explaining the process of a marker element detection means. It is a figure explaining the process of a marker unit detection means. It is a figure explaining the coordinate system used for a camera position and orientation detection. It is a perspective view which shows the example which has arrange | positioned the several marker unit to the object. It is a figure which shows schematic structure of a marker unit creation assistance system. It is a figure which shows the function structure of the computer of a marker unit creation assistance system. It is a flowchart which shows the process sequence by the computer of a marker unit creation assistance system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Mixed reality system 11 Processing apparatus 12 Camera 21 Information storage means 22 Image input means 23 Marker element detection means 24 Marker unit detection means 30 Marker unit creation support system 38 Computer (processing apparatus)
39 Camera (photographing device)
40 Printer (output device)
42 Image Input Means 43 Target Area Extraction Means 44 Marker Unit Presenting Means 45 Selection Acceptance Means 47 Decision Acceptance Means MU Marker Unit ME Marker Element R Object

Claims (8)

A marker unit that is used in a mixed reality system that recognizes the position of an object in a real environment and displays virtual information in a superimposed manner, and is used as an index for recognizing the position of the object by being arranged on the object,
A marker unit for use in a mixed reality system, characterized in that it is colored similar to the color of the object and comprises a plurality of marker elements that constitute identification information by a combination of its outer shape and arrangement.   The marker unit used for the mixed reality system according to claim 1, wherein the color of the marker element is the same color system as the color of the object and the density is approximate.   The marker unit used for the mixed reality system according to claim 1, wherein the color of the marker element is similar to the color of the object and the color system is approximate.   The marker unit used for the mixed reality system according to claim 1, wherein at least a part of the outer shape of the marker element has a shape along the outer shape of the object. A mixed reality system using the marker unit according to any one of claims 1 to 4,
Storage means for storing color information of the object and the marker element arranged on the object, and combination information of the outer shape and arrangement of the marker unit;
Marker element detection means for detecting the marker element based on the color information from a photographed image of the object;
Marker unit detection means for detecting a marker unit constituted by the marker element by determining whether a combination of the detected outer shape and arrangement of the marker elements corresponds to the combination information;
A mixed reality system characterized by A marker unit creation support system for creating a marker unit according to any one of claims 1 to 4 corresponding to an object,
A photographing device for photographing the object, and a processing device for performing a determination process of a marker unit used for the object,
The processing device is
An image input means for inputting an image of the photographed object;
Target area extraction means for extracting the area of the object in which the marker unit is arranged from the input image;
Marker unit determining means for determining a marker unit that can be arranged on the object according to the color and shape of the extracted object;
A marker unit creation support system comprising: Marker unit determination means
Presenting means for presenting a plurality of marker unit candidates that can be placed on the object;
Selection accepting means for accepting a user's selection from a plurality of marker unit candidates;
The marker unit creation support system according to claim 6. A marker unit creation support program for creating a marker unit according to any one of claims 1 to 4 corresponding to an object,
Image input means for inputting a captured image of an object;
Target area extraction means for extracting the area of the object where the marker unit is placed from the input image, and a marker for determining the marker unit that can be placed on the object according to the color and shape of the extracted object Unit determination means,
Marker unit creation support program characterized by causing a computer to function as

Images