JP4540329B2 - Information presentation device - Google Patents

Description

  The present invention relates to an information presenting apparatus that enables an image observer to present information at a high density by simultaneously displaying information related to a photographed image while superimposing the information on the photographed image.

A device using a two-dimensional code is very well known as an information presentation device that outputs predetermined related information to objects and / or predetermined markers in the real world. Among them, as a device for outputting related information using spatial localization information of an object and / or a predetermined marker, a two-dimensional code as a marker printed on a business card as an object is read by a camera, and a computer If the ID of the two-dimensional code is analyzed by the program in the program, the face photograph of the person corresponding to the ID is read from the image file specified by the ID, and it is as if next to the two-dimensional code on the business card A device for displaying on a computer display has been proposed (see, for example, Patent Document 1).
JP 2000-82107 A

  In Patent Document 1, a two-dimensional code is recognized and a position for displaying a face photograph is calculated based on two-dimensional code coordinate data. The display position of the face photograph is always the two-dimensional code. It is a fixed positional relationship. In the case of a business card card, the two-dimensional code and the display position are in the same space, and there is no problem.

  On the other hand, when applied to a large-scale structure such as a plant or factory equipment, for example, an image of an object in a real space such as a building inputted by an image input means such as a CCD photographing device, and the object It is assumed that related information based on markers placed on an object, for example, design information, is superimposed and displayed. When applied to such a large-scale structure such as a plant or factory facility, it is necessary to accurately install the marker according to the design information. However, in reality, it is difficult to set such markers accurately.

  In this way, when the reference marker itself is different from the design information, the real space that is the coordinate system unique to the object is associated with the model space that is the coordinate system unique to the design information of the object (registration If this is not done, it is not possible to display the object image and the object model created from the design information in a superimposed manner.

  The present invention has been made in view of the above points, and an object thereof is to provide an information presenting apparatus that can easily perform a registration operation.

In order to achieve the above object, one aspect of the information presentation apparatus of the present invention is:
Image input means for inputting an image of an object placed in real space;
3D position / orientation relationship output means for detecting and outputting a relative 3D position / orientation relationship between the image input means and the object from an image of a marker placed on the object input from the image input means When,
Shape information storage means for storing shape information of the object;
Based on the position and orientation of the object relative to the image input means detected by the three-dimensional position and orientation relation output means , the shape information of the object stored in the shape information storage means is transformed, and the image input means Error comparison detection means for detecting a difference between the shape at the actual position and orientation of the input object and the deformed shape information of the stored object;
The relative three-dimensional position / orientation relationship between the image input means and the object detected by the three-dimensional position / orientation output means is calculated with respect to the shape information of the object using the difference value from the error comparison / detection means. Three-dimensional position and orientation information management means for correcting by moving the position and orientation of the marker shape information;
Using the relative 3D position and orientation relationship between the image input means and the object detected by the 3D position and orientation relationship output means modified by the 3D position and orientation information management means, the image input means is input. Superimposed image generation means for generating an image obtained by superimposing the image of the target object and the shape information of the target object stored in the shape information storage means,
Display means for displaying an image generated by the superimposed image generation means;
It is characterized by comprising.

  ADVANTAGE OF THE INVENTION According to this invention, the information presentation apparatus which can perform a registration operation | work easily can be provided.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration of an information presentation apparatus and an information presentation system using the information presentation apparatus according to the first embodiment of the present invention.

  This information presentation system includes a small information presentation apparatus 100 that can be carried by a user such as a PDA with a camera or a notebook personal computer, and a three-dimensional CAD 200 installed in an office or the like.

  Here, the information presenting apparatus 100 includes the image input unit 101 such as a camera that inputs (photographs) an image of an object in real space, and the image of the object input from the image input unit 101. A three-dimensional position / orientation relationship output unit 102 for detecting a relative three-dimensional position / orientation relationship between the input unit 101 and the object; a shape information storage unit 103 such as a memory for storing shape information of the object; The position and orientation of the object detected by the position and orientation relation output unit 102 is compared with the value stored in the shape information storage unit 103 corresponding to the part of the object input from the image input unit 101 and input. An error comparison detection unit 104 that detects a difference between the actual position and orientation of the target object and the stored shape information of the target object, and a difference value from the error comparison detection unit 104 is used as the shape information storage unit 103. In case The shape information management unit 105 to be reflected in the shape information, the image of the object input from the image input unit 101, and the shape information of the object stored in the shape information storage unit 103 are superimposed. A superimposed image generating unit 106 that generates an image and a display unit 107 such as a liquid crystal display that displays an image generated by the superimposed image generating unit 106 are configured. Furthermore, the difference value from the error comparison detection unit 104 is reflected in the relative three-dimensional position / orientation relationship between the image input unit 101 and the object detected by the three-dimensional position / orientation relationship output unit 102. A three-dimensional position and orientation information management unit 108 is provided. The three-dimensional position / orientation relationship output unit 102 and the three-dimensional position / orientation information management unit 108 constitute a three-dimensional position / orientation relationship detection unit 120.

  On the other hand, the three-dimensional CAD 200 includes a three-dimensional design information storage unit 201 that is a large-capacity storage device such as a hard disk that stores three-dimensional design information of an object, and a three-dimensional design information stored in the three-dimensional design information storage unit 201. And a shape information generation unit 202 that generates shape information from the design information.

  The shape report information stored in the shape information storage unit 103 of the information presentation device is generated by the shape information generation unit 202 of the three-dimensional CAD 200. The transfer of the shape information from the shape information generation unit 202 to the shape information storage unit 103 may be performed online by wireless or wired communication, or may be performed offline via some storage medium.

  Here, the “three-dimensional design information of an object” is information necessary for manufacturing and installing the object such as shape, color, material, part name, and the like. The shape information generation unit 202 extracts only shape information from the three-dimensional design information of such an object. The object and the shape information each have a unique coordinate system. The coordinate system for the object is referred to as a real space, and the coordinate system for the shape information is referred to as a model space.

  Further, the shape information management unit 105 of the information presenting apparatus 100 uses the difference value from the error comparison detection unit 104 as the above-described 3 as with the extracted shape information stored in the shape information storage unit 103. The three-dimensional design information stored in the three-dimensional design information storage unit 201 of the two-dimensional CAD 200, that is, the original shape information is also reflected. Also in this case, information may be exchanged between the shape information management unit 105 and the three-dimensional design information storage unit 201 of the three-dimensional CAD 200 either online or offline.

Next, the operation in such a configuration will be described.
In the present embodiment, first, as preprocessing, an association (registration) operation between the real space and the model space is performed. That is, using the object itself, the object in the real space and the object model in the model space created from the shape information are displayed in a superimposed manner on the display unit 107, and the model space so that they overlap each other. By moving the object model above, the work of associating the real space with the model space is performed. Then, the moving amount of the object model at the time of the association work is recorded and used for associating the real space with the model space.

  After this associating work, since the association between the coordinate system of the real space and the model space has been completed, the object that is different in the superimposed display is not consistent with the object model (the object is It does not follow the shape information).

  Such an association (registration) work between the real space and the model space is performed as shown in the flowchart of FIG.

  That is, first, the object is photographed by the image input unit 101 (step S11). The object used for this registration is one whose shape information is known in advance to match between the real space and the model space. Further, it may be registered in advance as a landmark for registration.

  Next, the three-dimensional position / orientation output unit 102 detects the relative three-dimensional position / orientation between the image input unit 101 and the object from the image of the object input by the image input unit 101 (step S12). . At this time, since the association (registration) between the real space and the model space is not completed, there is a difference between the two.

  The detection of the three-dimensional position and orientation is performed by, for example, considering the object itself as a marker, and performing marker detection and position and orientation information estimation disclosed in Japanese Patent Laid-Open Nos. 2001-118187 and 2001-122601. This can be done by applying the method.

  Next, shape information corresponding to the object is selected from the shape information storage unit 103 (step S13). In this case, as described above, the object to be used for the registration uses the object whose shape information is known to match between the real space and the model space in advance, so it is automatically selected. Can do. The selected shape information is transformed based on the three-dimensional position / orientation detected by the three-dimensional position / orientation relationship output unit 102 and input to the error comparison / detection unit 104.

  Then, the error comparison detection unit 104 detects an error between the input image from the image input unit 101 and the shape information selected and deformed from the shape information storage unit 103 (step S14). The error at this time occurs because the real space and the model space are not accurately associated. That is, the superimposed image generation unit 106 generates an image obtained by superimposing the input image from the image input unit 101 and the selected shape information and displays the generated image on the display unit 107. On the screen 109 of the unit 107, as shown in FIG. 3A, the image 1A of the object 1 and the object model 2 created from the object shape information are displayed in a shifted manner.

  Therefore, the three-dimensional position / orientation relationship information management unit 108 determines the relative three-dimensional position between the image input unit 101 and the object 1 based on the error detected by the error comparison detection unit 104 so that the error is minimized. Information on the posture relationship is corrected (step S15).

  Then, the three-dimensional position / orientation relationship information management unit 108 supplies information regarding the corrected three-dimensional position / orientation relationship to the three-dimensional position / orientation relationship output unit 102, so that the coordinate system between the real space and the model space can be changed. The association work ends (step S16). Thereby, on the screen 109 of the display unit 107, as shown in FIG. 3B, the image 1A of the object 1 and the object model 2 are displayed in agreement with each other.

  After such pre-processing is completed, the error between the object in the real space and the design information (in this embodiment, shape information) is accurately grasped, and the design information and the actual object are matched. Will work.

  That is, as shown in the flowchart of FIG. 2B, first, the object 1 is photographed by the image input unit 101 (step S21).

  Next, the relative three-dimensional position and orientation between the image input unit 101 and the target object 1 are detected from the image 1A of the target object 1 input by the image input unit 101 by the three-dimensional position and orientation relation output unit 102. (Step S22). At this time, since the association (registration) between the real space and the model space has been completed, there is no difference between the two.

  Next, data corresponding to the three-dimensional position / orientation detected by the three-dimensional position / orientation relation output unit 102 is retrieved from the information on a plurality of shapes stored in the shape information storage unit 103 (step S23). . The retrieved shape information is input to the error comparison detection unit 104.

  Then, the error comparison detection unit 104 detects an error between the input image from the image input unit 101 and the shape information retrieved from the shape information storage unit 103 (step S24). The detected error is, for example, the amount of movement of shape information. Of course, this movement amount may be parallel movement, enlargement / reduction, rotation, or a combination thereof.

  Here, the shape information management unit 105 determines whether or not the detected error is within the allowable range (step S25), and if it is within the allowable range, it is not necessary to perform the subsequent operations.

  On the other hand, if the detected error is not within the allowable range (step S25), the detected error is added to the shape information retrieved from the shape information storage unit 103 by the shape information management unit 105 to obtain the shape information. Is changed (step S26). That is, since the shape of the object 1 that has already been made cannot be changed, the shape information is changed.

  The shape change information is stored in the shape information storage unit 103 to reflect the change (step S27).

  For example, as shown in FIG. 4A, if the object 11 is taken with the parts 12 and 13 attached to the main body 11, the shape information 31 of the main body 11 and the shape information 32 of the part 12 corresponding thereto are captured. , And the shape information 3 of the object 1 including the shape information 33 of the part 13 is retrieved and compared by the error comparison detection unit 104. At this time, the superimposed image generation unit 106 uses the object model 2 created from the image 1A of the object 1 and the shape information 3 (the model 21 of the main body 11, the model 22 of the part 12, and the model 23 of the part 13). Are generated and displayed on the display unit 107. Here, if there is an error between the actual mounting position of the part 13 and the shape information 33 of the part 13, as shown in the figure, an image 13A of the part 13 displayed on the screen 109 of the display unit 107 and its model are displayed. 23 will be displayed. This error is canceled by changing the shape information 33 of the component 13 to the shape information 33 ′ as shown in FIG. 4B by the operations of the above steps S26 and S27, and is displayed on the screen 109. The image 13A of the component 13 and the model 23 ′ after the change coincide with each other and are displayed in an overlapping manner.

  As described above, the shape change information from the shape information management unit 105 can be returned to the three-dimensional design information storage unit 201 of the three-dimensional CAD 200.

[Second Embodiment]
Next, a second embodiment of the present invention will be described.

  In the first embodiment, as the object 1 used in the association (registration) work between the real space and the model space, it is known in advance that the shape information matches between the real space and the model space. There was a need.

  However, such an object 1 may not always be prepared.

  Therefore, in this embodiment, at the time of such registration, since the object 1 whose shape information does not match between the real space and the model space is used in advance, a marker is placed on the object 1 by pasting or the like. Arrange to move.

  That is, when the three-dimensional position / orientation output unit 102 detects a three-dimensional position / orientation and the error comparison / detection unit 104 compares the image 1A of the object 1 with the shape information 3 of the object 1, An error is detected in between. In this case, since the marker is used as a reference, the marker image matches the marker shape information. Therefore, in this case, for example, as shown in FIG. 5A, the image 4A of the marker 4 displayed on the screen 109 of the display unit 107 matches the model 24 of the marker 4, but the object 1 The image 1A and the object model 2 are displayed in a shifted manner.

  Therefore, in the present embodiment, instead of using the three-dimensional position / orientation relationship information management unit 108 as in the first embodiment, as shown in FIG. By moving the position, the object 1 and the shape information 3 of the object 1 are matched. That is, the image observer moves the position of the marker 4 so that the two coincide with each other while observing the captured image 1A of the object 1 and the object model 2 displayed on the display unit 107. It is something to do.

  In this case, in order to facilitate the movement of the position of the marker 4, as shown in FIG. 6, the image 1 A of the object 1 and the image 4 A of the object model 2 and the marker 4 are displayed on the screen 109 of the display unit 107. In addition to the model 24 of the marker 4, a scale 25 may be added to the object model 2 and displayed. By displaying such a scale 25, the observer can confirm the amount of deviation from the scale 25 on the screen 109, and can move the marker 4 according to the confirmed amount of deviation.

[Third Embodiment]
Next, a third embodiment of the present invention will be described.

  Similarly to the second embodiment, the present embodiment uses the object 1 whose shape information does not match between the real space and the model space in advance at the time of registration. This is a case where the marker 4 is arranged on the object 1 by pasting or printing and the marker 4 cannot be moved.

  That is, when the three-dimensional position / orientation output unit 102 detects a three-dimensional position / orientation and the error comparison / detection unit 104 compares the image 1A of the object 1 with the shape information 3 of the object 1, An error is detected in between. In this case, since the marker 4 is used as a reference, the image 4A of the marker 4 matches the shape information 34 of the marker 4. Accordingly, in this case, as shown in FIG. 7A, the image 4A of the marker 4 displayed on the screen 109 of the display unit 107 matches the model 24 of the marker 4, but the image of the object 1 1A and the object model 2 are displayed in a shifted manner.

  Therefore, the error comparison / detection unit 104 changes the relative three-dimensional position / orientation relationship between the image input unit 101 and the object 1 detected by the three-dimensional position / orientation relationship output unit 102 by the three-dimensional position / orientation relationship information management unit 108. In this embodiment, the position and orientation of the shape information 34 of the marker 4 with respect to the shape information 3 of the object 1 are moved to obtain shape information 34 ′. To do. As a result, as shown in FIG. 7B, the image 4A of the marker 4 displayed on the screen 109 of the display unit 107 matches the model 24 ′ to which the marker 4 has been moved, and the object 1 Both the image 1A and the object model 2 will match.

  Further, when the object 1 is photographed from a plurality of directions by the image input unit 101 to be a still image, and the position and orientation of the shape information 34 of the marker 4 with respect to the shape information 3 of the object 1 is moved on any one still image. The movement is reflected in the paired still images. FIG. 8 shows before the movement of the shape information 34 of the marker 4 with respect to the shape information 3 of the object 1, and FIG. 9 shows the state after the movement of the shape information 34 ′ of the marker 4 with respect to the shape information 3 of the object 1 is completed. Show.

  Even when registration is performed by regarding the characteristic unevenness or pattern of the object 1 as a marker, matching can be performed in the same manner as in the present embodiment.

  Further, although the shape information 34 of the marker 4 is automatically moved, it is needless to say that the observer may specify the moving direction and the moving amount with a keyboard or the like. In that case, it is preferable that the scale 25 as described in the second embodiment is added to the object model 2 for display.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described.

  In the first embodiment, the shape information of the object is stored, and the error with the actually photographed object is detected and reflected. However, the attribute information of the object such as the part name is further added. May be stored, and the change of the attribute information may be reflected.

  Therefore, the information presentation system in this embodiment is configured as shown in FIG.

  That is, in addition to the configuration in the first embodiment, the three-dimensional CAD 200 further includes an attribute information generation unit 203 that generates attribute information from the three-dimensional design information of the object stored in the three-dimensional design information storage unit 201. Have. In addition, the information presenting apparatus 100 includes an attribute information storage unit 110 that stores attribute information generated by the attribute information generation unit 203 of the three-dimensional CAD 200, a keyboard 111 as input means, and attribute information input by the keyboard 111. And an attribute information management unit 112 that reflects the change in the attribute information stored in the attribute information storage unit 110.

  In addition to the object image input from the image input unit 101 and the shape information of the object stored in the shape information storage unit 103, the superimposed image generation unit 106 of the information presentation apparatus 100 includes the attribute information storage unit. The attribute information stored in 110 is also superimposed and displayed on the display unit 107.

  Further, the transfer of attribute information from the attribute information generation unit 203 to the attribute information storage unit 110 may be performed online by wireless or wired communication, or may be performed offline via some storage medium.

  Further, the attribute information management unit 112 of the information presentation device 100 changes the attribute information input by the keyboard 111 in the same manner as the attribute information stored in the attribute information storage unit 110. The three-dimensional design information stored in the three-dimensional design information storage unit 201 of the two-dimensional CAD 200, that is, the original attribute information is also reflected. Also in this case, information may be transferred between the attribute information management unit 112 and the three-dimensional design information storage unit 201 of the three-dimensional CAD 200 either online or offline.

[Fifth Embodiment]
Next, a fifth embodiment of the present invention will be described.

  As shown in FIG. 11, the information presentation system in the present embodiment includes a keyboard 111 instead of the error comparison detection unit 104 in the configuration shown in FIG. 1, and a three-dimensional position and orientation relation information management unit from the keyboard 111. The movement amount of the shape information is directly input as a value to 108 and the shape information management unit 105.

  That is, in the above-described first to third embodiments, the superposition deviation is automatically corrected by the error comparison detection unit 104. In contrast, in this embodiment, the observer observes the superimposed screen, for example, from the keyboard 111 so that the deviation between the object image and the object model created from the object shape information is minimized. By instructing the dimension position / orientation relation information management unit 108 and the shape information management unit 105, the deviation is corrected.

  Even with such a configuration, the same effects as those of the first to third embodiments can be obtained.

  Similarly, as shown in FIG. 12, instead of the error comparison detection unit 104 in the configuration shown in FIG. 10, input is made from the keyboard 111 to the three-dimensional position / orientation relationship information management unit 108 and the shape information management unit 105. Of course, it may be. With such a configuration, it is possible to achieve the same effects as the fourth embodiment.

[Sixth Embodiment]
Next, a sixth embodiment of the present invention will be described.

  In the fifth embodiment, the observer observes the superimposed screen, for example, from the keyboard 111 so that the deviation between the object image and the object model created from the object shape information is minimized. By instructing the position / orientation relation information management unit 108 and the shape information management unit 105, the deviation is corrected.

  In the present embodiment, not only the position and orientation but also the shape information management unit 105 is instructed to change the shape or replace parts by linking with the three-dimensional CAD 200.

  The configuration of the information presentation system in the present embodiment is the same as that in the fifth embodiment.

  In the sixth embodiment, for example, as shown in FIG. 13A, if a subject 11 with parts 14 and 15 attached to the main body 11 is photographed as the target object 1, the shape information 31 of the main body 11 corresponding thereto. The shape information 3 of the object 1 including the shape information 35 of the part 14 and the shape information 36 of the part 15 is retrieved, and is generated from the image 1A of the object 1 and the shape information 3 by the superimposed image generation unit 106. An image in which the target object model 2 (the model 21 of the main body 11, the model 26 of the component 14, the model 27 of the component 15) is superimposed is generated and displayed on the display unit 107. Here, if there is an error between the actual shape of the component 14 and the shape information 35 of the component 14, as shown in the figure, an image 14A of the component 14 displayed on the screen 109 of the display unit 107 and its model 26 are displayed. Will be displayed. Similarly, if there is an error between the actual shape of the component 15 and the shape information 36 of the component 15, the image 15A of the component 15 displayed on the screen 109 of the display unit 107 and its model 27 are displayed.

  That is, in the example of this figure, the design information includes the shape information 35 of the straight pipe (long) and the shape information 36 of the straight pipe (short). However, in the object in the real space, the part 14 is an elbow pipe, part Since 15 is a straight pipe (long), in the superimposed image displayed on the screen 109 of the display unit 107, they are displayed shifted.

  Therefore, in such a case, while the observer observes the superimposed screen, for example, the value representing the shape change of the shape information from the keyboard 111 to the shape information management unit 105 or the part change of the shape information Enter a value that represents. As a result, the shape information management unit 105 replaces the straight pipe (long) shape information 35 with the elbow pipe shape information 37 and replaces the straight pipe (short) shape information 36 as shown in FIG. The real space and the target information are matched by performing the operation of changing the length (referred to as the changed shape information 36 ′). That is, the shape information 35 of the component 14 is changed to the shape information 37, and the shape information 36 of the component 15 is changed to the shape information 36 ′, so that the deviation is canceled and the component 14 displayed on the screen 109 is displayed. The image 14A and the model 28 after the change match, and the image 15A of the part 15 and the model 27 ′ after the change match and are displayed in an overlapping manner.

  Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention.

  For example, as shown in FIGS. 14 and 15, the three-dimensional design information storage unit 201 of the three-dimensional CAD 200 by the shape information management unit 105 and / or the attribute information management unit 112 in the first embodiment and / or the fourth embodiment. There is no need to reflect it.

  Also in this case, as described in the fifth embodiment, as shown in FIGS. 16 and 17, instead of the error comparison detection unit 104, the three-dimensional position / orientation relationship information management unit 108 and the shape information management are performed from the keyboard 111. Of course, it does not matter even if it inputs to the part 105.

  Further, the input means is not limited to the keyboard 111, and may of course be a mouse, a trackball, a touch panel, or the like.

(Appendix)
The invention having the following configuration can be extracted from the specific embodiment.

(1) image input means for inputting an image of an object placed in real space;
3D position and orientation relationship output means for detecting and outputting a relative 3D position and orientation relationship between the image input means and the object from the image of the object input from the image input means;
Shape information storage means for storing shape information of the object;
Compare the position and orientation of the object detected by the three-dimensional position and orientation relation output means with the value stored in the shape information storage means corresponding to the part of the object input from the image input means, and input Error comparison detection means for detecting a difference between the actual position and orientation of the target object and the shape information of the stored object;
Three-dimensional position / orientation information to be reflected in the relative three-dimensional position / orientation relationship between the image input means and the object detected by the three-dimensional position / orientation relation output means using the difference value from the error comparison / detection means. Management means;
Superimposed image generation means for generating an image obtained by superimposing the image of the object input from the image input means and the shape information of the object stored in the shape information storage means;
Display means for displaying an image generated by the superimposed image generation means;
An information presentation apparatus comprising:

(Corresponding embodiment)
The first to third embodiments correspond to the embodiment related to the information presentation apparatus described in (1).

(Function and effect)
The information presentation device according to (1) detects a difference between the actual position and orientation of the input object and the shape information of the stored object, and uses the difference value to determine a three-dimensional position and orientation relationship. It is reflected in the relative three-dimensional position and orientation relationship between the image input means detected by the output means and the object.

  Therefore, according to the information presentation apparatus described in (1), the registration work can be easily performed.

  (2) The information presenting apparatus according to (1), wherein the difference detected by the error comparison detection means is a movement amount of shape information.

(Corresponding embodiment)
The first to third embodiments correspond to the embodiment related to the information presentation apparatus described in (2).

(Function and effect)
The information presentation device according to (2) detects the movement amount of the shape information as a difference between the actual position and orientation of the input target object and the stored shape information of the target object.

  (3) It further comprises shape information management means for reflecting the difference value from the error comparison detection means in the shape information stored in the shape information storage means (1) or (2) The information presentation device described in 1.

(Corresponding embodiment)
The first to third embodiments correspond to the embodiment related to the information presentation device described in (3).

(Function and effect)
The information presenting device described in (3) stores only the shape information of the object, and calculates a difference value between the actual position and orientation of the input object and the stored shape information of the object. It is reflected in the stored shape information.

  Therefore, according to the information presenting device described in (3), only the shape information that is a part of the design information, not the entire design information, is stored and matched with the actual object. No device or high-performance display device is required, and it can be realized by a portable device.

  (4) The three-dimensional position / orientation information management means includes a first difference value detected by the error comparison detection means based on an image obtained by photographing the object from the first direction, and the first direction. The information according to any one of (1) to (3), characterized by using a second difference value detected by the error comparison detection means based on images taken from different second directions. Presentation device.

(Corresponding embodiment)
The embodiment relating to the information presentation device described in (4) corresponds to the third embodiment.

(Function and effect)
The information presenting device described in (4) manages the three-dimensional position / orientation information using each difference value detected by the error comparison / detection means based on each of the images taken from a plurality of directions.

  Therefore, according to the information presenting apparatus described in (4), the difference can be detected from images in a plurality of directions, so that the registration work can be performed more accurately.

  (5) The display means includes a first image generated by the superimposed image generation means based on an image obtained by photographing the object from the first direction, and a second direction different from the first direction. The information presentation apparatus according to any one of (1) to (4), wherein the second image generated by the superimposed image generation unit is displayed based on the captured image.

(Corresponding embodiment)
The embodiment relating to the information presentation device described in (5) corresponds to the third embodiment.

(Function and effect)
The information presentation device according to (5) displays an image generated by the superimposed image generation unit based on each of the images taken from a plurality of directions.

  Therefore, according to the information presenting device described in (5), the superimposed image in a plurality of directions can be displayed, so that the result of the registration work can be confirmed in more detail from the plurality of directions.

(6) image input means for inputting an image of an object placed in real space;
3D position and orientation relationship output means for detecting and outputting a relative 3D position and orientation relationship between the image input means and the object from the image of the object input from the image input means;
Shape information storage means for storing shape information of the object;
Input means;
Three-dimensional position / orientation information management means for reflecting the relative three-dimensional position / orientation relationship between the image input means and the object detected by the three-dimensional position / orientation relation output means using a value input from the input means. When,
Superimposed image generation means for generating an image obtained by superimposing the image of the object input from the image input means and the shape information of the object stored in the shape information storage means;
Display means for displaying an image generated by the superimposed image generation means;
An information presentation apparatus comprising:

(Corresponding embodiment)
The fifth embodiment corresponds to the embodiment related to the information presentation device described in (6).

(Function and effect)
The information presenting device described in (6) uses a value that minimizes the overlay displacement on the screen, which is input by the observer from an input means such as a keyboard while the observer observes the superimposed screen, This is reflected in the relative three-dimensional position and orientation relationship between the image input means detected by the three-dimensional position and orientation relation output means and the object.

  Therefore, according to the information presentation apparatus described in (6), the registration work can be easily performed. Further, since the observer corrects the deviation by inputting from the input means such as a keyboard while observing the superimposed screen, the error comparison detecting means is not required.

  (7) The information presentation apparatus according to (6), wherein the value input from the input unit is a movement amount of shape information.

(Corresponding embodiment)
The fifth embodiment corresponds to the embodiment related to the information presentation device described in (7).

(Function and effect)
The information presentation device described in (7) directly inputs the movement amount of the shape information as a value.

  (8) The method according to (6) or (7), further comprising shape information management means for reflecting the shape information stored in the shape information storage means using the value input from the input means. Information presentation device.

(Corresponding embodiment)
The fifth embodiment corresponds to the embodiment related to the information presentation device described in (8).

(Function and effect)
The information presentation device described in (8) stores only the shape information of the object, and reflects the value input from the input means in the stored shape information.

  Therefore, according to the information presentation device described in (8), not only all design information but only shape information as a part thereof is stored, and this is matched with an actual object, so that a large-capacity memory is stored. No device or high-performance display device is required, and it can be realized by a portable device.

  (9) The three-dimensional position / orientation information management unit includes a first movement amount of the shape information input from the input unit based on an image obtained by photographing the object from the first direction, and the first direction. (6) to (8), wherein a second movement amount of shape information input from the input unit based on an image photographed from a second direction different from the first direction is used. Information presentation device.

(Corresponding embodiment)
The fifth embodiment corresponds to the embodiment related to the information presentation device described in (9).

(Function and effect)
The information presenting device described in (9) manages the three-dimensional position and orientation information using the movement amount of the shape information input from the input unit based on each of the images taken from a plurality of directions.

  Therefore, according to the information presentation apparatus described in (9), since the movement amount can be input while observing images in a plurality of directions, the registration work can be performed more accurately.

  (10) The display unit is configured to generate a first image generated by the superimposed image generation unit based on an image obtained by capturing the object from the first direction, and a second direction different from the first direction. The information presentation apparatus according to any one of (6) to (9), wherein the second image generated by the superimposed image generation unit is displayed based on the captured image.

(Corresponding embodiment)
The fifth embodiment corresponds to the embodiment related to the information presentation device described in (10).

(Function and effect)
The information presentation device according to (10) displays an image generated by the superimposed image generation unit based on each of images captured from a plurality of directions.

  Therefore, according to the information presentation device described in (10), since the superimposed images in a plurality of directions can be displayed, the result of the registration work can be confirmed in more detail from the plurality of directions.

  (11) The information presentation apparatus according to (6), wherein the value input from the input unit is a value expressing a shape change of the shape information.

(Corresponding embodiment)
The sixth embodiment corresponds to the embodiment related to the information presentation device described in (11).

(Function and effect)
According to the information presenting device described in (11), a value expressing the shape change of the shape information can be directly input.

  (12) The information presentation apparatus according to (6), wherein the value input from the input unit is a value expressing a part change of the shape information.

(Corresponding embodiment)
The sixth embodiment corresponds to the embodiment related to the information presentation apparatus described in (12).

(Function and effect)
According to the information presenting apparatus described in (12), a value expressing the part change of the shape information can be directly input.

  (13) The method according to (11) or (12), further comprising shape information management means for reflecting the shape information stored in the shape information storage means using the value input from the input means. Information presentation device.

(Corresponding embodiment)
The sixth embodiment corresponds to the embodiment related to the information presentation apparatus described in (13).

(Function and effect)
The information presenting device described in (13) stores only the shape information of the object, and reflects the value input from the input means in the stored shape information.

  Therefore, according to the information presenting apparatus described in (13), not only all design information but only part of the shape information is stored and matched with the actual object. No device or high-performance display device is required, and it can be realized by a portable device.

It is a block diagram which shows the structure of the information presentation apparatus which concerns on 1st Embodiment of this invention, and an information presentation system using the same. (A) is a flowchart showing a procedure of association (registration) work between the real space and the model space, and (B) shows a work procedure for matching the object in the real space with the design information (shape information). It is a flowchart. (A) is a figure which shows the target object and display state before completion of registration work, (B) is a figure which shows the target object and display state after completion of registration work. (A) is a figure which shows the target object before the operation completion which makes the target object and design information (shape information) in real space correspond, a display state, and the shape information of a target object, (B) is in real space It is a figure which shows the target object after the completion of the operation | work which makes the target object, and design information (shape information) correspond, a display state, and the shape information of a target object. (A) is a figure which shows the target object and display state before completion of the registration work in 2nd Embodiment of this invention, (B) is a figure which shows the target object and display state after completion of registration work. is there. It is a figure which shows the target object and display state before and after the completion of the registration operation for explaining a modification of the second embodiment. (A) is a figure which shows the target object before the registration work completion in 3rd Embodiment of this invention, a display state, and the shape information of a target object, (B) is the target object and display after completion of a registration work. It is a figure which shows a state and the shape information of a target object. It is a figure which shows the target object before the registration operation completion for describing the modification of 3rd Embodiment, a display state, and the shape information of a target object. It is a figure which shows the target object after the registration operation completion for describing the modification of 3rd Embodiment, a display state, and the shape information of a target object. It is a block diagram which shows the structure of the information presentation apparatus which concerns on 4th Embodiment of this invention, and an information presentation system using the same. It is a block diagram which shows the structure of the information presentation apparatus which concerns on 5th Embodiment of this invention, and an information presentation system using the same. It is a block block diagram of the information presentation apparatus for demonstrating the modification of 5th Embodiment, and an information presentation system using the same. (A) is the information presentation apparatus and the information presentation system using the information presentation apparatus according to the sixth embodiment of the present invention, and the object and display before completion of the work for matching the object in the real space with the design information (shape information). It is a figure which shows a shape and the shape information of a target object, (B) is the target object after completion | finish of the operation | work which matches the target object and design information (shape information) in real space, and the shape information of a target object FIG. It is a block block diagram of the information presentation apparatus for demonstrating the modification of 1st Embodiment, and an information presentation system using the same. It is a block block diagram of the information presentation apparatus for demonstrating the modification of 4th Embodiment, and an information presentation system using the same. It is a block block diagram of the information presentation apparatus for demonstrating the structure which further modified the modification of FIG. 14, and an information presentation system using the same. It is a block block diagram of the information presentation apparatus for demonstrating the structure which further modified the modification of FIG. 15, and an information presentation system using the same.

Explanation of symbols

    DESCRIPTION OF SYMBOLS 1 ... Object, 1A ... Image of object, 2 ... Object model, 3 ... Shape information of object, 4 ... Marker, 4A ... Image of marker, 11 ... Main body, 12, 13, 14, 15 ... Parts, 13A ... Image of part 13 14A ... Image of part 14 15A ... Image of part 15 21 ... Model of main body 22 ... Model of part 12 23 ... Model of part 13 23 '... Part 13 after change Model 24 ... Marker model 24 '... Moved marker model 25 ... Scale 26 ... Part 14 model 27 ... Part 15 model 27', 28 ... Modified model 31 ... Main body Shape information, 32 ... Shape information of part 12, 33 ... Shape information of part 13, 33 '... Shape information of part 13 after change, 34 ... Shape information of marker, 34' ... Shape of moved marker Shape information 35 ... Shape information of the part 14 36 ... Shape information of the part 15 36 '... Shape information after change, 37 ... Shape information of the elbow tube, 100 ... Information presentation device, 101 ... Image input unit, 102 ... Three-dimensional position / orientation relationship output unit 103 ... Shape information storage unit 104 ... Error comparison detection unit 105 ... Shape information management unit 106 ... Superimposed image generation unit 107 ... Display unit 108 ... Three-dimensional position / orientation relationship information management 109: Screen, 110 ... Attribute information storage unit, 111 ... Keyboard, 112 ... Attribute information management unit, 120 ... Three-dimensional position and orientation relation detection unit, 200 ... Three-dimensional CAD, 201 ... Three-dimensional design information storage unit, 202 ... shape information generation unit, 203 ... attribute information generation unit.

Claims (3)

Image input means for inputting an image of an object placed in real space;
3D position / orientation relationship output means for detecting and outputting a relative 3D position / orientation relationship between the image input means and the object from an image of a marker placed on the object input from the image input means When,
Shape information storage means for storing shape information of the object;
Based on the position and orientation of the object relative to the image input means detected by the three-dimensional position and orientation relation output means , the shape information of the object stored in the shape information storage means is transformed, and the image input means Error comparison detection means for detecting a difference between the shape at the actual position and orientation of the input object and the deformed shape information of the stored object;
The relative three-dimensional position / orientation relationship between the image input means and the object detected by the three-dimensional position / orientation output means is calculated with respect to the shape information of the object using the difference value from the error comparison / detection means. Three-dimensional position and orientation information management means for correcting by moving the position and orientation of the marker shape information;
Using the relative 3D position and orientation relationship between the image input means and the object detected by the 3D position and orientation relationship output means modified by the 3D position and orientation information management means, the image input means is input. Superimposed image generation means for generating an image obtained by superimposing the image of the target object and the shape information of the target object stored in the shape information storage means,
Display means for displaying an image generated by the superimposed image generation means;
An information presentation apparatus comprising: The difference detected by the error comparison detection means is the deformed shape information of the stored object relative to the shape of the object in the actual position and orientation input by the image input means when the marker is used as a reference. The information presenting apparatus according to claim 1, wherein the information presenting apparatus is an amount of movement. Further comprising shape information management means for reflecting the shape information stored in the shape information storage means using the difference value from the error comparison detection means;
The superimposed image generation means includes an object image input from the image input means, and shape information of the object after being reflected using the difference value stored in the shape information storage means, The information presentation apparatus according to claim 1, wherein an image in which the images are superimposed is generated.

Images