JP2022153509A - Measurement support system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a measurement system and a user interface device that can improve operability with respect to a measuring machine.

SOLUTION: A measurement system of the present invention comprises: a measuring machine that measures a measurement object; a measurement control unit that controls the measuring machine on the basis of instructions by users; a three-dimensional sensor unit that detects a three-dimensional coordinate of a prescribed object in a real three-dimensional space; a button setting area in the real three-dimensional space; a command setting unit that sets association of the button setting area with a command; a moving body detection unit that detects a moving body from the three-dimensional coordinate detected by the three-dimensional sensor unit; and a command implementation unit that, when the three-dimensional coordinate of the moving body detected by the moving body detection unit is located in the button setting area, performs the command associated with the button setting area.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to a measuring system and a user interface device with improved operability for a measuring machine.

2. Description of the Related Art In a measuring machine that measures the dimensions and three-dimensional position of an object to be measured (work), a user uses a measuring head to perform an operation of specifying a measurement point on the work. Depending on the shape of the workpiece, the user may have to handle the measuring head in an unnatural posture. A computer is connected to the measuring machine, and various information such as workpiece information, measuring procedures, and measurement results are displayed on the display. However, if the work and the computer are separated, it is not possible to efficiently perform both the measurement work and the display reference.

In the position measuring device disclosed in Patent Literature 1, a design model read into CAD software running on a computer is displayed on the display of the computer. Then, the data obtained by the measurement is displayed on the design model of the display.

Further, the coordinate measuring machine described in Patent Document 2 is provided with a device including an image projector, and visual guide information and measurement results are projected onto the component from the image projector. Further, Patent Documents 3 to 6 disclose a measurement system and a user interface device using a virtual reality space.

Japanese Patent Application Publication No. 2011-519419 Japanese Patent Publication No. 2013-517500 JP-A-06-241754 Japanese Patent Publication No. 2011-521318 Japanese Patent Publication No. 11-513157 Japanese translation of PCT publication No. 2009-521985

However, none of the techniques can be said to be a system that fully considers usability for the user. For example, when measuring a work, the user's hands are often occupied, and it is difficult to control the measuring machine while performing the measurement work. Also, depending on the work, the user may move to a place suitable for measurement and work there. When the user is away from the control device (computer) of the measuring machine, it becomes more difficult to control the measuring machine while the measuring operation is being performed.

SUMMARY OF THE INVENTION An object of the present invention is to provide a measuring system and a user interface device that can improve the operability of a measuring machine.

In order to solve the above problems, the measurement system of the present invention includes a measuring machine that measures an object to be measured, a measuring machine control unit that controls the measuring machine based on instructions from a user, and a predetermined object in a real three-dimensional space. a three-dimensional sensor unit for detecting the three-dimensional coordinates of the three-dimensional coordinates detected by the three-dimensional sensor unit; and a command execution unit for executing a command associated with the button setting area when the three-dimensional coordinates of the moving object detected by the moving object detection unit are positioned within the button setting area. characterized by

According to such a configuration, a desired position in the real three-dimensional space is set as an area functioning as a button by the command setting section. Further, by detecting a moving object by the moving object detection unit, a moving object (moving object) such as a user is detected. Accordingly, it is possible to detect that a button setting area in the real three-dimensional space has been selected from the action of the moving body, and execute the command associated with the button setting area.

In the measuring system of the present invention, the measuring machine has a measuring head that acquires the three-dimensional coordinates of the object to be measured, and an image output unit that is provided on the measuring head and projects an image. You may make it output the image of the button setting area|region set by . With such a configuration, it is possible to project the image of the button setting area output by the video output unit.

In the measurement system of the present invention, the moving object detection section may detect the movement of the measurement head, and the command execution section may execute the command according to the three-dimensional coordinates of the measurement head detected by the movement detection section. With such a configuration, it is possible to detect that a button setting area has been selected from the operation of the measuring head, and execute a command associated with the button setting area.

In the measurement system of the present invention, the moving object detection unit detects motion of at least one of the user's hand and/or foot, and the command execution unit detects at least one of the user's hand and/or foot detected by the moving object detection unit in three dimensions. A command may be executed according to the coordinates. With such a configuration, it is possible to detect that a button setting area has been selected from the motion of the user's hand or foot, and execute a command associated with the button setting area.

In the measurement system of the present invention, the command execution unit executes the first command when the three-dimensional coordinates of the moving object detected by the moving object detection unit remain within the button setting area for a certain period of time. is moving along the button setting area, the second command may be executed. With such a configuration, commands can be switched according to the type of movement of the moving object.

In the measurement system of the present invention, a three-dimensional imaging unit that acquires a three-dimensional image in a real three-dimensional space, a display unit that displays the three-dimensional image acquired by the three-dimensional imaging unit, and a three-dimensional image in the real three-dimensional space. A display control unit that synthesizes the image corresponding to the button setting area and displays it on the display unit may be further provided. With such a configuration, it is possible to perform display in which the image of the button setting area is combined with the three-dimensional video displayed in the real three-dimensional space.

In the measurement system of the present invention, the command execution section may select and execute one of a plurality of commands according to the shape of the user's hand detected by the moving object detection section. With such a configuration, it is possible to switch display/non-display of the button setting area according to the shape of the user's hand, and to execute a command corresponding to the button setting area.

The measuring system of the present invention includes a measuring machine that measures an object to be measured, a measuring machine control unit that controls the measuring machine based on an instruction from a user, and a three-dimensional coordinate of a predetermined object in a real three-dimensional space. A three-dimensional sensor, a moving body detector that detects user actions from the three-dimensional coordinates detected by the three-dimensional sensor, a display that displays an image of the measuring machine in a virtual three-dimensional space, and a display that displays the image. a display control unit for moving the display position of the image of the measuring machine in the virtual three-dimensional space according to the user's motion detected by the moving object detection unit.

According to such a configuration, an image of the measuring machine is displayed in the virtual three-dimensional space. Also, the user's motion can be detected by the motion detection unit, and the image of the measuring machine in the virtual three-dimensional space can be moved in accordance with the user's motion.

The measurement system of the present invention may further include a measurement operation storage unit that records the measurement operation along the display position of the image of the measuring machine moved by the user within the virtual three-dimensional space displayed on the display unit. . With such a configuration, the measurement operation can be recorded by moving the image of the measuring machine within the virtual three-dimensional space.

In the measuring system of the present invention, the measuring machine control section may control the measuring machine based on the measuring operation stored in the measuring operation storage section. With such a configuration, it is possible to perform measurement with the actual measuring machine based on the measurement operation recorded by moving the image of the measuring machine within the virtual three-dimensional space.

The measurement system of the present invention further includes an image storage unit for storing CAD (Computer Aided Design) images of the measuring machine, and the display unit displays the CAD images stored in the image storage unit in a virtual three-dimensional space. can be With such a configuration, a CAD image of the measuring machine can be displayed in the virtual three-dimensional space.

The measurement system of the present invention further includes an imaging unit that acquires an image of the measuring instrument in the real three-dimensional space, and the display control unit displays the image of the measuring instrument acquired by the imaging unit in the virtual three-dimensional space of the display unit. You can let it run. With such a configuration, it is possible to display an image of the actual measuring machine in the virtual three-dimensional space.

In the measurement system of the present invention, the display control section may move the image of the measuring machine displayed in the virtual three-dimensional space of the display section in accordance with a preset measurement operation. With such a configuration, it is possible to confirm the preset measurement operation of the measuring machine in the virtual three-dimensional space.

In the measurement system of the present invention, the display control unit may synthesize the image of the object to be measured and the measurement result and display them in the virtual three-dimensional space of the display unit. With such a configuration, it is possible to synthesize the measurement result of the measurement object with the image of the measurement object in the virtual three-dimensional space and refer to it.

The measurement system of the present invention further comprises a remote control unit connected to the measuring device control unit via a network, the remote control unit performing measurement operations specified by the user in the virtual three-dimensional space displayed on the display unit. The command may be transmitted to the measuring device control section via the network based on the above. With such a configuration, it is possible to refer to the measurement operation specified by the user in the virtual three-dimensional space at a position (remote location) away from the measuring machine. Commands can also be sent to the measuring instrument from a remote location via the network.

The measuring system of the present invention includes a measuring machine that measures an object to be measured, a measuring machine control unit that controls the measuring machine based on an instruction from a user, an imaging unit that acquires an image of the measuring machine in a real three-dimensional space, and a measuring machine. An image storage unit that stores images related to the measuring machine, a display unit that displays images in a real three-dimensional space, images of the measuring machine acquired by the imaging unit, and images stored in the image storage unit are stored in the real three-dimensional space. and a display control unit for synthesizing and displaying on a display unit. According to such a configuration, it is possible to synthesize and display an image of an actual measuring machine and an image such as a graphic in an actual three-dimensional space.

In the measurement system of the present invention, the display control unit may synthesize an image representing a predetermined measurement procedure with the image of the measuring device acquired by the imaging unit. With such a configuration, an actual image of the measuring machine and an image of the measuring machine, such as a graphic representing a predetermined measurement procedure, can be synthesized and displayed in a three-dimensional space.

In the measurement system of the present invention, the display control section may synthesize an image corresponding to a predetermined abnormality with the image of the measuring device acquired by the imaging section. With such a configuration, the actual image of the measuring machine and the image of the measuring machine, such as graphics corresponding to a predetermined abnormality, can be synthesized and displayed in a three-dimensional space.

In the measurement system of the present invention, the display control section may synthesize an image corresponding to a predetermined guidance with the image of the measuring device acquired by the imaging section. With such a configuration, it is possible to synthesize and display the actual image of the measuring machine and the image of the measuring machine such as graphics corresponding to the predetermined guidance in the three-dimensional space.

A measurement system of the present invention is a measurement system comprising a user-side control device connected to a measuring machine that measures an object to be measured, and a supporter-side control device connected to the user-side control device via a network. be.

The user-side control device in this measurement system includes a measuring machine control section that controls the measuring machine based on instructions from the user, and a first three-dimensional sensor section that detects the three-dimensional coordinates of a predetermined object in the real three-dimensional space. a first moving body detection unit that detects a moving body from the three-dimensional coordinates detected by the first three-dimensional sensor unit; a three-dimensional imaging unit that acquires a three-dimensional image in a real three-dimensional space; and a three-dimensional imaging unit. A first display section for displaying the acquired 3D video, and a first display control section for synthesizing the 3D video and an image sent from the supporter side control device and displaying the result on the first display section.

The supporter side control device in this measurement system includes a second three-dimensional sensor section for detecting the three-dimensional coordinates of the supporter in the real three-dimensional space, and a support system based on the three-dimensional coordinates detected by the second three-dimensional sensor section. a second moving object detection unit for detecting a motion of a person; a second display unit for displaying an image based on the 3D image acquired by the 3D imaging unit in a virtual 3D space; and 3D coordinates in the virtual 3D space. A second display control unit that displays an image sent from the user-side control device according to the time on the second display unit, and an image related to the supporter's motion detected by the second moving body detection unit is transmitted via the network to the user-side control device. and a motion information transmission unit for sending to.

According to such a configuration, the user side where the measuring machine is installed and the supporter side of the measuring machine are connected to each other via a network. The user side can refer to the combination of the image of the measuring device displayed on the first display unit and the image sent from the supporter side. On the side of the supporter, in the virtual three-dimensional space displayed on the second display unit, it is possible to refer to the composition of the image of the measuring machine, such as graphics, and the image sent from the user side.

In the measurement system of the present invention, the first display control unit causes the first display unit to display a supporter-side image corresponding to the supporter based on the image sent from the supporter-side control device, and the second display control unit Alternatively, a user-side image corresponding to the user may be displayed on the second display unit based on the image sent from the user-side control device. With such a configuration, the user side can refer to the supporter side image sent from the supporter side on the first display section, and the supporter side can refer to the user side image sent from the user side on the second display section. can do.

The measurement system of the present invention includes a measuring machine that measures an object to be measured, a measurement result storage unit that stores the measurement results obtained by the measuring machine, an imaging unit that acquires an image of the object to be measured, and an image obtained by the imaging unit. A feature point extraction unit for extracting feature points, a display unit for displaying an image of an object to be measured obtained from the image acquired by the imaging unit or the feature points extracted by the feature extraction unit, and the feature points extracted by the feature point extraction unit. A measurement result reading unit that reads out the measurement results corresponding to a predetermined position of the object to be measured from the measurement result storage unit based on the above, and displays the measurement results read out by the measurement result reading unit by synthesizing them with a video or image of the object to be measured. and a display control unit for displaying on the unit.

According to such a configuration, by projecting the image of the object to be measured, the characteristic points of the object to be measured are extracted, and based on these characteristic points, the measurement result corresponding to the predetermined position of the object to be measured is displayed on the object to be measured. can be displayed by combining it with the video or image of

In the measurement system of the present invention, the display control section may synthesize the measurement result with the video or image in accordance with the imaging angle of the video of the object to be measured acquired by the imaging section. According to such a configuration, it is possible to photograph the object to be measured at a desired angle and display the measurement result.

The measurement system of the present invention comprises a measuring machine that measures an object to be measured, a measurement result storage unit that stores measurement results obtained by the measuring machine, an imaging unit that acquires an image of a marker, and an image of the marker that is acquired by the imaging unit. a marker recognition unit that recognizes a marker; a measurement result reading unit that reads the measurement result corresponding to the marker recognized by the marker recognition unit from the measurement result storage unit; and a display control unit that causes the display unit to display the measurement result read out by the unit.

According to such a configuration, by acquiring the image of the marker, the measurement result corresponding to this marker can be read, and the measurement result can be displayed on the display unit while the image of the marker is being captured. .

In the measurement system of the present invention, the imaging unit acquires a background image with markers attached thereto, and the display control unit synthesizes the measurement results corresponding to the markers with the background image and causes the display unit to display the result. good too. With such a configuration, the measurement result can be combined with the background image and displayed on the display unit.

The user interface device of the present invention comprises a three-dimensional sensor unit for detecting three-dimensional coordinates of a predetermined object in a real three-dimensional space, a button setting area in the real three-dimensional space, and correspondence between the button setting area and a command. , a moving object detection unit for detecting a moving object from the three-dimensional coordinates detected by the three-dimensional sensor unit, and a button when the three-dimensional coordinates of the moving object detected by the moving object detection unit are positioned within the button setting area. a command execution unit that executes a command associated with a set region; a 3D imaging unit that acquires a 3D image in a real 3D space; a display unit that displays the 3D image acquired by the 3D imaging unit; a display control unit for synthesizing a three-dimensional image and an image corresponding to the button setting area and displaying the result on a display unit; One of the commands is selected and executed.

According to such a configuration, it is possible to switch display/non-display of the button setting area according to the shape of the user's hand, and to execute a command corresponding to the button setting area.

The user interface device of the present invention includes an imaging unit that acquires an image of an object to be measured, a feature point extraction unit that extracts feature points from the image acquired by the imaging unit, and an image acquired by the imaging unit or the feature extraction unit. a display unit for displaying an image of the object to be measured obtained from the feature points extracted by the feature point extraction unit; and a display control unit for synthesizing the dimension information acquired by the dimension acquisition unit with a video or image of the object to be measured and displaying the result on a display unit.

According to such a configuration, by projecting the image of the object to be measured, the characteristic points of the object to be measured are extracted, and based on these characteristic points, the measurement result corresponding to the predetermined position of the object to be measured is displayed on the object to be measured. can be displayed by combining it with the video or image of

The user interface device of the present invention includes an imaging unit that acquires an image of a background to which a marker is added, a marker recognition unit that recognizes the marker from the image of the marker acquired by the imaging unit, and a marker recognized by the marker recognition unit that corresponds to the marker. and a display control unit that synthesizes the measurement result obtained by the measurement result obtaining unit with the background image and displays it on the display unit while the image of the marker is being obtained by the imaging unit. and

According to such a configuration, by acquiring the image of the marker, the measurement result corresponding to this marker can be read, and the measurement result can be displayed on the display unit while the image of the marker is being captured. .

1A and 1B are configuration diagrams illustrating a measurement system according to a first embodiment; FIG. It is a schematic diagram which shows the example of an operation. (a) and (b) are schematic diagrams illustrating button setting areas. FIG. 11 is a schematic diagram illustrating another button setting area; FIG. 11 is a schematic diagram illustrating another button setting area; FIG. 2 is a configuration diagram illustrating a measurement system according to a second embodiment; FIG. It is a perspective view which illustrates a head mounted display. It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. (a) and (b) are configuration diagrams illustrating a measurement system according to a third embodiment. It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. FIG. 11 is a configuration diagram illustrating a measurement system according to a fourth embodiment; It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. It is a schematic diagram which shows the example of an operation. 6 is a flowchart illustrating the operation of helper display; It is a schematic diagram which shows the example of a display of a helper. It is a schematic diagram which shows the example of a display of a helper. It is a schematic diagram which shows the example of a display of a helper. It is a schematic diagram which shows the example of a display of a helper. FIG. 11 is a configuration diagram illustrating a measurement system according to a fifth embodiment; It is a schematic diagram which shows the example of an operation. FIG. 11 is a configuration diagram illustrating a measurement system according to a sixth embodiment; 7 is a flow chart showing an example of processing for combined display of measurement results; It is a schematic diagram which shows the example of a synthetic|combination display of a measurement result. (a) to (e) are schematic diagrams showing other composite display examples. FIG. 11 is a configuration diagram illustrating a measurement system according to a sixth embodiment; 4 is a flowchart illustrating marker registration and setting operations; 4 is a flow chart illustrating marker recognition and composite display; FIG. 10 is a schematic diagram showing a display example of measurement results by marker recognition; FIG. 10 is a schematic diagram showing a display example of measurement results by marker recognition; FIG. 10 is a schematic diagram showing a display example of measurement results by marker recognition;

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below with reference to the drawings. In the following description, the same members are denoted by the same reference numerals, and the description of members that have already been described will be omitted as appropriate.

(First embodiment)
FIGS. 1A and 1B are configuration diagrams illustrating the measurement system according to the first embodiment.
A measuring system 1A according to this embodiment includes a computer 100, a measuring machine M controlled by the computer 100, and a 3D sensor (three-dimensional sensor) SR. The computer 100 has a CPU (Central Processing Unit) 10 , a storage section 20 , a calculation section 30 , a measuring instrument control section 40 , a display control section 50 , a 3D sensor input section 70 and a position/distance/orientation recognition section 75 . Furthermore, a projector PR may be connected to the computer 100 . In this case, the computer 100 has a projected graphic generation section 80 and a projector output section 85 .

The measuring machine M is a device that measures the three-dimensional coordinates of an object to be measured, the length of a predetermined position, and the like. Examples of the measuring machine M include a three-dimensional position measuring machine and an image measuring machine. The CPU 10 executes predetermined programs to control each part and perform predetermined calculations. Storage unit 20 includes a main storage unit and a sub storage unit. The calculation unit 30 includes a calculation circuit that performs predetermined calculations.

The measuring machine control section 40 controls the measuring machine M based on instructions from the user. The display control unit 50 controls display of predetermined information on the display D. FIG. The input/output control unit 60 controls input devices such as a keyboard K and a mouse MS, and controls a touch panel (not shown).

The 3D sensor input section 70 is an interface section that takes in information acquired by the 3D sensor SR into the computer 100 . Here, the 3D sensor SR is a sensor that detects the three-dimensional coordinates of a predetermined object in the real three-dimensional space. The position/distance/orientation recognition unit 75 recognizes the position (three-dimensional coordinates) of the object, the distance from the 3D sensor SR to the object, and the orientation of the object based on the information acquired by the 3D sensor SR. conduct.

The projected graphic generation unit 80 performs processing for generating a graphic to be projected by the projector PR. The projector output section 85 is an interface section that sends the graphic generated by the projected graphic generation section 80 to the projector PR.

The measurement system 1A further includes a command setting section 101, a moving object detection section 102, and a command execution section 103 as programs executed by the CPU 10. FIG. The command setting unit 101 performs a process of setting a button setting area in the real three-dimensional space and a correspondence between the button setting area and the command. A button setting area is an area in which an arbitrary position in the real three-dimensional space is specified. For example, it may be a predetermined rectangular area in a real three-dimensional space, a predetermined three-dimensional area, or a predetermined object area.

The command setting unit 101 performs processing for setting three-dimensional coordinate information for representing such a button setting area. The command setting unit 101 also associates the button setting area with the command. The command is associated with the three-dimensional coordinate information of the button setting area and stored in the storage unit 20 as table data, for example.

The moving object detection unit 102 performs processing for detecting a moving object from the three-dimensional coordinates of the object detected by the 3D sensor SR. Specifically, the position of the object recognized by the position/distance/orientation recognition unit 75 is tracked to detect a moving object (moving object). As a result, a moving object can be detected in a real three-dimensional space that can be detected by the 3D sensor SR, and the position of the moving object can be tracked.

The command execution unit 103 executes a command associated with the button setting area when the three-dimensional coordinates of the moving object detected by the moving object detection unit 102 are positioned within the button setting area. The moving object detection unit 102 can track the position of a moving object (three-dimensional coordinates of the moving object) in the real three-dimensional space. The command execution unit 103 acquires information on the three-dimensional coordinates of the moving object from the moving object detection unit 102 .

Then, when the three-dimensional coordinates of the moving object are positioned within the button setting area, the command execution unit 103 executes the command associated with the button setting area. Accordingly, it is possible to detect that a button setting area in the real three-dimensional space has been selected from the action of the moving body, and execute the command associated with the button setting area.

FIG. 2 is a schematic diagram showing an operation example.
The 3D sensor SR detects the position of an object in real three-dimensional space. In the example shown in FIG. 2, for example, the measuring machine M, the user 800 of the measuring machine M, the screen SL on which the image projected from the projector PR is projected, the whiteboard WB, etc. are arranged in the real three-dimensional space. A 3D sensor SR detects these positions.

Further, in the example shown in FIG. 2, button setting areas B1 to B4 are provided on the whiteboard WB. The positions (three-dimensional coordinates) of the button setting areas B1 to B4 on the whiteboard WB are set by the command setting unit 101 in advance. A command is associated with each of the button setting areas B1 to B4. For example, the button setting area B1 is associated with the "start" command, the button setting area B2 is associated with the "end" command, the button setting area B3 is associated with the "save" command, and the button setting area B4 is associated with the "print" command.

A user 800 of the measuring machine M measures an object to be measured using the measuring head 501 of the measuring machine M, for example. When the measuring head 501 is provided with a projector PR, images based on various information such as design drawings of the object to be measured can be displayed on the screen SL from the projector PR.

The motion of the user 800 is detected by the moving object detection unit 102 based on the information acquired by the 3D sensor SR. The position/distance/orientation recognition unit 75 recognizes the movement of the hand 801 of the user 800, for example. Then, when the hand 801 of the user 800 is positioned in any one of the button setting areas B1 to B4, the command executing section 103 executes the command associated with the button setting areas B1 to B4 at the position of the hand 801. FIG.

For example, when the hand 801 of the user 800 touches the button setting area B1 on the whiteboard WB, the "start" command associated with the button setting area B1 is executed. Similarly, when the hand 8001 of the user 800 touches the button setting area B2, the "end" command associated with the button setting area B2 is executed. In this way, the user 800 can execute desired commands depending on the position of his hand 801 .

FIGS. 3A and 3B are schematic diagrams illustrating button setting areas.
FIG. 3A shows button setting areas B1 to B4 set on the whiteboard WB. The button setting areas B1 to B4 are appropriately written on the surface of the whiteboard WB. Here, four rectangles are written on the surface of the whiteboard WB.

The command setting unit 101 sets the positions of the button setting areas B1 to B4 by registering the three-dimensional coordinates of the even portions of each rectangle. For example, the button setting area B1 is set by three-dimensional coordinates of four even parts P1 to P4. The three-dimensional coordinates of the even parts P1-P4 can be captured by the 3D sensor SR. Also, the measurement head 501 may be used to capture the three-dimensional coordinates of the even portions P1 to P4.

FIG. 3B shows a three-dimensional button setting area B5. The button setting area B5 is, for example, a cubic area. The region of the volume is determined by four points P17, P18, P19, P20 in the real three-dimensional space. The command setting unit 101 sets the position of the button setting area B5 by registering the three-dimensional coordinates of the four points P17, P18, P19, and P20.

The command setting unit 101 associates a command with each of the button setting areas B1 to B5. The button setting areas B1 to B5 can be set anywhere in the real three-dimensional space. For example, even if the button setting areas B1 to B4 are set at a position where some object exists, such as a whiteboard WB, the command setting unit 101 sets the button setting area B5 in an area without an object in the real three-dimensional space. You may

4 and 5 are schematic diagrams illustrating other button setting areas.
In the example shown in FIG. 4, the image projected by the projector PR is the button setting area B6. For example, if the measuring head 501 is provided with a projector PR, the measuring head 501 is directed in a direction different from the screen SL. As a result, the image of the button setting area B6 is output from the projector PR. In the example shown in FIG. 4, the image of the button setting area B6 output from the projector PR is projected on the floor FL.

When projecting the image of the button setting area B6 with the projector PR, the projection position of the button setting area B6 is detected by the 3D sensor SR and recognized by the position/distance/orientation recognition section 75 . The position/distance/orientation recognition unit 75 can recognize the position of the button setting area B6 from the information detected by the 3D sensor SR by registering the features of the button setting area B6 in advance. Further, even if the projection position of the button setting area B6 moves, it can be tracked by the moving object detection unit 102. FIG.

When the button setting area B6 is projected onto the floor FL, the user 800 can select the button setting area B6 with his or her foot 802, for example. The position of the foot 802 of the user 800 is detected by the 3D sensor SR, and the movement of the foot 802 is tracked by the moving object detection section 102 . Accordingly, it is possible to detect that the foot 802 of the user 800 is positioned in the button setting area B6, and execute the command associated with the button setting area B6 by the command execution unit 103. FIG.

For the user 800 who holds the measuring head 501 by hand and performs a measuring operation, it may be difficult to give a command by hand. Since the button setting area B6 projected on the floor FL can be selected with the foot 802, the user 800 can designate and execute a desired command with the foot 802 while holding the measuring head 501 with the hand.

In the example shown in FIG. 5, the image of the button setting area B7 is projected on the screen SL. The projector PR projects, for example, the design information of the object to be measured on the screen SL, and also projects the image of the button setting area B7. Similar to the example shown in FIG. 4, the position of the button setting area B7 is detected by the 3D sensor SR and tracked by the moving object detection unit 102. FIG.

The user 800 can select the button setting area B7 projected on the screen SL with his/her own hand 801, for example. When detecting that the hand 801 of the user 800 is positioned in the button setting area B7, the command execution unit 103 executes the command associated with the button setting area B7.

Also, the command execution unit 103 may switch commands according to the motion of the hand 801 or foot 802 of the user 800 . For example, when the hand 801 of the user 800 remains in the button setting area B1 for a certain period of time (touch operation), the first command is executed, and when the hand 801 moves along the button setting area B1 (slide action) executes the second command. That is, the command may be switched depending on whether the button setting area B1 is touched or slid with the hand 801. FIG.

According to the measurement system 1A according to the first embodiment, the button setting areas B1 to B7 can be set at desired positions in the real three-dimensional space. Also, the user 800 can execute a desired command by selecting the button setting areas B1 to B7 with his or her hand 801 or foot 802. FIG. Therefore, the user 800 can easily execute a desired command while performing measurement work with the measuring instrument M. FIG. For example, even if the computer 100 is located away from the measuring instrument M, the user 800 can send a desired command to the computer 100 without leaving the measuring instrument M.

Also, the user 800 can set the button setting areas B1 to B7 at desired positions. As a result, the button setting areas B1 to B7 can be arranged at positions where the user 800 can work efficiently, and the efficiency of the measurement work can be improved.

(Second embodiment)
FIG. 6 is a configuration diagram illustrating a measurement system according to the second embodiment.
A measuring system 1B according to this embodiment includes a computer 100, and a measuring machine M and a 3D sensor SR controlled by the computer 100. FIG. The computer 100 has a CPU 10 , a storage section 20 , a calculation section 30 , a measuring instrument control section 40 , a display control section 50 , a 3D sensor input section 70 and a position/distance/orientation recognition section 75 . Furthermore, the computer 100 includes a 3D video input/output unit 51, a 3D camera control unit 52, a stereoscopic video generation unit 53, a head mounted display output unit 54, a control generation unit 55, a headphone microphone input/output unit 61, an audio input/output unit 62, It has a control panel input/output unit 65 and a control input recognition unit 76 . A control panel CT, a headphone microphone HPM, a head mounted display HD and a 3D camera CM are connected to the computer 100 .

A configuration different from that of the measurement system 1A according to the first embodiment will be described below.
The control panel input/output unit 65 is an interface portion for inputting/outputting information to/from the control panel CT. The user 800 can control the measuring machine M using the control panel CT.

The 3D camera CM is a 3D imaging unit that acquires a 3D image in a real 3D space. The head-mounted display HD is a display that displays a three-dimensional image captured by the 3D camera CM. Here, in the embodiment, "image" means image information captured by an imaging device such as a 3D camera CM, and "image" refers to graphics generated from design data such as CAD or prepared in advance. information such as graphics.

FIG. 7 is a perspective view illustrating a head mounted display.
Head mounted display HD is worn on the head of user 800 . A user 800 can refer to a three-dimensional image projected on the head-mounted display HD. A 3D camera CM is attached to this head mount display HD. That is, the head mounted display HD is provided integrally with the 3D camera CM.

The 3D camera control unit 52 is a part that controls the 3D camera CM. The 3D image input/output unit 51 is an interface portion that takes in the 3D image acquired by the 3D camera CM into the computer 100 . The head-mounted display output section 54 is an interface section that transmits video signals to the head-mounted display HD. The stereoscopic image generation unit 53 is a part that generates a 3D image to be displayed on the head-mounted display HD. The stereoscopic video generating unit 53 includes a captured video synthesizing unit 531 .

The captured image synthesizing unit 531 synthesizes the 3D image acquired by the 3D camera CM and the image corresponding to the button setting area, and performs processing for displaying the synthesized image on the head mounted display HD.

The control generator 55 is a part that associates a predetermined action of the user 800 with a command. The correspondence between the predetermined actions of the user 800 and the commands is stored in the storage unit 20 as table data, for example.

The headphone microphone HPM is a combination of a headphone and a microphone, and is worn on the head of the user 800 . The headphone microphone input/output section 61 is an interface section for inputting/outputting information to/from the headphone microphone HPM. The audio input/output unit 62 is a part that generates audio to be sent to the headphone microphone HPM and processes audio information captured from the headphone microphone HPM. Note that the headphone microphone HPM may be provided as necessary in the present embodiment.

The control input recognition section 76 is a section that recognizes a predetermined action of the user 800 . The control input recognition section 76 performs processing for recognizing a predetermined action of the user 800 based on the information sent from the position/distance/orientation recognition section 75 . In such a measurement system 1B, an image of the real three-dimensional space is displayed on the head-mounted display HD, and an image of the button setting area can be combined with this image and displayed.

8 to 11 are schematic diagrams showing operation examples.
As shown in FIG. 8, a user 800 wears a head mounted display HD. Also, the 3D sensor SR detects the positions of objects such as the user 800 and the measuring machine M in the real three-dimensional space. A 3D camera CM provided in the head-mounted display HD acquires 3D images of a real three-dimensional space.

The user 800 can refer to the 3D video captured by the 3D camera CM on the head mounted display HD. An image of the button setting area (button image BG) is displayed on the head-mounted display HD in accordance with the actual three-dimensional space of the 3D image. That is, the photographed image synthesizing unit 531 synthesizes the button image BG with the 3D image acquired by the 3D camera. The button image BG is displayed together with the three-dimensional coordinates in the display of the head mounted display HD based on the three-dimensional coordinates in the real three-dimensional space of the button setting area registered in advance. As a result, even if the angle of view of the 3D camera CM changes, the combining position of the button image BG on the 3D image does not change.

The motion of the user 800 is detected by the moving object detection unit 102 based on the information acquired by the 3D sensor SR. The position/distance/orientation recognition unit 75 recognizes the movement of the hand 801 or the foot 802 of the user 800, for example. Then, when the hand 801 or foot 802 of the user 800 is positioned in the button setting area corresponding to the button image BG, the command execution unit 103 executes the command associated with the button setting area at the position of the hand 801 or foot 802. . As a result, the user 800 can execute desired commands according to the positions of his/her own hands 801 and feet 802 .

The button image BG can be displayed at any position on the 3D image displayed on the head-mounted display HD. For example, the button image BG is displayed at a position (in the air) where no object exists in the real three-dimensional space where the D image is displayed, or the button image BG is displayed on the 3D image of the floor FL as if it were placed on the floor FL. A button image BG can be displayed.

FIG. 9 shows an example of synthesizing the button image BG on the 3D image of the floor FL. A user 800 holds a measuring head 501 of a measuring machine M with a hand 801 to measure a workpiece W, which is an object to be measured. The user 800 performs measurement by operating the measuring head 501 while referring to the 3D images of the measuring machine M and the workpiece W displayed on the head-mounted display HD.

Also, on the head-mounted display HD, a 3D image of the floor FL is displayed in a state in which the button image BG is synthesized. Buttons are not arranged on the actual floor FL, but button images BG are displayed on the display of the head-mounted display HD. This allows the user 800 to refer to the button image BG as if the button were arranged on the floor FL. User 800 selects button image BG on floor FL with foot 802 . As a result, the command of the button setting area corresponding to the selected button image BG is executed.

FIG. 10 shows an example of synthesizing a computer image CAL with a 3D image. In this example, the image CAL of the computer is synthesized near the 3D image of the work W displayed on the head-mounted display HD. A user 800 operates a computer image CAL with a hand 801, for example. The motion of the hand 801 is tracked by the moving object detection unit 102 to determine which key region of the computer image CAL the hand 801 is positioned. Thereby, the command execution unit 103 executes the command corresponding to the key of the image CAL of the computer. The user 800 can refer to the virtual computer image CAL displayed on the head-mounted display HD and perform calculations as if operating a real computer.

The command execution unit 103 may select and execute one of a plurality of commands according to the shape of the hand 801 of the user 800 detected by the moving object detection unit 102 . FIG. 11 shows an example of switching commands according to the shape of hand 801 .

As an example, when a certain period of time elapses with the hand 801 open in a specific direction (H1), a command to display a home button (an image in which a plurality of button images BG are aggregated) is executed. Further, when the home button is displayed at a position out of reach of the hand 801 on the display of the head mounted display HD, when the hand 801 is grasped (H2), a command to display the distant home button closer is executed. . Also, when the home button is picked up with the hand 801 and moved (H3), a command for moving the display position of the home button is executed.

Also, when the index finger of the hand 801 is directed in a specific direction (H4), a command to hide the home button is executed. When the hand 801 is pressed against the wall (H5), a command to move the home button to the wall is executed. When the palm of the hand 801 is turned upward (H6), a command to move the home button to the palm of the hand 801 is executed. When the user presses the hand 801 toward the floor (H7), a command is executed to move the home button so as to stick it on the floor. Note that the correspondence between the shape and motion of the hand 801 and the command is an example, and the present invention is not limited to this.

By switching commands according to the shape of the hand 801 in this manner, a desired command can be executed only by a gesture using the hand 801 without using an input device such as a keyboard K or a mouse MS. Become.

In the above example, the user 800 selects the button image BG with the hand 801 or the foot 802 to execute the command, but it is also possible to select the button image BG with the measurement head 501 and execute the command.

Further, the computer 100, the 3D sensor SR and the 3D camera CM in the measurement system 1B according to the present embodiment may constitute a user interface device. According to this user interface device, it is possible to switch display/non-display of the button image BG according to the shape of the hand 801 of the user 800, and to execute a command corresponding to the button image BG.

(Third Embodiment)
FIGS. 12A and 12B are configuration diagrams illustrating the measurement system according to the third embodiment.
A measuring system 1C according to this embodiment includes a computer 100, a measuring machine M controlled by the computer 100, a 3D sensor SR, and a head mounted display HD. The computer 100 has a CPU 10 , a storage section 20 , a calculation section 30 , a measuring instrument control section 40 , a display control section 50 , a 3D sensor input section 70 and a position/distance/orientation recognition section 75 . Further, the computer 100 has a stereoscopic video generation unit 53 , a head mounted display output unit 54 , a headphone microphone input/output unit 61 , an audio input/output unit 62 and a console input/output unit 65 . A control panel CT and a headphone microphone HPM are connected to the computer 100 .

In such a measuring system 1C, an image of the measuring machine M is displayed in the virtual three-dimensional space of the head-mounted display HD. As the image of the measuring machine M, for example, a CAD image is used. A CAD image is stored in the storage unit 20 . In addition, the motion of the user can be detected by the motion detection unit, and the image of the measuring instrument M in the virtual three-dimensional space can be moved in accordance with the motion of the user 800 .

The measuring system 1C according to the present embodiment is particularly suitable for application to a measuring machine M with many automatic measuring functions. For example, a CNC (Computerized Numerically Controlled) measuring machine can perform automatic measurement according to a preset program (for example, a part program), so that highly accurate and efficient measurement can be performed. However, handling a CNC measuring machine requires specialized skills and knowledge, such as operating methods and creating and modifying programs. In the measuring system 1C according to the present embodiment, it is possible to improve the user interface of such a measuring instrument M with many functions.

13 to 17 are schematic diagrams showing operation examples.
As shown in FIG. 13, a user 800 wears a head mounted display HD and a headphone microphone HPM. The 3D sensor SR detects the positions of objects such as the user 800 and the measuring machine M in the real three-dimensional space.

An image of the measuring machine M (measuring machine image MG) is displayed in the virtual three-dimensional space on the head-mounted display HD. The user 800 can move the measuring machine image MG within the virtual three-dimensional space by referring to the measuring machine image MG within the virtual three-dimensional space displayed on the head-mounted display HD.

That is, the 3D sensor SR detects the position of, for example, the hand 801 of the user 800, and the motion of the hand 801 is tracked by the moving object detection unit . The position and movement of the hand 801 are reflected in the motion of the measuring machine image MG displayed in the virtual three-dimensional space. For example, when the user 800 extends the hand 801 to the position of the image of the measuring head 501 (measuring head image 501G) in the virtual three-dimensional space, the measuring head image 501G can be moved in the virtual three-dimensional space.

The stereoscopic image generation unit 53 generates an image such that the display position of the measuring machine image MG in the virtual three-dimensional space is moved according to the movement of the hand 801 of the user 800 detected by the moving object detection unit 102 . This allows the user 800 to freely operate the measuring machine image MG within the virtual three-dimensional space displayed on the head-mounted display HD.

In the measurement system 1C according to this embodiment, the measurement operation storage section 104 may be provided as a program executed by the CPU 10. FIG. The measuring operation storage unit 104 performs a process of recording the measuring operation along the display position of the measuring machine image MG moved by the user 800 within the virtual three-dimensional space displayed on the head-mounted display HD.

Then, the measuring machine control unit 40 measures the workpiece W using the actual measuring machine M according to the measuring operation stored in the measuring operation storage unit 104 . Thereby, the user 800 can record an actual measurement procedure by moving the measuring machine image MG, for example, the measuring head image 501G, in the virtual three-dimensional space.

At this time, the actual position of the work W is detected by the 3D sensor SR, and the work image WG is synthesized in accordance with the position of the measuring machine image MG in the virtual three-dimensional space. When moving the measuring head image 501G, the user 800 can record the measurement procedure while confirming interference between the work image WG and the measuring head image 501G. For example, if interference with the workpiece image WG occurs due to the orientation of the measurement head image 501G, a warning may be displayed on the head mounted display HD. Such recording of the measurement procedure allows the movement of the measurement head 501 to be programmed without specialized knowledge.

FIG. 14 shows an example of displaying an actual image IMG in a virtual three-dimensional space. The measuring system 1C may include a camera CM1 that acquires the image IMG of the measuring machine M. An image IMG of the measuring machine M acquired by the camera CM1 is displayed at a predetermined position in the virtual three-dimensional space of the head-mounted display HD.

The user 800 can refer to the image IMG of the actual measuring machine M together with the measuring machine image MG displayed on the head-mounted display HD. As a result, it is possible to perform work while confirming the actual operation of the measuring machine M in the virtual three-dimensional space.

FIG. 15 shows an example of a part program correction operation. The head-mounted display HD displays the motion of the measuring head 501 according to the existing part program in a virtual three-dimensional space. User 800 reads an existing part program. For example, an existing part program may be read out by voice recognition by inputting voice to the headphone microphone HPM.

The measuring head image 501G in the virtual three-dimensional space operates in the same manner as when the read part program is executed. The user 800 can refer to the operation of the measuring head image 501G displayed in the virtual three-dimensional space, and can control the stop and restart of the operation by voice recognition. Also, the hand 801 can be used to move the measuring head image 501G.

In the virtual three-dimensional space, the numerical values of the movement order of the measuring head image 501G and the button image BG may be displayed. When correcting the part program, the operation is paused and the hand 801 moves the measuring head image 501G.

Here, as an example, the operation when adding a part program step will be described. In this example, as shown in FIG. 15, for a part program in which the measuring head 501 operates in the order of (1) to (4), the operation of adding (2') between (2) and (3) is performed. explain.

First, the user 800 operates the part program step by step, for example, by voice recognition. When the user 800 inputs, for example, "Go to the first step", the part program advances by one step, and the measuring head image 501G in the virtual three-dimensional space is displayed to move to position (1).

Next, the user 800 inputs, for example, "Go to the next step". As a result, the part program proceeds to the next step, and the measuring head image 501G in the virtual three-dimensional space is displayed so as to move to the position (2).

Here, the user 800 inputs, for example, "Add the step" in order to add a step. This executes the addition mode of the steps of the part program. Next, the user 800 moves the measuring head image 501G in the virtual three-dimensional space with the hand 801 and places it at the position (2'). In this state, for example, when the voice "Continue" is input, the position (2') is added, and the measuring head image 501G advances to the next position (3).

Here, the operation and modification of the part program are instructed by voice recognition, but the same operation and modification may be instructed by designating the button image BG. Such an operation enables the user 800 to intuitively modify the part program without specialized knowledge.

FIG. 16 shows an example of displaying information about measurements in a virtual three-dimensional space. A user 800 refers to an image in a virtual three-dimensional space by using the head-mounted display HD worn by the user. A workpiece image WG and a measuring head image 501G are displayed in the virtual three-dimensional space. The work image WG and the measuring head image 501G are CAD images and computer graphics. In the example shown in FIG. 16, an example is shown in which the measurement results are combined with the work image WG and displayed. In addition, when the measurement result does not fall within a predetermined allowable range, a display to that effect may be added to the work image WG (such as a display in which the work image WG is colored).

The user 800 inputs a voice such as "Show result" from the headphone microphone HPM. As a result, a display in which the measurement result is combined with the work image WG in the virtual three-dimensional space is performed. The user 800 can check the measurement result without referring to the display D of the computer 100 by referring to the contents displayed in the virtual three-dimensional space.

An example of measuring and teaching over network N is shown in FIG.
In the example shown in FIG. 17, the measurement system 1C further includes a remote control section 90 connected to the measuring device control section 40 via the network N. In the example shown in FIG. The remote control unit 90 performs a process of sending a command given by a user 800 who is away from the measuring device M to the measuring device control unit 40 via the network N. FIG.

A user 800 at a remote location can confirm the actual operation of the measuring machine M while referring to the measuring machine image MG in the virtual three-dimensional space displayed on the head-mounted display HD worn by the user. Also, the measuring machine image MG and the measuring head image 501G displayed in the virtual three-dimensional space can be moved by the hand 801 of the user 800 .

As described above, the operation of the measuring head 501 according to the part program can be confirmed by the operation of the measuring head image 501G displayed in the virtual three-dimensional space. Furthermore, the part program can be modified within the virtual three-dimensional space.

By transmitting instructions from a remote user 800 via the network N to the measuring device control unit 40 by the remote control unit 90, the user 800 can view images in the virtual three-dimensional space regardless of the position of the measuring device M. The measuring machine M can be handled by referring to

That is, the user 800 can handle the measuring device M from a remote location as if the user were in the vicinity of the measuring device M. FIG. In this measurement system 1C, for example, when the user 800 who is accustomed to operating the measuring machine M is not near the measuring machine M, it is possible to let a knowledgeable staff member in a remote location operate the measuring machine M via the network N. can. Moreover, even if the user 800 cannot approach the measuring device M under a severe environment, the user 800 can operate the measuring device M as if he or she were there.

(Fourth embodiment)
FIG. 18 is a configuration diagram illustrating a measurement system according to the fourth embodiment.
A measuring system 1D according to this embodiment includes a computer 100, and a measuring machine M and a 3D sensor SR controlled by the computer 100. FIG. The computer 100 has a CPU 10 , a storage section 20 , a calculation section 30 , a measuring instrument control section 40 , a display control section 50 , a 3D sensor input section 70 and a position/distance/orientation recognition section 75 . Furthermore, the computer 100 includes a 3D video input/output unit 51, a 3D camera control unit 52, a stereoscopic video generation unit 53, a photographed video synthesis unit 531, a head-mounted display output unit 54, a headphone microphone input/output unit 61, and an audio input/output unit 62. , a sound reproduction unit 621 , a console input/output unit 65 and a helper generation unit 95 . A control panel CT, a headphone microphone HPM, a head mounted display HD and a 3D camera CM are connected to the computer 100 .

In such a measurement system 1D, the image of the actual measuring machine M and the measuring machine image MG such as a CAD image or computer graphics can be combined and displayed in the virtual three-dimensional space of the head-mounted display HD. . The measuring instrument image MG is stored in advance in the image storage section 201 of the storage section 20 .

Further, the helper generator 95 generates an image representing a predetermined measurement procedure of the measuring machine M. FIG. Accordingly, the captured image synthesizing unit 531 of the stereoscopic image generating unit 53 can perform a process of synthesizing the image representing the measurement procedure generated by the helper generating unit 95 with the image captured by the 3D camera CM.

Further, the helper generating section 95 may generate an image corresponding to a predetermined abnormality of the measuring instrument M. FIG. Thus, the captured image synthesizing unit 531 of the stereoscopic image generating unit 53 can perform a process of synthesizing an image generated by the helper generating unit 95 and corresponding to the predetermined abnormality with the image captured by the 3D camera CM.

Further, the helper generating section 95 may generate an image corresponding to predetermined guidance of the measuring instrument M. FIG. Accordingly, the captured image synthesizing unit 531 of the stereoscopic image generating unit 53 can perform a process of synthesizing an image corresponding to the predetermined guidance generated by the helper generating unit 95 with the image captured by the 3D camera CM.

Here, if the user 800 does not know how to operate the measuring instrument M, the user 800 simulates an image of actual operation using the measuring instrument M while checking the instruction manual of the measuring instrument M. However, it is difficult to memorize all operations, and users are forced to perform troublesome tasks such as checking the instruction manual again.

Further, for example, when replacing the measurement probe, if the procedure is not followed, problems such as breakage may occur. Therefore, replacement of the measurement probe requires careful work. The user 800 who is not accustomed to the work will perform the work while referring to the instruction manual each time.

Further, for example, to find the origin of a non-orthogonal three-dimensional measuring machine, it is necessary to perform an ON/OFF operation of each limit switch of the seven-axis arm of the measuring machine M to acquire the origin. A user 800 who is unfamiliar with this work often finds it difficult to intuitively understand the axis for which the origin has not been obtained and the operation for obtaining the origin, and is at a loss about the operation.

Also, for example, when connecting the cable of the measuring instrument M to the dedicated controller, it is difficult to know which cable to insert into which connection port of the controller, and it is necessary to check the instruction manual each time.

Further, for example, in the case of a non-orthogonal three-dimensional measuring machine, measurement cannot be performed when the limit switch is ON. It is difficult to know whether the limit switch is in the ON state or in the measurable state, and the same place may be re-measured many times.

Further, for example, when an abnormality occurs in the measuring instrument M, a message is displayed on the display D of the computer 100, or a warning is given by sound. However, when the user 800 is concentrating on the measurement, the display D of the computer 100 may not be seen much, or the display D may be difficult to see, making it difficult to recognize.

In addition, for example, images such as videos may be reproduced for the purpose of educating beginners on measurement methods and the like. However, there are times when you don't know what to do when actually measuring, or when you make a mistake and start over from the beginning, and it takes a lot of time to learn.

In the measurement system 1D according to the present embodiment, by synthesizing and displaying the image of the actual measuring machine M and various images in the virtual three-dimensional space of the head-mounted display HD worn by the user 800, efficient operations can be performed.

19 to 23 are schematic diagrams showing operation examples.
FIG. 19 shows an example of providing technical support to the user 800. In FIG.
As shown in FIG. 19, a user 800 wears a head mounted display HD. Also, the 3D sensor SR detects the positions of objects such as the user 800 and the measuring machine M in the real three-dimensional space. A 3D camera CM provided in the head-mounted display HD acquires 3D images of a real three-dimensional space.

The user 800 can refer to the 3D video captured by the 3D camera CM on the head mounted display HD. In the example shown in FIG. 19, an image IMG of the measuring instrument M is displayed as a 3D image. Various images G related to the measuring instrument M are displayed on the head-mounted display HD in accordance with the actual three-dimensional space of the 3D image.

The display position of the image G is determined based on the position of the image IMG of the measuring instrument M. FIG. Accordingly, even if the angle of view of the 3D camera CM changes, the display position of the image G with respect to the measuring device M does not change. In the example shown in FIG. 19, an image G in accordance with the origin search procedure is displayed in a state of being synthesized with the image IMG of the measuring instrument M. In the example shown in FIG. The user 800 can refer to the image G in accordance with the origin search procedure while referring to the image IMG of the actual measuring machine M acquired by the 3D camera CM.

By displaying such an image G in combination with the image IMG of the actual measuring machine M, the user 800 can determine which axis of the measuring machine M the origin of which axis needs to be acquired, and the origin can be acquired. You can intuitively understand the operation procedures necessary to do so.

FIG. 20 shows an operation example when teaching how to operate the measuring instrument M. In FIG. FIG. 20 shows images G1 to G7 showing the replacement procedure of the measurement probe as an example. On the head-mounted display HD worn by the user 800, images G1 to G7 are displayed in order. These images G1 to G7 are combined with an image IMG (not shown in FIG. 20) of the actual measuring machine M captured by the 3D camera CM. As the display of the images G1 to G7, a step display in which the images are sequentially displayed one by one may be performed. Operations such as playback and stop of the step display are instructed by the voice of the user 800, the motion of the hand 801, and the like.

By referring to the images G1 to G7 displayed on the head-mounted display HD, the user 800 can comprehend the operation procedure while comparing the image IMG of the actual measuring machine M with the images G1 to G7.

FIG. 21 shows an operation example for teaching how to connect cables. On the head-mounted display HD worn by the user 800, an image IMG of the actual measuring machine M captured by the 3D camera CM and an image PC-IMG of the personal computer are displayed. Furthermore, the image G of the cable and the guidance is displayed in a state in which it is combined with the image IMG and the PC-IMG. The image G of the cable is synthesized and displayed according to the connection position of the video PC-IMG of the actual personal computer.

In this manner, the image G when the cable is connected is synthesized and displayed on the image IMG of the actual measuring machine M and the image PC-IMG of the personal computer, so that the user 800 intuitively imagines the cable connection method. be able to.

FIG. 22 shows an operation example of displaying an image G relating to a predetermined abnormality. A head-mounted display HD worn by the user 800 displays an image IMG of the actual measuring machine M captured by the 3D camera CM. Here, when some abnormality occurs in the measuring head 501, an image G is synthesized with the image IMG on the head-mounted display HD to indicate the abnormality.

In the example shown in FIG. 22, the image IMG of the actual measuring machine M is overlaid with an image G that prompts a warning such as "abnormal cable connection" or "limit ON". As a result, the user 800 can visually grasp what kind of abnormality occurs in what position in accordance with the image IMG of the actual measuring machine M. FIG.

FIG. 23 shows an example of displaying an image G of an instruction manual. A head-mounted display HD worn by the user 800 displays an image IMG of the actual measuring machine M captured by the 3D camera CM. When the user 800 transmits a predetermined command to the computer 100 in this state, the image G of the instruction manual is synthesized and displayed near the image IMG of the measuring instrument M on the head-mounted display HD.

While referring to the image IMG of the actual measuring machine M, the user 800 can also refer to the instruction manual image G appearing on the display space of the head-mounted display HD. In addition, the page of the image G of the instruction manual can be turned according to the instruction of the user 800 . For example, the motion of the hand 801 of the user 800 is detected by the moving object detection unit 102 to track the motion of the hand 801 . Then, when a page-turning action is recognized from the movement of the hand 801, the image G of the instruction manual is displayed as if the page is turned.

Further, the user 800 can enlarge and reduce the image G of the instruction manual by a predetermined instruction. This operation can also be performed by detecting the movement of the hand 801 .

FIG. 24 is a flow chart illustrating the operation of the helper display.
Helper display operation is performed by the helper generation unit 95 . The helper generator 95 starts operating when the user 800 calls the helper or when some error occurs.

First, as shown in step S101, a helper is displayed. A helper is an image corresponding to predetermined guidance, and may be a human-shaped image or a character image. The helper is displayed within the virtual three-dimensional space of the head-mounted display HD.

Next, as shown in step S102, the helper's response is output. For example, greetings and questions to the user 800 such as "Hello", "Error Occurred!", and "May I help you?" The helper's response changes the image displayed on the head-mounted display HD or is output as sound to the headphone microphone HPM.

Next, as shown in step S103, voice commands are recognized. The user 800 verbally asks the headphone microphone HPM. A command is recognized based on this voice. Next, as shown in step S104, the helper's response is output. The helper outputs a response corresponding to the command specified by the user 800 . For example, images and voices corresponding to commands from the user 800 such as "I teach you..." and "Please..." are output.

As shown in step S105, it is determined whether or not the response from the helper is acceptable. If it is good, the process proceeds to step S106 to delete the helper. On the other hand, if a further response is required, the process returns to step S102 and the subsequent processes are repeated.

25 to 28 are schematic diagrams showing display examples of helpers.
FIG. 25 shows a display example of a helper that guides the operating procedure of the measuring instrument M. As shown in FIG.
First, the user 800 makes a helper call. For example, the user 800 verbally asks "OK Mitutoyo." A helper is displayed in response. An image IMG of the actual measuring machine M is displayed in the three-dimensional space of the head-mounted display HD, and an image of the helper (helper image HG) is displayed in the vicinity thereof.

Next, the helper image HG responds with, for example, "Hello" by image display and voice. Next, the user 800 asks a question by voice. For example, in order to hear the operation procedure of the measuring instrument M, a question such as "How to..." is spoken.

Next, in response to the user's 800 question, the helper image HG responds, for example, "I teach you...,operating it." Subsequently, the user 800 asks, for example, "Teach me step by step." In this way, guidance for the operation procedure of the measuring instrument M is provided by repeating the question and response between the user 800 and the helper image HG. At this time, the measuring machine image MG may be displayed to represent the progress of the procedure with the measuring machine image MG.

FIG. 26 shows a display example of a helper that guides the measurement probe replacement procedure.
First, the user 800 calls a helper by asking, for example, "OK Mitutoyo." A helper image HG is displayed in response to this. The helper image HG responds by displaying, for example, "Hello" and displaying an image and sound.

Next, the user 800 asks, for example, "How to change the probe." In response to the user's 800 question, the helper image HG responds with, for example, "Please do it this way." Along with this response, the measurement head image 501G is displayed to guide the measurement probe replacement procedure using a video or the like.

FIG. 27 shows a display example of a helper that provides guidance for cable connection errors.
First, the user 800 calls a helper by asking, for example, "OK Mitutoyo." A helper image HG is displayed in response to this. The helper image HG responds by displaying, for example, "Hello" and displaying an image and sound.

Next, user 800 asks, for example, "Teach me connection error." In response to the user's 800 inquiry, the helper image HG responds with, for example, "Please check connection of this cable." Along with this response, the measuring machine image MG and the personal computer image PC-G are displayed. Furthermore, an image G of the cable prompting the check is displayed. By referring to this image, the user 800 can visually recognize which cable connection has an error.

FIG. 28 shows an example of helper display when an error occurs.
If any error occurs in the measurement by the measuring machine M, the helper image HG is automatically displayed. The helper image HG notifies the occurrence of an error by responding, for example, "Error occurred."

Next, the user 800 asks, for example, "Teach me about error." The helper image HG responds to a question from the user 800 and outputs details of the error that has occurred, such as "I teach you...".

In this way, by displaying the helper image HG and interacting with the helper image HG, the user 800 can solve the problem as if they were directly receiving guidance from a service person.

(Fifth embodiment)
FIG. 29 is a configuration diagram illustrating a measurement system according to the fifth embodiment.
A measurement system 1E according to the present embodiment has a configuration in which a computer 100A, which is a user-side control device, and a computer 100B, which is a supporter-side control device, are connected to each other via a network.

Computer 100A is connected with measuring instrument M, operation panel CT, display D1, keyboard K1, mouse MS1, head mounted display HD1, headphone microphone HPM1, 3D camera CM and 3D sensor SR1.

The computer 100A also includes a CPU 10A, a storage unit 20A, a calculation unit 30A, a measuring instrument control unit 40, a display control unit 50A, an image input/output unit 51A, a camera control unit 52A, a stereoscopic image generation unit 53A, a photographed image synthesis unit 531A, It has a head mounted display output section 54A, a headphone microphone input/output section 61A, a voice input/output section 62A, a sound reproduction section 621A, a 3D sensor input section 70A, a position/distance/orientation recognition section 75A, and a communication control section 101A.

A display D2, a keyboard K2, a mouse MS2, a head mounted display HD2, a headphone microphone HPM2 and a 3D sensor SR2 are connected to the computer 100B.

The computer 100B also includes a CPU 10B, a storage unit 20B, a calculation unit 30B, a display control unit 50B, a stereoscopic image generation unit 53B, a head-mounted display output unit 54B, a headphone microphone input/output unit 61B, an audio input/output unit 62B, and an audio reproduction unit. 621B, 3D sensor input unit 70B, position/distance/orientation recognition unit 75B, and communication control unit 101B.

In such a measurement system 1E, the user side where the measuring machine M is installed and the supporter side of the measuring machine M are connected to each other via a network N. The user side can refer to the combination of the image IMG of the measuring device M displayed on the head-mounted display HD1 and the image sent from the supporter side.

On the side of the supporter, in the virtual three-dimensional space displayed on the head-mounted display HD2, it is possible to refer to the composition of the measuring machine image MG such as graphics and the image sent from the user side. In other words, the user and the supporter, who are separated from each other, refer to the display of the head-mounted displays HD1 and HD2, respectively, to perform operations with reference to 3D images as if they were close to each other. will be able to

FIG. 30 is a schematic diagram showing an operation example.
As shown in FIG. 30, a user 800 wears a head mounted display HD1 and a headphone microphone HPM1. The 3D sensor SR1 detects the positions of objects such as the user 800, the measuring machine M, and the workpiece W in the user's real three-dimensional space. A 3D camera CM provided in the head-mounted display HD1 captures 3D images of the measuring instrument M, the workpiece W, and the like. The information of the measuring device M on the user side, the information detected by the 3D sensor SR1, the information acquired by the 3D camera CM, the information acquired by the headphone microphone HPM1, etc. are transmitted from the communication control unit 101A to the supporter side via the network N. be done.

On the other hand, the support person 900 wears a head mounted display HD2 and a headphone microphone HPM2. The 3D sensor SR2 detects the position of an object such as the supporter 900 in the supporter's real three-dimensional space. Information detected by the 3D sensor SR2 on the supporter side, information acquired by the headphone microphone HPM2, and the like are transmitted from the communication control unit 101B through the network N to the user side.

Next, a display example of the head mounted display HD1 of the user 800 will be described. A 3D image captured by the 3D camera CM is displayed on the head-mounted display HD1 of the user 800 . For example, images of the measuring instrument M, the workpiece W, the display D1 of the computer 100A, and the like are displayed on the head-mounted display HD1. In addition, a helper image HG1 is displayed on the head-mounted display HD1 as a supporter side image.

An image imitating the supporter 900 is used as the helper image HG1. The helper image HG1 may be a character image or the like other than an image imitating a person. Helper image HG1 changes according to the movement of supporter 900 .

Specifically, the movement of the supporter 900 is detected by the 3D sensor SR2 on the supporter side, and the helper image HG1 is moved to follow the movement of the supporter 900 based on the detection result. As a result, the user 800 can refer to an image as if the supporter 900 were at the place where the measuring machine M is located.

The supporter 900 sends advice to the user 800 with gestures and voice. The motion of the helper image HG1 referred to by the user 800 changes according to this motion. Also, the sound from the supporter 900 is output from the headphone microphone HPM1 of the user 800 .

Next, a display example of the head-mounted display HD2 of the supporter 900 will be described. An image of a virtual three-dimensional space is displayed on the head-mounted display HD2 of the supporter 900 based on information transmitted via the network N from the user side. A measuring machine image MG used by the user 800 and an image of the work W (work image) WG are displayed in the virtual three-dimensional space. Also, the image D-IMG on the display D1 of the computer 100A of the user 800 is synthesized and displayed in the virtual three-dimensional space.

Furthermore, a user image YG is displayed on the head-mounted display HD2 as a user-side image. An image simulating the user 800 is used as the user image YG. The user image YG may be a character image or the like other than an image imitating a person's shape. User image YG changes according to the movement of user 800 .

That is, the movement of the user 800 is detected by the 3D sensor SR1 on the user side, and the user image YG is moved to follow the movement of the user 800 based on the detection result. As a result, the supporter 900 can feel as if he or she is at the measurement location of the user 800 from the measuring machine image MG, the workpiece image WG, and the user image YG displayed in the virtual three-dimensional space.

For example, the user 800 can refer to the image of the measuring instrument M displayed on the head-mounted display HD1 and the helper image HG1 synthesized with this image, and perform the measurement work while receiving advice from the helper image HG1. .

On the other hand, the support person 900 refers to the measuring machine image MG in the virtual three-dimensional space displayed on the head-mounted display HD2 and the user image YG synthesized with this measuring machine image MG, and instructs the user 800 on the operation procedure. You can send advice such as instructions.

In this way, both the user 800 and the supporter 900 can communicate with each other using voice and gestures while referring to each other's images, and even though they are in remote locations, they can feel as if they are actually there. You will be able to proceed with your work with such a feeling.

(Sixth embodiment)
FIG. 31 is a configuration diagram illustrating a measurement system according to the sixth embodiment.
As shown in FIG. 31, the measurement system 1F according to this embodiment includes a computer 100 and a display device 300 connected to the computer 100 via wireless communication. Computer 100 is connected with measuring machine M, control panel CT, camera C1, display D, keyboard K and mouse MS. The computer 100 includes a CPU 10, a storage unit 20, a feature point generation unit 35, a measurement result synthesis setting unit 36, a measuring machine control unit 40, a display control unit 50, an input/output control unit 60, an operation panel input/output unit 65, and a communication control unit. 101A.

The feature point generation unit 35 performs processing for extracting feature points from the image of the measurement object (work) acquired by the camera C1. A feature point is an even portion or a side (outline) necessary to form the outer shape of a work. The measurement result synthesis setting unit 36 is a part that associates the feature points generated by the feature point generation unit 35 with the workpiece measurement results. For example, the distance between two adjacent feature points is associated with the measurement result of the workpiece corresponding to that position.

The display device 300 is provided with a camera C2 and a display panel DP such as a liquid crystal panel. Display device 300 has CPU 310 , storage unit 320 , feature point extraction/tracking unit 350 , input/output unit 360 , measurement result synthesis unit 370 and communication control unit 301 .

The feature point extraction/tracking unit 350 extracts feature points from the image of the workpiece captured by the camera C2, and performs processing for tracking the extracted feature points in the image. Feature points are extracted and tracked while the workpiece is being photographed by camera C2.

The measurement result synthesizing section 370 performs a process of synthesizing the measurement result with the image of the work taken by the camera C2 or the image of the work based on this image. That is, based on the feature points extracted by the feature point extraction/tracking unit 350, the measurement result synthesizing unit 370 reads the measurement results associated with the feature points from the computer 100, and converts the measurement results into video or images of the workpiece. Synthesize.

As a result, on the display panel DP of the display device 300, a video or image of the workpiece is displayed in which the measurement results corresponding to the feature points are synthesized. In other words, when the user of the display device 300 acquires the image of the workpiece with the camera C2, the user can refer to the image of the workpiece or the display in which the measurement result is combined with the image.

In addition, in the measurement system 1F according to the present embodiment, it is also possible to refer to the measurement results using a plurality of display devices 300. FIG. By acquiring the image of the work W by each display device 300, the image or image of the work and the measurement result are synthesized and displayed on each display device 300. FIG.

FIG. 32 is a flow chart showing an example of the measurement result composite display process.
First, as shown in step S201, an image of the work is acquired by camera C2. Next, as shown in step S202, it is determined whether or not feature points have been extracted. If not already extracted, the feature points are extracted in step S203. After the feature points are extracted from the image of the workpiece acquired by the camera C2, the feature points are tracked as shown in step S204. Feature point extraction and tracking are performed by the feature point extraction/tracking unit 350 .

Next, as shown in step S205, the measurement results are combined. Here, in accordance with the feature points of the work being extracted and tracked, processing is performed to synthesize an image of the measurement results corresponding to the feature points with the image of the work. The measurement result synthesizing process is performed by the measurement result synthesizing section 370 .

FIG. 33 is a schematic diagram showing an example of a combined display of measurement results.
In the example shown in FIG. 33, a mobile terminal is used as the display device 300 . A mobile terminal is provided with a camera C2 and a display panel DP. When the work W is photographed by the camera C2, an image IMG of the work W is displayed on the display panel DP.

The feature point extraction/tracking unit 350 extracts the feature points of the work W from the image IMG of the work W captured by the camera C2, and tracks the feature points while the image is being captured. In the example shown in FIG. 33, feature points CP1 to CP4 of the workpiece W are extracted and tracked.

The measurement result synthesizing unit 370 synthesizes the measurement results corresponding to the extracted and tracked feature points CP1 to CP4 with the image. For example, the measurement result of the distance between the feature points CP1 and CP2 is displayed together with the dimension line between the feature points CP1 and CP2 in the image IMG of the workpiece W. Also, the measurement result of the distance between the feature points CP2 and CP3 is displayed together with the dimension line between the feature points CP2 and CP3 in the image IMG of the work W. FIG. Furthermore, the measurement result of the diameter of the hole obtained from the characteristic point CP4 is displayed at the position of the hole indicated by the characteristic point CP4 in the image IMG of the workpiece W together with the dimension line.

Such composite display is continued while the work W is photographed by the display device 300 . Even if the angle of view for photographing the work W changes, the display position of the measurement result also changes so as to follow the change in the image IMG of the work W because the characteristic points CP1 to CP4 of the work W are tracked. displayed. Therefore, by photographing the workpiece W at a desired angle, the user can refer to the image IMG of the workpiece W viewed from a predetermined angle and the display of the corresponding measurement results.

FIGS. 34A to 34E are schematic diagrams showing other composite display examples.
FIG. 34(a) shows an image IMG of the work W photographed by the camera C2. Also, FIG. 34(b) shows feature points extracted and tracked from the image IMG. Here, feature points are indicated by circles. FIG. 34(c) shows an example in which the measurement result is synthesized and displayed on the image IMG. Here, preset measurement results are synthesized and displayed at the positions of the feature points.

On the other hand, FIG. 34(d) shows an example of the work image WG fitted to the CAD model based on the image IMG. That is, by extracting feature points from the image IMG shown in FIG. Based on this, the CAD model of the work W is fitted and the work image WG is displayed. Then, as shown in FIG. 34(e), the measurement result is combined with the workpiece image WG of the CAD model and displayed.

The user selects whether to synthesize the measurement result with the image IMG or synthesize the measurement result with the work image WG of the CAD model. By synthesizing the measurement results with the image IMG, the actually measured work W and the measurement results of the work W can be referred to.

In addition, if the measurement results are combined with the work image WG of the CAD model, design information based on the CAD model and the display method of the CAD model (surface model, wire frame model, section display, etc.) can be selected and adjusted according to the display. can be used to synthesize measurement results.

In the work image WG of the CAD model, the display method such as the display color can be changed based on the difference between the design value and the measured value. For example, if the dimension exceeds the allowable range, it can be displayed in red, or the display mode can be changed between the portion within the allowable range and the portion not within the allowable range.

Further, the computer 100 and the display device 300 of the measurement system 1F according to the present embodiment may constitute a user interface device. According to this user interface device, the feature points of the work W are extracted by photographing the image of the work W with the camera C2. It can be displayed by combining it with the image IMG.

(Seventh embodiment)
FIG. 35 is a configuration diagram illustrating a measurement system according to the sixth embodiment.
As shown in FIG. 35, the measurement system 1G according to this embodiment includes a computer 100 and a display device 300 connected to the computer 100 via wireless communication. Computer 100 is connected with measuring machine M, control panel CT, camera C1, display D, keyboard K and mouse MS. The computer 100 includes a CPU 10, a storage unit 20, a marker recognition registration unit 38, a marker measurement result correspondence setting unit 39, a measuring machine control unit 40, a display control unit 50, an input/output control unit 60, an operation panel input/output unit 65, and communication control. It has a part 101A.

The marker recognition registration unit 38 recognizes and registers information specific to the marker (shape recognition result, identification information, etc.) from the image of the marker acquired by the camera C1. Here, as a marker, a mark such as a figure or a photograph that can be distinguished from others is used. The marker measurement result correspondence setting unit 39 is a part that associates and registers the markers registered by the marker recognition registration unit 38 and predetermined measurement results. These pieces of registration information are stored in the storage unit 20 .

The display device 300 is provided with a camera C2 and a display panel DP such as a liquid crystal panel. Display device 300 has CPU 310 , storage section 320 , input/output section 360 , marker recognition section 380 , marker measurement result synthesis section 390 and communication control section 301 .

The marker recognition unit 380 performs processing for recognizing information unique to the marker from the image of the marker captured by the camera C2. The marker measurement result synthesizing unit 390 performs a process of synthesizing the measurement result associated with the marker-specific information recognized by the marker recognition unit 380 with the background image.

That is, the marker measurement result synthesizing unit 390 reads the measurement result associated with the marker-specific information recognized by the marker recognition unit 380 from the computer 100 and synthesizes the measurement result with the background image.

In such a measurement system 1G, the display panel DP of the display device 300 displays a background image and the measurement results combined with the positions of the markers. That is, when the user of the display device 300 acquires the image of the marker with the camera C2, the user can refer to the display in which the measurement result associated with the marker is combined with the background image.

In addition, in the measurement system 1G according to the present embodiment, it is also possible to refer to the measurement results by a plurality of display devices 300. FIG. By acquiring the image of the marker by each display device 300, the background and the measurement result are synthesized and displayed on the display panel DP of each display device 300. FIG.

FIG. 36 is a flowchart illustrating marker registration and setting operations. This operation is performed by computer 100 .
First, as shown in step S301, the image of the marker is acquired by the camera C1. Next, as shown in step S302, recognition of the image of the marker is performed. For example, a process of recognizing a photographed marker of a specific shape, a specific figure, a photograph, etc., adding identification information specific to the marker, and registering it is performed. This processing is performed by the marker recognition registration unit 38 .

Next, as shown in step S303, a process of setting correspondence between markers and measurement results is performed. That is, a process of associating the identification information of the marker recognized in the previous step S302 with the predetermined measurement result is performed. This processing is performed by the marker measurement result correspondence setting unit 39 .

Next, as shown in step S304, identification information unique to the marker and measurement results associated with this identification information are registered. The correspondence between the identification information unique to the marker and the measurement result is stored in the storage unit 20 as marker setting information. This processing is performed by the marker measurement result correspondence setting unit 39 .

FIG. 37 is a flow chart illustrating marker recognition and composite display. This operation is performed by the display device 300 .
First, as shown in step S401, marker setting information is received from the storage unit 20 of the computer 100 via wireless communication. Next, as shown in step S402, an image of the marker captured by camera C2 is acquired. Next, as shown in step S403, identification information unique to the marker is recognized from the acquired image of the marker. This processing is performed by the marker recognition unit 380 .

Next, as shown in step S404, a process of synthesizing the measurement results associated with the markers is performed. That is, from the marker-specific identification information recognized in the previous step S403, the measurement result associated with this identification information is read from the marker setting information. Then, a process of synthesizing the measurement result with the background image captured by the camera C2 is performed. This processing is performed by the marker measurement result synthesizing section 390 .

Next, as shown in step S405, it is determined whether or not all marks have been recognized and synthesized. If not completed, the process returns to step S402, and the subsequent processes are repeated. As a result, the measurement results are synthesized and displayed for all the markers captured by the camera C2.

38 to 40 are schematic diagrams showing display examples of measurement results by marker recognition.
FIG. 38 shows an example in which composite display is performed on a mobile terminal. In this example, a mobile terminal is used as the display device 300 . Also, the paper PP is provided with a plurality of markers MK1 to MK5.

Markers MK1 to MK5 may be specific graphics or specific photographs. Each marker MK1 to MK5 is associated with marker-specific identification information and a measurement result. For example, the markers MK1, MK2 and MK3 are associated with predetermined measurement results of the first measuring device M1. Further, predetermined measurement results of the second measuring device M2 are associated with the markers MK4 and MK5. These associations are stored in the storage unit 20 .

The user takes an image of the paper PP with the camera C<b>2 of the mobile terminal, which is the display device 300 . When the images of the markers MK1 to MK5 are captured by this photographing, the identification information of each of the markers MK1 to MK5 is recognized. Then, based on this identification information, the measurement results associated with the markers MK1 to MK5 are read out from the storage section 20 of the computer 100. FIG.

Then, the images of the measurement results associated with the respective markers MK1 to MK5 are synthesized in the display areas of the captured markers MK1 to MK5 and displayed on the display panel DP. For example, each of the markers MK1 to MK3 is synthesized with an image of the measurement result by the first measuring device M1, and each of the markers MK4 and MK5 is synthesized with an image of the measurement result by the second measuring device M2.

Images of measurement results are displayed in various formats such as a table format and a graph format. When synthesizing the images, the size and angle may be adjusted so as to match the areas indicated by the markers MK1 to MK5. Also, a three-dimensional image may be synthesized and displayed so that the image of the measurement result stands out on the image of the planar paper PP.

In this way, the user can refer to the measurement results laid out at the positions of the markers MK1 to MK5 simply by photographing the paper PP with the markers MK1 to MK5 with the portable terminal.

39 and 40 show another display example.
First, FIG. 39 shows a display example before composite display. As in the previous example, the paper PP is provided with a plurality of markers MK1 to MK5. Also, in this example, a glasses-type display device 300 is used.

In this display device 300, a display panel DP is provided on the lens portion of the glasses. Also, a camera C2 is provided on the temple portion of the eyeglasses. By wearing the glasses-type display device 300, the user can capture an image in front of the glasses with the camera C2 and can refer to information displayed on the display panel DP in the lens portion.

FIG. 40 shows a display example after composite display. That is, when the markers MK1 to MK5 are photographed by the camera C2 of the glasses-type display device 300, the identification information of each of the markers MK1 to ML5 is recognized. Then, the measurement results associated with the markers MK1 to MK5 are read out from the storage section 20 based on the identification information. Then, the images of the measurement results associated with the respective markers MK1 to MK5 are synthesized in the display areas of the captured markers MK1 to MK5 and displayed on the display panel DP.

A user wearing the glasses-type display device 300 can refer to a display in which the measurement result is combined with the background image displayed on the display panel DP in the lens portion of the glasses. In this example, the display areas of the markers MK1, MK2 and MK3 are combined with the measurement results of the first measuring machine M1, and the display areas of the markers MK4 and MK5 are combined with the measurement results of the second measuring machine M2.

By changing the setting of the correspondence between the markers MK1 to MK5 and the measurement results, it is possible to display different measurement results even with the same markers MK1 to MK5. That is, different measurement results can be displayed by creating one layout form for the markers MK1 to MK5.

Also, one marker may be associated with a plurality of measurement results. In this case, the user may arbitrarily switch which measurement result among the plurality of measurement results corresponding to one marker is synthesized and displayed. Accordingly, by capturing an image of one marker, it is possible to display different measurement results by switching to the position of the marker.

The image of the measurement result may be a moving image. As a result, a moving image is synthesized on the image of the paper PP, and the mere paper PP can be treated like a virtual moving image display device.

Further, the computer 100 and the display device 300 of the measurement system 1G according to the present embodiment may constitute a user interface device. According to this user interface device, by photographing the images of the markers MK1 to MK5 with the camera C2, the information corresponding to the markers MK1 to MK5 is read out, and while the images of the markers MK1 to MK5 are being photographed, The read information can be displayed on the display panel DP.

As described above, according to the measuring systems 1A, 1B, 1C, 1D, 1E, 1F and 1G according to the present embodiments, it is possible to improve the operability of the measuring instrument M.

Although the present embodiment has been described above, the present invention is not limited to these examples. For example, in a system using a head-mounted display HD to which a 3D camera CM is attached, the method is not limited to the method of synthesizing and displaying an image of the real space captured by the 3D camera CM and computer graphics as described above. For example, computer graphics may be displayed on the head-mounted display HD in accordance with real space coordinates. In this case, instead of the 3D camera CM, or in addition to the 3D camera CM, a 3D sensor is mounted on the head-mounted display HD, the coordinate system of the real space is grasped in real time by the 3D sensor, and the virtual space is generated by computer processing. It may be constructed and displayed with computer graphics updated in real time as if it actually existed in that space. In addition, additions, deletions, and design changes made by those skilled in the art to the above-described embodiments, and combinations of features of the embodiments as appropriate, do not include the gist of the present invention. so long as they are included in the scope of the present invention.

1A, 1B, 1C, 1D, 1E, 1F, 1G... Measurement system 10, 10A, 10B, 310... CPU
20, 20A, 20B, 320 Storage units 30, 30A, 30B Operation unit 35 Characteristic point generation unit 36 Measurement result synthesis setting unit 38 Marker recognition registration unit 39 Marker measurement result correspondence setting unit 40 Measuring instrument control Units 50, 50A, 50B Display control units 51, 51A Video input/output unit 52 3D camera control unit 52A Camera control units 53, 53A, 53B Stereoscopic video generation units 54, 54A, 54B Head mount display output unit 55 Control generation unit 60 Input/output control units 61, 61A, 61B Headphone microphone input/output units 62, 62A, 62B Audio input/output unit 65 Control panel input/output units 70, 70A, 70B 3D sensor input unit 75 , 75A, 75B Posture recognition unit 76 Control input recognition unit 80 Projected graphic generation unit 85 Projector output unit 90 Remote control unit 95 Helper generation units 100, 100a, 100B Computer 101 Command setting units 101A, 101B , 301 communication control unit 102 moving object detection unit 103 command execution unit 104 measurement operation storage unit 201 image storage unit 300 display device 350 tracking unit 360 input/output unit 370 measurement result synthesis unit 380 marker recognition Unit 390 Marker measurement result synthesizing unit 501 Measuring head 501G Measuring head images 531, 531A Photographed image synthesizing unit 621, 621A, 621B Sound reproducing unit 800 User 801 Hand 802 Foot 900 Supporters B1 to B7 ...Button setting area BG...Button images C1, C2, CM1...Camera CM...3D camera CM1...Cameras CP1 to CP4...Characteristic points D, D1, D2...Display D-IMG, IMG...Video DP...Display panel FL...Floor G , G1 to G7... Image HD, HD1, HD2... Head mounted display HG, HG1... Helper image HPM, HPM1, HPM2... Headphone microphone K, K1, K2... Keyboard M, M1, M2... Measuring machine MG... Measuring machine image MK1 ~MK5... Markers MS, MS1, MS2... Mouse N... Network P1 to P4... Even part PC-G... Image PC-IMG... Image PP... Paper PR... Projector SR, SR1, SR2... 3D sensor W... Work WB... White Board WG: work image YG: user image

Claims (28)

a measuring instrument for measuring an object to be measured;
a measuring machine control unit that controls the measuring machine based on an instruction from a user;
a three-dimensional sensor unit for detecting three-dimensional coordinates of a predetermined object in a real three-dimensional space;
a command setting unit for setting a button setting area in the real three-dimensional space and a correspondence between the button setting area and a command;
a moving object detection unit that detects a moving object from the three-dimensional coordinates detected by the three-dimensional sensor unit;
a command execution unit that executes the command associated with the button setting area when the three-dimensional coordinates of the moving object detected by the moving object detection unit are positioned within the button setting area;
A measurement system comprising: The measuring machine has a measuring head that acquires the three-dimensional coordinates of the object to be measured, and an image output unit that is provided on the measuring head and projects an image,
2. The measurement system according to claim 1, wherein the image output section outputs an image of the button setting area set by the command setting section. The moving object detection unit detects an operation of the measurement head,
3. The measurement system according to claim 2, wherein the command execution section executes the command according to the three-dimensional coordinates of the measurement head detected by the moving body detection section. The moving body detection unit detects motion of at least one of the hand and foot of the user,
4. The command execution unit according to any one of claims 1 to 3, wherein the command execution unit executes the command according to the three-dimensional coordinates of at least one of the user's hands and feet detected by the moving object detection unit. A measurement system as described in Clause 1. The command execution section executes a first command when the three-dimensional coordinates of the moving body detected by the moving body detection section remain within the button setting area for a predetermined period of time. 5. The measurement system according to any one of claims 1 to 4, wherein the second command is executed when moving along the button setting area. a three-dimensional imaging unit that acquires a three-dimensional image in the real three-dimensional space;
a display unit for displaying the three-dimensional image acquired by the three-dimensional imaging unit;
a display control unit for synthesizing the three-dimensional image in the real three-dimensional space and an image corresponding to the button setting area and displaying the image on the display unit;
The measurement system according to any one of claims 1 to 5, further comprising: 7. The command execution unit selects and executes one of the plurality of commands according to the hand shape of the user detected by the moving object detection unit. A measurement system as described in Clause 1. a measuring instrument for measuring an object to be measured;
a measuring machine control unit that controls the measuring machine based on an instruction from a user;
a three-dimensional sensor unit for detecting three-dimensional coordinates of a predetermined object in a real three-dimensional space;
a moving object detection unit that detects the motion of the user from the three-dimensional coordinates detected by the three-dimensional sensor unit;
a display unit that displays an image of the measuring machine in a virtual three-dimensional space;
a display control unit that moves the display position of the image of the measuring machine in the virtual three-dimensional space displayed on the display unit in accordance with the user's motion detected by the moving object detection unit;
A measurement system comprising: A measurement operation storage unit for recording a measurement operation along a display position of the image of the measuring device moved by the user in the virtual three-dimensional space displayed on the display unit. Item 9. The measurement system according to item 8. 10. The measuring system according to claim 9, wherein said measuring machine control unit controls said measuring machine based on said measuring operation stored in said measuring operation storage unit. further comprising an image storage unit that stores a CAD image of the measuring machine;
11. The measurement system according to any one of claims 8 to 10, wherein the display unit displays the CAD image stored in the image storage unit in the virtual three-dimensional space. further comprising an imaging unit that acquires an image of the measuring device in the real three-dimensional space;
The measurement according to any one of claims 8 to 11, wherein the display control unit displays an image of the measuring device acquired by the imaging unit in the virtual three-dimensional space of the display unit. system. 13. The display controller according to any one of claims 8 to 12, wherein the display control unit moves the image of the measuring instrument displayed in the virtual three-dimensional space of the display unit in accordance with a preset measurement operation. 2. The measurement system according to item 1. 14. The display control unit according to any one of claims 8 to 13, wherein the display control unit synthesizes the image of the object to be measured and the measurement result and displays them in the virtual three-dimensional space of the display unit. measurement system. further comprising a remote control unit connected to the measuring device control unit via a network;
The remote control unit transmits a command to the measuring instrument control unit via the network based on a measurement operation designated by the user in the virtual three-dimensional space displayed on the display unit. Measuring system according to any one of claims 8-14. a measuring instrument for measuring an object to be measured;
a measuring machine control unit that controls the measuring machine based on an instruction from a user;
an imaging unit that acquires an image of the measuring device in a real three-dimensional space;
an image storage unit that stores an image related to the measuring machine;
a display unit that displays in the real three-dimensional space;
a display control unit for synthesizing the image of the measuring instrument acquired by the imaging unit and the image stored in the image storage unit in the real three-dimensional space and displaying the image on the display unit;
A measurement system comprising: 17. The measurement system according to claim 16, wherein the display control unit synthesizes an image representing a predetermined measurement procedure with the image of the measuring device acquired by the imaging unit. 18. The measuring system according to claim 16, wherein the display control unit synthesizes an image corresponding to a predetermined abnormality with the image of the measuring device acquired by the imaging unit. The measuring system according to any one of claims 16 to 18, wherein the display control unit synthesizes an image corresponding to predetermined guidance with the image of the measuring device acquired by the imaging unit. a user-side control device connected to a measuring machine for measuring an object to be measured;
a supporter-side control device connected to the user-side control device via a network;
A measurement system comprising
The user-side control device
a measuring machine control unit that controls the measuring machine based on an instruction from a user;
a first three-dimensional sensor unit for detecting three-dimensional coordinates of a predetermined object in a real three-dimensional space;
a first moving object detection unit that detects a moving object from the three-dimensional coordinates detected by the first three-dimensional sensor unit;
a three-dimensional imaging unit that acquires a three-dimensional image in the real three-dimensional space;
a first display unit that displays the three-dimensional image acquired by the three-dimensional imaging unit;
a first display control unit for synthesizing the three-dimensional video and the image sent from the supporter side control device and displaying it on the first display unit;
The supporter side control device,
a second three-dimensional sensor unit that detects the three-dimensional coordinates of the supporter in the real three-dimensional space;
a second moving body detection unit that detects the motion of the supporter from the three-dimensional coordinates detected by the second three-dimensional sensor unit;
a second display unit that displays an image based on the three-dimensional video acquired by the three-dimensional imaging unit in a virtual three-dimensional space;
a second display control unit that causes the second display unit to display an image based on information sent from the user-side control device in accordance with the three-dimensional coordinates of the virtual three-dimensional space;
and a motion information transmission unit that transmits an image related to the motion of the supporter detected by the second moving body detection unit to the user-side control device via the network. The first display control unit causes the first display unit to display a supporter-side image corresponding to the supporter based on the image sent from the supporter-side control device,
21. The measurement system according to claim 20, wherein the second display control unit displays a user-side image corresponding to the user on the second display unit based on the image sent from the user-side control device. a measuring instrument for measuring an object to be measured;
a measurement result storage unit that stores measurement results obtained by the measuring machine;
an imaging unit that acquires an image of the object to be measured;
a feature point extracting unit that extracts feature points from the image acquired by the imaging unit;
a display unit for displaying an image of the measurement object obtained from the video image acquired by the imaging unit or the feature points extracted by the feature point extraction unit;
a measurement result reading unit that reads the measurement result corresponding to a predetermined position of the measurement object from the measurement result storage unit based on the feature points extracted by the feature point extraction unit;
a display control unit for synthesizing the measurement result read by the measurement result reading unit with the video or the image of the measurement object and displaying the result on the display unit;
A measurement system comprising: 23. The measurement system according to claim 22, wherein the display control unit synthesizes the measurement result with the video or the image in accordance with the imaging angle of the video of the object to be measured acquired by the imaging unit. a measuring instrument for measuring an object to be measured;
a measurement result storage unit that stores measurement results obtained by the measuring machine;
an imaging unit that acquires an image of the marker;
a marker recognition unit that recognizes the marker from the image of the marker acquired by the imaging unit;
a measurement result reading unit that reads the measurement result corresponding to the marker recognized by the marker recognition unit from the measurement result storage unit;
a display control unit that causes a display unit to display the measurement result read by the measurement result reading unit while the image of the marker is being acquired by the imaging unit;
A measurement system comprising: The imaging unit acquires an image of the background to which the marker is attached,
25. The measurement system according to claim 24, wherein the display control unit synthesizes the measurement result corresponding to the marker with the image of the background and displays the result on the display unit. a three-dimensional sensor unit for detecting three-dimensional coordinates of a predetermined object in a real three-dimensional space;
a command setting unit for setting a button setting area in the real three-dimensional space and a correspondence between the button setting area and a command;
a moving object detection unit that detects a moving object from the three-dimensional coordinates detected by the three-dimensional sensor unit;
a command execution unit that executes the command associated with the button setting area when the three-dimensional coordinates of the moving object detected by the moving object detection unit are positioned within the button setting area;
a three-dimensional imaging unit that acquires a three-dimensional image in the real three-dimensional space;
a display unit for displaying the three-dimensional image acquired by the three-dimensional imaging unit;
a display control unit for synthesizing the three-dimensional image and an image corresponding to the button setting area and displaying the result on the display unit;
with
The user interface device, wherein the command execution unit selects and executes one of the plurality of commands according to the shape of the user's hand detected by the moving object detection unit. an imaging unit that acquires an image of an object to be measured;
a feature point extracting unit that extracts feature points from the image acquired by the imaging unit;
a display unit for displaying an image of the measurement object obtained from the video image acquired by the imaging unit or the feature points extracted by the feature point extraction unit;
a dimension acquisition unit that acquires dimension information corresponding to a predetermined position of the measurement object based on the feature points extracted by the feature point extraction unit;
a display control unit for synthesizing the dimension information acquired by the dimension acquisition unit with the video or the image of the measurement object and displaying the result on the display unit;
A user interface device comprising: an imaging unit that acquires an image of the background to which the marker is attached;
a marker recognition unit that recognizes the marker from the image of the marker acquired by the imaging unit;
a measurement result acquisition unit that acquires a measurement result corresponding to the marker recognized by the marker recognition unit;
a display control unit for synthesizing the measurement result obtained by the measurement result obtaining unit with the background image and displaying the result on a display unit while the image of the marker is being obtained by the imaging unit;
A user interface device comprising:

Images