JP2020073872A - Control system - Google Patents

Abstract

To provide a measurement system which can increase operability of a measuring machine, and a user interface device.SOLUTION: A measurement system of the present invention comprises: a measuring machine which measures an object to be measured; a measuring machine control unit which controls the measuring machine on the basis of an instruction by a user; a three-dimensional sensor unit which detects a three-dimensional coordinate of a predetermined object in a real three-dimensional space; a command set unit which sets a button set region in the real three-dimensional space and association between the button set region and a command; a moving body detection unit which detects a moving body from the three-dimensional coordinate detected by the three-dimensional sensor unit; and a command execution unit which, when the three-dimensional coordinate of the moving body detected by the moving body detection unit is positioned in the button set region, executes the command associated with the button set region.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a measurement system and a user interface device that improve the operability of a measuring machine.

  In a measuring machine that measures the dimensions and three-dimensional position of an object to be measured (workpiece), a user performs an operation of designating a measurement point of a work by using a measuring head. Depending on the shape of the work, the user may have to handle the measuring head in an unnatural posture. A computer is connected to the measuring machine, and various information such as work information, measurement procedures, and measurement results are displayed on the display. However, if the work and the computer are separated from each other, both the measurement work and the display reference cannot be performed efficiently.

  In the position measuring device described in Patent Document 1, the design model read by the CAD software running on the computer is displayed on the display of the computer. Then, the data obtained by the measurement is displayed on the design model of the display.

  Further, the coordinate measuring machine described in Patent Document 2 includes a device including an image projector, and projects visual guidance information and measurement results on the component from the image projector. Further, Patent Documents 3 to 6 disclose measurement systems and user interface devices that utilize virtual reality space.

Special table 2011-519419 gazette Special table 2013-517500 gazette JP, 06-241754, A Special table 2011-521318 gazette Japanese Patent Publication No. 11-513157 Japanese Patent Publication No. 2009-521985

  However, none of the technologies can be said to be a system in which the usability of the user is fully taken into consideration. For example, when measuring a work, both hands of the user are often blocked, and it is difficult to control the measuring machine while performing the measurement work. In addition, depending on the work, the user may move to a place suitable for measurement to perform the work. When the user is away from the control device (computer) of the measuring machine, it becomes more difficult to control the measuring machine while performing the measurement work.

  An object of the present invention is to provide a measurement system and a user interface device that can improve the operability of a measuring machine.

  In order to solve the above problems, a measuring system of the present invention includes a measuring machine that measures an object to be measured, a measuring machine controller that controls the measuring machine based on an instruction from a user, and a predetermined object in an actual three-dimensional space. Three-dimensional sensor unit for detecting the three-dimensional coordinates of the button, a button setting region in the actual three-dimensional space, a command setting unit for setting the correspondence between the button setting region and a command, and the three-dimensional coordinates detected by the three-dimensional sensor unit And a command execution unit that executes a command associated with the button setting area when the three-dimensional coordinates of the moving object detected by the moving object detection unit are located within the button setting area. It is characterized by

  According to such a configuration, the command setting unit sets a desired position in the actual three-dimensional space as an area functioning as a button. In addition, by detecting the moving body by the moving body detection unit, a moving body (moving body) such as a user is detected. Accordingly, it is possible to detect that the button setting area in the actual three-dimensional space is selected from the motion of the moving body and execute the command associated with the button setting area.

  In the measuring system of the present invention, the measuring machine has a measuring head for acquiring the three-dimensional coordinates of the object to be measured, and a video output unit provided on the measuring head for projecting a video, and the video output unit is a command setting unit. You may make it output the image of the button setting area set by. With such a configuration, the image of the button setting area output by the video output unit can be projected.

  In the measurement system of the present invention, the moving body detection unit may detect the operation of the measurement head, and the command execution unit may execute the command according to the three-dimensional coordinates of the measurement head detected by the moving body detection unit. With such a configuration, it is possible to detect that the button setting area is selected from the operation of the measuring head and execute the command associated with the button setting area.

  In the measurement system of the present invention, the moving body detection unit detects at least one of the hand and foot of the user, and the command execution unit three-dimensionally includes at least one of the hand and foot of the user detected by the moving body detection unit. The command may be executed according to the coordinates. With such a configuration, it is possible to detect that the button setting area is selected from the motion of the user's hand or foot and execute the command associated with the button setting area.

  In the measurement system of the present invention, the command execution unit executes the first command when the three-dimensional coordinates of the moving body detected by the moving body detection unit remain in the button setting area for a certain period of time, and the three-dimensional coordinates of the moving body. The second command may be executed when is moving along the button setting area. With such a configuration, commands can be switched according to the type of motion of the moving body.

  In the measurement system of the present invention, a three-dimensional image capturing unit that acquires a three-dimensional image in an actual three-dimensional space, a display unit that displays the three-dimensional image acquired by the three-dimensional image capturing unit, and a three-dimensional image in an actual three-dimensional space. The display control unit may further include a display control unit that combines the image corresponding to the button setting area and displays the combined image on the display unit. With such a configuration, it is possible to perform display in which the image of the button setting area is combined with the 3D video displayed in the actual 3D space.

  In the measurement system of the present invention, the command execution unit may select and execute any one of the plurality of commands according to the shape of the user's hand detected by the moving body detection unit. With such a configuration, display / non-display of the button setting area can be switched according to the shape of the user's hand, and a command corresponding to the button setting area can be executed.

  The measuring system of the present invention detects a measuring machine for measuring an object to be measured, a measuring machine controller for controlling the measuring machine based on an instruction from a user, and three-dimensional coordinates of a predetermined object in an actual three-dimensional space. The three-dimensional sensor unit, the moving body detection unit that detects the user's motion from the three-dimensional coordinates detected by the three-dimensional sensor unit, the display unit that displays the image of the measuring machine in the virtual three-dimensional space, and the display unit that displays the image. And a display control unit for moving the display position of the image of the measuring machine in the virtual three-dimensional space according to the user's motion detected by the moving body detection unit.

  With such a configuration, the image of the measuring machine is displayed in the virtual three-dimensional space. Further, it is possible to detect the motion of the user by the motion detection unit and move the image of the measuring machine in the virtual three-dimensional space according to the motion of the user.

  The measurement system of the present invention may further include a measurement operation storage unit that records the measurement operation along the display position of the image of the measuring machine moved by the user in the virtual three-dimensional space displayed on the display unit. .. With such a configuration, the measurement operation can be recorded by moving the image of the measuring machine in the virtual three-dimensional space.

  In the measuring system of the present invention, the measuring machine control unit may control the measuring machine based on the measuring operation stored in the measuring operation storage unit. With such a configuration, it is possible to perform measurement with an actual measuring machine based on the measurement operation recorded by moving the image of the measuring machine in the virtual three-dimensional space.

  The measuring system of the present invention further includes an image storage unit for storing a CAD (Computer Aided Design) image of the measuring machine, and the display unit displays the CAD image stored in the image storage unit in a virtual three-dimensional space. You can With such a configuration, a CAD image of the measuring machine can be displayed in the virtual three-dimensional space.

  The measurement system of the present invention further includes an imaging unit that acquires an image of the measuring machine in the actual three-dimensional space, and the display control unit displays the image of the measuring machine acquired by the imaging unit in the virtual three-dimensional space of the display unit. It may be allowed to. With such a configuration, an image of an actual measuring machine can be displayed in the virtual three-dimensional space.

  In the measurement system of the present invention, the display control unit may move the image of the measuring machine displayed in the virtual three-dimensional space of the display unit in accordance with the preset measurement operation. With such a configuration, it is possible to confirm the preset measurement operation of the measuring machine in the virtual three-dimensional space.

  In the measurement system of the present invention, the display control unit may combine the image of the measurement object and the measurement result and display them in the virtual three-dimensional space of the display unit. With such a configuration, the measurement result of the measurement target can be combined with the image of the measurement target in the virtual three-dimensional space and referred to.

  The measurement system of the present invention further includes a remote control unit connected to the measuring machine control unit via a network, and the remote control unit is a measurement operation designated by a user in the virtual three-dimensional space displayed on the display unit. Based on the above, the command may be transmitted to the measuring instrument control unit via the network. With such a configuration, it is possible to refer to the measurement operation designated by the user in the virtual three-dimensional space at a position (remote location) apart from the measuring machine. Further, a command can be transmitted from a remote place to the measuring instrument via the network.

  The measuring system of the present invention includes a measuring machine for measuring an object to be measured, a measuring machine control section for controlling the measuring machine based on an instruction from a user, an imaging section for acquiring an image of the measuring machine in an actual three-dimensional space, and a measuring system. An image storage unit for storing images related to the machine, a display unit for displaying in a real three-dimensional space, an image of the measuring machine acquired by the imaging unit, and an image stored in the image storage unit in a real three-dimensional space. And a display control unit for combining and displaying on the display unit. With such a configuration, it is possible to synthesize and display an image of an actual measuring machine and an image such as a graphic in an actual three-dimensional space.

  In the measurement system of the present invention, the display control unit may combine an image representing the predetermined measurement procedure with the video of the measuring machine acquired by the imaging unit. With such a configuration, an actual image of the measuring machine and an image of the measuring machine such as a graphic showing a predetermined measurement procedure can be combined and displayed in the three-dimensional space.

  In the measurement system of the present invention, the display control unit may combine an image corresponding to a predetermined abnormality with the image of the measuring machine acquired by the imaging unit. With such a configuration, an actual image of the measuring machine and an image of the measuring machine such as a graphic corresponding to a predetermined abnormality can be combined and displayed in the three-dimensional space.

  In the measurement system of the present invention, the display control unit may combine an image corresponding to a predetermined guidance with the video of the measuring machine acquired by the imaging unit. With such a configuration, an actual image of the measuring machine and an image of the measuring machine such as a graphic corresponding to a predetermined guidance can be combined and displayed in the three-dimensional space.

  The measurement system of the present invention is a measurement system including a user-side control device connected to a measuring device that measures an object to be measured, and a support-side control device connected to the user-side control device via a network. is there.

  The user-side control device in this measurement system includes a measuring machine control unit that controls a measuring machine based on an instruction from a user, and a first three-dimensional sensor unit that detects the three-dimensional coordinates of a predetermined object in an actual three-dimensional space. And a first moving body detecting unit that detects a moving body from the three-dimensional coordinates detected by the first three-dimensional sensor unit, a three-dimensional image capturing unit that acquires a three-dimensional image in an actual three-dimensional space, and a three-dimensional image capturing unit. A first display unit that displays the acquired three-dimensional image, and a first display control unit that combines the three-dimensional image and the image sent from the supporter-side control device and displays the combined image on the first display unit.

  The supporter-side control device in this measurement system supports from the second three-dimensional sensor unit that detects the three-dimensional coordinates of the supporter in the actual three-dimensional space and the three-dimensional coordinates detected by the second three-dimensional sensor unit. A second moving body detection unit that detects a person's motion, a second display unit that displays an image based on the three-dimensional image acquired by the three-dimensional imaging unit in a virtual three-dimensional space, and three-dimensional coordinates of the virtual three-dimensional space. A second display control unit for displaying the image sent from the user side control device on the second display unit in accordance with the above, and an image related to the operation of the supporter detected by the second moving body detection unit via the network through the user side control device. And an operation information transmission unit for sending to

  According to such a configuration, the user side in which the measuring instrument is installed and the supporter side of the measuring instrument are connected to each other via a network. The user side can refer to the combination of the image of the measuring instrument displayed on the first display unit and the image sent from the supporter side. The supporter side can refer to the combination of the image of the measuring machine such as a graphic and the image sent from the user side in the virtual three-dimensional space displayed on the second display unit.

  In the measurement system of the present invention, the first display control unit causes the first display unit to display the supporter-side image corresponding to the supporter based on the image sent from the supporter-side control device, and the second display control unit The user-side image corresponding to the user may be displayed on the second display unit based on the image sent from the user-side control device. With such a configuration, the user side can refer to the supporter side image sent from the supporter side on the first display unit, and the supporter side can refer to the user side image sent from the user side on the second display unit. can do.

  The measuring system of the present invention includes a measuring machine for measuring a measuring object, a measurement result storing section for storing a measurement result of the measuring machine, an image capturing section for acquiring an image of the measuring object, and an image acquired by the image capturing section. A feature point extraction unit that extracts feature points, a display unit that displays an image of the measurement target obtained from the image acquired by the imaging unit or the feature points extracted by the feature extraction unit, and the feature points extracted by the feature point extraction unit The measurement result reading unit that reads the measurement result corresponding to the predetermined position of the measurement object from the measurement result storage unit based on the above, and the measurement result read by the measurement result reading unit is combined with the image or image of the measurement object and displayed. And a display control unit for displaying on the unit.

  According to such a configuration, by projecting an image of the measurement target, the feature points of the measurement target are extracted, and the measurement result corresponding to the predetermined position of the measurement target is obtained based on the feature points. It can be combined with the video or image of and displayed.

  In the measurement system of the present invention, the display control unit may combine the measurement result with the image or image corresponding to the image capturing angle of the image of the measurement target acquired by the image capturing unit. According to such a configuration, it is possible to display the measurement result of the measurement target by shooting at the desired angle.

  The measuring system of the present invention includes a measuring machine for measuring an object to be measured, a measurement result storing section for storing a measurement result of the measuring machine, an image capturing section for acquiring a marker image, and a marker image acquired by the image capturing section. A marker recognition unit that recognizes the marker, a measurement result reading unit that reads the measurement result corresponding to the marker recognized by the marker recognition unit from the measurement result storage unit, and a measurement result read while the image of the marker is being acquired by the imaging unit And a display control unit for displaying the measurement result read by the unit on the display unit.

  With such a configuration, by acquiring the image of the marker, the measurement result corresponding to the marker is read out, and the measurement result can be displayed on the display unit while the image of the marker is captured. ..

  In the measurement system of the present invention, the imaging unit acquires the background image with the marker, and the display control unit synthesizes the background image with the measurement result corresponding to the marker and displays the result on the display unit. Good. With such a configuration, the measurement result can be combined with the background image and displayed on the display unit.

  The user interface device of the present invention associates a three-dimensional sensor unit that detects three-dimensional coordinates of a predetermined object in a real three-dimensional space, a button setting area in the real three-dimensional space, and a button setting area with a command. A command setting section for setting a moving body, a moving body detecting section for detecting a moving body from the three-dimensional coordinates detected by the three-dimensional sensor section, and a button when the three-dimensional coordinates of the moving body detected by the moving body detecting section are located in the button setting area. A command execution unit that executes a command associated with the setting area, a three-dimensional imaging unit that acquires a three-dimensional image in an actual three-dimensional space, and a display unit that displays the three-dimensional image acquired by the three-dimensional imaging unit, And a display control unit configured to combine a three-dimensional image and an image corresponding to the button setting area and display the combined image on the display unit, and the command execution unit has a shape of the user's hand detected by the moving body detection unit. Flip and wherein the selecting and executing one of a plurality of commands.

  With such a configuration, it is possible to switch the display / non-display of the button setting area according to the shape of the user's hand and to execute the command corresponding to the button setting area.

  The user interface device of the present invention includes an image capturing unit that acquires an image of a measurement target, a feature point extracting unit that extracts a feature point from the image acquired by the image capturing unit, and an image acquired by the image capturing unit or extracted by the feature extracting unit. A display unit that displays an image of the measurement target obtained from the feature points, and a dimension acquisition unit that acquires information about the dimensions corresponding to the predetermined positions of the measurement target based on the feature points extracted by the feature point extraction unit. And a display control unit configured to combine the dimension information acquired by the dimension acquisition unit with a video or image of the measurement target and display the image on the display unit.

  According to such a configuration, by projecting an image of the measurement target, the feature points of the measurement target are extracted, and the measurement result corresponding to the predetermined position of the measurement target is obtained based on the feature points. It can be combined with the video or image of and displayed.

  The user interface device of the present invention corresponds to an imaging unit that acquires a background image with a marker, a marker recognition unit that recognizes the marker from the image of the marker acquired by the imaging unit, and a marker that is recognized by the marker recognition unit. A display control unit that combines the measurement result acquired by the measurement result acquisition unit with the background image and displays it on the display unit while the image of the marker is acquired by the measurement result acquisition unit and the imaging unit. And are provided.

  With such a configuration, by acquiring the image of the marker, the measurement result corresponding to the marker is read out, and the measurement result can be displayed on the display unit while the image of the marker is captured. ..

(A) And (b) is a block diagram which illustrates the measuring system which concerns on 1st Embodiment. It is a schematic diagram which shows an operation example. (A) And (b) is a schematic diagram which illustrates a button setting area. It is a schematic diagram which illustrates another button setting area. It is a schematic diagram which illustrates another button setting area. It is a block diagram which illustrates the measuring system which concerns on 2nd Embodiment. It is a perspective view which illustrates a head mounted display. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. (A) And (b) is a block diagram which illustrates the measuring system which concerns on 3rd Embodiment. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. It is a block diagram which illustrates the measuring system which concerns on 4th Embodiment. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. It is a schematic diagram which shows an operation example. It is a flow chart which illustrates operation of helper display. It is a schematic diagram which shows the example of a display of a helper. It is a schematic diagram which shows the example of a display of a helper. It is a schematic diagram which shows the example of a display of a helper. It is a schematic diagram which shows the example of a display of a helper. It is a block diagram which illustrates the measuring system which concerns on 5th Embodiment. It is a schematic diagram which shows an operation example. It is a block diagram which illustrates the measuring system which concerns on 6th Embodiment. It is a flow chart which shows an example of processing of measurement result synthetic display. It is a schematic diagram which shows the synthetic display example of a measurement result. (A)-(e) is a schematic diagram which shows another example of a synthetic display. It is a block diagram which illustrates the measuring system which concerns on 6th Embodiment. 7 is a flowchart illustrating a marker registration and setting operation. It is a flow chart which illustrates marker recognition and synthetic display. It is a schematic diagram which shows the example of a display of the measurement result by marker recognition. It is a schematic diagram which shows the example of a display of the measurement result by marker recognition. It is a schematic diagram which shows the example of a display of the measurement result by marker recognition.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same members are designated by the same reference numerals, and the description of the members once described is omitted as appropriate.

(First embodiment)
1A and 1B are configuration diagrams illustrating a measurement system according to the first embodiment.
The measurement system 1A according to this embodiment includes a computer 100, a measuring machine M controlled by the computer 100, and a 3D sensor (three-dimensional sensor) SR. The computer 100 includes a CPU (Central Processing Unit) 10, a storage unit 20, a calculation unit 30, a measuring machine control unit 40, a display control unit 50, a 3D sensor input unit 70, and a position / distance / orientation recognition unit 75. Further, the projector PR may be connected to the computer 100. In this case, the computer 100 has a projected figure generation unit 80 and a projector output unit 85.

  The measuring machine M is a device that measures the three-dimensional coordinates of a measurement target, the length of a predetermined position, and the like. Examples of the measuring machine M include a three-dimensional position measuring machine and an image measuring machine. The CPU 10 controls each part and performs a predetermined calculation by executing a predetermined program. The storage unit 20 includes a main storage unit and a sub storage unit. The arithmetic unit 30 includes an arithmetic circuit that performs a predetermined arithmetic operation.

  The measuring machine control unit 40 controls the measuring machine M based on an instruction from the user. The display control unit 50 controls to display predetermined information on the display D. The input / output control unit 60 controls input devices such as the keyboard K and mouse MS, and controls a touch panel (not shown).

  The 3D sensor input unit 70 is an interface unit that captures information acquired by the 3D sensor SR into the computer 100. Here, the 3D sensor SR is a sensor that detects the three-dimensional coordinates of a predetermined target object in the actual three-dimensional space. The position / distance / posture recognition unit 75 performs processing for recognizing the position (three-dimensional coordinates) of the target object, the distance from the 3D sensor SR to the target object, and the target object posture based on the information acquired by the 3D sensor SR. To do.

  The projection figure generation unit 80 performs a process of generating a figure to be projected by the projector PR. The projector output unit 85 is an interface unit that sends the graphic generated by the projection graphic generation unit 80 to the projector PR.

  The measurement system 1A further includes a command setting unit 101, a moving object detection unit 102, and a command execution unit 103 as programs executed by the CPU 10. The command setting unit 101 performs a process of setting a button setting area in the real three-dimensional space and a correspondence between the button setting area and a command. The button setting area is an area in which an arbitrary position is designated in the actual three-dimensional space. For example, it may be a predetermined rectangular area in a real three-dimensional space, a predetermined three-dimensional area, or a predetermined target area.

  The command setting unit 101 performs processing for setting three-dimensional coordinate information for representing such a button setting area. The command setting unit 101 also associates a button setting area with a command. The command is associated with the three-dimensional coordinate information of the button setting area and is stored in the storage unit 20 as table data, for example.

  The moving body detection unit 102 performs processing for detecting a moving body from the three-dimensional coordinates of the target object detected by the 3D sensor SR. Specifically, the position / distance / orientation recognition unit 75 tracks the position of the object recognized to detect a moving object (moving object). Accordingly, it is possible to detect the moving body in the real three-dimensional space that can be detected by the 3D sensor SR and track the position of the moving body.

  The command execution unit 103 executes a process of executing a command associated with the button setting area when the three-dimensional coordinates of the moving object detected by the moving object detection unit 102 are located within the button setting area. The moving body detection unit 102 can track the position of the moving object (three-dimensional coordinates of the moving body) in the actual three-dimensional space. The command execution unit 103 acquires information on the three-dimensional coordinates of the moving body from the moving body detection unit 102.

  Then, when the three-dimensional coordinates of the moving body are located within the button setting area, the command executing unit 103 executes the process of executing the command associated with the button setting area. Accordingly, it is possible to detect that the button setting area in the actual three-dimensional space is selected from the motion of the moving body and execute the command associated with the button setting area.

FIG. 2 is a schematic diagram showing an operation example.
The 3D sensor SR detects the position of the object in the actual three-dimensional space. In the example shown in FIG. 2, for example, the measuring machine M, the user 800 of the measuring machine M, the screen SL displaying the image projected from the projector PR, the whiteboard WB, and the like are arranged in the real three-dimensional space. The 3D sensor SR detects these positions.

  Further, in the example shown in FIG. 2, button setting areas B1 to B4 are provided on the whiteboard WB. The positions (three-dimensional coordinates) of the button setting areas B1 to B4 on the whiteboard WB are preset by the command setting unit 101. A command is associated with each of the button setting areas B1 to B4. For example, a "start" command is associated with the button setting area B1, an "end" command is associated with the button setting area B2, a "save" command is associated with the button setting area B3, and a "print" command is associated with the button setting area B4.

  The user 800 of the measuring machine M uses the measuring head 501 of the measuring machine M, for example, to measure an object to be measured. When the measurement head 501 is provided with the projector PR, an image based on various information such as a design drawing of the measurement target can be displayed on the screen SL from the projector PR.

  The motion of the user 800 is detected by the moving body detection unit 102 based on the information acquired by the 3D sensor SR. The position / distance / posture recognition unit 75 recognizes the movement of the hand 801 of the user 800, for example. Then, when the hand 801 of the user 800 is located in any of the button setting regions B1 to B4, the command execution unit 103 executes the command associated with the button setting region B1 to B4 at the position of the hand 801.

  For example, when the hand 801 of the user 800 touches the button setting area B1 on the whiteboard WB, the “start” command associated with the button setting area B1 is executed. Similarly, when the hand 8001 of the user 800 touches the button setting area B2, the “end” command associated with the button setting area B2 is executed. In this way, the user 800 can execute a desired command depending on the position of his / her hand 801.

FIGS. 3A and 3B are schematic views illustrating the button setting area.
FIG. 3A shows button setting areas B1 to B4 set on the whiteboard WB. The button setting areas B1 to B4 are appropriately written on the surface of the whiteboard WB. Here, four rectangles are written on the surface of the whiteboard WB.

  The command setting unit 101 sets the positions of the button setting areas B1 to B4 by registering the three-dimensional coordinates of the even part of each rectangle. For example, the button setting area B1 is set by the three-dimensional coordinates of the four even parts P1 to P4. The three-dimensional coordinates of the even parts P1 to P4 can be captured by the 3D sensor SR. Further, the measurement head 501 may be used to capture the three-dimensional coordinates of the even parts P1 to P4.

  FIG. 3B shows a three-dimensional button setting area B5. The button setting area B5 is, for example, a cubic area. The area of the solid body is determined by four points P17, P18, P19, and P20 in the real three-dimensional space. The command setting unit 101 sets the position of the button setting area B5 by registering the three-dimensional coordinates of the four points P17, P18, P19, and P20.

  The command setting unit 101 associates a command with each of the button setting areas B1 to B5. The button setting areas B1 to B5 can be set anywhere in the actual three-dimensional space. For example, the command setting unit 101 sets the button setting areas B1 to B4 at positions where there are some objects such as the whiteboard WB, but also sets the button setting areas B5 at areas where there are no objects in the actual three-dimensional space. You may.

4 and 5 are schematic views illustrating other button setting areas.
In the example shown in FIG. 4, the image displayed by the projector PR is the button setting area B6. For example, when the measurement head 501 is provided with the projector PR, the measurement head 501 is directed in a direction different from the screen SL. As a result, the image of the button setting area B6 is output from the projector PR. In the example shown in FIG. 4, the image of the button setting area B6 output from the projector PR is displayed on the floor FL.

  When the image of the button setting area B6 is displayed by the projector PR, the projection position of the button setting area B6 is detected by the 3D sensor SR, and the position / distance / posture recognition unit 75 recognizes it. The position / distance / posture recognition unit 75 can recognize the position of the button setting area B6 from the information detected by the 3D sensor SR by registering the features of the button setting area B6 in advance. Further, even if the projection position of the button setting area B6 moves, it can be tracked by the moving body detection unit 102.

  When the button setting area B6 is projected on the floor FL, the user 800 can select the button setting area B6 with his / her foot 802, for example. The position of the foot 802 of the user 800 is detected by the 3D sensor SR, and the movement of the foot 802 is tracked by the moving body detection unit 102. Accordingly, it is possible to detect that the foot 802 of the user 800 is at the position of the button setting area B6 and execute the command associated with the button setting area B6 by the command execution unit 103.

  It may be difficult for a user 800 who holds the measuring head 501 by hand to perform a measurement operation and manually issue a command. Since the button setting area B6 displayed on the floor FL can be selected with the foot 802, the user 800 can specify and execute a desired command with the foot 802 while holding the measuring head 501 by hand.

  In the example shown in FIG. 5, the image of the button setting area B7 is projected on the screen SL. The projector PR displays, for example, the design information of the measurement target on the screen SL and also displays the image of the button setting area B7. Similar to the example shown in FIG. 4, the position of the button setting area B7 is detected by the 3D sensor SR and tracked by the moving body detection unit 102.

  The user 800 can select, for example, the button setting area B7 projected on the screen SL by his / her hand 801. When it is detected that the hand 801 of the user 800 is at the position of the button setting area B7, the command execution unit 103 executes the command associated with the button setting area B7.

  In addition, the command execution unit 103 may switch the command according to the operation of the hand 801 or the foot 802 of the user 800. For example, when the hand 801 of the user 800 stays in the button setting area B1 for a certain period of time (touch operation), the first command is executed, and when the hand 801 moves along the button setting area B1 (slide). The second command is executed for (operation). That is, the command may be switched by touching or sliding the button setting area B1 with the hand 801.

  According to the measurement system 1A of the first embodiment, the button setting areas B1 to B7 can be set at desired positions in the actual three-dimensional space. Further, the user 800 can execute a desired command by selecting the button setting areas B1 to B7 with his / her hand 801 or foot 802. Therefore, the user 800 can easily execute a desired command while performing measurement work by the measuring machine M. For example, even when the computer 100 is located at a position away from the measuring machine M, the user 800 can send a desired command to the computer 100 without leaving the position of the measuring machine M.

  Also, the user 800 can set the button setting areas B1 to B7 at desired positions. As a result, the button setting areas B1 to B7 can be arranged at positions where the user 800 can work efficiently, and the efficiency of the measurement work can be improved.

(Second embodiment)
FIG. 6 is a configuration diagram illustrating a measurement system according to the second embodiment.
The measurement system 1B according to this embodiment includes a computer 100, a measuring machine M controlled by the computer 100, and a 3D sensor SR. The computer 100 includes a CPU 10, a storage unit 20, a calculation unit 30, a measuring machine control unit 40, a display control unit 50, a 3D sensor input unit 70, and a position / distance / posture recognition unit 75. Further, the computer 100 includes a 3D video input / output unit 51, a 3D camera control unit 52, a stereoscopic video generation unit 53, a head mounted display output unit 54, a control generation unit 55, a headphone microphone input / output unit 61, an audio input / output unit 62, It has an operation panel input / output unit 65 and a control input recognition unit 76. An operation panel CT, a headphone microphone HPM, a head mounted display HD, and a 3D camera CM are connected to the computer 100.

Hereinafter, a configuration different from that of the measurement system 1A according to the first embodiment will be described.
The operation panel input / output unit 65 is an interface portion that inputs and outputs information to and from the operation panel CT. The user 800 can control the measuring machine M using the operation panel CT.

  The 3D camera CM is a three-dimensional imaging unit that acquires a three-dimensional image in a real three-dimensional space. The head mounted display HD is a display that displays a three-dimensional image acquired by the 3D camera CM. Here, in the embodiment, “video” means information on an image captured by an imaging device such as a 3D camera CM, and “image” is a graphic generated from design data such as CAD or prepared in advance. It means information such as graphics.

FIG. 7 is a perspective view illustrating a head mounted display.
The head mounted display HD is mounted on the head of the user 800. The user 800 can refer to the three-dimensional image displayed on the head mounted display HD. A 3D camera CM is attached to the head mounted display HD. That is, the head mounted display HD is provided integrally with the 3D camera CM.

  The 3D camera control unit 52 is a part that controls the 3D camera CM. The 3D video input / output unit 51 is an interface unit that loads the 3D video acquired by the 3D camera CM into the computer 100. The head mounted display output unit 54 is an interface unit that transmits a video signal to the head mounted display HD. The stereoscopic image generation unit 53 is a unit that generates a 3D image displayed on the head mounted display HD. The stereoscopic video generation unit 53 includes a captured video synthesis unit 531.

  The captured image synthesis unit 531 performs a process of synthesizing the 3D image acquired by the 3D camera CM and the image corresponding to the button setting area and displaying the synthesized image on the head mounted display HD.

  The control generation unit 55 is a unit that associates a predetermined action of the user 800 with a command. The association between the predetermined action of the user 800 and the command is stored in the storage unit 20 as table data, for example.

  The headphone microphone HPM is a combination of a headphone and a microphone and is attached to the head of the user 800. The headphone microphone input / output unit 61 is an interface portion that inputs / outputs information to / from the headphone microphone HPM. The voice input / output unit 62 is a unit that generates a voice to be sent to the headphone microphone HPM and processes information of the voice taken from the headphone microphone HPM. In this embodiment, the headphone microphone HPM may be provided as needed.

  The control input recognition unit 76 is a unit that recognizes a predetermined motion of the user 800. The control input recognition unit 76 performs a process of recognizing a predetermined motion of the user 800 based on the information sent from the position / distance / orientation recognition unit 75. In such a measurement system 1B, an image of the actual three-dimensional space is displayed on the head mounted display HD, and an image of the button setting area can be combined with the image and displayed.

8 to 11 are schematic diagrams showing an operation example.
As shown in FIG. 8, the user 800 wears the head mounted display HD. Further, the 3D sensor SR detects the position of an object such as the user 800 or the measuring machine M in the real three-dimensional space. The 3D camera CM provided in the head mounted display HD acquires a 3D image in a real 3D space.

  The user 800 can refer to the 3D image acquired by the 3D camera CM by the head mounted display HD. Further, an image (button image BG) of the button setting area is displayed on the head mounted display HD in accordance with the actual three-dimensional space of the 3D video. That is, the captured image synthesis unit 531 synthesizes the button image BG with the 3D image acquired by the 3D camera. The button image BG is displayed together with the three-dimensional coordinates in the display of the head mounted display HD based on the three-dimensional coordinates in the actual three-dimensional space of the button setting area registered in advance. As a result, even if the angle of view of the 3D camera CM changes, the composite position of the button image BG on the 3D image does not change.

  The motion of the user 800 is detected by the moving body detection unit 102 based on the information acquired by the 3D sensor SR. The position / distance / posture recognition unit 75 recognizes the movement of the hand 801 or the foot 802 of the user 800, for example. Then, when the hand 801 or the foot 802 of the user 800 is located in the button setting area corresponding to the button image BG, the command execution unit 103 executes the command associated with the button setting area at the position of the hand 801 or the foot 802. .. This allows the user 800 to execute a desired command depending on the position of his / her hand 801 or foot 802.

  The button image BG can be displayed at an arbitrary position on the 3D video displayed on the head mounted display HD. For example, the button image BG is displayed at a position (in the air) where an object in the actual three-dimensional space where the D image is displayed does not exist, or the button image BG is displayed on the 3D image of the floor FL as if it were placed on the floor FL. The button image BG can be displayed.

  FIG. 9 shows an example in which the button image BG is combined with the 3D image of the floor FL. The user 800 holds the measuring head 501 of the measuring machine M with his / her hand 801 and measures the workpiece W, which is an object to be measured. The user 800 operates the measurement head 501 to perform measurement while referring to the 3D image of the measuring machine M or the work W displayed on the head mounted display HD.

  Further, on the head mounted display HD, the button image BG is displayed in a state of being combined with the 3D image of the floor FL. Although no button is arranged on the actual floor FL, the button image BG is displayed on the head mounted display HD. Accordingly, the user 800 can refer to the button image BG as if the button is arranged on the floor FL. The user 800 selects the button image BG on the floor FL with the foot 802. As a result, the command in the button setting area corresponding to the selected button image BG is executed.

  FIG. 10 shows an example in which a computer image CAL is combined with a 3D video. In this example, a computer image CAL is synthesized in the vicinity of the 3D image of the work W displayed on the head mounted display HD. The user 800 operates the image CAL of the computer with, for example, the hand 801. The motion of the hand 801 is tracked by the moving body detection unit 102 to determine which key area of the image CAL of the computer the hand 801 is located. As a result, the command execution unit 103 executes the command corresponding to the key of the image CAL of the computer. The user 800 can refer to the image CAL of the virtual computer displayed on the head mounted display HD and perform the calculation as if he / she is operating an actual computer.

  The command execution unit 103 may select and execute any one of a plurality of commands according to the shape of the hand 801 of the user 800 detected by the moving body detection unit 102. FIG. 11 shows an example of switching commands according to the shape of the hand 801.

  As an example, when a certain period of time elapses with the hand 801 opened in a specific direction (H1), a command for displaying a home button (an image in which a plurality of button images BG is collected) is executed. Further, when the home button is displayed at a position where the hand 801 cannot reach in the display of the head mounted display HD, when the user holds the hand 801 (H2), a command for displaying a distant home button nearby is executed. .. When the home button is picked with the hand 801 and moved (H3), a command for moving the display position of the home button is executed.

  Further, when the operation of pointing the index finger of the hand 801 in a specific direction is performed (H4), a command for hiding the home button is executed. When the hand 801 is pressed against the wall (H5), a command for moving the home button to be stuck on the wall is executed. When the palm of the hand 801 is turned up (H6), a command for moving the home button to the palm of the hand 801 is executed. When the hand 801 is pressed against the floor (H7), a command for moving the home button to be stuck on the floor is executed. It should be noted that the correspondence between the shape and motion of the hand 801 and the command is an example, and the present invention is not limited to this.

  As described above, if the commands are switched according to the shape of the hand 801, the desired command can be executed only by the gesture using the hand 801 without using the input device such as the keyboard K or the mouse MS. Become.

  In the above example, the user 800 executes the command by selecting the button image BG with the hand 801 or the foot 802, but the user head 800 may select the button image BG and execute the command.

  Further, in the measurement system 1B according to this embodiment, the computer 100, the 3D sensor SR, and the 3D camera CM may constitute a user interface device. According to this user interface device, the display / non-display of the button image BG can be switched according to the shape of the hand 801 of the user 800, and the command corresponding to the button image BG can be executed.

(Third Embodiment)
12A and 12B are configuration diagrams illustrating a measurement system according to the third embodiment.
The measurement system 1C according to this embodiment includes a computer 100, a measuring machine M controlled by the computer 100, a 3D sensor SR, and a head mounted display HD. The computer 100 includes a CPU 10, a storage unit 20, a calculation unit 30, a measuring machine control unit 40, a display control unit 50, a 3D sensor input unit 70, and a position / distance / posture recognition unit 75. Further, the computer 100 includes a stereoscopic image generation unit 53, a head mounted display output unit 54, a headphone microphone input / output unit 61, a voice input / output unit 62, and an operation panel input / output unit 65. An operation panel CT and a headphone microphone HPM are connected to the computer 100.

  In such a measuring system 1C, the image of the measuring machine M is displayed in the virtual three-dimensional space of the head mounted display HD. As the image of the measuring machine M, for example, a CAD image is used. The CAD image is stored in the storage unit 20. In addition, the motion of the user can be detected by the motion detection unit, and the image of the measuring machine M in the virtual three-dimensional space can be moved according to the motion of the user 800.

  The measurement system 1C according to the present embodiment is particularly suitable for application to the measuring machine M having many automatic measurement functions. For example, a CNC (Computerized Numerically Controlled) measuring device can perform automatic measurement according to a preset program (for example, a part program), and thus can perform highly accurate and efficient measurement. However, in order to handle a CNC measuring machine, specialized techniques and knowledge such as operating methods and creating / modifying programs are required. In the measurement system 1C according to this embodiment, it is possible to improve the user interface of the measuring machine M having such a rich function.

13 to 17 are schematic diagrams showing an operation example.
As shown in FIG. 13, the user 800 wears the head mounted display HD and the headphone microphone HPM. The 3D sensor SR detects the positions of objects such as the user 800 and the measuring machine M in the actual three-dimensional space.

  The image of the measuring machine M (measuring machine image MG) is displayed in the virtual three-dimensional space on the head mounted display HD. The user 800 can refer to the measuring machine image MG in the virtual three-dimensional space displayed on the head mounted display HD to move the measuring machine image MG in the virtual three-dimensional space.

  That is, the position of, for example, the hand 801 of the user 800 is detected by the 3D sensor SR, and the movement of the hand 801 is tracked by the moving body detection unit 102. The position and movement of the hand 801 are reflected in the movement of the measuring machine image MG displayed in the virtual three-dimensional space. For example, when the user 800 extends the hand 801 to the position of the image of the measurement head 501 (measurement head image 501G) in the virtual three-dimensional space, the measurement head image 501G in the virtual three-dimensional space can be moved.

  The stereoscopic image generation unit 53 generates an image so that the display position of the measuring machine image MG in the virtual three-dimensional space is moved according to the motion of the hand 801 of the user 800 detected by the moving body detection unit 102. As a result, the user 800 can freely operate the measuring machine image MG in the virtual three-dimensional space displayed on the head mounted display HD.

  The measurement system 1C according to the present embodiment may include the measurement operation storage unit 104 as a program executed by the CPU 10. The measurement operation storage unit 104 performs a process of recording the measurement operation along the display position of the measuring machine image MG moved by the user 800 in the virtual three-dimensional space displayed on the head mounted display HD.

  Then, the measuring machine control unit 40 measures the work W using the actual measuring machine M according to the measuring operation stored in the measuring operation storage unit 104. Accordingly, the user 800 can record the actual measurement procedure by moving, for example, the measurement head image 501G of the measuring machine image MG in the virtual three-dimensional space.

  At this time, the actual position of the work W is detected by the 3D sensor SR, and the work image WG is combined according to the position of the measuring machine image MG in the virtual three-dimensional space. When moving the measurement head image 501G, the user 800 can record the measurement procedure while confirming the interference between the work image WG and the measurement head image 501G. For example, when the orientation of the measurement head image 501G causes interference with the work image WG, a warning may be displayed on the head mounted display HD. According to such recording of the measuring procedure, the movement of the measuring head 501 can be programmed without any specialized knowledge.

  FIG. 14 shows an example of displaying an actual image IMG in the virtual three-dimensional space. The measurement system 1C may include a camera CM1 that acquires an image IMG of the measuring machine M. The image IMG of the measuring machine M acquired by the camera CM1 is displayed at a predetermined position in the virtual three-dimensional space of the head mounted display HD.

  The user 800 can refer to the image IMG of the actual measuring machine M as well as the measuring machine image MG displayed on the head mounted display HD. As a result, it becomes possible to work while confirming the actual operation of the measuring machine M in the virtual three-dimensional space.

  FIG. 15 shows an example of the correction operation of the part program. On the head mounted display HD, the operation of the measuring head 501 according to the existing part program is displayed in the virtual three-dimensional space. The user 800 reads an existing part program. For example, an existing part program may be read by voice recognition by inputting voice into the headphone microphone HPM.

  The measurement head image 501G in the virtual three-dimensional space operates in the same manner as when the read part program is executed. The user 800 can refer to the operation of the measurement head image 501G displayed in the virtual three-dimensional space, and can control stop and restart of the operation by voice recognition. The measurement head image 501G can also be moved using the hand 801.

  Numerical values in the moving order of the measurement head image 501G and button images BG may be displayed in the virtual three-dimensional space. When modifying the part program, the operation is temporarily stopped and the measuring head image 501G is moved by the hand 801.

  Here, as an example, the operation when adding a step of the part program will be described. In this example, as shown in FIG. 15, for the part program in which the measuring head 501 operates in the order of (1) to (4), the operation of adding (2 ′) between (2) and (3) is performed. explain.

  First, the user 800 operates the part program step by step by, for example, voice recognition. When the user 800 inputs, for example, a voice "Go to the first step", the part program advances by one step, and the measurement head image 501G in the virtual three-dimensional space is displayed so as to move to the position (1).

  Next, the user 800 inputs, for example, a voice “Go to the next step”. As a result, the part program proceeds to the next step, and the measurement head image 501G in the virtual three-dimensional space is displayed so as to move to the position (2).

  Here, the user 800 inputs, for example, a voice “Add the step” to add a step. As a result, the step addition mode of the part program is executed. Next, the user 800 moves the measurement head image 501G in the virtual three-dimensional space by the hand 801 and arranges it at the position (2 ′). In this state, for example, when the voice "Continue" is input, the position (2 ') is added, and the measurement head image 501G advances to the next position (3).

  Although the operation and modification of the part program are instructed by voice recognition here, the same operation and modification may be instructed by designating the button image BG. By such an operation, the user 800 can intuitively modify the part program without any specialized knowledge.

  FIG. 16 shows an example of displaying the information regarding the measurement in the virtual three-dimensional space. The user 800 refers to the image in the virtual three-dimensional space by the mounted head mounted display HD. The work image WG and the measurement head image 501G are displayed in the virtual three-dimensional space. The work image WG and the measurement head image 501G are CAD images and computer graphics. In the example shown in FIG. 16, an example in which the measurement result is combined and displayed on the work image WG is shown. Further, when the measurement result does not fall within the predetermined permissible range, a display to that effect (a display for coloring the work image WG, etc.) may be added to the work image WG.

  The user 800 inputs a voice such as “Show result” from the headphone microphone HPM. As a result, a display in which the measurement result is combined with the work image WG in the virtual three-dimensional space is performed. The user 800 can confirm the measurement result without referring to the display D of the computer 100 by referring to the content displayed in the virtual three-dimensional space.

FIG. 17 shows an example of performing measurement and teaching via the network N.
In the example shown in FIG. 17, the measurement system 1C further includes a remote control unit 90 connected to the measuring machine control unit 40 via the network N. The remote control unit 90 performs a process of sending a command instructed by the user 800 located at a position away from the measuring machine M to the measuring machine control unit 40 via the network N.

  The user 800 at a remote location can confirm the actual operation of the measuring machine M while referring to the measuring machine image MG in the virtual three-dimensional space displayed on the mounted head mounted display HD. The measuring machine image MG and the measuring head image 501G displayed in the virtual three-dimensional space can be moved by the hand 801 of the user 800.

  As described above, the operation of the measuring head 501 by the part program can be confirmed by the operation of the measuring head image 501G displayed in the virtual three-dimensional space. Further, the part program can be modified in the virtual three-dimensional space.

  By transmitting an instruction from the user 800 at a remote location to the measuring machine control section 40 via the network N by the remote control section 90, the user 800 can obtain an image of the virtual three-dimensional space at any position of the measuring machine M. Can handle the measuring machine M.

  That is, the user 800 can handle the measuring machine M from a remote place as if he / she were near the measuring machine M. In this measurement system 1C, for example, when a user 800 who is accustomed to operating the measuring machine M is not near the measuring machine M, a knowledgeable staff member in a remote place can operate the measuring machine M via the network N. it can. Further, even when the measuring machine M is in a harsh environment and the user 800 cannot approach it, it is possible to operate the measuring machine M as if the user 800 were there.

(Fourth Embodiment)
FIG. 18 is a configuration diagram illustrating a measurement system according to the fourth embodiment.
The measurement system 1D according to this embodiment includes a computer 100, a measuring machine M controlled by the computer 100, and a 3D sensor SR. The computer 100 includes a CPU 10, a storage unit 20, a calculation unit 30, a measuring machine control unit 40, a display control unit 50, a 3D sensor input unit 70, and a position / distance / posture recognition unit 75. Further, the computer 100 includes the 3D video input / output unit 51, the 3D camera control unit 52, the stereoscopic video generation unit 53, the captured video synthesis unit 531, the head mounted display output unit 54, the headphone microphone input / output unit 61, and the audio input / output unit 62. The sound reproduction unit 621, the operation panel input / output unit 65, and the helper generation unit 95. An operation panel CT, a headphone microphone HPM, a head mounted display HD, and a 3D camera CM are connected to the computer 100.

  In such a measuring system 1D, an image of the actual measuring machine M and a measuring machine image MG such as a CAD image or a computer graphic can be combined and displayed in the virtual three-dimensional space of the head mounted display HD. .. The measuring machine image MG is stored in advance in the image storage unit 201 of the storage unit 20.

  Further, the helper generation unit 95 generates an image showing a predetermined measurement procedure of the measuring machine M. Accordingly, the captured image synthesis unit 531 of the stereoscopic image generation unit 53 can perform a process of combining the image captured by the 3D camera CM with the image representing the measurement procedure generated by the helper generation unit 95.

  Further, the helper generation unit 95 may generate an image corresponding to a predetermined abnormality of the measuring machine M. Accordingly, the captured image synthesis unit 531 of the stereoscopic image generation unit 53 can perform a process of combining the image captured by the 3D camera CM with the image corresponding to the predetermined abnormality generated by the helper generation unit 95.

  In addition, the helper generation unit 95 may generate an image corresponding to the predetermined guidance of the measuring machine M. Accordingly, the captured image synthesis unit 531 of the stereoscopic image generation unit 53 can perform a process of synthesizing the image captured by the 3D camera CM with the image corresponding to the predetermined guidance generated by the helper generation unit 95.

  Here, when the operation of the measuring machine M is unknown, the user 800 simulates the operation image of the actual measuring machine M in mind while checking the instruction manual of the measuring machine M. However, it is difficult to memorize all the operations, and it is troublesome work such as checking the instruction manual again.

  In addition, for example, when the measurement probe is replaced, if it is not performed according to the procedure, there is a risk of causing a defect such as damage. Therefore, replacement of the measurement probe requires careful work. The user 800 who is not accustomed to the work will perform the work with reference to the instruction manual each time.

  Further, for example, in order to find the origin of the non-orthogonal three-dimensional coordinate measuring machine, it is necessary to perform an operation of turning on / off each limit switch of the 7-axis arm of the measuring machine M to obtain the origin. The user 800 who is not accustomed to this work is often confused about the operation because the axis for which the origin is not acquired and the operation for acquiring the origin are difficult to understand intuitively.

  Further, for example, when connecting the cable of the measuring machine M to the dedicated controller, it is difficult to know which cable should be inserted into which connection port of the controller, and it is necessary to check the instruction manual each time.

  Further, for example, in the case of a non-orthogonal coordinate measuring machine, measurement cannot be performed when the limit switch is ON. It is difficult to tell whether this limit switch is in the ON state or in the measurable state, and the same location may be repeatedly measured.

  In addition, for example, when an abnormality occurs in the measuring machine M, a message is displayed on the display D of the computer 100 or a sound is issued. However, when the user 800 concentrates on the measurement, the display D of the computer 100 is not often seen, or the display D is difficult to see, which may be difficult to recognize.

  In addition, for example, an image such as a video may be reproduced for the purpose of educating a beginner on the measuring method and the like. However, there are cases in which the time required for acquisition increases, such as when you do not know what to do when actually measuring, or when you make a mistake in the procedure and start over.

  In the measurement system 1D according to the present embodiment, by combining and displaying the image of the actual measuring machine M and various images in the virtual three-dimensional space of the head mounted display HD worn by the user 800, it is efficient. You can perform various operations.

19 to 23 are schematic diagrams showing an operation example.
FIG. 19 shows an example of providing technical assistance to the user 800.
As shown in FIG. 19, the user 800 wears the head mounted display HD. Further, the 3D sensor SR detects the position of an object such as the user 800 or the measuring machine M in the real three-dimensional space. The 3D camera CM provided in the head mounted display HD acquires a 3D image in a real 3D space.

  The user 800 can refer to the 3D image acquired by the 3D camera CM by the head mounted display HD. In the example shown in FIG. 19, the image IMG of the measuring machine M is displayed as a 3D image. Further, various images G relating to the measuring machine M are displayed on the head mounted display HD in accordance with the actual three-dimensional space of the 3D video.

  The display position of the image G is determined based on the position of the image IMG of the measuring machine M. As a result, even if the angle of view of the 3D camera CM changes, the display position of the image G on the measuring machine M does not change. In the example shown in FIG. 19, the image G following the procedure of origin search is displayed in a state of being combined with the image IMG of the measuring machine M. The user 800 can refer to the image G according to the procedure of origin search while referring to the image IMG of the actual measuring machine M acquired by the 3D camera CM.

  By combining and displaying the image G on the image IMG of the actual measuring machine M, the user 800 obtains the origin of which axis of the measuring machine M it is necessary to obtain. It is possible to intuitively understand the operation procedure required to do so.

  FIG. 20 shows an operation example when teaching the operation method of the measuring machine M. In FIG. 20, images G1 to G7 showing the procedure for replacing the measurement probe are shown as an example. On the head mounted display HD worn by the user 800, the images G1 to G7 are displayed in a state of being arranged in order. These images G1 to G7 are combined with the image IMG (not shown in FIG. 20) of the actual measuring machine M captured by the 3D camera CM. As the display of the images G1 to G7, step display may be performed in which the images G1 to G7 are sequentially displayed one by one. Operations such as reproduction and stop of the step display are instructed by the voice of the user 800 or the operation of the hand 801.

  By referring to the images G1 to G7 displayed on the head mounted display HD, the user 800 can understand the operation procedure while comparing the image IMG of the actual measuring machine M and the images G1 to G7.

  FIG. 21 shows an operation example in the case of teaching a cable connecting method. On the head mounted display HD worn by the user 800, the image IMG of the actual measuring machine M captured by the 3D camera CM and the image PC-IMG of the personal computer are displayed. Further, the image G of the cable and the guidance is displayed in a combined state on the video IMG and PC-IMG. The image G of the cable is synthesized and displayed according to the connection position of the video PC-IMG of the actual personal computer.

  In this way, since the image G when the cable is connected is compositely displayed on the image IMG of the actual measuring machine M and the image PC-IMG of the personal computer, the user 800 intuitively imagines the method of connecting the cable. be able to.

  FIG. 22 shows an operation example when displaying an image G relating to a predetermined abnormality. The image IMG of the actual measuring machine M captured by the 3D camera CM is displayed on the head mounted display HD worn by the user 800. Here, when some abnormality occurs in the measuring head 501, an image G indicating the abnormality is displayed by combining with the image IMG of the head mounted display HD.

  In the example shown in FIG. 22, an image G that urges a warning such as “cable connection abnormality” or “limit ON” is synthetically displayed on the image IMG of the actual measuring machine M. As a result, the user 800 can visually understand what kind of abnormality occurs at what position in accordance with the image IMG of the actual measuring machine M.

  FIG. 23 shows an example of displaying the image G of the instruction manual. The image IMG of the actual measuring machine M captured by the 3D camera CM is displayed on the head mounted display HD worn by the user 800. When the user 800 sends a predetermined command to the computer 100 in this state, the image G of the instruction manual is composited and displayed near the image IMG of the measuring machine M on the head mounted display HD.

  The user 800 can also refer to the image G of the instruction manual appearing in the display space of the head mounted display HD while referring to the image IMG of the actual measuring machine M. Also, the page of the image G of the instruction manual can be turned over by the instruction of the user 800. For example, the movement of the hand 801 of the user 800 is detected by the moving body detection unit 102, so that the movement of the hand 801 is tracked. Then, when a page-turning operation is recognized from the movement of the hand 801, a page-turning display of the image G of the instruction manual is displayed.

  Further, the user 800 can enlarge or reduce the image G of the instruction manual according to a predetermined instruction. This operation can also be performed by detecting the movement of the hand 801.

FIG. 24 is a flowchart illustrating the operation of the helper display.
The helper display operation is performed by the helper generation unit 95. The helper generation unit 95 starts the operation when the user 800 calls the helper or when some error occurs.

  First, as shown in step S101, a helper is displayed. The helper is an image corresponding to a predetermined guidance, and may be an image imitating a human shape or a character image. The helper is displayed in the virtual three-dimensional space of the head mounted display HD.

  Next, as shown in step S102, the helper response is output. For example, "Hello", "Error Occurred!", "May I help you?" The helper's response changes the image displayed on the head mounted display HD or is output as sound to the headphone microphone HPM.

  Next, as shown in step S103, the voice command is recognized. The user 800 makes a voice inquiry to the headphone microphone HPM. The command is recognized based on this voice. Next, as shown in step S104, the helper response is output. The helper outputs a response corresponding to the command instructed by the user 800. For example, an image or sound corresponding to the command of the user 800 such as “I teach you ...” or “Please ...” is output.

  As shown in step S105, it is determined whether the response from the helper is acceptable. If it is good, the process proceeds to step S106, and the helper is deleted. On the other hand, if a further response is required, the process returns to step S102 and the subsequent processing is repeated.

25 to 28 are schematic diagrams showing display examples of helpers.
FIG. 25 shows a display example of a helper for guiding the operating procedure of the measuring machine M.
First, the user 800 calls a helper. For example, the user 800 makes a voice inquiry “OK Mitutoyo.”. In response, the helper is displayed. An image IMG of the actual measuring machine M is displayed in the three-dimensional space of the head mounted display HD, and a helper image (helper image HG) is displayed in the vicinity thereof.

  Next, the helper image HG responds with, for example, “Hello”, image display, and voice. Next, the user 800 makes an inquiry by voice. For example, in order to hear the operating procedure of the measuring machine M, a voice inquiry such as “How to ...” is performed.

  Next, in response to the inquiry of the user 800, the helper image HG responds with, for example, “I teach you ..., operating it.”. Subsequently, the user 800 asks, for example, "Teach me step by step." In this way, the guidance of the operation procedure of the measuring machine M is performed by repeating the inquiry and the response between the user 800 and the helper image HG. At this time, the measuring machine image MG may be displayed so that the progress of the procedure is represented by the measuring machine image MG.

FIG. 26 shows a display example of a helper that guides the procedure for replacing the measurement probe.
First, the user 800 calls the helper by, for example, asking "OK Mitutoyo." By voice. In response to this, the helper image HG is displayed. The helper image HG responds by displaying, for example, "Hello" and displaying an image and voice.

  Next, the user 800 asks, for example, "How to change the probe." In response to the inquiry of the user 800, the helper image HG responds, for example, with "Please do it this way." The measurement head image 501G is displayed together with this response, and guidance of the measurement probe replacement procedure is provided in a moving image or the like.

FIG. 27 shows a display example of a helper for guiding a cable connection error.
First, the user 800 calls the helper by, for example, asking "OK Mitutoyo." By voice. In response to this, the helper image HG is displayed. The helper image HG responds by displaying, for example, "Hello" and displaying an image and voice.

  Next, the user 800 asks, for example, “Teach me connection error.”. In response to the question from the user 800, the helper image HG responds, for example, with "Please check connection of this cable." The measurement machine image MG and the image PC-G of the personal computer are displayed together with this response. Further, an image G of the cable for prompting the check is displayed. By referring to this image, the user 800 can visually recognize which cable connection has an error.

FIG. 28 shows a display example of the helper when an error occurs.
If any error occurs in the measurement by the measuring machine M, the helper image HG is automatically displayed. The helper image HG notifies the occurrence of an error by responding, for example, “Error occurred.”.

  Next, the user 800 asks, for example, "Teach me about error." The helper image HG responds to the inquiry from the user 800, and outputs details of the error that has occurred such as "I teach you ...".

  As described above, by displaying the helper image HG and interacting with the helper image HG, the user 800 can solve the problem as if the user 800 were receiving the guidance directly from the service person.

(Fifth Embodiment)
FIG. 29 is a configuration diagram illustrating a measurement system according to the fifth embodiment.
The measurement system 1E according to the present embodiment has a configuration in which a computer 100A that is a user-side control device and a computer 100B that is a supporter-side control device are connected to each other via a network.

  To the computer 100A, a measuring machine M, a control panel CT, a display D1, a keyboard K1, a mouse MS1, a head mounted display HD1, a headphone microphone HPM1, a 3D camera CM and a 3D sensor SR1 are connected.

  Further, the computer 100A includes a CPU 10A, a storage unit 20A, a calculation unit 30A, a measuring machine control unit 40, a display control unit 50A, a video input / output unit 51A, a camera control unit 52A, a stereoscopic video generation unit 53A, a captured video synthesis unit 531A, It has a head mounted display output unit 54A, a headphone microphone input / output unit 61A, a voice input / output unit 62A, a sound reproduction unit 621A, a 3D sensor input unit 70A, a position / distance / posture recognition unit 75A, and a communication control unit 101A.

  A display D2, a keyboard K2, a mouse MS2, a head mounted display HD2, a headphone microphone HPM2 and a 3D sensor SR2 are connected to the computer 100B.

  The computer 100B also includes a CPU 10B, a storage unit 20B, a calculation unit 30B, a display control unit 50B, a stereoscopic image generation unit 53B, a head mounted display output unit 54B, a headphone microphone input / output unit 61B, a sound input / output unit 62B, and a sound reproduction unit. 621B, 3D sensor input unit 70B, position / distance / orientation recognition unit 75B, and communication control unit 101B.

  In such a measurement system 1E, the user side in which the measuring machine M is installed and the supporter side of the measuring machine M are connected to each other via the network N. The user side can refer to the combination of the image IMG of the measuring machine M displayed on the head mounted display HD1 and the image sent from the supporter side.

  The supporter side can refer to the combination of the measuring machine image MG such as a graphic and the image sent from the user side in the virtual three-dimensional space displayed on the head mounted display HD2. That is, by referring to the displays of the head mounted displays HD1 and HD2 respectively for the user and the supporter who are located apart from each other, it is possible to perform the operation by referring to the 3D images as if they are close to each other. Will be able to.

FIG. 30 is a schematic diagram showing an operation example.
As shown in FIG. 30, the user 800 wears the head mounted display HD1 and the headphone microphone HPM1. The 3D sensor SR1 detects the positions of objects such as the user 800, the measuring machine M, and the work W in the user-side real three-dimensional space. Further, the 3D camera CM provided in the head mounted display HD1 captures a 3D image of the measuring machine M, the work W, or the like. Information of the measuring device M on the user side, information detected by the 3D sensor SR1, information acquired by the 3D camera CM, information acquired by the headphone microphone HPM1 and the like are transmitted from the communication control unit 101A to the supporter side via the network N. To be done.

  On the other hand, the supporter 900 wears the head mounted display HD2 and the headphone microphone HPM2. The 3D sensor SR2 detects the position of an object such as the supporter 900 in the actual three-dimensional space on the supporter side. Information detected by the 3D sensor SR2 on the supporter side, information acquired by the headphone microphone HPM2, and the like are transmitted from the communication control unit 101B to the user side via the network N.

  Next, a display example of the head mounted display HD1 of the user 800 will be described. A 3D image acquired by the 3D camera CM is displayed on the head mounted display HD1 of the user 800. For example, images of the measuring machine M, the work W, the display D1 of the computer 100A, and the like are displayed on the head-mounted display HD1. Further, a helper image HG1 is displayed on the head mounted display HD1 as a supporter side image.

  An image imitating the supporter 900 is used as the helper image HG1. The helper image HG1 may be a character image or the like other than the image simulating the shape of a person. The helper image HG1 changes according to the movement of the supporter 900.

  That is, the movement of the supporter 900 is detected by the 3D sensor SR2 on the supporter side, and the helper image HG1 is moved so as to follow the movement of the supporter 900 from the detection result. Accordingly, the user 800 can refer to the image as if the supporter 900 is present at the place where the measuring machine M is located.

  The supporter 900 sends advice to the user 800 with gestures and voices. The movement of the helper image HG1 referred to by the user 800 changes according to this operation. Further, the sound from the supporter 900 is output from the headphone microphone HPM1 of the user 800.

  Next, a display example of the head mount display HD2 of the supporter 900 will be described. On the head mounted display HD2 of the supporter 900, an image of the virtual three-dimensional space is displayed based on the information transmitted from the user side via the network N. In the virtual three-dimensional space, the measuring machine image MG used by the user 800 and the image of the work W (work image) WG are displayed. In addition, the image D-IMG on the display D1 of the computer 100A of the user 800 is combined and displayed in the virtual three-dimensional space.

  Furthermore, the user image YG is displayed on the head mounted display HD2 as a user side image. An image imitating the user 800 is used as the user image YG. The user image YG may be a character image or the like other than the image simulating the shape of a person. The user image YG changes according to the movement of the user 800.

  That is, the movement of the user 800 is detected by the 3D sensor SR1 on the user side, and the user image YG is moved so as to follow the movement of the user 800 from the detection result. As a result, the supporter 900 can feel as if he / she were at the measurement location of the user 800 by the measuring machine image MG, the work image WG, and the user image YG displayed in the virtual three-dimensional space.

  For example, the user 800 can refer to the image of the measuring machine M displayed on the head mounted display HD1 and the helper image HG1 combined with this image, and can perform the measurement work while receiving the advice of the helper image HG1. ..

  On the other hand, the supporter 900 refers to the measuring machine image MG in the virtual three-dimensional space displayed on the head mounted display HD2 and the user image YG to be combined with this measuring machine image MG, and instructs the user 800 to perform the operation procedure. You can send advice, such as giving instructions.

  In this way, both the user 800 and the supporter 900 can communicate with each other while referring to each other's images by using voices and gestures. You will be able to proceed with such a feeling.

(Sixth Embodiment)
FIG. 31 is a configuration diagram illustrating a measurement system according to the sixth embodiment.
As shown in FIG. 31, the measurement system 1F according to the present embodiment includes a computer 100 and a display device 300 connected to the computer 100 via wireless communication. The computer 100 is connected with a measuring machine M, a control panel CT, a camera C1, a display D, a keyboard K and a mouse MS. The computer 100 includes a CPU 10, a storage unit 20, a feature point generation unit 35, a measurement result combination setting unit 36, a measuring machine control unit 40, a display control unit 50, an input / output control unit 60, an operation panel input / output unit 65, and a communication control unit. Has 101A.

  The feature point generation unit 35 performs a process of extracting feature points from the image of the measurement target (work) acquired by the camera C1. The characteristic point is an even part or a side (outline) that is necessary to form the outer shape of the work. The measurement result combination setting unit 36 is a unit that associates the feature points generated by the feature point generation unit 35 with the measurement results of the work. For example, the distance between two adjacent feature points is associated with the measurement result of the work corresponding to the position.

  The display device 300 is provided with a camera C2 and a display panel DP such as a liquid crystal panel. The display device 300 includes a CPU 310, a storage unit 320, a feature point extraction / tracking unit 350, an input / output unit 360, a measurement result combining unit 370, and a communication control unit 301.

  The feature point extraction / tracking unit 350 performs a process of extracting feature points from the image of the work captured by the camera C2 and tracking the extracted feature points in the image. Feature points are extracted and tracked while the work is photographed by the camera C2.

  The measurement result synthesizing unit 370 performs a process of synthesizing the measurement result with the image of the work captured by the camera C2 or the image of the work based on this image. That is, the measurement result synthesizing unit 370 reads out the measurement result associated with the feature point from the computer 100 based on the feature point extracted by the feature point extracting / tracking unit 350, and outputs the measurement result to the image or image of the work. To synthesize.

  As a result, on the display panel DP of the display device 300, a display in which the measurement result corresponding to the feature point is combined with the video or image of the work is displayed. That is, when the user of the display device 300 acquires the image of the work with the camera C2, the user can refer to the display in which the measurement result is combined with the image or the image of the work.

  In the measurement system 1F according to this embodiment, the measurement result can be referred to by the plurality of display devices 300. By acquiring the image of the work W by each display device 300, the image or image of the work and the measurement result are combined and displayed on each display device 300.

FIG. 32 is a flowchart showing an example of the processing of measurement result combination display.
First, as shown in step S201, an image of a work is acquired by the camera C2. Next, as shown in step S202, it is determined whether the characteristic points have been extracted. If not already extracted, the feature points shown in step S203 are extracted. After extracting the feature points from the image of the work acquired by the camera C2, the feature points are tracked as shown in step S204. The feature point extraction / tracking is performed by the process of the feature point extraction / tracking unit 350.

  Next, as shown in step S205, the measurement results are combined. Here, in accordance with the feature points of the workpiece that are being extracted and tracked, processing of synthesizing the image of the measurement result corresponding to the feature points with the video of the workpiece is performed. The measurement result combining process is performed by the measurement result combining unit 370.

FIG. 33 is a schematic diagram showing a combined display example of measurement results.
In the example shown in FIG. 33, a mobile terminal is used as the display device 300. The mobile terminal is provided with a camera C2 and a display panel DP. When the camera W captures the work W, the image IMG of the work W is displayed on the display panel DP.

  The feature point extraction / tracking unit 350 extracts the feature points of the work W from the image IMG of the work W captured by the camera C2, and also tracks the feature points during shooting. In the example shown in FIG. 33, the characteristic points CP1 to CP4 of the work W are extracted and tracked.

  The measurement result combining unit 370 combines the measurement results corresponding to the extracted and tracked characteristic points CP1 to CP4 into an image. For example, the measurement result of the distance between the characteristic points CP1 and CP2 is displayed together with the dimension line between the characteristic points CP1 and CP2 of the image IMG of the work W. Further, the measurement result of the distance between the characteristic points CP2 and CP3 is displayed together with the dimension line between the characteristic points CP2 and CP3 of the image IMG of the work W. Further, the measurement result of the diameter of the hole obtained from the characteristic point CP4 is displayed along with the dimension line at the position of the hole indicated by the characteristic point CP4 of the image IMG of the work W.

  Such a composite display is continued while the work W is photographed by the display device 300. Even if the angle of view for photographing the work W is changed, since the characteristic points CP1 to CP4 of the work W are tracked, the display position of the measurement result is changed so as to follow the change of the image IMG of the work W. Is displayed. Therefore, the user can refer to the image IMG of the work W viewed from a predetermined angle and the display of the measurement result corresponding thereto by photographing the work W at a desired angle.

34 (a) to 34 (e) are schematic diagrams showing other examples of combined display.
FIG. 34A shows an image IMG of the work W captured by the camera C2. Further, FIG. 34B shows feature points extracted and tracked from the image IMG. Here, the feature points are indicated by circles. FIG. 34C shows an example in which the measurement result is combined and displayed on the image IMG. Here, the measurement result set in advance is combined and displayed at the position of the feature point.

  On the other hand, FIG. 34D shows an example of the work image WG fitted to the CAD model based on the image IMG. That is, by extracting the characteristic points from the image IMG shown in FIG. 34B, the shooting location and the angle of the work W can be grasped. Based on this, the CAD model of the work W is fitted and the work image WG is displayed. Then, as shown in FIG. 34E, the measurement results are combined and displayed on the work image WG of the CAD model.

  The user can select whether to combine the measurement result with the image IMG or the measurement result with the work image WG of the CAD model. If the measurement result is combined with the image IMG, the actually measured work W and the measurement result of the work W can be referred to.

  Moreover, if the measurement result is combined with the work image WG of the CAD model, the design information based on the CAD model and the display method of the CAD model (surface model, wire frame model, section display, etc.) can be selected and matched to the display. The measurement result can be synthesized.

  In the work image WG of the CAD model, the display method such as the display color can be changed based on the difference between the design value and the measured value. For example, it is possible to display in red when the size exceeds the allowable range, or to change the display mode between the part that is within the allowable range and the part that is not within the allowable range.

  Further, in the measurement system 1F according to the present embodiment, the user interface device may be configured by the computer 100 and the display device 300. According to this user interface device, the feature point of the work W is extracted by capturing the image of the work W by the camera C2, and the information corresponding to the predetermined position of the work W is obtained based on the feature point. It can be combined with the image IMG and displayed.

(Seventh embodiment)
FIG. 35 is a configuration diagram illustrating a measurement system according to the sixth embodiment.
As shown in FIG. 35, a measurement system 1G according to this embodiment includes a computer 100 and a display device 300 connected to the computer 100 via wireless communication. The computer 100 is connected with a measuring machine M, a control panel CT, a camera C1, a display D, a keyboard K and a mouse MS. The computer 100 includes a CPU 10, a storage unit 20, a marker recognition registration unit 38, a marker measurement result correspondence setting unit 39, a measuring machine control unit 40, a display control unit 50, an input / output control unit 60, an operation panel input / output unit 65, and communication control. It has a part 101A.

  The marker recognition registration unit 38 recognizes marker-specific information (shape recognition result, identification information, etc.) from the image of the marker acquired by the camera C1, and performs registration. Here, as the marker, a marker such as a figure or a photograph that can be distinguished from others is used. The marker measurement result correspondence setting unit 39 is a unit that registers the marker registered by the marker recognition registration unit 38 and a predetermined measurement result in association with each other. The registration information is stored in the storage unit 20.

  The display device 300 is provided with a camera C2 and a display panel DP such as a liquid crystal panel. The display device 300 includes a CPU 310, a storage unit 320, an input / output unit 360, a marker recognition unit 380, a marker measurement result combining unit 390, and a communication control unit 301.

  The marker recognition unit 380 performs a process of recognizing information unique to the marker from the image of the marker captured by the camera C2. The marker measurement result synthesizing unit 390 performs a process of synthesizing the measurement result associated with the marker-specific information recognized by the marker recognizing unit 380 with the background image.

  That is, the marker measurement result synthesizing unit 390 reads the measurement result associated with the information based on the marker-specific information recognized by the marker recognizing unit 380 from the computer 100, and synthesizes the measurement result with the background image.

  In such a measurement system 1G, the display panel DP of the display device 300 displays the background image and the measurement result combined at the position of the marker. That is, when the user of the display device 300 acquires the image of the marker with the camera C2, the user can refer to the display in which the measurement result associated with the marker is combined with the background image.

  In the measurement system 1G according to this embodiment, the measurement result can be referred to by the plurality of display devices 300. By acquiring the image of the marker with each display device 300, the background and the measurement result are combined and displayed on the display panel DP of each display device 300.

FIG. 36 is a flowchart illustrating the marker registration and setting operation. This operation is performed by the computer 100.
First, as shown in step S301, the image of the marker is acquired by the camera C1. Next, as shown in step S302, the marker image is recognized. For example, a process of recognizing a photographed marker having a specific shape, a specific figure, a photograph, and the like, adding identification information unique to the marker to the marker, and performing registration processing is performed. This processing is performed by the marker recognition registration unit 38.

  Next, as shown in step S303, a process of setting the association between the marker and the measurement result is performed. That is, the process of associating the marker identification information recognized in step S302 with the predetermined measurement result is performed. This processing is performed by the marker measurement result correspondence setting unit 39.

  Next, as shown in step S304, the identification information unique to the marker and the measurement result associated with this identification information are registered. The association between the identification information unique to the marker and the measurement result is stored in the storage unit 20 as the marker setting information. This processing is performed by the marker measurement result correspondence setting unit 39.

FIG. 37 is a flowchart illustrating the marker recognition and the composite display. This operation is performed by the display device 300.
First, as shown in step S401, the marker setting information is received from the storage unit 20 of the computer 100 via wireless communication. Next, as shown in step S402, the image of the marker captured by the camera C2 is acquired. Next, as shown in step S403, the identification information unique to the marker is recognized from the acquired image of the marker. This processing is performed by the marker recognition unit 380.

  Next, as shown in step S404, a process of combining the measurement results associated with the markers is performed. That is, from the marker-specific identification information recognized in step S403, the measurement result associated with this identification information is read from the marker setting information. Then, a process of combining the measurement result with the background image taken by the camera C2 is performed. This processing is performed by the marker measurement result combining unit 390.

  Next, as shown in step S405, it is determined whether recognition and combination display of all the marks have been completed. If not completed, the process returns to step S402 to repeat the subsequent processes. As a result, the measurement results are combined and displayed corresponding to all the markers photographed by the camera C2.

38 to 40 are schematic diagrams showing display examples of measurement results by marker recognition.
FIG. 38 shows an example in which composite display is performed on the mobile terminal. In this example, a mobile terminal is used as the display device 300. Further, the paper PP is provided with a plurality of markers MK1 to MK5.

  The markers MK1 to MK5 may be specific figures or specific photographs. The marker-specific identification information and the measurement result are associated with each of the markers MK1 to MK5. For example, the markers MK1, MK2, and MK3 are associated with predetermined measurement results of the first measuring machine M1. Further, the markers MK4 and MK5 are associated with a predetermined measurement result of the second measuring machine M2. These associations are stored in the storage unit 20.

  The user photographs the sheet PP with the camera C2 of the mobile terminal which is the display device 300. When the images of the markers MK1 to MK5 are captured by this photographing, the identification information of each of the markers MK1 to MK5 is recognized. Then, the measurement result associated with each of the markers MK1 to MK5 based on this identification information is read from the storage unit 20 of the computer 100.

  Then, the images of the measurement results associated with the respective markers MK1 to MK5 are combined in the display areas of the photographed markers MK1 to MK5 and displayed on the display panel DP. For example, each of the markers MK1 to MK3 is combined with the image of the measurement result obtained by the first measuring machine M1, and each of the markers MK4 and MK5 is combined with the image of the measurement result obtained by the second measuring machine M2.

  The image of the measurement result is displayed in various formats such as a tabular format and a graph format. When synthesizing the images, the size and the angle may be adjusted so as to be displayed in accordance with the regions indicated by the markers MK1 to MK5. Further, the image of the measurement result may be combined and displayed as a stereoscopic image on the image of the flat sheet PP.

  In this way, the user can refer to the measurement results laid out at the positions of the markers MK1 to MK5 simply by photographing the paper PP with the markers MK1 to MK5 on the mobile terminal.

39 and 40 show another display example.
First, FIG. 39 shows a display example before composite display. Similar to the previous example, the paper PP is provided with a plurality of markers MK1 to MK5. Further, in this example, the glasses-type display device 300 is used.

  In this display device 300, the display panel DP is provided on the lens portion of the glasses. Further, a camera C2 is provided on the temple of the glasses. By wearing the glasses-type display device 300, the user can shoot an image in front of the glasses with the camera C2 and can refer to the information displayed on the display panel DP of the lens portion.

  FIG. 40 shows a display example after the composite display. That is, when the markers CK1 to MK5 are photographed by the camera C2 of the glasses-type display device 300, the respective identification information of the markers MK1 to ML5 is recognized. Then, the measurement result associated with each of the markers MK1 to MK5 based on the identification information is read from the storage unit 20. Then, the images of the measurement results associated with the respective markers MK1 to MK5 are combined in the display areas of the photographed markers MK1 to MK5 and displayed on the display panel DP.

  The user wearing the glasses-type display device 300 can refer to the display in which the measurement result is combined with the background image displayed on the display panel DP of the lens portion of the glasses. In this example, the measurement results of the first measuring machine M1 are combined in the display areas of the markers MK1, MK2, and MK3, and the measurement results of the second measuring machine M2 are combined in the display areas of the markers MK4 and MK5.

  By changing the setting of the correspondence between the markers MK1 to MK5 and the measurement results, different measurement results can be displayed even for the same markers MK1 to MK5. That is, different measurement results can be displayed by creating one layout form of the markers MK1 to MK5.

  Moreover, a plurality of measurement results may be associated with one marker. In this case, the user may arbitrarily switch which of the plurality of measurement results corresponding to one marker is to be combined and displayed. Thus, by shooting the image of one marker, different measurement results can be displayed by switching to the position of the marker.

  The image of the measurement result may be a moving image. As a result, the moving image is combined with the image on the sheet PP, and the simple sheet PP can be treated like a virtual moving image display device.

  Further, in the measurement system 1G according to the present embodiment, the user interface device may be configured by the computer 100 and the display device 300. According to this user interface device, by shooting the images of the markers MK1 to MK5 with the camera C2, the information corresponding to the markers MK1 to MK5 is read out, and while shooting the images of the markers MK1 to MK5, The read information can be displayed on the display panel DP.

  As described above, according to the measurement systems 1A, 1B, 1C, 1D, 1E, 1F and 1G of the present embodiment, it is possible to improve the operability of the measuring machine M.

  Although the present embodiment has been described above, the present invention is not limited to these examples. For example, in the system using the head mounted display HD to which the 3D camera CM is attached, the method of synthesizing and displaying the image of the physical space captured by the 3D camera CM and the computer graphics as described above is not limited. For example, computer graphics may be displayed on the head mounted display HD in accordance with the real space coordinates. In this case, instead of the 3D camera CM or in addition to the 3D camera CM, a 3D sensor is mounted on the head mounted display HD, the coordinate system of the real space is grasped in real time by the 3D sensor, and the virtual space is calculated by computer processing. It may be constructed and the computer graphics may be updated and displayed in real time as if it were actually present in the space. Further, those skilled in the art appropriately added, deleted, and changed the design of each of the above-described embodiments, and those obtained by appropriately combining the features of the respective embodiments have the gist of the present invention. As long as it is included in the scope of the present invention.

1A, 1B, 1C, 1D, 1E, 1F, 1G ... Measuring system 10, 10A, 10B, 310 ... CPU
20, 20A, 20B, 320 ... Storage unit 30, 30A, 30B ... Calculation unit 35 ... Feature point generation unit 36 ... Measurement result composition setting unit 38 ... Marker recognition registration unit 39 ... Marker measurement result correspondence setting unit 40 ... Measuring machine control Units 50, 50A, 50B ... Display control units 51, 51A ... Video input / output unit 52 ... 3D camera control unit 52A ... Camera control units 53, 53A, 53B ... Stereoscopic image generation units 54, 54A, 54B ... Head mounted display output unit 55 ... Control generation unit 60 ... Input / output control units 61, 61A, 61B ... Headphone microphone input / output units 62, 62A, 62B ... Voice input / output unit 65 ... Operation panel input / output units 70, 70A, 70B ... 3D sensor input unit 75 , 75A, 75B ... Attitude recognition unit 76 ... Control input recognition unit 80 ... Projected figure generation unit 85 ... Projector output unit 90 ... Remote control unit 5 ... Helper generation unit 100, 100a, 100B ... Computer 101 ... Command setting unit 101A, 101B, 301 ... Communication control unit 102 ... Moving object detection unit 103 ... Command execution unit 104 ... Measurement operation storage unit 201 ... Image storage unit 300 ... Display Device 350 ... Tracking unit 360 ... Input / output unit 370 ... Measurement result combining unit 380 ... Marker recognition unit 390 ... Marker measurement result combining unit 501 ... Measuring head 501G ... Measuring head image 531,531A ... Shooting video combining unit 621,621A, 621B Sound reproduction unit 800 User 801, Hand 802 Foot 900 Supporters B1 to B7 Button setting area BG Button images C1, C2, CM1 Camera CM 3D camera CM1 Camera CP1 CP4 Feature points D, D1, D2 ... Display D-IMG, IMG ... Image DP ... Display Channel FL ... Floor G, G1-G7 ... Image HD, HD1, HD2 ... Head mounted display HG, HG1 ... Helper image HPM, HPM1, HPM2 ... Headphone microphone K, K1, K2 ... Keyboard M, M1, M2 ... Measuring machine MG ... Measuring machine images MK1 to MK5 ... Markers MS, MS1, MS2 ... Mouse N ... Networks P1 to P4 ... Even part PC-G ... Image PC-IMG ... Image PP ... Paper PR ... Projector SR, SR1, SR2 ... 3D sensor W ... Work WB ... Whiteboard WG ... Work image YG ... User image

Claims (28)

A measuring machine for measuring the measuring object,
A measuring machine control unit for controlling the measuring machine based on an instruction from a user,
A three-dimensional sensor unit for detecting the three-dimensional coordinates of a predetermined object in an actual three-dimensional space,
A button setting area in the actual three-dimensional space, and a command setting section that sets the correspondence between the button setting area and a command
A moving body detection unit that detects a moving body from the three-dimensional coordinates detected by the three-dimensional sensor unit;
A command execution unit that executes the command associated with the button setting area when the three-dimensional coordinates of the moving object detected by the moving object detection unit are located in the button setting area;
A measuring system characterized by having. The measuring machine has a measuring head for acquiring the three-dimensional coordinates of the measuring object, and an image output unit provided on the measuring head for projecting an image,
The measurement system according to claim 1, wherein the video output unit outputs a video of the button setting area set by the command setting unit. The moving body detection unit detects the operation of the measuring head,
The measurement system according to claim 2, wherein the command execution unit executes the command according to the three-dimensional coordinates of the measurement head detected by the moving body detection unit. The moving body detection unit detects a motion of at least one of a hand and a foot of the user,
4. The command execution unit executes the command according to the three-dimensional coordinate of at least one of the hand and foot of the user detected by the moving body detection unit. The measurement system according to the item.   The command execution unit executes the first command when the three-dimensional coordinates of the moving body detected by the moving body detection unit remain in the button setting area for a certain period of time, and the three-dimensional coordinates of the moving body are The measurement system according to any one of claims 1 to 4, wherein the second command is executed when moving along the button setting area. A three-dimensional imaging unit for acquiring a three-dimensional image in the actual three-dimensional space,
A display unit that displays the three-dimensional image acquired by the three-dimensional imaging unit;
A display control unit configured to combine the 3D image in the actual 3D space and an image corresponding to the button setting area and display the combined image on the display unit;
The measurement system according to any one of claims 1 to 5, further comprising:   7. The command execution unit selects and executes any one of the plurality of commands according to the shape of the user's hand detected by the moving body detection unit, and executes the selected command. The measurement system according to the item. A measuring machine for measuring the measuring object,
A measuring machine control unit for controlling the measuring machine based on an instruction from a user,
A three-dimensional sensor unit for detecting the three-dimensional coordinates of a predetermined object in an actual three-dimensional space,
A moving body detection unit that detects the motion of the user from the three-dimensional coordinates detected by the three-dimensional sensor unit;
A display unit for displaying an image of the measuring machine in a virtual three-dimensional space;
A display control unit that moves the display position of the image of the measuring machine in the virtual three-dimensional space displayed on the display unit according to the user's motion detected by the moving body detection unit;
A measuring system characterized by having.   The apparatus further comprises a measurement operation storage unit that records a measurement operation along a display position of an image of the measuring machine moved by the user in the virtual three-dimensional space displayed on the display unit. Item 8. The measurement system according to Item 8.   The measuring system according to claim 9, wherein the measuring machine control unit controls the measuring machine based on the measuring operation stored in the measuring operation storage unit. An image storage unit for storing the CAD image of the measuring machine is further provided,
The measurement system according to claim 8, wherein the display unit displays the CAD image stored in the image storage unit in the virtual three-dimensional space. Further comprising an imaging unit that acquires an image of the measuring machine in the real three-dimensional space,
The said display control part displays the image | video of the said measuring device acquired by the said imaging part in the said virtual three-dimensional space of the said display part, The measurement of any one of Claims 8-11 characterized by the above-mentioned. system.   13. The display control unit moves the image of the measuring machine displayed in the virtual three-dimensional space of the display unit in accordance with a preset measurement operation. The measurement system according to item 1.   14. The display control unit synthesizes an image of the measurement object and a measurement result and displays the synthesized image in the virtual three-dimensional space of the display unit. Measuring system. Further comprising a remote control unit connected to the measuring instrument control unit via a network,
The remote control unit transmits a command to the measuring machine control unit via the network based on a measurement operation designated by the user in the virtual three-dimensional space displayed on the display unit. The measurement system according to any one of claims 8 to 14. A measuring machine for measuring the measuring object,
A measuring machine control unit for controlling the measuring machine based on an instruction from a user,
An imaging unit for acquiring an image of the measuring machine in a real three-dimensional space;
An image storage unit that stores an image related to the measuring machine,
A display unit for displaying in the real three-dimensional space,
A display control unit configured to combine the image of the measuring device acquired by the imaging unit and the image stored in the image storage unit in the real three-dimensional space and display the combined image on the display unit;
A measuring system characterized by having.   17. The measurement system according to claim 16, wherein the display control unit synthesizes an image representing a predetermined measurement procedure with the image of the measuring device acquired by the imaging unit.   18. The measurement system according to claim 16, wherein the display control unit synthesizes an image corresponding to a predetermined abnormality with the image of the measuring machine acquired by the imaging unit.   The said display control part synthesize | combines the image corresponding to predetermined guidance with the image | video of the said measuring device acquired by the said imaging part, The measuring system in any one of Claims 16-18. A user-side control device connected to the measuring machine for measuring the measuring object,
A supporter-side control device connected to the user-side control device via a network;
A measurement system comprising:
The user-side control device,
A measuring machine control unit for controlling the measuring machine based on an instruction from a user,
A first three-dimensional sensor unit for detecting the three-dimensional coordinates of a predetermined object in a real three-dimensional space;
A first moving body detection unit that detects a moving body from three-dimensional coordinates detected by the first three-dimensional sensor unit;
A three-dimensional imaging unit for acquiring a three-dimensional image in the actual three-dimensional space,
A first display unit for displaying the three-dimensional image acquired by the three-dimensional imaging unit;
A first display control unit configured to combine the three-dimensional image and an image sent from the supporter-side control device and display the combined image on the first display unit;
The supporter side control device,
A second three-dimensional sensor unit for detecting the three-dimensional coordinates of the supporter in the actual three-dimensional space,
A second moving body detection unit that detects the motion of the supporter from the three-dimensional coordinates detected by the second three-dimensional sensor unit;
A second display unit that displays an image based on the three-dimensional image acquired by the three-dimensional imaging unit in a virtual three-dimensional space;
A second display control unit that causes the second display unit to display an image based on the information sent from the user-side control device in accordance with the three-dimensional coordinates of the virtual three-dimensional space;
A motion information transmission unit that transmits an image regarding the motion of the supporter detected by the second moving object detection unit to the user-side control device via the network, the measurement system. The first display control unit causes the first display unit to display a supporter-side image corresponding to a supporter based on an image sent from the supporter-side control device,
21. The measurement system according to claim 20, wherein the second display control unit causes the second display unit to display a user-side image corresponding to the user based on the image sent from the user-side control device. A measuring machine for measuring the measuring object,
A measurement result storage unit for storing the measurement result by the measuring machine,
An image capturing unit that acquires an image of the measurement target,
A feature point extraction unit that extracts feature points from the image acquired by the imaging unit,
A display unit for displaying an image of the measurement object obtained from the image acquired by the image capturing unit or the feature points extracted by the feature point extraction unit;
A measurement result reading unit that reads out the measurement result corresponding to a predetermined position of the measurement object from the measurement result storage unit based on the feature points extracted by the feature point extracting unit,
A display control unit configured to combine the measurement result read by the measurement result reading unit with the image or the image of the measurement object and display the display unit on the display unit;
A measuring system characterized by having.   23. The measurement system according to claim 22, wherein the display control unit combines the measurement result with the image or the image in correspondence with the image capturing angle of the image of the measurement target acquired by the image capturing unit. A measuring machine for measuring the measuring object,
A measurement result storage unit for storing the measurement result by the measuring machine,
An imaging unit that acquires the image of the marker,
A marker recognition unit that recognizes the marker from the image of the marker acquired by the imaging unit,
A measurement result reading unit that reads out the measurement result corresponding to the marker recognized by the marker recognition unit from the measurement result storage unit;
A display control unit for displaying the measurement result read by the measurement result reading unit on a display unit while the image of the marker is being acquired by the imaging unit;
A measuring system characterized by having. The imaging unit acquires a background image with the marker attached,
25. The measurement system according to claim 24, wherein the display control unit synthesizes the measurement result corresponding to the marker with the background image and displays the combined result on the display unit. A three-dimensional sensor unit for detecting the three-dimensional coordinates of a predetermined object in an actual three-dimensional space,
A button setting area in the actual three-dimensional space, and a command setting section that sets the correspondence between the button setting area and a command
A moving body detection unit that detects a moving body from the three-dimensional coordinates detected by the three-dimensional sensor unit;
A command execution unit that executes the command associated with the button setting area when the three-dimensional coordinates of the moving object detected by the moving object detection unit are located in the button setting area;
A three-dimensional imaging unit for acquiring a three-dimensional image in the actual three-dimensional space,
A display unit that displays the three-dimensional image acquired by the three-dimensional imaging unit;
A display control unit that combines the three-dimensional image and an image corresponding to the button setting area and displays the combined image on the display unit;
Equipped with
The user interface device, wherein the command execution unit selects and executes any one of the plurality of commands according to the shape of the user's hand detected by the moving body detection unit. An imaging unit that acquires an image of the measurement target,
A feature point extraction unit that extracts feature points from the image acquired by the imaging unit,
A display unit for displaying an image of the measurement object obtained from the image acquired by the image capturing unit or the feature points extracted by the feature point extraction unit;
A dimension acquisition unit that acquires information on a dimension corresponding to a predetermined position of the measurement object based on the characteristic points extracted by the characteristic point extraction unit,
A display control unit that synthesizes the information of the dimension acquired by the dimension acquisition unit with the image or the image of the measurement object and displays the image on the display unit;
A user interface device comprising: An imaging unit that acquires a background image with a marker attached,
A marker recognition unit that recognizes the marker from the image of the marker acquired by the imaging unit,
A measurement result acquisition unit that acquires a measurement result corresponding to the marker recognized by the marker recognition unit;
While acquiring the image of the marker in the imaging unit, a display control unit that combines the measurement result acquired by the measurement result acquisition unit with the image of the background and causes the display unit to display the result.
A user interface device comprising: