JP6837109B2 - Control system - Google Patents

Description

The present invention relates to a measurement system and a user interface device having improved operability for a measuring machine.

In a measuring machine that measures the dimensions and three-dimensional position of an object (workpiece) to be measured, the user performs an operation of designating a measurement point of the work using a measuring head. Depending on the shape of the work, the user may have to handle the measuring head in an unnatural posture. A computer is connected to the measuring machine, and various information such as work information, measurement procedures, and measurement results are displayed on the display. However, if the work and the computer are separated from each other, both the measurement work and the reference of the display cannot be performed efficiently.

In the position measuring device described in Patent Document 1, a design model loaded into CAD software running on a computer is displayed on a computer display. Then, the data obtained by the measurement is displayed on the design model of the display.

Further, the coordinate measuring machine described in Patent Document 2 includes a device including an image projector, and projects visual guide information and measurement results from the image projector onto parts. Further, Patent Documents 3 to 6 disclose a measurement system and a user interface device using a virtual reality space.

Japanese Patent Application Laid-Open No. 2011-516419 Special Table 2013-517500 Japanese Unexamined Patent Publication No. 06-241754 Japanese Patent Publication No. 2011-521318 Special Table No. 11-513157 Special Table 2009-521985

However, none of the technologies can be said to be a system that fully considers user-friendliness. For example, when measuring a work, both hands of the user are often blocked, and it is difficult to control the measuring machine while performing the measuring work. Further, depending on the work, the user may move to a place suitable for measurement to perform the work. When the user is separated from the control device (computer) of the measuring machine, it becomes more difficult to control the measuring machine while performing the measuring work.

An object of the present invention is to provide a measurement system and a user interface device capable of improving the operability of a measuring machine.

In order to solve the above problems, the measuring system of the present invention includes a measuring machine that measures a measuring object, a measuring machine control unit that controls the measuring machine based on an instruction by a user, and a predetermined object in a real three-dimensional space. 3D sensor unit that detects the 3D coordinates of, the button setting area in the actual 3D space, the command setting unit that sets the correspondence between the button setting area and the command, and the 3D coordinates detected by the 3D sensor unit. It is provided with a moving object detection unit that detects a moving object from the object, and a command execution unit that executes a command associated with the button setting area when the three-dimensional coordinates of the moving object detected by the moving object detection unit are located in the button setting area. It is characterized by that.

According to such a configuration, the command setting unit sets a desired position in the real three-dimensional space as an area that functions as a button. Further, by detecting a moving object by the moving object detecting unit, a moving object (moving object) such as a user is detected. As a result, it is possible to detect that the button setting area in the real three-dimensional space is selected from the movement of the moving object and execute the command associated with the button setting area.

In the measurement system of the present invention, the measuring machine has a measuring head for acquiring the three-dimensional coordinates of the object to be measured and a video output unit for projecting an image provided on the measuring head, and the video output unit is a command setting unit. The image of the button setting area set in may be output. With such a configuration, the image of the button setting area output by the video output unit can be projected.

In the measurement system of the present invention, the moving object detecting unit may detect the operation of the measuring head, and the command executing unit may execute the command according to the three-dimensional coordinates of the measuring head detected by the moving object detecting unit. With such a configuration, it is possible to detect that the button setting area is selected from the operation of the measurement head and execute the command associated with the button setting area.

In the measurement system of the present invention, the motion detection unit detects the movement of at least one of the user's hand and foot, and the command execution unit detects at least one of the user's hand and foot in three dimensions. The command may be executed according to the coordinates. With such a configuration, it is possible to detect that the button setting area is selected from the movements of the user's hand or foot and execute the command associated with the button setting area.

In the measurement system of the present invention, the command execution unit executes the first command when the three-dimensional coordinates of the moving object detected by the moving object detecting unit are stopped in the button setting area for a certain period of time, and the three-dimensional coordinates of the moving object. The second command may be executed when is moving along the button setting area. With such a configuration, it is possible to switch commands according to the type of movement of the moving object.

In the measurement system of the present invention, a three-dimensional image pickup unit that acquires a three-dimensional image in a real three-dimensional space, a display unit that displays a three-dimensional image acquired by the three-dimensional image pickup unit, and a three-dimensional image in the real three-dimensional space. It may further include a display control unit that synthesizes an image corresponding to the button setting area and displays it on the display unit. With such a configuration, it is possible to display a composite of the image of the button setting area with the three-dimensional image displayed in the actual three-dimensional space.

In the measurement system of the present invention, the command execution unit may select and execute one of a plurality of commands according to the shape of the user's hand detected by the motion detection unit. With such a configuration, it is possible to switch the display / non-display of the button setting area according to the shape of the user's hand, and to execute the command corresponding to the button setting area.

The measuring system of the present invention detects a measuring machine that measures a measuring object, a measuring machine control unit that controls the measuring machine based on an instruction by a user, and three-dimensional coordinates of a predetermined object in a real three-dimensional space. It is displayed on the 3D sensor unit, the moving object detection unit that detects the user's movement from the 3D coordinates detected by the 3D sensor unit, the display unit that displays the image of the measuring machine in the virtual 3D space, and the display unit. It is characterized by including a display control unit that moves the display position of the image of the measuring machine in the virtual three-dimensional space according to the movement of the user detected by the moving object detection unit.

According to such a configuration, the image of the measuring machine is displayed in the virtual three-dimensional space. Further, the motion of the user can be detected by the motion detection unit, and the image of the measuring machine in the virtual three-dimensional space can be moved according to the motion of the user.

The measurement system of the present invention may further include a measurement operation storage unit that records the measurement operation along the display position of the image of the measuring machine moved by the user in the virtual three-dimensional space displayed on the display unit. .. With such a configuration, the measurement operation can be recorded by moving the image of the measuring machine in the virtual three-dimensional space.

In the measurement system of the present invention, the measuring machine control unit may control the measuring machine based on the measurement operation stored in the measurement operation storage unit. With such a configuration, it is possible to perform measurement with an actual measuring machine based on the measurement operation recorded by moving the image of the measuring machine in the virtual three-dimensional space.

The measurement system of the present invention further includes an image storage unit that stores a CAD (Computer Aided Design) image of the measuring machine, and the display unit displays the CAD image stored in the image storage unit in a virtual three-dimensional space. You may do it. With such a configuration, the CAD image of the measuring machine can be displayed in the virtual three-dimensional space.

The measuring system of the present invention further includes an imaging unit that acquires an image of the measuring machine in the actual three-dimensional space, and the display control unit displays the image of the measuring machine acquired by the imaging unit in the virtual three-dimensional space of the display unit. You may let it. With such a configuration, the image of the actual measuring machine can be displayed in the virtual three-dimensional space.

In the measurement system of the present invention, the display control unit may move the image of the measuring machine displayed in the virtual three-dimensional space of the display unit according to a preset measurement operation. With such a configuration, it is possible to confirm the measurement operation of the preset measuring machine in the virtual three-dimensional space.

In the measurement system of the present invention, the display control unit may combine the image of the measurement object and the measurement result and display them in the virtual three-dimensional space of the display unit. With such a configuration, it is possible to synthesize and refer to the measurement result of the measurement object with the image of the measurement object in the virtual three-dimensional space.

The measurement system of the present invention further includes a remote control unit connected to the measuring machine control unit via a network, and the remote control unit is a measurement operation specified by a user in a virtual three-dimensional space displayed on the display unit. A command may be transmitted to the measuring instrument control unit via the network based on the above. With such a configuration, it is possible to refer to the measurement operation specified by the user in the virtual three-dimensional space at a position (remote location) away from the measuring machine. In addition, commands can be sent to the measuring instrument from a remote location via the network.

The measuring system of the present invention includes a measuring machine that measures an object to be measured, a measuring machine control unit that controls the measuring machine based on a user's instruction, an imaging unit that acquires an image of the measuring machine in a real three-dimensional space, and a measurement unit. An image storage unit that stores images related to the machine, a display unit that displays in the actual 3D space, an image of the measuring machine acquired by the imaging unit, and an image stored in the image storage unit are stored in the actual 3D space. It is characterized in that it is provided with a display control unit that is combined with and displayed on the display unit. According to such a configuration, the image of the actual measuring machine and the image such as a graphic can be combined and displayed in the actual three-dimensional space.

In the measurement system of the present invention, the display control unit may synthesize an image representing a predetermined measurement procedure with the image of the measuring machine acquired by the imaging unit. With such a configuration, the actual image of the measuring machine and the image of the measuring machine such as a graphic showing a predetermined measurement procedure can be combined and displayed in the three-dimensional space.

In the measurement system of the present invention, the display control unit may synthesize an image corresponding to a predetermined abnormality with the image of the measuring machine acquired by the imaging unit. With such a configuration, the actual image of the measuring machine and the image of the measuring machine such as a graphic corresponding to a predetermined abnormality can be combined and displayed in the three-dimensional space.

In the measurement system of the present invention, the display control unit may synthesize an image corresponding to a predetermined guidance with the image of the measuring machine acquired by the imaging unit. With such a configuration, the actual image of the measuring machine and the image of the measuring machine such as a graphic corresponding to a predetermined guidance can be combined and displayed in the three-dimensional space.

The measurement system of the present invention is a measurement system including a user-side control device connected to a measuring device for measuring an object to be measured, and a supporter-side control device connected to the user-side control device via a network. is there.

The user-side control device in this measurement system includes a measuring machine control unit that controls the measuring machine based on an instruction by the user, and a first three-dimensional sensor unit that detects the three-dimensional coordinates of a predetermined object in the actual three-dimensional space. With the first moving object detecting unit that detects a moving object from the 3D coordinates detected by the first 3D sensor unit, the 3D imaging unit that acquires a 3D image in the actual 3D space, and the 3D imaging unit. It includes a first display unit that displays the acquired three-dimensional image, and a first display control unit that combines the three-dimensional image and the image sent from the supporter side control device and displays them on the first display unit.

The supporter-side control device in this measurement system supports from the second 3D sensor unit that detects the 3D coordinates of the supporter in the actual 3D space and the 3D coordinates detected by the 2nd 3D sensor unit. A second moving object detection unit that detects the movement of a person, a second display unit that displays an image based on the three-dimensional image acquired by the three-dimensional imaging unit in a virtual three-dimensional space, and three-dimensional coordinates of the virtual three-dimensional space. The second display control unit that displays the image sent from the user side control device on the second display unit, and the user side control device that displays the image related to the supporter's operation detected by the second moving object detection unit via the network. It is provided with an operation information transmission unit for sending to.

According to such a configuration, the user side on which the measuring machine is installed and the supporter side of the measuring machine are connected to each other by a network. On the user side, it is possible to refer to the composition of the image of the measuring machine displayed on the first display unit and the image sent from the supporter side. On the supporter side, it is possible to refer to the composition of the image of the measuring machine such as a graphic and the image sent from the user side in the virtual three-dimensional space displayed on the second display unit.

In the measurement system of the present invention, the first display control unit displays the supporter side image corresponding to the supporter on the first display unit based on the image sent from the supporter side control device, and the second display control unit displays the supporter side image corresponding to the supporter. , The user-side image corresponding to the user may be displayed on the second display unit based on the image sent from the user-side control device. With such a configuration, the user side can refer to the supporter side image sent from the supporter side in the first display unit, and the supporter side can refer to the user side image sent from the user side in the second display unit. can do.

The measurement system of the present invention comprises a measuring machine that measures a measurement object, a measurement result storage unit that stores the measurement result by the measuring machine, an imaging unit that acquires an image of the measurement object, and an image acquired by the imaging unit. A feature point extraction unit that extracts feature points, a display unit that displays an image of an object to be measured obtained from an image acquired by the imaging unit or a feature point extracted by the feature extraction unit, and a feature point extraction unit that extracts the feature points. The measurement result reading unit that reads the measurement result corresponding to the predetermined position of the measurement object from the measurement result storage unit and the measurement result read by the measurement result reading unit are combined and displayed with the image or image of the measurement object. It is characterized by having a display control unit for displaying on the unit.

According to such a configuration, the feature points of the measurement target are extracted by projecting the image of the measurement target, and the measurement result corresponding to the predetermined position of the measurement target is obtained based on the feature points. It can be combined and displayed with the video or image of.

In the measurement system of the present invention, the display control unit may synthesize the measurement result into the image or the image according to the imaging angle of the image of the measurement object acquired by the image pickup unit. According to such a configuration, it is possible to take a picture at a desired angle and display the measurement result of the measurement object.

The measurement system of the present invention is composed of a measuring machine for measuring an object to be measured, a measurement result storage unit for storing the measurement result by the measuring machine, an imaging unit for acquiring a marker image, and a marker image acquired by the imaging unit. The marker recognition unit that recognizes the marker, the measurement result reading unit that reads the measurement result corresponding to the marker recognized by the marker recognition unit from the measurement result storage unit, and the measurement result reading unit that reads the measurement result while the imaging unit acquires the image of the marker. It is characterized by including a display control unit for displaying the measurement result read by the unit on the display unit.

According to such a configuration, by acquiring the image of the marker, the measurement result corresponding to the marker is read out, and the measurement result can be displayed on the display unit while the image of the marker is being captured. ..

In the measurement system of the present invention, the imaging unit acquires a background image with a marker, and the display control unit synthesizes the background image with the measurement result corresponding to the marker and displays it on the display unit. May be good. With such a configuration, the measurement result can be combined with the background image and displayed on the display unit.

The user interface device of the present invention associates a three-dimensional sensor unit that detects the three-dimensional coordinates of a predetermined object in the real three-dimensional space, a button setting area in the real three-dimensional space, and a button setting area and a command. A button when the 3D coordinates of the moving object detected by the moving object detection unit and the 3D coordinate detected by the 3D sensor unit and the 3D coordinate detected by the 3D sensor unit are located in the button setting area. A command execution unit that executes a command associated with the setting area, a 3D imaging unit that acquires a 3D image in the actual 3D space, and a display unit that displays the 3D image acquired by the 3D imaging unit. It is equipped with a display control unit that combines a three-dimensional image and an image corresponding to the button setting area and displays it on the display unit, and the command execution unit has a plurality of command execution units according to the shape of the user's hand detected by the motion detection unit. It is characterized in that one of the commands is selected and executed.

According to such a configuration, it is possible to switch the display / non-display of the button setting area according to the shape of the user's hand, and to execute the command corresponding to the button setting area.

The user interface device of the present invention has an imaging unit that acquires an image of an object to be measured, a feature point extraction unit that extracts feature points from the image acquired by the imaging unit, and an image or feature extraction unit that extracts the image acquired by the imaging unit. A display unit that displays an image of the measurement object obtained from the feature points, and a dimension acquisition unit that acquires dimensional information corresponding to a predetermined position of the measurement object based on the feature points extracted by the feature point extraction unit. It is characterized by including a display control unit that synthesizes the dimensional information acquired by the dimensional acquisition unit with an image or an image of a measurement object and displays it on the display unit.

According to such a configuration, the feature points of the measurement target are extracted by projecting the image of the measurement target, and the measurement result corresponding to the predetermined position of the measurement target is obtained based on the feature points. It can be combined and displayed with the video or image of.

The user interface device of the present invention corresponds to an imaging unit that acquires a background image with a marker, a marker recognition unit that recognizes a marker from the marker image acquired by the imaging unit, and a marker recognized by the marker recognition unit. A display control unit that combines the measurement result acquired by the measurement result acquisition unit with the background image and displays it on the display unit while the measurement result acquisition unit that acquires the measurement result and the image pickup unit are acquiring the marker image. It is characterized by having.

According to such a configuration, by acquiring the image of the marker, the measurement result corresponding to the marker is read out, and the measurement result can be displayed on the display unit while the image of the marker is being captured. ..

(A) and (b) are block diagrams illustrating the measurement system according to the first embodiment. It is a schematic diagram which shows the operation example. (A) and (b) are schematic diagrams illustrating the button setting area. It is a schematic diagram which illustrates other button setting area. It is a schematic diagram which illustrates other button setting area. It is a block diagram which illustrates the measurement system which concerns on 2nd Embodiment. It is a perspective view which illustrates the head-mounted display. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. (A) and (b) are block diagrams illustrating the measurement system according to the third embodiment. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. It is a block diagram which illustrates the measurement system which concerns on 4th Embodiment. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. It is a schematic diagram which shows the operation example. It is a flowchart which illustrates the operation of a helper display. It is a schematic diagram which shows the display example of a helper. It is a schematic diagram which shows the display example of a helper. It is a schematic diagram which shows the display example of a helper. It is a schematic diagram which shows the display example of a helper. It is a block diagram which illustrates the measurement system which concerns on 5th Embodiment. It is a schematic diagram which shows the operation example. It is a block diagram which illustrates the measurement system which concerns on 6th Embodiment. It is a flowchart which shows an example of the process of the measurement result composite display. It is a schematic diagram which shows the synthetic display example of the measurement result. (A) to (e) are schematic diagrams showing other synthetic display examples. It is a block diagram which illustrates the measurement system which concerns on 6th Embodiment. It is a flowchart which illustrates the marker registration and setting operation. It is a flowchart which exemplifies a marker recognition and a composite display. It is a schematic diagram which shows the display example of the measurement result by marker recognition. It is a schematic diagram which shows the display example of the measurement result by marker recognition. It is a schematic diagram which shows the display example of the measurement result by marker recognition.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same members are designated by the same reference numerals, and the description of the members once described will be omitted as appropriate.

(First Embodiment)
1 (a) and 1 (b) are block diagrams illustrating the measurement system according to the first embodiment.
The measurement system 1A according to the present embodiment includes a computer 100, a measuring machine M controlled by the computer 100, and a 3D sensor (three-dimensional sensor) SR. The computer 100 includes a CPU (Central Processing Unit) 10, a storage unit 20, a calculation unit 30, a measuring machine control unit 40, a display control unit 50, a 3D sensor input unit 70, and a position / distance / attitude recognition unit 75. Further, a projector PR may be connected to the computer 100. In this case, the computer 100 has a projection figure generation unit 80 and a projector output unit 85.

The measuring machine M is a device that measures the three-dimensional coordinates of an object to be measured, the length of a predetermined position, and the like. Examples of the measuring machine M include a three-dimensional position measuring machine and an image measuring machine. The CPU 10 controls each part and performs a predetermined calculation by executing a predetermined program. The storage unit 20 includes a main storage unit and a sub storage unit. The calculation unit 30 includes a calculation circuit that performs a predetermined calculation.

The measuring machine control unit 40 controls the measuring machine M based on an instruction by the user. The display control unit 50 controls to display predetermined information on the display D. The input / output control unit 60 controls an input device such as a keyboard K or a mouse MS, or controls a touch panel (not shown).

The 3D sensor input unit 70 is an interface portion that takes in the information acquired by the 3D sensor SR into the computer 100. Here, the 3D sensor SR is a sensor that detects the three-dimensional coordinates of a predetermined object in the real three-dimensional space. The position / distance / posture recognition unit 75 performs a process of recognizing the position (three-dimensional coordinates) of the object, the distance from the 3D sensor SR to the object, and the posture of the object based on the information acquired by the 3D sensor SR. Do.

The projection figure generation unit 80 performs a process of generating a figure to be projected by the projector PR. The projector output unit 85 is an interface portion that sends the figure generated by the projection figure generation unit 80 to the projector PR.

The measurement system 1A further includes a command setting unit 101, a moving object detection unit 102, and a command execution unit 103 as programs executed by the CPU 10. The command setting unit 101 performs a process of setting the association between the button setting area in the real three-dimensional space, the button setting area, and the command. The button setting area is an area in which an arbitrary position is specified in the real three-dimensional space. For example, it may be a predetermined rectangular region in a real three-dimensional space, a predetermined three-dimensional region, or a predetermined object region.

The command setting unit 101 performs a process of setting three-dimensional coordinate information for representing such a button setting area. The command setting unit 101 also associates the button setting area with the command. The command is associated with the three-dimensional coordinate information of the button setting area, and is stored in the storage unit 20 as table data, for example.

The moving object detection unit 102 performs a process of detecting a moving object from the three-dimensional coordinates of the object detected by the 3D sensor SR. Specifically, the position of the object recognized by the position / distance / posture recognition unit 75 is tracked to detect a moving object (moving object). This makes it possible to detect a moving object in a real three-dimensional space that can be detected by the 3D sensor SR and track the position of the moving object.

The command execution unit 103 performs a process of executing a command associated with the button setting area when the three-dimensional coordinates of the moving object detected by the moving object detection unit 102 are located in the button setting area. The moving object detection unit 102 can track the position of an object moving in the real three-dimensional space (three-dimensional coordinates of the moving object). The command execution unit 103 acquires information on the three-dimensional coordinates of the moving object from the moving object detecting unit 102.

Then, when the three-dimensional coordinates of the moving object are located in the button setting area, the command execution unit 103 executes a process of executing the command associated with the button setting area. As a result, it is possible to detect that the button setting area in the real three-dimensional space is selected from the movement of the moving object and execute the command associated with the button setting area.

FIG. 2 is a schematic diagram showing an operation example.
The 3D sensor SR detects the position of an object in real three-dimensional space. In the example shown in FIG. 2, for example, the measuring machine M, the user 800 of the measuring machine M, the screen SL for projecting the image projected from the projector PR, the whiteboard WB, and the like are arranged in the actual three-dimensional space. The 3D sensor SR detects these positions.

Further, in the example shown in FIG. 2, button setting areas B1 to B4 are provided on the whiteboard WB. The positions (three-dimensional coordinates) of the button setting areas B1 to B4 on the whiteboard WB are set in advance by the command setting unit 101. A command is associated with each of the button setting areas B1 to B4. For example, the button setting area B1 is associated with the "start" command, the button setting area B2 is associated with the "end" command, the button setting area B3 is associated with the "save" command, and the button setting area B4 is associated with the "print" command.

The user 800 of the measuring machine M measures the object to be measured by using, for example, the measuring head 501 of the measuring machine M. When the measuring head 501 is provided with the projector PR, an image based on various information such as a design drawing of a measurement object can be displayed on the screen SL from the projector PR.

The movement of the user 800 is detected by the moving object detection unit 102 based on the information acquired by the 3D sensor SR. The position / distance / posture recognition unit 75 recognizes, for example, the movement of the hand 801 of the user 800. Then, when the hand 801 of the user 800 is located in any of the button setting areas B1 to B4, the command execution unit 103 executes the command associated with the button setting areas B1 to B4 at the position of the hand 801.

For example, when the hand 801 of the user 800 touches the button setting area B1 on the whiteboard WB, the "start" command associated with the button setting area B1 is executed. Similarly, when the user 800's hand 8001 touches the button setting area B2, the "end" command associated with the button setting area B2 is executed. In this way, the user 800 can execute a desired command depending on the position of his / her hand 801.

3A and 3B are schematic views illustrating the button setting area.
FIG. 3A shows button setting areas B1 to B4 set on the whiteboard WB. The button setting areas B1 to B4 are appropriately written on the surface of the whiteboard WB. Here, four rectangles are written on the surface of the whiteboard WB.

The command setting unit 101 sets the positions of the button setting areas B1 to B4 by registering the three-dimensional coordinates of the even part of each rectangle. For example, the button setting area B1 is set by the three-dimensional coordinates of the four even parts P1 to P4. The three-dimensional coordinates of the even portions P1 to P4 can be captured by the 3D sensor SR. Further, the measurement head 501 may be used to capture the three-dimensional coordinates of the even portions P1 to P4.

FIG. 3B shows a three-dimensional button setting area B5. The button setting area B5 is, for example, a cube area. The region of the solid body is determined by four points P17, P18, P19, P20 in the real three-dimensional space. The command setting unit 101 sets the position of the button setting area B5 by registering the three-dimensional coordinates of the four points P17, P18, P19, and P20.

The command setting unit 101 associates a command with each of such button setting areas B1 to B5. The button setting areas B1 to B5 can be set at any place in the real three-dimensional space. Even if the command setting unit 101 sets the button setting areas B1 to B4 at a position where some object exists, such as the whiteboard WB, the command setting unit 101 sets the button setting area B5 in the area where there is no object in the real three-dimensional space. You may.

4 and 5 are schematic views illustrating other button setting areas.
In the example shown in FIG. 4, the image projected by the projector PR is designated as the button setting area B6. For example, when the measuring head 501 is provided with the projector PR, the measuring head 501 is directed in a direction different from that of the screen SL. As a result, the image of the button setting area B6 is output from the projector PR. In the example shown in FIG. 4, the image of the button setting area B6 output from the projector PR is projected on the floor FL.

When the image of the button setting area B6 is projected by the projector PR, the projection position of the button setting area B6 is detected by the 3D sensor SR and recognized by the position / distance / posture recognition unit 75. By registering the features of the button setting area B6 in advance, the position / distance / posture recognition unit 75 can recognize the position of the button setting area B6 from the information detected by the 3D sensor SR. Further, even if the projection position of the button setting area B6 moves, it can be tracked by the moving object detection unit 102.

When the button setting area B6 is projected on the floor FL, the user 800 can select the button setting area B6 by, for example, his / her own foot 802. The position of the foot 802 of the user 800 is detected by the 3D sensor SR, and the movement of the foot 802 is tracked by the moving object detection unit 102. As a result, it is possible to detect that the foot 802 of the user 800 is at the position of the button setting area B6 and execute the command associated with the button setting area B6 by the command execution unit 103.

For the user 800 who holds the measurement head 501 by hand and performs the measurement work, it may be difficult to give a command by hand. By selecting the button setting area B6 projected on the floor FL with the foot 802, the user 800 can specify and execute a desired command with the foot 802 while holding the measurement head 501 by hand.

In the example shown in FIG. 5, the image of the button setting area B7 is projected on the screen SL. The projector PR, for example, projects design information of an object to be measured on the screen SL and also projects an image of the button setting area B7. Similar to the example shown in FIG. 4, the position of the button setting area B7 is detected by the 3D sensor SR and tracked by the moving object detection unit 102.

The user 800 can select the button setting area B7 projected on the screen SL by his / her own hand 801 for example. When it is detected that the hand 801 of the user 800 is at the position of the button setting area B7, the command execution unit 103 executes the command associated with the button setting area B7.

Further, the command execution unit 103 may switch commands according to the movements of the hand 801 and the foot 802 of the user 800. For example, when the hand 801 of the user 800 is stopped in the button setting area B1 for a certain period of time (touch operation), the first command is executed, and when the hand 801 is moving along the button setting area B1 (slide). Operation), the second command is executed. That is, the command may be switched depending on whether the button setting area B1 is touched or slid by the hand 801.

According to the measurement system 1A according to the first embodiment, the button setting areas B1 to B7 can be set at desired positions in the actual three-dimensional space. Further, the user 800 can execute a desired command by selecting the button setting areas B1 to B7 with his / her hand 801 or foot 802. Therefore, the user 800 can easily execute a desired command while performing the measurement work by the measuring machine M. For example, even when the computer 100 is located at a position away from the measuring machine M, the user 800 can send a desired command to the computer 100 without moving away from the position of the measuring machine M.

Further, the user 800 can set the button setting areas B1 to B7 at desired positions. As a result, the button setting areas B1 to B7 can be arranged at positions where the user 800 can work efficiently, and the efficiency of the measurement work can be improved.

(Second Embodiment)
FIG. 6 is a configuration diagram illustrating the measurement system according to the second embodiment.
The measurement system 1B according to the present embodiment includes a computer 100, a measuring machine M controlled by the computer 100, and a 3D sensor SR. The computer 100 includes a CPU 10, a storage unit 20, a calculation unit 30, a measuring machine control unit 40, a display control unit 50, a 3D sensor input unit 70, and a position / distance / attitude recognition unit 75. Further, the computer 100 includes a 3D video input / output unit 51, a 3D camera control unit 52, a stereoscopic image generation unit 53, a head mount display output unit 54, a control generation unit 55, a headphone microphone input / output unit 61, and an audio input / output unit 62. It has an operation panel input / output unit 65 and a control input recognition unit 76. An operation panel CT, a headphone microphone HPM, a head-mounted display HD, and a 3D camera CM are connected to the computer 100.

Hereinafter, a configuration different from the measurement system 1A according to the first embodiment will be described.
The operation panel input / output unit 65 is an interface portion for inputting / outputting information to the operation panel CT. The user 800 can control the measuring machine M by using the operation panel CT.

The 3D camera CM is a three-dimensional imaging unit that acquires a three-dimensional image in a real three-dimensional space. The head-mounted display HD is a display that displays a three-dimensional image acquired by a 3D camera CM. Here, in the embodiment, the "image" means the information of the image captured by the imaging device such as the 3D camera CM, and the "image" is a graphic generated from the design data such as CAD or prepared in advance. It means information such as graphics.

FIG. 7 is a perspective view illustrating a head-mounted display.
The head-mounted display HD is mounted on the head of the user 800. The user 800 can refer to the three-dimensional image displayed on the head-mounted display HD. A 3D camera CM is attached to this head-mounted display HD. That is, the head-mounted display HD is provided integrally with the 3D camera CM.

The 3D camera control unit 52 is a part that controls the 3D camera CM. The 3D video input / output unit 51 is an interface portion that captures the 3D video acquired by the 3D camera CM into the computer 100. The head-mounted display output unit 54 is an interface portion that transmits a video signal to the head-mounted display HD. The stereoscopic image generation unit 53 is a part that generates a 3D image to be displayed on the head-mounted display HD. The stereoscopic image generation unit 53 includes a captured image composition unit 531.

The captured image compositing unit 531 performs a process of synthesizing the 3D image acquired by the 3D camera CM and the image corresponding to the button setting area and displaying them on the head-mounted display HD.

The control generation unit 55 is a part that associates a predetermined operation of the user 800 with a command. The correspondence between the predetermined operation of the user 800 and the command is stored in the storage unit 20 as, for example, table data.

The headphone microphone HPM is an integrated headphone and microphone, and is attached to the head of the user 800. The headphone / microphone input / output unit 61 is an interface part that inputs / outputs information to / from the headphone / microphone HPM. The voice input / output unit 62 is a part that generates voice to be sent to the headphone microphone HPM and processes voice information captured from the headphone microphone HPM. In this embodiment, the headphone microphone HPM may be provided as needed.

The control input recognition unit 76 is a part that recognizes a predetermined operation of the user 800. The control input recognition unit 76 performs a process of recognizing a predetermined operation of the user 800 based on the information sent from the position / distance / posture recognition unit 75. In such a measurement system 1B, an image in a real three-dimensional space is displayed on the head-mounted display HD, and an image in a button setting area can be combined and displayed on the image.

8 to 11 are schematic views showing an operation example.
As shown in FIG. 8, the user 800 wears the head-mounted display HD. Further, the 3D sensor SR detects the position of an object such as the user 800 or the measuring machine M in the real three-dimensional space. The 3D camera CM provided in the head-mounted display HD acquires a 3D image in a real three-dimensional space.

The user 800 can refer to the 3D image acquired by the 3D camera CM by the head-mounted display HD. Further, the head-mounted display HD displays an image (button image BG) of the button setting area according to the actual three-dimensional space of the 3D image. That is, the captured image compositing unit 531 synthesizes the button image BG with the 3D image acquired by the 3D camera. The button image BG is displayed together with the three-dimensional coordinates in the display of the head-mounted display HD based on the three-dimensional coordinates in the real three-dimensional space of the button setting area registered in advance. As a result, even if the angle of view of the 3D camera CM changes, the composite position of the button image BG on the 3D image does not change.

The movement of the user 800 is detected by the moving object detection unit 102 based on the information acquired by the 3D sensor SR. The position / distance / posture recognition unit 75 recognizes, for example, the movements of the hand 801 and the foot 802 of the user 800. Then, when the hand 801 or foot 802 of the user 800 is located in the button setting area corresponding to the button image BG, the command execution unit 103 executes the command associated with the button setting area at the position of the hand 801 or foot 802. .. As a result, the user 800 can execute a desired command depending on the position of his / her hand 801 or foot 802.

The button image BG can be displayed at an arbitrary position on the 3D image displayed on the head-mounted display HD. For example, the button image BG is displayed at a position (in the air) where an object does not exist in the real 3D space where the D image is displayed, or the button image BG is displayed on the 3D image of the floor FL as if it were placed on the floor FL. The button image BG can be displayed.

FIG. 9 shows an example in which the button image BG is synthesized on the 3D image of the floor FL. The user 800 holds the measuring head 501 of the measuring machine M in his / her hand and measures the work W which is the object to be measured. The user 800 operates the measuring head 501 while referring to the 3D images of the measuring machine M and the work W displayed on the head-mounted display HD to perform the measurement.

Further, on the head-mounted display HD, the button image BG is displayed in a state of being combined with the 3D image of the floor FL. Although the buttons are not arranged on the actual floor FL, the button image BG is displayed on the display of the head-mounted display HD. As a result, the user 800 can refer to the button image BG as if the button is arranged on the floor FL. User 800 selects the button image BG on the floor FL with the foot 802. As a result, the command in the button setting area corresponding to the selected button image BG is executed.

FIG. 10 shows an example in which a computer image CAL is combined with a 3D image. In this example, the computer image CAL is synthesized in the vicinity of the 3D image of the work W displayed on the head-mounted display HD. The user 800 operates the image CAL of the calculator by, for example, the hand 801. The motion detection unit 102 tracks the operation of the hand 801 to determine in which key area of the image CAL of the computer the hand 801 is located. As a result, the command execution unit 103 executes the command corresponding to the key of the image CAL of the computer. The user 800 can refer to the virtual computer image CAL displayed on the head-mounted display HD and perform calculations as if he / she is operating an actual computer.

The command execution unit 103 may select and execute one of a plurality of commands according to the shape of the user 800's hand 801 detected by the motion detection unit 102. FIG. 11 shows an example of switching commands according to the shape of the hand 801.

As an example, when a certain period of time elapses with the hand 801 opened in a specific direction (H1), a command for displaying a home button (an image in which a plurality of button image BGs are aggregated) is executed. In addition, when the home button is displayed in a position out of reach of the hand 801 on the display of the head-mounted display HD, when the hand 801 is held (H2), a command to display the distant home button near is executed. .. Further, when the home button is picked up by the hand 801 and moved (H3), a command for moving the display position of the home button is executed.

Further, when the index finger of the hand 801 is directed in a specific direction (H4), a command to hide the home button is executed. When the hand 801 is pressed against the wall (H5), a command to move the home button to be attached to the wall is executed. When the palm of the hand 801 is turned upward (H6), a command to move the home button on the palm of the hand 801 is executed. When the hand 801 is pressed toward the floor (H7), a command to move the home button so as to be pasted on the floor is executed. The correspondence between the shape and operation of the hand 801 and the command is an example, and the present invention is not limited to this.

In this way, by switching commands according to the shape of the hand 801 so that a desired command can be executed only by a gesture using the hand 801 without using an input device such as a keyboard K or a mouse MS. Become.

In the above example, the user 800 executes the command by selecting the button image BG by the hand 801 or the foot 802, but the user 800 may execute the command by selecting the button image BG by the measurement head 501.

Further, in the measurement system 1B according to the present embodiment, the user interface device may be configured by the computer 100, the 3D sensor SR, and the 3D camera CM. According to this user interface device, it is possible to switch the display / non-display of the button image BG according to the shape of the hand 801 of the user 800, and to execute the command corresponding to the button image BG.

(Third Embodiment)
12 (a) and 12 (b) are block diagrams illustrating the measurement system according to the third embodiment.
The measurement system 1C according to the present embodiment includes a computer 100, a measuring machine M controlled by the computer 100, a 3D sensor SR, and a head-mounted display HD. The computer 100 includes a CPU 10, a storage unit 20, a calculation unit 30, a measuring machine control unit 40, a display control unit 50, a 3D sensor input unit 70, and a position / distance / attitude recognition unit 75. Further, the computer 100 includes a stereoscopic image generation unit 53, a head-mounted display output unit 54, a headphone microphone input / output unit 61, an audio input / output unit 62, and an operation panel input / output unit 65. An operation panel CT and a headphone microphone HPM are connected to the computer 100.

In such a measurement system 1C, the image of the measuring machine M is displayed in the virtual three-dimensional space of the head-mounted display HD. As the image of the measuring machine M, for example, a CAD image is used. The CAD image is stored in the storage unit 20. Further, the motion of the user can be detected by the motion detection unit, and the image of the measuring machine M in the virtual three-dimensional space can be moved according to the motion of the user 800.

The measuring system 1C according to the present embodiment is particularly suitable for application to a measuring machine M having abundant automatic measuring functions. For example, a CNC (Computerized Numerically Controlled) measuring machine can perform automatic measurement according to a preset program (for example, a part program), so that highly accurate and efficient measurement can be performed. However, handling a CNC measuring machine requires specialized skills and knowledge such as operation methods and program creation / modification. In the measurement system 1C according to the present embodiment, it is possible to improve the user interface of the measuring machine M having such abundant functions.

13 to 17 are schematic views showing an operation example.
As shown in FIG. 13, the user 800 wears a head-mounted display HD and a head-mounted microphone HPM. The 3D sensor SR detects the position of an object such as the user 800 or the measuring machine M in the real three-dimensional space.

An image of the measuring machine M (measuring machine image MG) is displayed on the head-mounted display HD in a virtual three-dimensional space. The user 800 can move the measuring machine image MG in the virtual three-dimensional space by referring to the measuring machine image MG in the virtual three-dimensional space displayed on the head-mounted display HD.

That is, the position of the user 800, for example, the hand 801 is detected by the 3D sensor SR, and the movement of the hand 801 is tracked by the moving object detection unit 102. The position and movement of the hand 801 are reflected in the movement of the measuring machine image MG displayed in the virtual three-dimensional space. For example, when the user 800 extends the hand 801 to the position of the image of the measurement head 501 (measurement head image 501G) in the virtual three-dimensional space, the measurement head image 501G in the virtual three-dimensional space can be moved.

The stereoscopic image generation unit 53 generates an image so that the display position of the measuring machine image MG in the virtual three-dimensional space is moved according to the operation of the user 800's hand 801 detected by the moving object detection unit 102. As a result, the user 800 can freely operate the measuring machine image MG in the virtual three-dimensional space displayed on the head-mounted display HD.

In the measurement system 1C according to the present embodiment, the measurement operation storage unit 104 may be provided as a program executed by the CPU 10. The measurement operation storage unit 104 performs a process of recording the measurement operation along the display position of the measuring machine image MG moved by the user 800 in the virtual three-dimensional space displayed on the head-mounted display HD.

Then, the measuring machine control unit 40 measures the work W using the actual measuring machine M according to the measuring operation stored in the measuring operation storage unit 104. As a result, the user 800 can record the actual measurement procedure by moving the measuring machine image MG, for example, the measuring head image 501G, in the virtual three-dimensional space.

At this time, the actual position of the work W is detected by the 3D sensor SR, and the work image WG is synthesized according to the position of the measuring machine image MG in the virtual three-dimensional space. When moving the measurement head image 501G, the user 800 can record the measurement procedure while confirming the interference between the work image WG and the measurement head image 501G. For example, if interference with the work image WG occurs depending on the orientation of the measurement head image 501G, a warning may be displayed on the head-mounted display HD. According to the record of such a measurement procedure, the movement of the measurement head 501 can be programmed without any specialized knowledge.

FIG. 14 shows an example of displaying an actual video IMG in a virtual three-dimensional space. The measuring system 1C may include a camera CM1 that acquires a video IMG of the measuring machine M. The video IMG of the measuring machine M acquired by the camera CM1 is displayed at a predetermined position in the virtual three-dimensional space of the head-mounted display HD.

The user 800 can refer to the video IMG of the actual measuring machine M together with the measuring machine image MG displayed on the head-mounted display HD. As a result, the work can be performed while confirming the actual operation of the measuring machine M in the virtual three-dimensional space.

FIG. 15 shows an example of the modification operation of the part program. On the head-mounted display HD, the operation of the measurement head 501 by the existing part program is displayed in the virtual three-dimensional space. User 800 reads an existing part program. For example, by inputting voice to the headphone microphone HPM, the existing part program may be read by voice recognition.

The measurement head image 501G in the virtual three-dimensional space operates in the same manner as when the read part program is executed. The user 800 can refer to the operation of the measurement head image 501G displayed in the virtual three-dimensional space, and can control the stop and restart of the operation by voice recognition. It is also possible to move the measurement head image 501G using the hand 801.

In the virtual three-dimensional space, a numerical value in the moving order of the measurement head image 501G or a button image BG may be displayed. When modifying the part program, the operation is paused and the measurement head image 501G is moved by the hand 801.

Here, as an example, the operation when adding a step of a part program will be described. In this example, as shown in FIG. 15, for a part program in which the measurement head 501 operates in the order of (1) to (4), the operation of adding (2') between (2) and (3) is performed. explain.

First, the user 800 operates the part program step by step by voice recognition, for example. When the user 800 inputs, for example, the voice "Go to the first step", the part program advances by one step, and the measurement head image 501G in the virtual three-dimensional space is displayed to move to the position (1).

Next, the user 800 inputs a voice such as "Go to the next step". As a result, the part program proceeds to the next step, and the measurement head image 501G in the virtual three-dimensional space is displayed so as to move to the position (2).

Here, the user 800 inputs, for example, a voice such as "Add the step" in order to add a step. As a result, the additional mode of the step of the part program is executed. Next, the user 800 moves the measurement head image 501G in the virtual three-dimensional space by the hand 801 and arranges it at the position (2'). In this state, for example, if the voice "Continue" is input, the position (2') is added, and the measurement head image 501G advances to the next position (3).

Although the operation and modification of the part program are instructed by voice recognition here, the same operation and modification may be instructed by designating the button image BG. By such an operation, the user 800 can intuitively modify the part program without any specialized knowledge.

FIG. 16 shows an example of displaying information about measurement in a virtual three-dimensional space. The user 800 refers to an image in the virtual three-dimensional space by the worn head-mounted display HD. The work image WG and the measurement head image 501G are displayed in the virtual three-dimensional space. The work image WG and the measurement head image 501G are CAD images and computer graphics. In the example shown in FIG. 16, an example in which the measurement results are combined and displayed on the work image WG is shown. Further, when the measurement result is not within the predetermined allowable range, a display indicating that fact (such as a display for coloring the work image WG) may be added to the work image WG.

The user 800 inputs a voice such as "Show result" from the headphone microphone HPM. As a result, the measurement result is combined and displayed on the work image WG in the virtual three-dimensional space. The user 800 can confirm the measurement result without referring to the display D of the computer 100 by referring to the contents displayed in the virtual three-dimensional space.

FIG. 17 shows an example of performing measurement and teaching via the network N.
In the example shown in FIG. 17, the measurement system 1C further includes a remote control unit 90 connected to the measuring machine control unit 40 via the network N. The remote control unit 90 performs a process of sending a command instructed by the user 800 located at a position away from the measuring machine M to the measuring machine control unit 40 via the network N.

The user 800 at a remote location can confirm the actual operation of the measuring machine M while referring to the measuring machine image MG in the virtual three-dimensional space displayed on the worn head-mounted display HD. Further, the measuring machine image MG and the measuring head image 501G displayed in the virtual three-dimensional space can be moved by the hand 801 of the user 800.

As described above, the operation of the measurement head 501 by the part program can be confirmed by the operation of the measurement head image 501G displayed in the virtual three-dimensional space. Further, the part program can be modified in the virtual three-dimensional space.

By transmitting the instruction of the user 800 in the remote location to the measuring machine control unit 40 via the network N by the remote control unit 90, the user 800 is an image of the virtual three-dimensional space regardless of the position of the measuring machine M. The measuring machine M can be handled with reference to.

That is, the user 800 can handle the measuring machine M from a remote location as if he / she is near the measuring machine M. In this measurement system 1C, for example, when a user 800 who is accustomed to operating the measuring machine M is not near the measuring machine M, a knowledgeable staff in a remote place can operate the measuring machine M via the network N. it can. Further, even when the measuring machine M is in a harsh environment and the user 800 cannot approach the measuring machine M, the measuring machine M can be operated as if the user 800 is there.

(Fourth Embodiment)
FIG. 18 is a configuration diagram illustrating the measurement system according to the fourth embodiment.
The measurement system 1D according to the present embodiment includes a computer 100, a measuring machine M controlled by the computer 100, and a 3D sensor SR. The computer 100 includes a CPU 10, a storage unit 20, a calculation unit 30, a measuring machine control unit 40, a display control unit 50, a 3D sensor input unit 70, and a position / distance / attitude recognition unit 75. Further, the computer 100 includes a 3D video input / output unit 51, a 3D camera control unit 52, a stereoscopic image generation unit 53, a captured video synthesis unit 531, a head-mounted display output unit 54, a headphone microphone input / output unit 61, and an audio input / output unit 62. , Sound reproduction unit 621, operation panel input / output unit 65, and helper generation unit 95. An operation panel CT, a headphone microphone HPM, a head-mounted display HD, and a 3D camera CM are connected to the computer 100.

In such a measurement system 1D, the image of the actual measuring machine M and the measuring machine image MG such as a CAD image or computer graphic can be combined and displayed in the virtual three-dimensional space of the head-mounted display HD. .. The measuring machine image MG is stored in advance in the image storage unit 201 of the storage unit 20.

In addition, the helper generation unit 95 generates an image showing a predetermined measurement procedure of the measuring machine M. As a result, the captured image compositing unit 531 of the stereoscopic image generation unit 53 can perform a process of synthesizing the image representing the measurement procedure generated by the helper generation unit 95 with the image captured by the 3D camera CM.

Further, the helper generation unit 95 may generate an image corresponding to a predetermined abnormality of the measuring machine M. As a result, the captured image compositing unit 531 of the stereoscopic image generation unit 53 can perform a process of synthesizing an image corresponding to a predetermined abnormality generated by the helper generation unit 95 with the image captured by the 3D camera CM.

Further, the helper generation unit 95 may generate an image corresponding to a predetermined guidance of the measuring machine M. As a result, the captured image compositing unit 531 of the stereoscopic image generation unit 53 can perform a process of synthesizing an image corresponding to a predetermined guidance generated by the helper generation unit 95 with the image captured by the 3D camera CM.

Here, when the operation of the measuring machine M is unknown, the user 800 simulates the actual operation image of the measuring machine M in his / her mind while checking the instruction manual of the measuring machine M. However, it is difficult to memorize all the operations, and it is forced to perform troublesome work such as checking the instruction manual again.

Further, for example, when the measurement probe is replaced, if it is not performed according to the procedure, a problem such as breakage may occur. Therefore, careful work is required to replace the measurement probe. The user 800 who is not accustomed to the work will perform the work while referring to the instruction manual each time.

Further, for example, in order to set the origin of the non-orthogonal coordinate measuring machine, it is necessary to obtain the origin by turning on / off each limit switch of the 7-axis arm of the measuring machine M. The user 800 who is not accustomed to this work often finds it difficult to intuitively understand the axis for which the origin has not been acquired and the operation for acquiring the origin, and is often confused about the operation.

Further, for example, when connecting the cable of the measuring device M to the dedicated controller, it is difficult to know which cable should be inserted into which connection port of the controller, and it becomes necessary to check the instruction manual each time.

Further, for example, in the case of a non-orthogonal coordinate measuring machine, measurement cannot be performed when the limit switch is ON. It is difficult to tell whether this limit switch is in the ON state or in a measurable state, and the same place may be remeasured many times.

Further, for example, when an abnormality occurs in the measuring device M, a message is displayed on the display D of the computer 100 or a sound is warned. However, when the user 800 is concentrating on the measurement, the display D of the computer 100 may not be seen so much, or the display D may be difficult to see, so that it may be difficult to recognize.

Further, for example, a video or the like may be reproduced for the purpose of educating a beginner about a measurement method or the like. However, there are cases where the time required for learning increases, such as when you do not know what to do when actually measuring, or when you make a mistake in the procedure and start over.

The measurement system 1D according to the present embodiment is efficient by combining and displaying the image of the actual measuring machine M and various images in the virtual three-dimensional space of the head-mounted display HD worn by the user 800. You will be able to perform various operations.

19 to 23 are schematic views showing an operation example.
FIG. 19 shows an example of providing technical assistance to the user 800.
As shown in FIG. 19, the user 800 wears a head-mounted display HD. Further, the 3D sensor SR detects the position of an object such as the user 800 or the measuring machine M in the real three-dimensional space. The 3D camera CM provided in the head-mounted display HD acquires a 3D image in a real three-dimensional space.

The user 800 can refer to the 3D image acquired by the 3D camera CM by the head-mounted display HD. In the example shown in FIG. 19, the image IMG of the measuring machine M is displayed as a 3D image. Further, on the head-mounted display HD, various images G related to the measuring machine M are displayed according to the actual three-dimensional space of the 3D image.

The display position of the image G is determined based on the position of the image IMG of the measuring machine M. As a result, even if the angle of view of the 3D camera CM changes, the display position of the image G with respect to the measuring machine M does not change. In the example shown in FIG. 19, the image G according to the procedure of origin determination is displayed in a state of being combined with the image IMG of the measuring machine M. The user 800 can refer to the image G according to the origin setting procedure while referring to the image IMG of the actual measuring machine M acquired by the 3D camera CM.

By synthesizing and displaying such an image G on the video IMG of the actual measuring machine M, the user 800 can obtain the origin of which axis of the measuring machine M needs to be acquired and the origin. It is possible to intuitively grasp the operation procedure required to do so.

FIG. 20 shows an operation example when teaching the operation method of the measuring machine M. FIG. 20 shows images G1 to G7 showing the procedure for exchanging the measurement probe as an example. Images G1 to G7 are displayed in a state of being arranged in order on the head-mounted display HD worn by the user 800. These images G1 to G7 are combined with the image IMG (not shown in FIG. 20) of the actual measuring machine M taken by the 3D camera CM. As the display of the images G1 to G7, a step display may be performed in which the images G1 to G7 are displayed one by one. Operations such as playing and stopping the step display are instructed by the voice of the user 800, the operation of the hand 801 and the like.

By referring to the images G1 to G7 displayed on the head-mounted display HD, the user 800 can grasp the operation procedure while comparing the image IMG of the actual measuring machine M with the images G1 to G7.

FIG. 21 shows an operation example when teaching a cable connection method. On the head-mounted display HD worn by the user 800, the image IMG of the actual measuring machine M taken by the 3D camera CM and the image PC-IMG of the personal computer are displayed. Further, the image G of the cable and the guidance is displayed in a state of being combined with the video IMG and the PC-IMG. The image G of the cable is compositely displayed according to the connection position of the video PC-IMG of the actual personal computer.

In this way, since the image G when the cable is connected is compositely displayed on the image IMG of the actual measuring machine M and the image PC-IMG of the personal computer, the user 800 intuitively imagines the cable connection method. be able to.

FIG. 22 shows an operation example when displaying an image G relating to a predetermined abnormality. On the head-mounted display HD worn by the user 800, the image IMG of the actual measuring machine M taken by the 3D camera CM is displayed. Here, when some abnormality has occurred in the measurement head 501, the image G for notifying the abnormality is displayed by synthesizing with the image IMG of the head-mounted display HD.

In the example shown in FIG. 22, an image G prompting a warning such as "cable connection abnormality" or "limit ON" is compositely displayed on the video IMG of the actual measuring machine M. As a result, the user 800 can visually grasp what kind of abnormality is occurring at what position according to the image IMG of the actual measuring machine M.

FIG. 23 shows an example of displaying the image G of the instruction manual. On the head-mounted display HD worn by the user 800, the image IMG of the actual measuring machine M taken by the 3D camera CM is displayed. When the user 800 transmits a predetermined command to the computer 100 in this state, the image G of the instruction manual is compositely displayed in the vicinity of the image IMG of the measuring device M of the head-mounted display HD.

The user 800 can also refer to the image G of the instruction manual appearing on the display space of the head-mounted display HD while referring to the video IMG of the actual measuring machine M. Further, the page of the image G of the instruction manual can be turned by the instruction of the user 800. For example, the movement of the hand 801 of the user 800 is detected by the moving object detection unit 102 to track the movement of the hand 801. Then, when the movement of turning the page is recognized from the movement of the hand 801, the display of turning the page of the image G of the instruction manual is performed.

In addition, the user 800 can enlarge or reduce the image G of the instruction manual according to a predetermined instruction. This operation can also be performed by detecting the movement of the hand 801.

FIG. 24 is a flowchart illustrating the operation of the helper display.
The helper display operation is performed by the helper generation unit 95. The helper generation unit 95 starts the operation when the helper is called by the user 800 or when some error occurs.

First, as shown in step S101, the helper is displayed. The helper is an image corresponding to a predetermined guidance, and may be an image imitating a human shape or a character image. The helper is displayed in the virtual three-dimensional space of the head-mounted display HD.

Next, as shown in step S102, the helper response is output. For example, a door greeting or a question is given to the user 800 such as "Hello", "Error Occurred!", "May I help you?". The helper's response changes the image displayed on the head-mounted display HD, or is output as voice to the head-mounted microphone HPM.

Next, as shown in step S103, the voice command is recognized. The user 800 asks the headphone microphone HPM by voice. The command is recognized based on this voice. Next, as shown in step S104, the helper response is output. The helper outputs a response corresponding to the command instructed by the user 800. For example, an image or sound corresponding to a command of the user 800 such as "I teach you ..." or "Please ..." is output.

As shown in step S105, it is determined whether or not the helper's response is acceptable. If it is good, the process proceeds to step S106 and the helper is deleted. On the other hand, when a further response is required, the process returns to step S102 and the subsequent processing is repeated.

25 to 28 are schematic views showing a display example of the helper.
FIG. 25 shows a display example of a helper that guides the operation procedure of the measuring machine M.
First, the user 800 calls the helper. For example, the user 800 asks "OK Mitutoyo." By voice. A helper is displayed in response. The image IMG of the actual measuring machine M is displayed in the three-dimensional space of the head-mounted display HD, and a helper image (helper image HG) is displayed in the vicinity thereof.

Next, the helper image HG responds with, for example, "Hello", an image display, and a voice. Next, the user 800 asks a question by voice. For example, in order to hear the operation procedure of the measuring machine M, a question is asked by voice such as "How to ...".

Next, in response to the question of the user 800, the helper image HG responds, for example, "I teach you ..., operating it." Subsequently, the user 800 asks, for example, "Teach me step by step." In this way, guidance on the operation procedure of the measuring device M is performed by repeating the question and response between the user 800 and the helper image HG. At this time, the measuring machine image MG may be displayed and the progress of the procedure may be expressed by the measuring machine image MG.

FIG. 26 shows a display example of a helper that guides the procedure for replacing the measurement probe.
First, the user 800 calls the helper by asking, for example, "OK Mitutoyo." By voice. In response to this, the helper image HG is displayed. The helper image HG responds with, for example, "Hello", an image display, and a voice together with the display.

Next, the user 800 asks, for example, "How to change the probe." In response to the user 800's question, the helper image HG responds, for example, "Please do it this way." Along with this response, the measurement head image 501G is displayed, and the procedure for replacing the measurement probe is guided by a moving image or the like.

FIG. 27 shows a display example of a helper that guides a cable connection error.
First, the user 800 calls the helper by asking, for example, "OK Mitutoyo." By voice. In response to this, the helper image HG is displayed. The helper image HG responds with, for example, "Hello", an image display, and a voice together with the display.

Next, the user 800 asks, for example, "Teach me connection error." In response to the user 800's question, the helper image HG responds, for example, "Please check connection of this cable." Along with this response, the measuring machine image MG and the personal computer image PC-G are displayed. Further, the image G of the cable prompting the check is displayed. By referring to this image, the user 800 can visually recognize which cable connection has an error.

FIG. 28 shows a display example of the helper when an error occurs.
If any error occurs in the measurement by the measuring machine M, the helper image HG is automatically displayed. The helper image HG responds with, for example, "Error occurred." To notify the occurrence of an error.

Next, the user 800 asks, for example, "Teach me about error." The helper image HG responds to a question from the user 800 and outputs the details of the error that occurred, for example, "I teach you ...".

In this way, by displaying the helper image HG and interacting with the helper image HG, the user 800 can solve the problem as if receiving guidance directly from the serviceman.

(Fifth Embodiment)
FIG. 29 is a configuration diagram illustrating the measurement system according to the fifth embodiment.
The measurement system 1E according to the present embodiment includes a configuration in which a computer 100A, which is a user-side control device, and a computer 100B, which is a supporter-side control device, are connected to each other via a network.

A measuring machine M, an operation panel CT, a display D1, a keyboard K1, a mouse MS1, a head-mounted display HD1, a head-mounted microphone HPM1, a 3D camera CM, and a 3D sensor SR1 are connected to the computer 100A.

Further, the computer 100A includes a CPU 10A, a storage unit 20A, a calculation unit 30A, a measuring machine control unit 40, a display control unit 50A, a video input / output unit 51A, a camera control unit 52A, a stereoscopic image generation unit 53A, and a captured image synthesis unit 531A. It has a head-mounted display output unit 54A, a head-mounted microphone input / output unit 61A, an audio input / output unit 62A, a sound reproduction unit 621A, a 3D sensor input unit 70A, a position / distance / attitude recognition unit 75A, and a communication control unit 101A.

A display D2, a keyboard K2, a mouse MS2, a head-mounted display HD2, a headphone microphone HPM2, and a 3D sensor SR2 are connected to the computer 100B.

Further, the computer 100B includes a CPU 10B, a storage unit 20B, a calculation unit 30B, a display control unit 50B, a stereoscopic image generation unit 53B, a head-mounted display output unit 54B, a head-mounted microphone input / output unit 61B, an audio input / output unit 62B, and an audio reproduction unit. It has a 3D sensor input unit 70B, a position / distance / attitude recognition unit 75B, and a communication control unit 101B.

In such a measurement system 1E, the user side on which the measuring machine M is installed and the supporter side of the measuring machine M are connected to each other by a network N. On the user side, it is possible to refer to the composition of the image IMG of the measuring device M displayed on the head-mounted display HD1 and the image sent from the supporter side.

On the supporter side, it is possible to refer to the composition of the measuring machine image MG such as a graphic and the image sent from the user side in the virtual three-dimensional space displayed on the head-mounted display HD2. That is, by referring to the displays of the head-mounted displays HD1 and HD2, each of the user and the supporter who are separated from each other can perform the operation by referring to the 3D image as if they are close to each other. Will be able to.

FIG. 30 is a schematic diagram showing an operation example.
As shown in FIG. 30, the user 800 wears the head-mounted display HD1 and the head-mounted microphone HPM1. The 3D sensor SR1 detects the position of an object such as the user 800, the measuring machine M, or the work W in the real three-dimensional space on the user side. Further, the 3D camera CM provided in the head-mounted display HD1 captures 3D images of the measuring machine M, the work W, and the like. Information of the measuring machine M on the user side, information detected by the 3D sensor SR1, information acquired by the 3D camera CM, information acquired by the headphone microphone HPM1, etc. are transmitted from the communication control unit 101A to the supporter side via the network N. Will be done.

On the other hand, the supporter 900 wears the head-mounted display HD2 and the headphone microphone HPM2. The 3D sensor SR2 detects the position of an object such as the supporter 900 in the real three-dimensional space on the supporter side. The information detected by the 3D sensor SR2 on the supporter side, the information acquired by the headphone microphone HPM2, and the like are transmitted from the communication control unit 101B to the user side via the network N.

Next, a display example of the head-mounted display HD1 of the user 800 will be described. The 3D image acquired by the 3D camera CM is displayed on the head-mounted display HD1 of the user 800. For example, the head-mounted display HD1 displays images of the measuring device M, the work W, the display D1 of the computer 100A, and the like. Further, the helper image HG1 is displayed on the head-mounted display HD1 as a supporter side image.

As the helper image HG1, an image imitating the supporter 900 is used. The helper image HG1 may be a character image or the like in addition to an image imitating a human shape. The helper image HG1 changes in response to the movement of the supporter 900.

That is, the movement of the supporter 900 is detected by the 3D sensor SR2 on the supporter side, and the helper image HG1 is moved so as to follow the movement of the supporter 900 from the detection result. As a result, the user 800 can refer to the image as if the supporter 900 is present at the place where the measuring device M is located.

The supporter 900 sends advice to the user 800 with gestures and voice. The movement of the helper image HG1 referenced by the user 800 changes according to this operation. Further, the headphone microphone HPM1 of the user 800 outputs the voice from the supporter 900.

Next, a display example of the head-mounted display HD2 of the supporter 900 will be described. An image of a virtual three-dimensional space is displayed on the head-mounted display HD2 of the supporter 900 based on the information transmitted from the user side via the network N. In the virtual three-dimensional space, the measuring machine image MG used by the user 800 and the image (work image) WG of the work W are displayed. Further, the video D-IMG of the display D1 of the computer 100A of the user 800 is compositely displayed in the virtual three-dimensional space.

Further, the user image YG is displayed on the head-mounted display HD2 as a user-side image. As the user image YG, an image imitating the user 800 is used. The user image YG may be a character image or the like in addition to an image imitating a human shape. The user image YG changes according to the movement of the user 800.

That is, the movement of the user 800 is detected by the 3D sensor SR1 on the user side, and the user image YG is moved so as to follow the movement of the user 800 from the detection result. As a result, the supporter 900 can obtain the feeling as if he / she is at the measurement location of the user 800 by the measuring machine image MG, the work image WG, and the user image YG displayed in the virtual three-dimensional space.

For example, the user 800 can refer to the image of the measuring machine M displayed on the head-mounted display HD1 and the helper image HG1 synthesized with this image, and perform the measurement work while receiving the advice of the helper image HG1. ..

On the other hand, the supporter 900 refers to the measuring machine image MG in the virtual three-dimensional space displayed on the head-mounted display HD2 and the user image YG synthesized in the measuring machine image MG, and gives the user 800 an operation procedure. You can send advice such as giving instructions.

In this way, both the user 800 and the supporter 900 can communicate with each other by referring to each other's images and using voices and gestures. You will be able to proceed with the work with such a feeling.

(Sixth Embodiment)
FIG. 31 is a configuration diagram illustrating the measurement system according to the sixth embodiment.
As shown in FIG. 31, the measurement system 1F according to the present embodiment includes a computer 100 and a display device 300 connected to the computer 100 via wireless communication. A measuring machine M, an operation panel CT, a camera C1, a display D, a keyboard K, and a mouse MS are connected to the computer 100. The computer 100 includes a CPU 10, a storage unit 20, a feature point generation unit 35, a measurement result synthesis setting unit 36, a measuring machine control unit 40, a display control unit 50, an input / output control unit 60, an operation panel input / output unit 65, and a communication control unit. It has 101A.

The feature point generation unit 35 performs a process of extracting feature points from the image of the measurement object (work) acquired by the camera C1. The feature points are even parts and sides (outer lines) required to form the outer shape of the work. The measurement result synthesis setting unit 36 is a part that associates the feature points generated by the feature point generation unit 35 with the measurement results of the work. For example, the distance between two adjacent feature points is associated with the measurement result of the work corresponding to the position.

The display device 300 is provided with a display panel DP such as a camera C2 and a liquid crystal panel. The display device 300 includes a CPU 310, a storage unit 320, a feature point extraction / tracking unit 350, an input / output unit 360, a measurement result synthesis unit 370, and a communication control unit 301.

The feature point extraction / tracking unit 350 extracts feature points from the image of the work captured by the camera C2, and performs a process of tracking the extracted feature points in the image. While the work is being photographed by the camera C2, feature points are extracted and tracked.

The measurement result synthesizing unit 370 performs a process of synthesizing the measurement result with the image of the work taken by the camera C2 or the image of the work based on the image. That is, the measurement result synthesis unit 370 reads out the measurement result associated with the feature point from the computer 100 based on the feature point extracted by the feature point extraction / tracking unit 350, and converts the measurement result into a video or image of the work. Synthesize.

As a result, the display panel DP of the display device 300 displays the image or image of the work in which the measurement results corresponding to the feature points are combined. That is, when the user of the display device 300 acquires the image of the work with the camera C2, he / she can refer to the display in which the measurement result is combined with the image or the image of the work.

In the measurement system 1F according to the present embodiment, the measurement results can be referred to by a plurality of display devices 300. By acquiring the image of the work W by each display device 300, the image or image of the work and the measurement result are combinedly displayed on each display device 300.

FIG. 32 is a flowchart showing an example of the processing of the measurement result composite display.
First, as shown in step S201, the image of the work is acquired by the camera C2. Next, as shown in step S202, it is determined whether or not the feature points have been extracted. If it has not been extracted, the feature points shown in step S203 are extracted. After the feature points are extracted from the image of the work acquired by the camera C2, the feature points are tracked as shown in step S204. The feature point extraction and tracking are performed by the process of the feature point extraction / tracking unit 350.

Next, as shown in step S205, the measurement results are synthesized. Here, according to the feature points of the work to be extracted and tracked, the process of synthesizing the image of the measurement result corresponding to the feature points into the image of the work is performed. The measurement result synthesis process is performed by the measurement result synthesis unit 370.

FIG. 33 is a schematic diagram showing a composite display example of the measurement results.
In the example shown in FIG. 33, a mobile terminal is used as the display device 300. The mobile terminal is provided with a camera C2 and a display panel DP. When the work W is photographed by the camera C2, the image IMG of the work W is displayed on the display panel DP.

The feature point extraction / tracking unit 350 extracts the feature points of the work W from the image IMG of the work W taken by the camera C2, and tracks the feature points during the shooting. In the example shown in FIG. 33, feature points CP1 to CP4 of the work W are extracted and tracked.

The measurement result synthesis unit 370 synthesizes the measurement results corresponding to the feature points CP1 to CP4 extracted and tracked into the video. For example, the measurement result of the distance between the feature points CP1 and CP2 is displayed together with the dimension line between the feature points CP1 and CP2 of the video IMG of the work W. Further, the measurement result of the distance between the feature points CP2 and CP3 is displayed together with the dimension line between the feature points CP2 and CP3 of the video IMG of the work W. Further, the measurement result of the diameter of the hole obtained from the feature point CP4 is displayed together with the dimension line at the position of the hole indicated by the feature point CP4 of the image IMG of the work W.

Such a composite display is continued while the work W is being photographed by the display device 300. Even if the angle of view for shooting the work W changes, since the feature points CP1 to CP4 of the work W are tracked, the display position of the measurement result also changes so as to follow the change of the image IMG of the work W. Is displayed. Therefore, by shooting the work W at a desired angle, the user can refer to the image IMG of the work W viewed from a predetermined angle and the display of the measurement result corresponding to the image IMG.

34 (a) to 34 (e) are schematic views showing other synthetic display examples.
FIG. 34A shows a video IMG of the work W taken by the camera C2. In addition, FIG. 34 (b) shows feature points extracted and tracked from the video IMG. Here, the feature points are indicated by circles. FIG. 34 (c) shows an example in which the measurement results are compositely displayed on the video IMG. Here, the preset measurement results are compositely displayed at the positions of the feature points.

On the other hand, FIG. 34 (d) shows an example of a work image WG fitted to a CAD model based on the video IMG. That is, by extracting the feature points from the video IMG shown in FIG. 34 (b), the shooting location and the angle of the work W can be grasped. Based on this, the CAD model of the work W is fitted and the work image WG is displayed. Then, as shown in FIG. 34 (e), the measurement result is combined and displayed on the work image WG of the CAD model.

Switching between synthesizing the measurement result with the video IMG and synthesizing the measurement result with the work image WG of the CAD model is performed by the user's selection. By synthesizing the measurement result with the video IMG, the actually measured work W and the measurement result of the work W can be referred to.

In addition, if the measurement result is combined with the work image WG of the CAD model, the design information based on the CAD model and the display method of the CAD model (surface model, wireframe model, section display, etc.) can be selected and matched to the display. The measurement results can be synthesized.

In the work image WG of the CAD model, the display method such as the display color can be changed based on the difference between the design value and the measured value. For example, when the dimension exceeds the permissible range, it can be displayed in red, or the display mode can be changed between the portion within the permissible range and the portion not within the permissible range.

Further, in the measurement system 1F according to the present embodiment, the user interface device may be configured by the computer 100 and the display device 300. According to this user interface device, by shooting the image of the work W with the camera C2, the feature points of the work W are extracted, and the information corresponding to the predetermined position of the work W based on the feature points of the work W is obtained. It can be combined and displayed on the video IMG.

(7th Embodiment)
FIG. 35 is a configuration diagram illustrating the measurement system according to the sixth embodiment.
As shown in FIG. 35, the measurement system 1G according to the present embodiment includes a computer 100 and a display device 300 connected to the computer 100 via wireless communication. A measuring machine M, an operation panel CT, a camera C1, a display D, a keyboard K, and a mouse MS are connected to the computer 100. The computer 100 includes a CPU 10, a storage unit 20, a marker recognition registration unit 38, a marker measurement result correspondence setting unit 39, a measuring machine control unit 40, a display control unit 50, an input / output control unit 60, an operation panel input / output unit 65, and communication control. It has a part 101A.

The marker recognition registration unit 38 recognizes marker-specific information (shape recognition result, identification information, etc.) from the marker image acquired by the camera C1 and registers the marker. Here, as the marker, a marker such as a figure or a photograph that can be distinguished from others is used. The marker measurement result correspondence setting unit 39 is a portion for registering the marker registered by the marker recognition registration unit 38 in association with a predetermined measurement result. These registration information are stored in the storage unit 20.

The display device 300 is provided with a display panel DP such as a camera C2 and a liquid crystal panel. The display device 300 includes a CPU 310, a storage unit 320, an input / output unit 360, a marker recognition unit 380, a marker measurement result synthesis unit 390, and a communication control unit 301.

The marker recognition unit 380 performs a process of recognizing information unique to the marker from the image of the marker captured by the camera C2. The marker measurement result synthesis unit 390 performs a process of synthesizing the measurement result associated with the marker-specific information recognized by the marker recognition unit 380 into the background image.

That is, the marker measurement result synthesis unit 390 reads the measurement result associated with the information from the computer 100 based on the marker-specific information recognized by the marker recognition unit 380, and synthesizes the measurement result with the background image.

In such a measurement system 1G, the background image and the measurement result synthesized at the position of the marker are displayed on the display panel DP of the display device 300. That is, when the user of the display device 300 acquires the image of the marker with the camera C2, the user can refer to the display in which the measurement result associated with the marker is combined with the background image.

In the measurement system 1G according to the present embodiment, the measurement results can be referred to by a plurality of display devices 300. By acquiring the image of the marker by each display device 300, the background and the measurement result are combinedly displayed on the display panel DP of each display device 300.

FIG. 36 is a flowchart illustrating the marker registration and setting operation. This operation is performed by the computer 100.
First, as shown in step S301, the camera C1 acquires the image of the marker. Next, as shown in step S302, the image of the marker is recognized. For example, a process of recognizing a photographed marker having a specific shape, a specific figure, a photograph, or the like, assigning identification information peculiar to the marker to the marker, and registering the marker is performed. This process is performed by the marker recognition registration unit 38.

Next, as shown in step S303, a process of setting the association between the marker and the measurement result is performed. That is, the process of associating the identification information of the marker recognized in the previous step S302 with the predetermined measurement result is performed. This process is performed by the marker measurement result correspondence setting unit 39.

Next, as shown in step S304, the marker-specific identification information and the measurement result associated with the identification information are registered. The association between the marker-specific identification information and the measurement result is stored in the storage unit 20 as marker setting information. This process is performed by the marker measurement result correspondence setting unit 39.

FIG. 37 is a flowchart illustrating marker recognition and composite display. This operation is performed by the display device 300.
First, as shown in step S401, marker setting information is received from the storage unit 20 of the computer 100 via wireless communication. Next, as shown in step S402, the image of the marker taken by the camera C2 is acquired. Next, as shown in step S403, the identification information peculiar to the marker is recognized from the acquired image of the marker. This process is performed by the marker recognition unit 380.

Next, as shown in step S404, a process of synthesizing the measurement results associated with the markers is performed. That is, from the marker-specific identification information recognized in the previous step S403, the measurement result associated with this identification information is read from the marker setting information. Then, a process of synthesizing the measurement result with the background image captured by the camera C2 is performed. This process is performed by the marker measurement result synthesis unit 390.

Next, as shown in step S405, it is determined whether or not the recognition and composite display of all the marks are completed. If it is not completed, the process returns to step S402 and the subsequent processing is repeated. As a result, the measurement results are compositely displayed corresponding to all the markers photographed by the camera C2.

38 to 40 are schematic views showing a display example of a measurement result by marker recognition.
FIG. 38 shows an example in which a composite display is performed on a mobile terminal. In this example, a mobile terminal is used as the display device 300. Further, the paper PP is provided with a plurality of markers MK1 to MK5.

The markers MK1 to MK5 may be a specific figure or a specific photograph. Each marker MK1 to MK5 is associated with a marker-specific identification information and a measurement result. For example, the markers MK1, MK2, and MK3 are associated with a predetermined measurement result of the first measuring machine M1. Further, the markers MK4 and MK5 are associated with a predetermined measurement result of the second measuring machine M2. These correspondences are stored in the storage unit 20.

The user takes a picture of the paper PP by the camera C2 of the mobile terminal which is the display device 300. When the images of the markers MK1 to MK5 are captured by this shooting, the identification information of the markers MK1 to MK5 is recognized. Then, based on this identification information, the measurement results associated with the markers MK1 to MK5 are read out from the storage unit 20 of the computer 100.

Then, the image of the measurement result associated with each marker MK1 to MK5 is synthesized in the display area of the photographed markers MK1 to MK5 and displayed on the display panel DP. For example, an image of the measurement result by the first measuring machine M1 is combined with each of the markers MK1 to MK3, and an image of the measurement result by the second measuring machine M2 is combined with each of the markers MK4 and MK5.

The image of the measurement result is displayed in various formats such as a tabular format and a graph format. When synthesizing the images, the size and angle may be adjusted and displayed so as to match the regions indicated by the markers MK1 to MK5. Further, the image of the measurement result may be compositely displayed as a three-dimensional image on the image of the flat paper PP so as to be raised.

In this way, the user can refer to the measurement result laid out at the positions of the markers MK1 to MK5 only by photographing the paper PP to which the markers MK1 to MK5 are attached with the mobile terminal.

39 and 40 show another display example.
First, FIG. 39 shows a display example before the composite display. Similar to the previous example, the paper PP is provided with a plurality of markers MK1 to MK5. Further, in this example, a glasses-type display device 300 is used.

In the display device 300, a display panel DP is provided on the lens portion of the glasses. Further, the camera C2 is provided on the temple portion of the glasses. By wearing the glasses-type display device 300, the user can take an image in front of the glasses with the camera C2 and can refer to the information displayed on the display panel DP of the lens portion.

FIG. 40 shows a display example after the composite display. That is, when the markers MK1 to MK5 are photographed by the camera C2 of the glasses-type display device 300, the identification information of the markers MK1 to ML5 is recognized. Then, the measurement results associated with the markers MK1 to MK5 are read out from the storage unit 20 based on the identification information. Then, the image of the measurement result associated with each marker MK1 to MK5 is synthesized in the display area of the photographed markers MK1 to MK5 and displayed on the display panel DP.

A user wearing the glasses-type display device 300 can refer to a display in which the measurement result is combined with the background image displayed on the display panel DP of the lens portion of the glasses. In this example, the measurement result of the first measuring machine M1 is synthesized in the display area of the markers MK1, MK2 and MK3, and the measurement result of the second measuring machine M2 is synthesized in the display area of the markers MK4 and MK5.

By changing the setting of the association between the markers MK1 to MK5 and the measurement result, different measurement results can be displayed even if the markers MK1 to MK5 are the same. That is, different measurement results can be displayed by creating one layout form of the markers MK1 to MK5.

Further, a plurality of measurement results may be associated with one marker. In this case, the user may arbitrarily switch which measurement result among the plurality of measurement results is to be combined and displayed corresponding to one marker. As a result, by capturing an image of one marker, different measurement results can be displayed by switching to the position of the marker.

The image of the measurement result may be a moving image. As a result, the moving image is synthesized on the image of the paper PP, and the simple paper PP can be treated like a virtual moving image display device.

Further, in the measurement system 1G according to the present embodiment, the user interface device may be configured by the computer 100 and the display device 300. According to this user interface device, by shooting the images of the markers MK1 to MK5 with the camera C2, the information corresponding to the markers MK1 to MK5 is read out, and while the images of the markers MK1 to MK5 are being photographed, The read information can be displayed on the display panel DP.

As described above, according to the measurement systems 1A, 1B, 1C, 1D, 1E, 1F and 1G according to the present embodiment, it is possible to improve the operability of the measuring machine M.

Although the present embodiment has been described above, the present invention is not limited to these examples. For example, in a system using a head-mounted display HD to which a 3D camera CM is attached, the method is not limited to a method of synthesizing and displaying a real space image taken by the 3D camera CM and computer graphics as described above. For example, the computer graphics may be displayed on the head-mounted display HD in accordance with the real space coordinates. In this case, instead of the 3D camera CM, or in addition to the 3D camera CM, a 3D sensor is mounted on the head-mounted display HD, the coordinate system of the real space is grasped in real time by the 3D sensor, and the virtual space is created by computer processing. It may be constructed and the computer graphics may be repositioned and displayed in real time as if they actually existed in the space. In addition, those skilled in the art appropriately adding, deleting, or changing the design of each of the above-described embodiments, and those in which the features of each embodiment are appropriately combined are also provided with the gist of the present invention. As long as it is, it is included in the scope of the present invention.

1A, 1B, 1C, 1D, 1E, 1F, 1G ... Measurement system 10, 10A, 10B, 310 ... CPU
20, 20A, 20B, 320 ... Storage unit 30, 30A, 30B ... Calculation unit 35 ... Feature point generation unit 36 ... Measurement result synthesis setting unit 38 ... Marker recognition registration unit 39 ... Marker measurement result correspondence setting unit 40 ... Measuring machine control Units 50, 50A, 50B ... Display control units 51, 51A ... Video input / output unit 52 ... 3D camera control unit 52A ... Camera control unit 53, 53A, 53B ... Stereoscopic image generation unit 54, 54A, 54B ... Head mount display output unit 55 ... Control generation unit 60 ... Input / output control unit 61, 61A, 61B ... Headphone microphone input / output unit 62, 62A, 62B ... Audio input / output unit 65 ... Operation panel input / output unit 70, 70A, 70B ... 3D sensor input unit 75 , 75A, 75B ... Attitude recognition unit 76 ... Control input recognition unit 80 ... Projection figure generation unit 85 ... Projector output unit 90 ... Remote control unit 95 ... Helper generation unit 100, 100a, 100B ... Computer 101 ... Command setting unit 101A, 101B , 301 ... Communication control unit 102 ... Motion detection unit 103 ... Command execution unit 104 ... Measurement operation storage unit 201 ... Image storage unit 300 ... Display device 350 ... Tracking unit 360 ... Input / output unit 370 ... Measurement result synthesis unit 380 ... Marker recognition Unit 390 ... Marker measurement result synthesis unit 501 ... Measurement head 501G ... Measurement head image 531 and 531A ... Captured image synthesis unit 621, 621A, 621B ... Sound reproduction unit 800 ... User 801 ... Hand 802 ... Foot 900 ... Supporters B1 to B7 ... Button setting area BG ... Button image C1, C2, CM1 ... Camera CM ... 3D camera CM1 ... Camera CP1 to CP4 ... Feature points D, D1, D2 ... Display D-IMG, IMG ... Video DP ... Display panel FL ... Floor G , G1 to G7 ... Image HD, HD1, HD2 ... Head mount display HG, HG1 ... Helper image HPM, HPM1, HPM2 ... Headphone microphone K, K1, K2 ... Keyboard M, M1, M2 ... Measuring machine MG ... Measuring machine image MK1 ~ MK5 ... Marker MS, MS1, MS2 ... Mouse N ... Network P1 to P4 ... Even part PC-G ... Image PC-IMG ... Video PP ... Paper PR ... Projector SR, SR1, SR2 ... 3D sensor W ... Work WB ... White Board WG ... Work image YG ... User image

Claims (5)

User side control device and
A supporter-side control device connected to the user-side control device via a network,
It is a control system equipped with
The user-side control device is
Connected to a measuring machine that measures the object to be measured,
The first moving object detection unit that detects the user's movement in the real three-dimensional space,
A first display unit that displays a helper image that changes in response to the supporter's movements based on the information about the supporter's movements sent from the supporter side control device.
A first transmission unit that sends information about the user's operation detected by the first moving object detection unit to the supporter side control device via the network, and a first transmission unit.
A measuring machine control unit that controls the measuring machine,
With
The supporter side control device is
A second moving object detection unit that detects the movement of the supporter in a real three-dimensional space,
A second display unit that displays an image of the measuring machine in a virtual three-dimensional space together with a user image that changes in response to the user's operation based on information about the user's operation sent from the user-side control device.
A display control unit that moves the display position of the measuring machine in the virtual three-dimensional space according to the movement of the supporter detected by the second moving object detection unit.
A second transmission unit that sends information about the operation of the supporter detected by the second moving object detection unit to the user-side control device via the network, and a second transmission unit.
Equipped with a,
The user-side control device is
The measurement procedure of the measuring machine is defined so as to move the measuring machine along the display position of the image of the measuring machine moved by the supporter in the virtual three-dimensional space displayed on the second display unit. A control system characterized by further equipped with a measurement motion storage unit for recording measurement motions. The user-side control device is
Further provided with an imaging unit for acquiring an image in the actual three-dimensional space,
The first transmission unit transmits the video imaged by the imaging unit to the supporter side control device.
The control according to claim 1, wherein in the supporter-side control device, the second display unit displays an image obtained by synthesizing the video and the user image sent from the user-side control device. system. The control system according to claim 1 or 2 , wherein the measuring machine control unit controls the measuring machine based on the measuring operation stored in the measuring operation storage unit. The control system according to any one of claims 1 to 3, wherein the first display unit displays an image of the measuring device together with the helper image. When the supporter moves the display position of the image of the measuring machine in the virtual three-dimensional space displayed on the second display unit, the image of the measurement object is moved according to the position of the image of the measuring machine. The control system according to any one of claims 1 to 4, wherein the control system is synthesized and displayed.