WO2020008936A1 - Image processing device, image processing method, system, and method for manufacturing article - Google Patents

Abstract

In the present invention, a first model indicating a target of measurement by a measurement instrument and a second model indicating a measurement range of the measurement instrument are displayed on a display screen. The position and orientation of the second model are changed in accordance with a user operation. The position and orientation of a region having a fixed position and orientation relative to the measurement instrument in the state where the position and orientation of the second model have been changed are obtained and output.

Description

Image processing apparatus, image processing method, system, article manufacturing method

The present invention relates to measurement technology.

位置 Before capturing an image of an object using the imaging device, it is necessary to set a positional relationship between the object and the imaging device in advance. This is because the field of view of the imaging device is defined by various factors including the focal length of the lens of the imaging device, the size of the imaging device such as a CCD, and the like, and the target object needs to be in that region. That's why. There is a technique for making this setting on a simulation device.

Patent Literature 1 discloses a technique that aims to know in advance by simulation what kind of area can be photographed before installing a camera in a real space. In this technique, an area image of an area to be a camera installation area is displayed, and a camera indicator is arranged and displayed on the area image. Then, an imaging range and a blind spot are presented based on information such as the type of camera and the angle of view.

According to Patent Literature 2, when viewing a video at a distance from the shooting site of a video camera, it is unclear where the video camera is looking from, and it is difficult to grasp the shooting site from the video and to give an instruction to the video camera. The task is difficult. On the other hand, in Patent Literature 2, a real world including an existing position of a video camera and a viewable area is prepared in advance in a virtual space, and the position, the line of sight, and the angle of view of the real world video camera are prepared in the virtual space. This problem is solved by displaying the area three-dimensionally in real time.

The purpose of the technology disclosed in Patent Literatures 1 and 2 is to know where in a real space an imaging device is to be installed to place a desired target or a desired area in a visual field.

JP 2009-105802 A JP 2008-5450A

On the other hand, for example, in a three-dimensional measuring device having an imaging device, there is a situation in which a sufficient effect cannot be obtained by the technology described in the background art. In general, in a measuring instrument, a predetermined area suitable for measurement is defined separately from a visual field area, and the area is naturally invisible in a real space. There may be a part with even higher accuracy in this predetermined area, and it is difficult to adjust the measurement target so as to come to that area or a specific part in the area.

例 An example of the actual adjustment method is described below. First, the user moves the measuring instrument to an approximate position where the measurement target enters an area suitable for measurement. Then, in order to know the actual relative positional relationship between the measuring instrument and the measurement target, the distance and the like are measured with a scale or the like. Then, the measuring instrument is moved based on the result, and the confirmation is performed again on the scale. Thereafter, the image photographed at the position of the measuring instrument is confirmed, and if it is not within the angle of view, for example, the adjustment and confirmation are performed again.

３The above-described three-dimensional measuring device may be used, for example, fixed to a robot or the like. In this case, there are further difficulties and troubles. First, the user does not control the position and orientation of the measuring device itself, but controls the position and orientation of the measuring device by operating the position and orientation of the robot equipped with the measuring device with a controller or the like. That is, even if the position and orientation of the measuring instrument are known, the position and orientation of the robot must be controlled so that the measuring instrument itself becomes the position and orientation. Further, the operation of the robot is often performed outside the safety fence, and when checking on a scale after the operation of the robot, it is necessary to enter and exit the safety fence, so that the work efficiency may be reduced. As described above, in setting the positional relationship between the measuring instrument and the measurement target, the technique described in the background art cannot provide a sufficient effect. The present invention provides a technique for quickly and easily setting the position and orientation of a measuring instrument for measuring a measurement target.

One embodiment of the present invention is a display unit that displays, on a display screen, a first model representing an object to be measured by a measuring instrument, a second model representing a measuring range of the measuring instrument, and A change unit for changing the position and orientation of the second model, and a position and orientation of a part whose position and orientation relative to the measuring device are constant when the position and orientation of the second model are changed. Output means.

According to the configuration of the present invention, it is possible to quickly and easily set the position and orientation of the measuring instrument for measuring the measurement target.

Other features and advantages of the present invention will become apparent from the following description with reference to the accompanying drawings. In the accompanying drawings, the same or similar components are denoted by the same reference numerals.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings are included in and constitute a part of the specification and illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
FIG. 1 is a diagram illustrating a configuration example of a system. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. FIG. 9 is a view showing a display example of a GUI. 4 is a flowchart of a process performed by the image processing apparatus 1. FIG. 2 is a diagram illustrating a control system including a measurement device and a robot arm.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific embodiments having the configuration described in the claims.

[First Embodiment]
First, a configuration example of a system according to the present embodiment will be described with reference to FIG. The measuring device 2 is a three-dimensional shape measuring device for measuring three-dimensional information relating to the measurement target 3 such as the position, posture, shape, and the like of the measurement target (target object) 3 mounted on the mounting table 10. The measuring device 2 includes, for example, an active type that includes a projection unit that projects pattern light onto the measurement target 3 and an imaging unit that captures the measurement target 3 onto which the pattern light is projected. Some of the measuring instruments 2 are of a passive type having a plurality of imaging devices for imaging the measurement target 3 at different positions and orientations.

様 々 Various objects (objects of various sizes and various materials) can be applied as the measurement target 3. If the measurement target 3 is relatively small, it may be placed in a bulk state, and if it is relatively large, it may be placed with only one individual. When the measurement targets 3 are placed in bulk, there are many cases where the measurement targets 3 are stored in containers such as pallets. In the present embodiment, the measurement target 3 includes not only the measurement target by the measuring device 2 but also a container in which the measurement target by the measuring device 2 is stored. In the present embodiment, it is assumed that the measurement target 3 is a small work that is piled on a pallet. As shown in FIG. 1, it is assumed that the measurement target 3 is mounted on an installation table 10 and is installed obliquely with respect to the direction of gravity.

The measuring instrument 2 has a measuring range 4 defined by its performance and specifications. For example, when the measuring device 2 includes an imaging device, a desired measurement cannot be performed (a desired measurement result cannot be obtained) unless at least a part of the measurement target 3 is included in the visual field of the imaging device. In addition, if the measurement target 3 is within the field of view of the imaging device, the relative position and orientation between the measurement device 2 and the measurement target 3 is good (that is, the measurement accuracy of the measurement target 3 by the measurement device 2 is sufficient). Not always. In other words, even if the measurement target 3 is in the field of view of the imaging device, there is a position or orientation of the imaging device suitable for measurement from the viewpoint of defocus and aberration. Further, for example, in the above-mentioned active type three-dimensional shape measuring device, there is a good position and orientation of the measurement target 3 which can be considered from the performance and specifications of an optical system for projecting a pattern. In such a three-dimensional shape measuring instrument, an area having good recognition performance and good accuracy is often defined. In the present embodiment, a truncated quadrangular pyramid having an area of 500 × 500 mm at a position 500 mm far from the measuring device 2 as a bottom surface and an area of 300 × 300 mm at a position of 300 mm far from the measuring device 2 as a top surface Is defined as a measurement range 4. In the present embodiment, it is assumed that the measurement range 4 is an area where the measuring instrument 2 can achieve a certain measurement performance. That is, the relative position and orientation (position and orientation relationship) between the measuring device 2 and the measurement range 4 is constant unless the parameters (focal length and the like) of the measuring device 2 are changed.

(4) In order to measure the measurement target 3 with high accuracy, the user performs adjustment so that the measurement target 3 matches a specific portion of the measurement range 4. Various states are assumed as the state of the workpieces 3 to be measured in bulk, and there are various patterns in the height position of the surface workpieces stacked in bulk. In this case, the user tries to make the bottom surface of the pallet containing the measurement target 3 coincide with the far end of the measurement range 4, that is, the above-described 500 × 500 mm area. With such a setting, the workpieces stacked thereon enter the truncated quadrangular pyramid-shaped area, and even if the state of the bulk loading is high, the workpieces existing on the bulk loading surface are farther than 300 mm farther than the measuring instrument 2. This allows the work to be contained in the measurement range 4.

In the present embodiment, as shown in FIG. 1, the measuring device 2 is attached (fixed) to the robot arm 12 via the mounter 11 and the like. The robot arm 12 is fixed to a robot base 13, and the position and posture of a specific portion of the robot arm 12 such as the movable part 5 are controlled by a robot controller 14. A command related to the control is transmitted from, for example, the programming pendant 15. When the user operates the programming pendant 15 to input an instruction to change the position or posture of the movable unit 5, the change instruction is transmitted to the robot controller 14, and the robot controller 14 changes the position or posture of the movable unit 5 according to the change instruction. change. Unless otherwise specified, the position and orientation handled below. It is assumed that the position and orientation are in a coordinate system (reference coordinate system) based on the position and orientation of the robot base 13. The reference coordinate system is, for example, a coordinate system in which the position of the robot base 13 is set as an origin, and three axes orthogonal to the origin are set as an X axis, a Y axis, and a Z axis, respectively. An example of the movable portion 5 is a flange portion at the tip of the robot arm 12. Alternatively, when a gripping mechanism is fixedly connected to the flange portion as a tool for gripping the measurement target 3 as shown in FIG. 1, the tip of the tool may be the movable portion 5. In this embodiment, the tip of the tool is the movable part 5.

Here, since the measuring instrument 2 and the movable section 5 are fixedly attached to the robot arm 12, the relative position and orientation between the measuring instrument 2 and the movable section 5 are independent of the position and orientation of the robot arm 12. Constant. The “relative position and orientation between the measuring device 2 and the movable unit 5” can be acquired based on design information of the measuring device 2, the mounter 11, the tool, and the like. A method is also known in which a marker or the like is photographed from a plurality of viewpoints with the measuring device 2 and is obtained by performing arithmetic processing.

It is assumed that the “relative position and orientation between the measuring device 2 and the movable unit 5” is stored in the storage unit 6 of the image processing apparatus 1. The image processing apparatus 1 further includes a calculation unit 9, a display unit 7, and an input unit 8.

The operation unit 9 includes one or more CPUs, processors, operation processing circuits, and the like. The arithmetic unit 9 controls the overall operation of the image processing apparatus 1 by executing processing using a computer program and data stored in the storage unit 6, and performs various operations described below as being performed by the image processing apparatus 1. Execute or control the process. For example, the calculation unit 9 executes or controls display control of a GUI (graphical user interface) described below and processes in accordance with an operation input to the GUI.

The storage unit 6 stores, in addition to the “relative position and orientation between the measuring device 2 and the movable unit 5”, information that the image processing apparatus 1 handles as known information in the following description. . In addition, the storage unit 6 stores computer programs and data for causing the arithmetic unit 9 to execute or control each process described below, which is performed by the image processing apparatus 1.

The input unit 8 is configured by a user interface such as a keyboard, a mouse, and a touch panel screen, and can input various instructions to the arithmetic unit 9 by operating the user.

The display unit 7 is configured by a liquid crystal screen, a touch panel screen, or the like, and can display a processing result of the calculation unit 9 as an image, characters, or the like. For example, the display unit 7 can display a GUI described below.

Next, FIG. 16 is a flowchart showing the same processing performed by the image processing apparatus 1 to obtain the position and orientation of the movable unit 5 such that the measurement target 3 falls within a desired area within the measurement range 4 measured by the measuring device 2. It will be described according to.

Here, it is assumed that the GUI illustrated in FIG. 2 is displayed on the display screen of the display unit 7 when the processing according to the flowchart in FIG. 16 is started. The display area 17 is an area for displaying an image of a “virtual space defined by a coordinate system (virtual coordinate system) that matches the above-described reference coordinate system” viewed from a viewpoint having a specified position and orientation. In the display area 17, an icon 18 representing each axis (X axis, Y axis, Z axis) of the virtual coordinate system is displayed.

In step S1601, the calculation unit 9 determines whether or not the “measurement target display” button 19 has been instructed by the user operating the display screen of the input unit 8 or the display unit 7 (already instructed). As a result of this determination, if the “measurement object display” button 19 has been instructed, the process proceeds to step S1602. If the “measurement object display” button 19 has not been instructed, the process proceeds to step S1613.

In step S1602, the calculation unit 9 generates a measurement target model that is a three-dimensional model imitating the geometric shape of the measurement target 3, and sets the generated measurement target model as the position and orientation of the measurement target model. The measurement target position and orientation are arranged in the virtual space. A specified position and orientation is set as the initial value of the position and orientation of the measurement target, and the position and orientation of the measurement target are updated in accordance with a user operation in step S1605 described below. In addition, the data of the measurement target model is created in advance and registered in the storage unit 6, and the calculation unit 9 generates the measurement target model based on the data of the measurement target model.

In step S1603, the calculation unit 9 generates an image of the “measurement target model arranged in step S1602” viewed from a viewpoint having a specified position and orientation, and displays the image in the display area 17. FIG. 3 shows a display example of the GUI in step S1603. As shown in FIG. 3, the display area 17 displays an image of the measurement target model 23 viewed from a viewpoint having a specified position and orientation. The measurement target position / posture is displayed in the measurement target position / posture column 21. In FIG. 3, an X coordinate position “0”, a Y coordinate position “300.0”, and a Z coordinate position “100.0” are displayed as measurement target positions (the unit is “mm”). Further, as measurement target postures (units are “degrees”), Rx (angle around the X axis in the virtual coordinate system) “0”, Ry (angle around the Y axis in the virtual coordinate system) “0”, Rz (virtual coordinates) (Angle around the Z axis in the system) “0” is displayed.

When there are a plurality of measurement targets 3, information on each measurement target 3 may be displayed in a list on the GUI, and selection of the measurement target 3 for displaying the measurement target model may be accepted. That is, the measurement target model of the measurement target 3 and the measurement target position and orientation of the measurement target model corresponding to the information selected by the user via the display screen of the input unit 8 or the display unit 7 from the information displayed in the list. It may be displayed on a GUI as illustrated in FIG.

In step S1604, the calculation unit 9 determines whether a change instruction for changing the measurement target position or the measurement target posture has been input. The user operates the input unit 8 to press a key for changing the position or orientation, or to perform a touch operation or a drag operation on the display area 17 to issue an instruction to change the measurement target position or the measurement target posture. Can be entered. Alternatively, the user may operate the input unit 8 or the display screen of the display unit 7 to directly change the numerical value in the column 21 of the measurement target position and orientation to change the position and orientation of the measurement target model.

It is assumed that the user has previously acquired the approximate position and orientation of the measurement target 3 in the reference coordinate system in the real space. The “rough position and orientation of the measurement target 3 in the reference coordinate system in the real space” is assumed to be obtained by measuring using a measure or the like in the real space, or estimated from the position and orientation of the installation table 10. Is done. However, the user operates the display screen of the input unit 8 or the display unit 7 to set “approximate position and orientation of the measurement target 3 in the reference coordinate system in the real space” as the measurement target position and orientation.

If the result of this determination is that an instruction to change the measurement target position or measurement target orientation has been input, the process proceeds to step S1605; otherwise, the process proceeds to step S1606.

In step S1605, the calculation unit 9 updates the measurement target position and the measurement target posture in accordance with the instruction to change the measurement target position and the measurement target posture. When the position and orientation of the measurement target model 23 are changed in the GUI of FIG. 3, an image of the measurement target model 23 arranged with the changed position and orientation is generated and displayed in the display area 17 as shown in FIG. .

Note that a prescribed position and orientation is set as the initial value of the position and orientation of the “viewpoint”, and the user operates the input unit 8 or performs a touch operation, a drag operation, or the like on the display area 17. , The position and orientation of the viewpoint can be changed. When the position and orientation of the viewpoint are changed, the relative position and orientation of the viewpoint with respect to the virtual coordinate system are changed. Therefore, the icon 18 is updated in the change. That is, the icon 18 in which the direction of each axis of the virtual coordinate system is updated is generated and displayed so as to indicate the relative posture between the viewpoint and the virtual coordinate system.

In step S1606, the arithmetic unit 9 determines whether the “measurement device display” button 20 has been instructed by the user operating the display screen of the input unit 8 or the display unit 7 (already instructed). As a result of this determination, when the “measurement device display” button 20 is instructed, the process proceeds to step S1607, and when the “measurement device display” button 20 is not instructed, the process proceeds to step S1613.

In step S1607, the calculation unit 9 generates a measuring instrument model that is a three-dimensional model imitating the geometric shape of the measuring instrument 2, and sets the generated measuring instrument model as the position and orientation of the measuring instrument model. The measurement device is placed in the virtual space with the position and orientation. A prescribed position and orientation is set as an initial value of the position and orientation of the measuring instrument, and the position and orientation of the measuring instrument are updated in accordance with a user operation in step S1610 described below. The data of the measuring instrument model is created in advance and registered in the storage unit 6, and the calculating unit 9 generates the measuring instrument model based on the data of the measuring instrument model. The calculation unit 9 generates a measurement range model that is a three-dimensional model representing the measurement range 4 of the measuring device 2. Then, the arithmetic unit 9 arranges the generated measurement range model in the virtual space with the measurement range position and orientation obtained by adding “the relative position and orientation of the measurement range 4 with respect to the measurement device 2” to the measurement device position and orientation. I do. That is, no matter what the position and orientation of the measuring instrument model is, the relative position and orientation between the measuring instrument model and the measurement range model does not change. It is assumed that this “relative position and orientation of the measurement range 4 with respect to the measuring device 2” is registered in the storage unit 6 in advance. The data of the measurement range model is created in advance and registered in the storage unit 6, and the calculation unit 9 generates a measurement range model based on the data of the measurement range model. In addition, the arithmetic unit 9 obtains the position and orientation of the movable unit 5 by adding the “relative position and orientation between the measuring unit 2 and the movable unit 5” to the position and orientation of the measuring unit 2.

In step S1608, the arithmetic unit 9 generates an image of the measuring instrument model arranged in step S1607 as viewed from the above viewpoint and an image of the measuring range model arranged in step S1607 as viewed from the above viewpoint, and generates each of the generated images. Is displayed in the display area 17. FIG. 5 shows a display example of the GUI in step S1608. As shown in FIG. 5, in the display area 17, an image of the measuring instrument model 24 and an image of the measurement range model 25 are displayed in addition to the image of the measurement target model 23 viewed from the viewpoint. That is, an image (virtual space image) of the virtual space in which the measurement target model, the measuring instrument model, and the measurement range model are arranged is generated from the above viewpoint and displayed in the display area 17.

{Circle around (2)} The position / posture of the movable unit 5 obtained in step S1607 is displayed in the column 22 of the movable unit position / posture. In FIG. 5, the X-coordinate position “87.2”, the Y-coordinate position “250.2”, and the Z-coordinate position “230.0” are displayed as the positions of the movable part 5 (the unit is “mm”). Further, as the posture (unit is “degrees”) of the movable portion 5, Rx (angle around the X axis in the virtual coordinate system) “178.0”, Ry (angle around the Y axis in the virtual coordinate system) “1.2 , Rz (the angle around the Z axis in the virtual coordinate system) “0.4”.

In the case where there are a plurality of measuring instruments 2, information on each measuring instrument 2 may be displayed in a list on the GUI, and selection of the measuring instrument 2 for displaying the measuring instrument model may be accepted. In other words, the measurement device model of the measurement device 2 corresponding to the information selected by the user via the display screen of the input unit 8 or the display unit 7 from the information displayed in the list, and the measurement range of the measurement range 4 of the measurement device 2 The model may be displayed on a GUI as illustrated in FIG.

In step S1609, the calculation unit 9 determines whether or not a change instruction for changing the position or orientation of the measuring instrument model or the measurement range model has been input by the user operating the display screen of the input unit 8 or the display unit 7. to decide.

When the measurement target model, measurement device model, and measurement range model are displayed, the user moves the measurement device model, measurement range model, and hand model so that part or all of the measurement target model fits inside the measurement range model. Operation (adjustment). For example, suppose that the pallet bottom of the measurement target 3 is to be aligned with the far end of the measurement range 4. In this case, first, the measuring instrument model and the measuring range model are moved such that the bottom surface of the truncated pyramid of the measuring range model coincides with the pallet bottom surface. Then, the truncated quadrangular pyramid of the measurement range model is moved in a direction parallel to the pallet bottom surface, and adjustment is made, for example, so that the center of the bottom surface of the quadrangular pyramid and the center of the pallet bottom surface approximately match. The operation for changing the position and orientation of the measuring instrument model and the measurement range model is the same as the above-described operation for changing the position and orientation of the measurement target model. For the above operation, the measurement range model is a translucent model, a wire frame model, or a model in which the measurement target model is not hidden by the measurement range model even if it overlaps with the measurement target model. Desirably. Further, the display form of each model is not limited to a specific display form as long as the same purpose can be achieved.

If the result of this determination is that an instruction to change the position or orientation of the measuring instrument model or the measurement range model has been input, the process proceeds to step S1610; otherwise, the process proceeds to step S1613.

In step S1610, if there is an instruction to change the measuring instrument position or the measuring instrument attitude, the arithmetic unit 9 updates the measuring instrument position or the measuring instrument attitude in accordance with the instruction to change the measuring instrument position or the measuring instrument attitude. For example, the measurement range position and the measurement range posture are updated according to the change instruction.

When the measuring instrument position / posture is updated, the arithmetic unit 9 measures the measuring range position / posture by adding “the relative position / posture of the measuring range 4 with respect to the measuring instrument 2” to the updated measuring instrument position / posture. Update to range position and orientation.

On the other hand, when the measurement range position / posture is updated, the arithmetic unit 9 calculates the measurement device position / posture by subtracting “the relative position / posture of the measurement range 4 with respect to the measurement device 2” from the updated measurement range position / posture. Update to container position and orientation.

That is, as described above, the relative position and orientation between the measuring instrument model and the measurement range model are maintained even if the position and orientation of the measuring instrument model and the measurement range model are changed. However, when one position and orientation is changed, the calculation unit 9 updates the other position and orientation so that the relative position and orientation are maintained.

If the measuring instrument position and orientation have been updated, in the next step S1607, the measuring instrument model is arranged in the virtual space using the updated measuring instrument position and orientation. Similarly, when the measurement range position and orientation are updated, in the next step S1607, the measurement range model is arranged in the virtual space with the updated measurement range position and orientation.

For example, when the position and orientation of the measuring instrument model 24 and the measurement range model 25 are changed in the GUI of FIG. 5, as shown in FIG. 6, the image of the measuring instrument model 24 and the measurement range model arranged in the changed position and orientation are changed. Twenty-five images are generated and displayed in the display area 17. In FIG. 6, the measurement target model 23 is included in the measurement range model 25.

Then, the user operates the input unit 8 or the display screen of the display unit 7 to move the measuring instrument model or the measurement range model, and the inclusion state of the measurement target model with respect to the measurement range model becomes a desired inclusion state (for example, the measurement range). (When all of the measurement target models fit inside the model). In this case, the user uses the programming pendant 15 to input the position and orientation displayed in the column 22 of the movable unit position and orientation at this time, and issues an instruction to move the movable unit 5 to the position and orientation using the programming pendant 15. Enter Thereby, the robot controller 14 operates the robot arm 12 to move the position and orientation of the movable unit 5 to the position and orientation input using the programming pendant 15. Thereby, the measuring device 2 can be moved so that the measurement target 3 has a desired position and orientation with respect to the measurement range 4.

Actually, the error between the position and orientation of the measurement target model set in the virtual space and the position and orientation of the measurement target 3 in the real space, and the error between the measurement unit 2 and the movable unit 5 stored in the storage unit 6 There may be a case where a slight shift occurs due to various factors such as an error of “relative position and orientation”. In this case, after the movement, photographing or measurement is performed using the measuring device 2 to check whether the measuring device 2 and the measurement target 3 have a desired relative position and orientation relationship, and move the movable section 5 to move the measuring device 2 It is also possible to set an optimum position by finely adjusting the position and orientation.

In step S1613, the calculation unit 9 determines whether the user operates the input unit 8 or the display screen of the display unit 7 to input an end instruction. As a result of this determination, when the user operates the input unit 8 or the display screen of the display unit 7 to input an end instruction, the process according to the flowchart of FIG. 16 ends. The process returns to step S1601.

As described above, in the present embodiment, it is not necessary to repeat the movement of the movable unit 5 and the measuring device 2 in the real space and the confirmation of the relative position and orientation between the measuring device 2 and the measurement target 3. According to the present embodiment, the relative position and orientation of the measuring instrument 2 and the measurement target 3 can be set to a desired position and orientation by a simpler operation than in the past with less labor than in the past.

The movable part 5 is merely an example of a specific part of the robot arm 12, and the position and orientation of a part other than the movable part 5 (a part having a constant relative position and orientation with respect to the measuring device 2) in the robot arm 12 is determined by the movable part Instead of the position and orientation of No. 5, it may be obtained and displayed.

<Modification>
In order to guide the moving direction of the measuring instrument model or the measurement range model, for example, an axis corresponding to the direction of the measuring instrument 2 in the measuring instrument model (if the measuring instrument 2 has an imaging section, Axis) or an axis corresponding to the direction of the measurement range model may be displayed on the GUI.

Further, in the above embodiment, the user performed an operation of moving the measuring instrument model 24 and the measuring range model 25, but the operation is not limited to this, and the relative position and orientation with respect to the measuring instrument 2 and the measuring range 4 may be different. It may be something that is fixed or predetermined. For example, the position and orientation of the movable unit 5 with respect to the measuring instrument model 24 and the measurement range model 25 are fixed. For this reason, as shown in FIG. 6, a coordinate system 22c representing the position and orientation of the movable unit 5 in which the relative position and orientation with respect to the measurement range model 25 is fixed is displayed on the display unit 7, and the user changes the position and orientation of the coordinate system 22c. You may operate to change it. Further, for example, the display unit 7 displays a robot model, which is a three-dimensional model imitating the geometric shape of the mounter 11 fixing the measuring device 2 and the movable unit 5, and the user views the robot model while viewing the robot model. May be changed.

[Second embodiment]
In each of the following embodiments and modifications including the present embodiment, differences from the first embodiment will be described, and unless otherwise specified below, the same as the first embodiment will be described. In the present embodiment, a GUI is provided that makes input and setting by a user easier than in the first embodiment.

In the present embodiment, the GUI of FIG. 7 is displayed on the display screen of the display unit 7 instead of the GUI of FIG. In the display area 17, an image captured by the measuring instrument 2 is displayed.

(4) Before performing various inputs and operations, the user sets the measurement target 3 at a desired position in the real space with a desired posture. In the present embodiment, the measurement target 3 is a relatively large rectangular solid. Next, the user operates the programming pendant 15 to move the movable unit 5 so that the measurement target 3 is approximately in the measurable area of the measuring device 2 (so that the measurement target 3 is within the minimum measurable range). Instruct movement. Since the image captured by the measuring device 2 is displayed in the display area 17, the user moves the movable unit 5 while observing the display area 17, and confirms that the measurement target 3 has been captured in the display area 17 in a measurable state. When confirmed, the user operates the programming pendant 15 to stop the movement.

{Circle around (4)} When the user confirms that the measurement target 3 is approximately in the measurable area of the measuring device 2, the user inputs the position and orientation of the movable unit 5 at this time in the movable unit position and orientation column 22. There are various methods for inputting the position and orientation of the movable unit 5 into the column 22 at this time, and there is no particular method. For example, the user may input the position and orientation of the movable unit 5 displayed on the programming pendant 15 to the field 22 by operating the input unit 8 and the display screen of the display unit 7. Further, the user may operate the programming pendant 15 to transfer the position and orientation displayed on the programming pendant 15 to the image processing apparatus 1 and display the position and orientation on the column 22.

Next, when the user operates the input section 8 or the display screen of the display section 7 to designate the “work measurement” button 26, the arithmetic section 9 performs the imaging based on the captured image displayed in the display area 17. The position and orientation of the measurement target 3 in the image are obtained (measured). There are various known methods for obtaining (measuring) the position and orientation of an object in a captured image, and any method may be employed. For example, the calculation unit 9 matches the result of performing image processing on the captured image with the “3D model for matching with respect to the measurement target 3” registered in the storage unit 6 in advance, and Find the position and orientation. Here, the position and orientation of the measurement target 3 obtained from the captured image by the calculation unit 9 is a position and orientation in a coordinate system based on the measuring device 2.

{Circle around (4)} After calculating the position and orientation of the measurement target 3 in the captured image, the calculation unit 9 displays a GUI illustrated in FIG. In the GUI of FIG. 8, the measurement target model 23 arranged with the obtained position and orientation is displayed in the display area 17 in a state of being superimposed on the captured image.

The calculation unit 9 displays the position and orientation of the measurement target 3 obtained from the captured image, the “relative position and orientation between the measuring device 2 and the movable unit 5” registered in the storage unit 6 in advance, in a column 22. The position and orientation of the measurement target 3 in the reference coordinate system (virtual coordinate system) are obtained based on the position and orientation being measured. For example, by adding “the relative position and orientation between the measuring device 2 and the movable unit 5” and “the position and orientation of the measurement target 3 obtained from the captured image” to the position and orientation displayed in the column 22, the reference is obtained. The position and orientation of the measurement target 3 in the coordinate system (virtual coordinate system) are obtained. Then, the calculation unit 9 displays the position and orientation of the measurement target 3 in the reference coordinate system (virtual coordinate system) in the measurement target position and orientation column 21.

Here, when the user operates the input unit 8 or the display screen of the display unit 7 to designate the “measurement target position / posture registration” button 27, the calculation unit 9 displays the GUI illustrated in FIG. To display. The display area 17 displays an image of the measurement target model 23, the measurement device model 24, and the measurement range model 25 as viewed from the above viewpoint, and the icon 18. The position and orientation of the measurement target model 23 is “the position and orientation of the measurement target 3 in the reference coordinate system (virtual coordinate system)” obtained by the process performed by pressing the “work measurement” 26. The position and orientation of the measuring instrument model 24 is obtained by adding “the relative position and orientation between the measuring instrument 2 and the movable unit 5” to the position and orientation displayed in the column 22. The position and orientation of the measurement range model 25 can be obtained by adding “the relative position and orientation of the measurement range 4 with respect to the measurement device 2” to the position and orientation of the measurement device model in the reference coordinate system.

After each model is displayed, as in the first embodiment, the user performs an operation of moving the measuring instrument model or the measurement range model so that a part or the whole of the measurement target model fits inside the measurement range model. . FIG. 10 shows a display example of the GUI in a state where the operation is completed.

In the example of FIG. 10, the surface portion of the measurement target model 23 is substantially aligned with the center of the measurement range model 25. The position and orientation of the movable unit 5 at that time are calculated by the calculation unit 9 in the same manner as in the first embodiment, and are displayed in the column 22. This allows the user to know the position and orientation of the movable part for performing measurement under desired conditions.

As described above, according to the present embodiment, the condition that the approximate position and orientation of the measurement target 3 in the reference coordinate system presupposed in the first embodiment is not required in advance. That is, the act of acquiring the position and orientation using a measure or the like in the real space and the act of acquiring the position and orientation from the position and orientation of the installation table 10 are not required, and the work is further simplified. In addition, if the measurement can be performed, the arrangement relationship of the measurement target model 23, the measurement device model 24, and the measurement range model 25 in the real space at that time can be displayed in the virtual space. The set load decreases.

[Third Embodiment]
In the present embodiment, an image (virtual space image) that is visible when a virtual space within the measurement range (view volume) indicated by the measurement range model is observed from the viewpoint having the position and orientation of the measuring instrument model, and is displayed on the GUI. . Since a technique for generating an image of a “space within a prescribed viewing volume” viewed from a viewpoint having a prescribed position and orientation is well known, detailed description thereof will be omitted. For example, the GUI of FIG. 11 is displayed instead of the GUI of FIG. In the GUI of FIG. 11, an image 29 that is visible when the space in the measurement range model 25 is observed from the position and orientation of the measurement device model 24 is displayed. The display mode of the image 29 is not limited to the above example, and various display modes can be considered. For example, in FIG. 11, the image 29 is displayed in the display area 17, but may be displayed in a different place.

As described above, according to the present embodiment, it is possible for the user to understand roughly how the measurement target 3 appears before performing imaging by the measuring device 2, and to capture a virtual captured image (image 29). It is possible to perform the adjustment efficiently and accurately while checking.

[Fourth embodiment]
In the present embodiment, information indicating the relative positional relationship between the measurement target model and the measurement range model is displayed on the GUI. In the present embodiment, the GUI of FIG. 12 is displayed instead of the GUI of FIG. The relative position column 30 displays a relative position between a specified position in the measurement range model 25 and a specified position in the measurement target model 23. In FIG. 12, as a relative position (unit: mm) between the specified position in the measurement range model 25 and the specified position in the measurement target model 23, the X coordinate position "3.2" and the Y coordinate position "2.1" ”And the Z coordinate position“ 1.2 ”are displayed. It is assumed that both the “specified position in the measurement range model 25” and the “specified position in the measurement target model 23” are specified in advance, and the specified positions are registered in the storage unit 6. For example, the position of the measurement range model corresponding to the center position of the bottom surface of the truncated pyramid represented by the measurement range 4 and the position of the measurement target model corresponding to the center position of the bottom surface of the pallet of the measurement target 3 are specified in advance. , Are registered in the storage unit 6. Accordingly, the calculation unit 9 obtains a relative position between the specified position in the measurement range model 25 and the specified position in the measurement target model 23, and displays the obtained position in the column 30.

The closer the numerical value (X coordinate position, Y coordinate position, Z coordinate position) displayed in the column 30 is to 0, the closer the specified position in the measurement range model 25 and the specified position in the measurement target model 23 are. ing. However, the user moves the measuring instrument model and the measurement range model while watching the numerical value displayed in the column 30 so that the numerical value approaches zero. In the above example, by moving the measuring instrument model and the measuring range model so that the numerical value displayed in the column 30 approaches 0, the measuring instrument model and the measuring instrument model are adjusted so that the far end of the measuring range 4 and the bottom of the pallet match. The measurement range model can be moved.

In addition, as information indicating the relative positional relationship between the measurement target model and the measurement range model, other information may be displayed in addition to or in place of the relative position. For example, a relative posture may be displayed in addition to the relative position. In the above example, the position of the measurement range model corresponding to the center coordinates of the far end plane of the measurement range 4 and the position of the measurement target model corresponding to the center coordinates of the bottom surface of the pallet of the measurement target 3 are specified. However, the position of the measurement range model corresponding to the barycenter coordinates of the measurement range 4 and the position of the measurement target model corresponding to the barycenter coordinates of the measurement target 3 may be specified. In addition, the position of the measuring instrument model corresponding to the center coordinates of the bottom surface of the measuring instrument 2 and the position of the measuring object model corresponding to the center coordinates of the bottom surface of the pallet of the measuring object 3 are specified. May be calculated, and the distance may be displayed instead of the column 30.

Further, in the GUI of FIG. 12, the relative relationship between the measurement target model 23 and the measurement range model 25 is indicated by numerical values. However, the position of the measurement target in the measurement range is indicated by color information of the model. May be. In this case, the GUI of FIG. 13 is displayed instead of the GUI of FIG.

13. In the GUI of FIG. 13, instead of the measurement target model 23, a measurement target model 31 indicating, by color, which region in the measurement range model 25 each part of the measurement target model 23 is displayed. For example, the calculation unit 9 determines at which position in the measurement range model 25 each element obtained by dividing the measurement target model 31 in a prescribed unit (for example, a certain volume) corresponds to the element corresponding to the determined position. Draw in the color you want. In FIG. 13, elements that are far from the measurement range model 25 (far as viewed from the measuring instrument model 24) among the elements constituting the measurement target model 31 are drawn in dark color, and are intermediate between the near end and the far end. The element at the position is drawn in light color. In the present embodiment as well, as in FIG. 6, the purpose of adjustment is to match the far end of the measurement range 4 with the pallet bottom surface, so that the color of the bottom surface of the measurement target model 31 becomes dark as shown in FIG. 13. The object is achieved by performing such adjustment.

Also, for example, instead of the measurement target model 23, a measurement target model indicating how much each part of the measurement target model 23 is apart from the specified position in the measuring instrument model 24 by color may be displayed.

着色 Also, there are various modes of coloring, and they may be represented by gradations of light and shade as in the above example, or may be represented by gradations of different colors. Further, for example, only the elements of the measurement target model at a specific position such as a far end plane of a predetermined area may be colored. Furthermore, coloring may be considered so as to emphasize elements of the measurement target model existing outside the measurement range model 25.

As described above, according to the present embodiment, the relative relationship between the measurement target model and the measuring instrument model or the measurement range model can be intuitively grasped by numerical values or coloring. The adjustment can be performed efficiently and accurately.

[Fifth Embodiment]
In the present embodiment, the GUI of FIG. 14 is displayed instead of the GUI of FIG. In general, the movable section 5 has a movable range and a posture restriction in the real space. However, condition information that defines the range of positions and postures that the movable unit 5 can take is created in advance and registered in the storage unit 6. After displaying the GUI, the calculation unit 9 determines whether or not the position and the posture of the movable unit 5 displayed in the column 22 are within the range of the position and the posture indicated by the condition information. Displays a warning on the display screen of the display unit 7. In the GUI of FIG. 14, a message 32 “out of robot posture restriction range” is displayed as a warning, and the user is notified that the posture of the movable unit 5 displayed in the column 22 exceeds the posture restriction range. Notify Thus, the user can know whether or not the arrangement relationship in the virtual space can be realized in the real space without actually trying to move the movable unit 5 in the real space.

In the above GUI, the measuring instrument model and the measuring range model are displayed by semi-transparently displaying the measuring instrument model and the range of the measuring range model (prohibited area) corresponding to the range of the position and orientation that cannot be taken by the movable section 5. The user can be notified of an area that should not be moved. For the same purpose, a message may be displayed on the display screen of the display unit 7 when the measuring instrument model or the measurement range model moves to the prohibited area.

If the measuring instrument 2 or the movable section 5 interferes with the measurement target 3, the device or the measurement target 3 may be damaged. Therefore, the calculation unit 9 determines interference between the measuring instrument model and the measurement target model, and displays a warning (for example, a message indicating that there is interference) on the display screen of the display unit 7 if there is interference. In addition, when a model imitating the geometric shape of the movable unit 5 is placed at the position and orientation of the movable unit 5, the calculation unit 9 performs interference determination between the model and the measurement target model, and warns if there is interference (for example, , A message indicating that there is interference) is displayed on the display screen of the display unit 7.

警告 Also, the display form of the warning is not limited to a specific display form. For example, when the position is not within the range of the position indicated by the condition information, the position displayed in the column 22 may be highlighted, such as displaying in red, or when the position is not within the range of the posture indicated by the condition information, The posture displayed in the column 22 may be highlighted, for example, displayed in red.

As described above, according to the present embodiment, the user can determine whether the movement can be realized in the real space or does not cause interference due to the movement without moving the movable unit 5 in the real space. It is possible and contributes to setting efficiency and prevention of physical damage to the device.

[Sixth Embodiment]
In the present embodiment, the position and orientation of the movable unit 5 with respect to the plurality of measurement targets 3 are obtained and displayed. In the present embodiment, a GUI shown in FIG. 15 is displayed instead of the GUI shown in FIG. In the display area 17, the measurement target models 23a, 23b, and 23c corresponding to the three measurement targets 3 (target 1, target 2, target 3) are displayed. The user sets the position and orientation of the measuring instrument model 24 and the measurement range model 25 with respect to one of the measurement target models 23a, 23b, and 23c (here, as an example, the measurement target model 23a) as in the first embodiment. It is assumed that it is adjusted to Here, the user operates the input unit 8 and the display screen of the display unit 7 to input the number and arrangement information of the measurement targets 3 in the measurement target arrangement column 33. In FIG. 15, the number of the measurement targets 3 (3 in FIG. 15), the distance between the measurement targets 3 in the X-axis direction (30 mm in FIG. 15), the distance in the Y-axis direction (−200 mm in FIG. 15), and the Z-axis direction (0 mm in FIG. 15). This means that the measurement target model 23b is located at a position shifted by 30 mm in the X axis direction, −200 mm in the Y axis direction, and 0 mm in the Z axis direction from the position of the measurement target model 23a. This also indicates that the measurement target model 23c is located at a position shifted by 30 mm in the X-axis direction, −200 mm in the Y-axis direction, and 0 mm in the Z-axis direction from the position of the measurement target model 23b. Here, for the sake of simplicity, the posture is assumed to be the same for all measurement target models.

When the user operates the input unit 8 or the display screen of the display unit 7 to indicate the “calculation” button 1501, the calculation unit 9 determines the position and orientation of the movable unit 5 obtained for the measurement target model 23a and the measurement target model 23a. The relative position / posture between and is referred to as a relative position / posture T. Then, the calculation unit 9 obtains a position and orientation such that the relative position and orientation with respect to the position and orientation of the measurement target model 23b becomes the relative position and orientation T, as “the position and orientation of the movable unit 5 corresponding to the measurement target model 23b”. Similarly, the calculation unit 9 obtains a position and orientation such that the relative position and orientation relative to the position and orientation of the measurement target model 23c is the relative position and orientation T, as "the position and orientation of the movable unit 5 corresponding to the measurement target model 23c". Since the position and orientation of the movable unit 5 corresponding to each of the target 1, the target 2, and the target 3 are obtained in this manner, the calculation unit 9 displays the obtained position and orientation of the movable unit 5 in the column 22. This makes it possible to obtain the position and orientation of the movable part 5 when measuring each individual model without adjusting the relative relationship between the measuring instrument model 24 and the measurement range model 25 with respect to the individual measurement target models. Here, the position and orientation of the movable unit 5 corresponding to the measurement target models 23a, 23b, and 23c in this order are obtained, but the order is not limited to this order.

The procedure for obtaining the position and orientation of the movable unit 5 corresponding to the plurality of measurement targets 3 is not limited to a specific procedure. For example, after setting the position and orientation of one measurement target model, the number and arrangement intervals of the measurement target models are input in the column 33 to determine the position and orientation of another measurement target model. After that, the position and orientation of the measuring instrument model 24 and the measurement range model 25 with respect to any one measurement target model are adjusted.

In FIG. 15, the information that can be entered in the column 33 is only the number of measurement target models and the arrangement intervals, but other information may be input. For example, information specifying the posture component of each measurement target model may be input.

In addition, the column 21 may display the position and orientation of each measurement target model. In addition, although the respective measurement targets 3 are described as being the same type of object, they may be different types of objects.

<Modification>
In the above embodiments and modifications, the measuring device 2 is a three-dimensional shape measuring device, but may be another measuring device. For example, the measuring device 2 may be a measuring device that calculates the distance from the measuring device 2 to a target, or may be a device that extracts a contour from a captured image and inspects the measuring target 3. . That is, it is possible to assume various forms of the measuring device 2 having a measuring range defined by its performance and specifications.

Also, in the above-described embodiment and the modified examples, the measurement target 3 is a small work stacked in bulk or a pallet for storing the small work, but is not limited thereto. For example, when there is no pallet, it is also possible to treat the approximate entire space where small piled works exist as the measurement target 3. In addition, it is often the case that only a single relatively large work is placed, and among them, there is a case where a specific range is measured. In that case, the specific range is set as the measurement target 3. good. That is, it does not depend on the size, material, or the like of the measurement target 3.

Also, in the above-described embodiment and modified examples, the measurement range 4 is a truncated quadrangular pyramid-shaped region in which the measurement device 2 can achieve a constant measurement performance, but is not limited thereto. First, as the definition of the measurement range 4, a working distance region that is generally desirable as an imaging range of a measuring instrument may be indicated without indicating measurement performance. Various shapes can be considered for the shape of the measurement range 4. For example, as the measurement range 4, in addition to the truncated quadrangular pyramid region, a truncated cone shape, a pyramid shape such as a quadrangular pyramid shape or a conical shape, a rectangular parallelepiped shape, or a cubic shape may be considered. In addition, the size of the measurement range 4 can be variously changed depending on various conditions such as the performance of the measurement device 2.

In the above-described embodiment and the modification, the movable portion 5 is the tip of the tool fixedly connected to the flange portion of the robot arm 12 for gripping the measurement target 3, but is not limited thereto. As described above, a part of the robot arm, for example, a flange portion of the robot arm 12 may be the movable portion 5. The movable part 5 may be any part having a specific relative relationship with the measuring instrument 2 irrespective of a change in position and orientation due to its movement, and various parts can be used within the definition.

The configuration of the image processing apparatus 1 is not limited to the configuration shown in FIG. For example, in FIG. 1, the storage unit 6 is provided inside the image processing apparatus 1, but may be provided outside the image processing apparatus 1. For example, the storage unit 6 may be incorporated in the robot controller 14. , May be incorporated in an external device that can communicate with the image processing apparatus 1. This is the same for the arithmetic unit 9.

Also, in the above embodiments and modifications, various GUI configurations have been described. However, these GUI configurations described above are merely examples, and various configurations are possible. For example, by displaying a three-dimensional model imitating the geometrical shape of the movable part 5 or an icon representing a coordinate system based on the movable part 5 in the display area 17, the movable part 5 calculated by the arithmetic unit 9 is displayed. The position and orientation may be visually represented.

In addition, a three-dimensional model of the mounter to which the measuring device 2 and the movable part 5 are fixed and the robot arm 12 are also displayed in the display area 17, and the positions and orientations of the models are operated with the operation of the measuring device model 24 and the measurement range model 25. May change. In this case, the interference between the three-dimensional model of the mounter or the robot arm 12 and the measurement target model 23 or the measuring instrument model 24 is determined, and if there is interference, a warning (for example, a message indicating that there is interference) is displayed. You may make it display on the display screen of the part 7. Further, the display of the measuring instrument model 24 can be omitted. In this case, the display area 17 shows the measurement target model 23 and the measurement range model 25, and the relative relationship between them is adjusted. However, the relative position and orientation between the measurement device 2 and the measurement range 4 are constant. Therefore, it is possible to calculate the position and orientation of the movable unit 5 as in the first embodiment.

撮 像 Alternatively, an imaging button or a display area for confirmation may be added to the GUI. In this case, after performing the above adjustment in the virtual space, the user operates the programming pendant 15 to move the movable unit 5 in the real space, and then operates the input unit 8 and the display screen of the display unit 7 to perform imaging. Press the button. Accordingly, the measuring device 2 performs imaging, sends the captured image obtained by the imaging to the image processing apparatus 1, and the arithmetic unit 9 displays the captured image in the display area for confirmation. Thereby, without changing to another screen, it is confirmed whether or not adjustment has been performed so that the image captured by the measuring device 2 becomes a desired captured image (the measurement target 3 is captured in a desired state). Can be done.

Further, in the above-described embodiment and the modified example, the position and orientation of the movable unit 5 are displayed in the column 22. However, the arithmetic unit 9 does not display the position and orientation of the movable unit 5 and the robot controller 14 or the programming pendant. 15 may be output.

The position and orientation of the measuring instrument model 24 and the measurement range model 25 may be displayed on a GUI. In that case, the position and orientation of the measuring instrument model 24 and the measurement range model 25 may be changed by the user operating the display screen of the input unit 8 and the display unit 7 to change the displayed position and orientation.

The method of expressing the position and orientation displayed on the GUI is not limited to a specific expression method, but can be expressed by various expression methods such as an Euler angle expression and a rotation angle expression. In the above-described GUI, an example in which the positions and orientations of six degrees of freedom are displayed in individual boxes has been presented. However, the position and orientation of six degrees of freedom may be displayed in one box by separating them with commas.

(4) The method of selecting a model to be operated on the display screen of the display unit 7 is not limited to a specific method. For example, the model may be selected by directly specifying the display area of the model to be operated, or a radio button corresponding to each model may be provided outside the display area 17 to operate the model corresponding to the radio button selected by the user. You may make it select as a target model.

There are also various operations on the display area 17. For example, the image of the virtual space displayed in the display area 17 may be enlarged or reduced by rotating a mouse wheel when a mouse is used as the input unit 8. Regarding the enlargement and reduction of the image in the virtual space, the image may be simply enlarged or reduced, or may be included in the image in the virtual space by moving the viewpoint in the virtual space in the direction of the line of sight / in the direction opposite to the line of sight. May be changed. Alternatively, the viewpoint may be moved in the virtual space by moving the mouse while right-clicking.

The display unit 7 may be a display device included in another device. For example, the display unit 7 may be a display device included in the programming pendant 15. Further, the display unit 7 may be a device separate from the image processing device 1 or a device integrated with the image processing device 1.

入 力 Also, the information input form is not limited to the above input form. For example, information input using a mouse or a keyboard may be notified by, for example, command communication from the robot controller 14, and the input form is arbitrary.

Also, in the above-described embodiments and modified examples, the order of operations performed by the user is specifically shown, but these orders are merely examples, and the present invention is not limited to this. For example, the display of the measurement target model 23 and the display of the measuring instrument model 24 may be in the reverse order instead of the order described in the above example.

In addition, in the above-described embodiment and the modified example, each time the measuring instrument model 24 or the measurement range model 25 is moved, the calculation unit 9 calculates the position and orientation of the movable unit 5 and displays the calculated position and orientation in the column 22. However, the calculation and display of the position and orientation of the movable unit 5 may be performed with a trigger triggered by the user's input of an instruction to complete the adjustment by operating the display screen of the input unit 8 or the display unit 7.

In addition, some procedures described in the above-described embodiment and modified examples can be omitted. For example, when the position and orientation are determined in advance such as when the measurement target 3 is installed on a jig in a real space, the position and orientation are stored in the storage unit 6 or the like in advance, and It is not necessary to set the position and orientation of. In this case, the measurement target model 23 is displayed in the display area 17 based on the stored position and orientation.

Note that the numerical values used in each of the above embodiments and modifications are merely examples, and are not intended to limit the numerical values. Further, in each of the above-described embodiments and modifications, the model imitates the geometric shape of the original real object. However, the present invention is not limited to this, as long as the original real object can be identified. However, the model is not necessarily limited to a model imitating the geometric shape of the original real object. If the model has been created in advance by CAD or the like, the previously created model may be obtained. In addition, the configurations and operation methods of the GUIs described in each of the above embodiments and modifications are merely examples, and are not intended to limit the configurations and operation methods.

In addition, some or all of the above-described embodiments and modifications may be used in appropriate combination. In addition, some or all of the above-described embodiments and modifications may be selectively used.

[Seventh embodiment]
The measuring device that is the above-described image processing device 1 can be used while being supported by a certain support member. In the present embodiment, as an example, a control system provided and used in a robot arm 5300 (gripping device) as shown in FIG. 17 will be described. The measuring device 5100 projects and captures an image of the pattern light on the test object 5210 placed on the support base 5350 to obtain an image. Then, the control unit of the measuring device 5100 or the control unit 5310 that has acquired the image data from the control unit of the measuring device 5100 obtains the position and orientation of the test object 5210 and controls the information of the obtained position and orientation. The unit 5310 acquires the information. The control unit 5310 controls the robot arm 5300 by sending a drive command to the robot arm 5300 based on the information on the position and the posture. The robot arm 5300 holds the test object 5210 with a robot hand or the like (grasping portion) at the tip, and moves such as translation or rotation. Further, by attaching the test object 5210 to another component by the robot arm 5300, an article composed of a plurality of components, for example, an electronic circuit board or a machine can be manufactured. In addition, by processing the moved test object 5210, an article can be manufactured. The control unit 5310 includes an arithmetic device such as a CPU and a storage device such as a memory. Note that a control unit for controlling the robot may be provided outside the control unit 5310. The measurement data measured by the measurement device 5100 and the obtained image may be displayed on a display unit 5320 such as a display.
(Other Examples)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. It can also be realized by the following processing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to make the scope of the present invention public.

This application claims priority based on Japanese Patent Application No. 2018-129460 filed on Jul. 6, 2018 and Japanese Patent Application No. 2019-093917 filed on May 17, 2019, The entire contents of the description are incorporated herein by reference.

2: Measuring instrument 3: Measurement target 4: Measurement range 5: Movable part 6: Storage part 7: Display part 8: Input part 9: Calculation part 10: Installation base 11: Mounter 12: Robot arm 13: Robot base 14: Robot Controller # 15: Programming pendant

Claims (27)

1. Display means for displaying, on a display screen, a first model representing an object to be measured by the measuring instrument, and a second model representing a measuring range of the measuring instrument;
   Changing means for changing the position and orientation of the second model in accordance with a user operation;
   An output unit configured to output a position and orientation of a part whose position and orientation relative to the measuring device are constant in a state where the position and orientation of the second model are changed.
2. The image processing apparatus according to claim 1, wherein the display unit displays an axis corresponding to the direction of the measuring device or the measurement range on the display screen.
3. The display means obtains a position and orientation of the measurement target based on a captured image of the measurement target by the measuring device, and displays the first model arranged at the obtained position and orientation on the display screen. The image processing apparatus according to claim 1 or 2, wherein
4. 4. The display device according to claim 1, wherein the display unit displays a relative position and / or posture between the first model and the second model on the display screen. 5. An image processing apparatus according to claim 1.
5. The said display means draws each element which comprises the said 1st model with the color according to the positional relationship with the said 2nd model, The Claims 1 to 4 characterized by the above-mentioned. Image processing device.
6. 6. The display device according to claim 1, wherein the display unit displays a warning on the display screen when the position and orientation of the part exceeds a predetermined range of restriction. An image processing apparatus according to the item.
7. The image processing apparatus according to any one of claims 1 to 6, wherein the display unit displays an image captured by the measuring device on the display screen.
8. The image processing apparatus according to any one of claims 1 to 7, wherein the output unit obtains and outputs the position and orientation of the part for each of a plurality of measurement targets.
9. The image processing apparatus according to any one of claims 1 to 8, wherein the output unit displays the position and orientation of the part on the display screen.
10. The image processing apparatus according to any one of claims 1 to 9, wherein the output unit outputs the position and orientation of the part to an apparatus for controlling the position and orientation of the part.
11. 11. The image processing apparatus according to claim 1, wherein the measuring device is attached to a robot arm, and the part is a part of the robot arm.
12. The measurement device according to claim 1, further comprising: a projection unit configured to project pattern light onto the measurement target; and an imaging unit configured to capture the measurement target onto which the pattern light is projected. Item 2. The image processing device according to item 1.
13. The image processing apparatus according to any one of claims 1 to 11, wherein the measuring device includes a plurality of imaging devices that image the measurement target at different positions and orientations.
14. The image processing apparatus according to any one of claims 1 to 13, wherein the second model is a translucent model.
15. The image processing apparatus according to any one of claims 1 to 13, wherein the second model is a wire frame model.
16. The display means displays a third model representing the measuring instrument on the display screen,
    The changing unit changes the position and orientation of the second model by operating the second model or the third model in response to a user operation,
    The said output means calculates | requires and outputs the position and orientation of the said part based on the position and orientation of the said 2nd model or the said 3rd model. The output of any one of Claims 1 thru | or 15 characterized by the above-mentioned. Image processing device.
17. The change unit is configured to, when one of the second model and the third model is operated in response to a user operation, change a relative position and orientation between the second model and the third model. The image processing apparatus according to claim 16, wherein the other position and orientation are changed so that is maintained.
18. The display means obtains the respective positions and orientations of the second model and the third model based on the position and orientation designated by the user as the position and orientation of the part, and arranges the positions in the respective positions and orientations. The image processing apparatus according to claim 16, wherein a second model and the third model are displayed on the display screen.
19. 19. The display according to claim 16, wherein the display unit displays, on the display screen, an image of a space in a measurement range represented by the second model, which is visible from a viewpoint having the position and orientation of the third model. The image processing device according to claim 1.
20. 20. The image processing apparatus according to claim 16, wherein the display unit displays a distance between the first model and the third model on the display screen.
21. The display means performs an interference determination between the first model and the third model, an interference determination between the model representing the site and the first model, and issues a warning on the display screen if there is interference. The image processing apparatus according to claim 16, wherein the image is displayed.
22. The display means displays on the display screen an object whose relative position and orientation are determined with respect to the second model,
    The image processing according to any one of claims 1 to 15, wherein the changing unit changes the position and orientation of the second model by operating the object in response to a user operation. apparatus.
23. An image processing apparatus according to any one of claims 1 to 22,
    A measuring device for measuring a test object based on the position and orientation output by the image processing device.
24. An image processing apparatus according to any one of claims 1 to 22,
    A measuring device for measuring the test object,
    The measuring instrument is fixed, and a robot that operates the test object based on a measurement result by the measuring instrument,
    A system, wherein the robot is moved based on the position and orientation output by the image processing device, and the test object is measured by the measuring device.
25. A step of measuring a test object based on the position and orientation output by the image processing apparatus according to any one of claims 1 to 22,
    Manufacturing an article by processing the test object based on the measurement result.
26. An image processing method performed by the image processing apparatus,
    A display step of displaying, on a display screen, a display unit of the image processing device, a first model representing a measurement target of the measuring instrument, and a second model representing a measurement range of the measuring instrument;
    Changing means for changing the position and orientation of the second model according to a user operation,
    An output step of outputting the position and orientation of a part whose position and orientation relative to the measuring device are constant in a state where the position and orientation of the second model are changed. An image processing method comprising:
27. A computer program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 22.

Images