WO2019182084A1

WO2019182084A1 - Method and device

Info

Publication number: WO2019182084A1
Application number: PCT/JP2019/011980
Authority: WO
Inventors: 陽粟田
Original assignee: アトラスコプコ株式会社
Priority date: 2018-03-23
Filing date: 2019-03-22
Publication date: 2019-09-26
Also published as: JP6596530B2; JP2019168304A

Abstract

In the present invention, a marker attached to an object is used to assist in an operation performed by an operator. Provided is a method comprising: a step in which the position and orientation, in a prescribed coordinate system, of an object marker attached to an object are acquired from an image in which the object marker is captured; a step in which the position, in the prescribed coordinate system, of a detection point of the object from the object marker in a marker coordinate system, and the orientation of the object, are calculated on the basis of the coordinates of the detection point and the position and orientation of the object marker; and a step in which an assessment is made as to whether the position of the detection point is within a prescribed position range in the prescribed coordinate system and whether the orientation of the object is within a prescribed orientation range in the prescribed coordinate system.

Description

Method and apparatus

The present disclosure relates to a method and an apparatus for detecting whether an object using a marker attached to an object such as a tool or a work on the object is correctly performed.

In order to detect the position of a tool for assembling a part at a production site, a system is known in which a marker attached to the tool is imaged and the obtained image is analyzed to detect the position of the tool. For example, in Patent Document 1, a marker attached near the tip of a tool is imaged and the position of the tool is detected.

Patent No. 5930708

However, in the conventional system as described above, when a screw is tightened using a tool, a tool having a special shape in which a marker can be attached to a position acting on a screw or the like that is the most desired position is used. There was a need. For example, in Patent Document 1, it is necessary to use a torque wrench capable of attaching a marker at a position close to a screw. However, depending on the shape and size of the tool, it is difficult to attach a marker at a position where the tool acts on a screw or the like. For example, when the tool is a driver, the position where the driver is fitted with the screw is narrow, and it is difficult to attach a marker at the position. Even if the marker can be attached at a position where the driver is fitted with the screw, the marker may be hidden by the hand or the driver, and the marker may not be identified. In addition, when assembling a part with screws, a marker cannot be attached to the screw hole of the part that is the most desired position. The present disclosure has been made in view of the above points, and is the most desired position of a tool or the like having various shapes and sizes (that is, in the case of a tool, the position where the tool contacts a screw or the like). In this case, the object marker is attached at a position different from the position where the component is acted on by a screw or the like. Then, one of the purposes is to obtain the position of the detection point that is the most desired position by using the object marker (in the above-described tool, the position where the tool contacts a screw or the like).

Also, in the conventional system as described above, depending on the posture of the tool, the posture of the part on which the tool or the like acts may be incorrect, and the work may not be performed correctly. For example, when the tool is a driver, if the posture of the tool is wrong, the screw attached to the tool enters obliquely with respect to the part or the like, and the screw cannot be inserted correctly. This indication is made in view of the above-mentioned point, and makes it one of other objects to support a worker's work so that a tool etc. can be used in a right position and posture.

In order to solve the above-described problem, one aspect of the present disclosure includes a step of acquiring the position and orientation of the object marker in a predetermined coordinate system from an image obtained by imaging an object marker attached to an object such as a tool; The predetermined coordinates based on the coordinates of the detection point of the object from the object marker in the marker coordinate system (in the above driver, the coordinates of the place where the screw is fitted), and the position and orientation of the object marker Calculating the position of the detection point and the posture of the object in the system, the position of the detection point is within a predetermined position range in the predetermined coordinate system, and the posture of the object is the predetermined coordinate Determining whether the position is within a predetermined posture range in the system. The predetermined position range is a position range in which a tool or the like can be moved in order to correctly assemble parts or the like. The predetermined posture range is a posture range in which a tool or the like can be moved in order to correctly assemble parts or the like.

In another aspect of the present invention, the position and orientation of the object marker in a predetermined coordinate system are detected from an image obtained by imaging an object marker attached to an object such as a tool, and the object marker in the marker coordinate system is detected. A position / orientation detection unit that detects the position of the detection point and the attitude of the object in the predetermined coordinate system based on the coordinates of the detection point of the object from the position of the object marker and the position and orientation of the object marker; A position and orientation comparison unit that determines whether the position of a point is within a predetermined position range in the predetermined coordinate system and the posture of the object is within a predetermined posture range in the predetermined coordinate system; It is a device provided.

1 is a schematic configuration diagram of an offset setting system according to an embodiment of the present disclosure. It is a figure which shows the relationship between each coordinate system which concerns on one Embodiment of this indication, an object marker, etc. FIG. 2 illustrates an example of a marker according to an embodiment of the present disclosure. 4 illustrates a configuration example of detection point data according to an embodiment of the present disclosure. 2 illustrates an example of an object according to an embodiment of the present disclosure. 2 illustrates an example of an object according to an embodiment of the present disclosure. 2 illustrates an example of an object according to an embodiment of the present disclosure. 2 illustrates an example of an object according to an embodiment of the present disclosure. It is a schematic structure figure of a control system concerning a 1st embodiment of this indication. It is a schematic structure figure containing a functional block of a control system concerning a 1st embodiment of this indication. 5 is a flowchart illustrating an operation of a control system according to an embodiment of the present disclosure. It is a figure which shows the relationship between each coordinate system which concerns on one Embodiment of this indication, an object marker, etc. FIG. 3 shows a configuration example of target coordinate posture data according to an embodiment of the present disclosure. 3 shows a configuration example of a sequence table according to an embodiment of the present disclosure. It is a figure which shows the relationship between each coordinate system which concerns on one Embodiment of this indication, an object marker, etc. FIG. 3 shows a schematic configuration of a control system according to a second embodiment of the present disclosure. 3 shows a schematic configuration of a control system according to a second embodiment of the present disclosure. 3 shows a schematic configuration of a control system according to a second embodiment of the present disclosure. 4 shows a schematic configuration of a control system according to a third embodiment of the present disclosure. 6 shows a schematic configuration of a control system according to a fourth embodiment of the present disclosure. 6 schematically shows a configuration of a control system according to a fifth embodiment of the present disclosure. It is a figure which shows the example of the setting tool which concerns on one Embodiment of this indication.

[Description of Embodiment of the Present Invention]
First, the contents of the embodiment of the present invention will be listed and described. One embodiment of the present invention has the following configuration.
(Item 1) A step of acquiring the position and orientation of the object marker in a predetermined coordinate system from an image obtained by imaging the object marker attached to the object, and the detection point of the object from the object marker in the marker coordinate system Calculating the position of the detection point and the posture of the object in the predetermined coordinate system based on the coordinates and the position and posture of the object marker; and the position of the detection point in the predetermined coordinate system. Determining whether the object is within a predetermined position range and the posture of the object is within a predetermined posture range in the predetermined coordinate system.

(Item 2) When the position of the detection point is within the predetermined position range and the posture of the object is within the predetermined posture range, a step of outputting a control signal for the control object is further included. The method according to Item 1, wherein the control signal is a signal that does not depend on a type of the control object.

(Item 3) The method according to item 2, further comprising a step of obtaining a response signal from the control object in response to the control signal, wherein the response signal is a signal independent of a type of the control object. Method.

(Item 4) When the predetermined position range and the predetermined posture range do not change with respect to the camera coordinate system of the camera that images the object marker, the predetermined coordinate system is a camera coordinate system. The method described in 1.

(Item 5) When the predetermined position range and the predetermined posture range change with respect to the camera coordinate system of the camera that captures the object marker, the predetermined coordinate system is a reference marker coordinate system. The method according to 1.

(Item 6) When the predetermined position range and the predetermined posture range change with respect to the camera coordinate system of the camera that captures the object marker, the target coordinates of the predetermined position range and the predetermined posture range Item 2. The method according to Item 1, wherein the target posture is corrected to the target coordinate and the target posture in the camera coordinate system.

(Item 7) The position and orientation of the object marker in a predetermined coordinate system is detected from an image obtained by imaging the object marker attached to the object, and the coordinates of the detection point of the object from the object marker in the marker coordinate system A position and orientation detection unit that detects the position of the detection point and the orientation of the object in the predetermined coordinate system based on the position and orientation of the object marker;
A position and orientation comparison unit that determines whether the position of the detection point is within a predetermined position range in the predetermined coordinate system and the posture of the object is in a predetermined target posture in the predetermined coordinate system;
A device comprising:

(Item 8) Generation of a control signal for outputting a control signal for a control object when the position of the detection point is within the predetermined position range and the posture of the object is within the predetermined posture range The apparatus according to item 7, further comprising: a control unit, wherein the control signal is a signal that does not depend on a type of the control object.

(Item 9) The control signal generation unit further acquires a response signal from the control object in accordance with the control signal, and the response signal is a signal that does not depend on the type of the control object. Item 9. The apparatus according to item 8.

(Item 10) calculating the position and orientation of the object marker in a predetermined coordinate system when the detection point of the object to which the object marker is attached overlaps the reference position of the setting tool;
Setting the coordinates of the detection point of the object based on the object marker in a marker coordinate system based on the reference position and the position and orientation of the object marker;
A method comprising:

(Item 11) The method according to item 10, wherein the reference position is arranged at a position distant from a setting marker attached to the setting tool.

[Details of the embodiment of the present invention]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, the same or similar elements are denoted by the same or similar reference numerals, and redundant description of the same or similar elements may be omitted in the description of each embodiment. Further, the features shown in each embodiment can be applied to other embodiments as long as they do not contradict each other. However, embodiments of the present disclosure are not necessarily limited to such an aspect. It will be apparent to those skilled in the art that the embodiments of the present disclosure can take various forms that fall within the scope of the claims.

According to the present disclosure, the three-dimensional position of the object marker is calculated based on the image obtained by imaging the object marker attached to the object. The object is a position and posture detected by an object marker, and includes tools, parts, labels, workers' hands, and the like. The tool is gripped and operated by an operator. The part receives an action from an operator or a tool, for example, a part having a screw hole. The label is moved or pasted by an operator. According to the present disclosure, from the three-dimensional position of the object marker, the three-dimensional position of the detection point located away from the object marker is calculated, and the three-dimensional position of the detection point is compared with a predetermined position range set in advance. The correctness of the worker's work is determined. When the worker's work is correct or incorrect, a signal is sent to the control object based on a preset sequence. The control object includes a tool operated by an operator, an indicator light controlled by a computer, a switch, and the like. A tool can be either an object or a controlled object.

The object marker only needs to be arranged at a position where it can be imaged from the camera, and is arranged at a position different from the detection point, usually on the object. The positional relationship between the object marker and the detection point of the object can be arbitrarily set.

First, a configuration for setting the relative positional relationship between an arbitrarily placed object marker and a detection point will be described with reference to FIGS. 1A and 1B.

FIG. 1A is a schematic configuration diagram of an offset setting system 100 according to an embodiment of the present disclosure. The offset setting system 100 includes a camera 110, an object 120 to which an object marker 122 is attached, a setting tool 130 to which a setting marker 132 is attached, and a computer 140. The camera 110 and the computer 140 are installed in the same place and connected to each other wirelessly or by wire. Alternatively, the camera 110 and the computer 140 are installed at remote locations and are connected to each other via a network (not shown) such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. May be. Although the object 120 is shown as a tool in FIG. 1A, the same applies to the object 120 such as a part other than the tool or a label.

The camera 110 is a camera that can capture images continuously or at regular intervals, for example, a web camera. The camera 110 is arranged at a position where the object marker 122 and the setting marker 132 can be imaged. The object marker 122 and the setting marker 132 are so-called AR (Augmented Reality) markers. By analyzing the marker image photographed by the camera 110, the marker position in the camera coordinate system Σc can be obtained from the marker size, and the marker posture in the camera coordinate system Σc can be obtained from the marker distortion method. The camera coordinate system Σc is a coordinate system with the center of the camera lens as the origin. FIG. 1C shows an example of the object marker 122 and the setting marker 132. Each marker is a quadrangular two-dimensional image in which a white pattern is arranged on a black background. Each marker is assigned a marker identification number and a different bit pattern in which an error correction code is encoded. The AR marker shown in FIG. 1C is an example, and the shape and color of the marker are not limited to this.

The object marker 122 is attached to the object 120 at an arbitrary position away from the detection point 124 and capable of being imaged from the camera 110. As an example, the object 120 may be a tool as shown in FIG. 1A, or may be other objects of various types as described later with reference to FIGS. 3A to 3C. The detection point 124 is set to a position on the object 120 whose position is to be detected. For example, when the object 120 is a tool as shown in FIG. 1A, the detection point 124 is set to the tip of the tool. Note that the object marker 122 may be imaged from the camera 110 and may not be provided on the object 120 as long as the positional relationship with the object 120 is fixed.

The setting tool 130 is used to set a relative positional relationship between the object marker 122 and the detection point 124. The setting tool 130 is provided with a setting marker 132 and a graphic indicating the reference position 134. The reference position 134 is arranged at a predetermined distance from the setting marker 132, for example, at a position away from the center of the setting marker 132 by a distance L in a certain direction. The setting tool 130 may be a plate as shown in FIG. 1A, or when the object 120 is a tool, as shown in FIG. 12, it may be a jig having a portion that can be fitted to the tip of the tool. Good.

The computer 140 includes a processor 142, a memory 144, and a display operation unit 146 as main components.

The processor 142 executes a series of instructions included in the program stored in the memory 144 based on a signal given to the computer 140 or when a predetermined condition is established. In one embodiment, the processor 142 is realized as a device such as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), or an FPGA (Field-Programmable Gate Array). The components included in the processor 142 are just one example of expressing functions performed by the processor 142 as specific modules. The functions of multiple components may be realized by a single component. The processor 142 may be configured to perform the functions of all components. In one example, as shown in FIG. 1A, the processor 142 includes a marker coordinate setting unit 150. The marker coordinate setting unit 150 includes a position detection unit 152 and a coordinate conversion unit 154.

The memory 144 stores programs and data. The program is loaded from the memory 144, for example. The data includes data input to the computer 140 and data generated by the processor 142. In one embodiment, the memory 144 is realized as a volatile memory such as a RAM (Random Access Memory) or a non-volatile memory such as a ROM (Read Only Memory). In one example, the memory 144 includes detection point data 162, as shown in FIG. 1A.

The display operation unit 146 displays the state of each unit and can accept operation input by an operator.

FIG. 1B shows an object marker 122, a detection point 124, a reference position 134 set on the setting tool 130, a setting marker 132, and each three-dimensional orthogonal coordinate system (marker coordinate system Σm (Xm, Ym, Zm). The relationship between the camera coordinate system Σc (Xc, Yc, Zc) and the setting marker coordinate system Σs (Xs, Ys, Zs) is shown in the offset coordinate system 100 in the marker coordinate system Σm with the object marker 122 as a reference. The position of the detection point 124 is obtained, and the marker coordinate system Σm has the origin at the object marker coordinate 126 set on the object marker 122. The posture of the marker coordinate system Σm is set based on the posture of the object marker 122, For example, the XY axis is set on the arrangement plane of the object marker 122, and the Z axis is set in a direction orthogonal to the arrangement plane. Over Ca coordinate system Σs may be set similarly to the posture marker coordinate system Σm of that. Set marker coordinate system Σs the origin setting marker coordinates set on the setting marker 132.

Hereinafter, a method for obtaining the position of the detection point 124 based on the object marker 122 will be described with reference to FIGS. 1A and 1B. Since the positional relationship between the object marker 122 and the detection point 124 is unknown, the setting marker 132 provided on the setting tool for which the positional relationship is determined and the setting marker 132 are provided at positions separated from each other. Using the reference position 134, the position of the detection point 124 with respect to the object marker 122 is obtained.

First, the operator overlaps the setting tool 130 and the object 120 in order to set the relative positional relationship between the object marker 122 and the detection point 124. At this time, if the detection point 124 is to be superimposed on the setting marker 132, the setting marker 132 is hidden by the object 120 depending on the shape and size of the object 120, and the camera 110 moves the setting marker 132. It becomes impossible to take an image. Therefore, in the present disclosure, the detection point 124 overlaps the reference position 134 that is separated from the setting marker 132 and has a positional relationship with the setting marker 132. Then, the reference position 134 is detected as the position of the detection point 124.

The processor 142 (position detection unit 152) analyzes an image including the object marker 122 and the setting marker 132 from the camera image when the detection point 124 and the reference position 134 are overlapped. Then, the position and orientation of each marker in the camera coordinate system Σc are calculated. At this time, as shown in FIG. 1B, the position of the detection point 124 in the camera coordinate system Σc is equal to the reference position 134 in the camera coordinate system Σc. The processor 142 calculates the reference position 134 in the camera coordinate system Σc based on the position and orientation of the setting marker 132 and the distance L between the setting marker 132 and the reference position 134.

Then, the processor 142 (coordinate conversion unit 154) obtains a rotation matrix from the posture of the object marker 122 in the camera coordinate system Σc, and determines the reference position 134 (that is, the coordinates of the detection point 124) in the camera coordinate system Σc as the object marker 122. The coordinate transformation is performed to the marker coordinate system Σm based on the object marker coordinates 126. As a method of converting the coordinates of the detection point 124 into the coordinates of the marker coordinate system Σm, a known three-dimensional coordinate conversion method is used, and detailed description thereof is omitted here. The offset coordinates of the detection point 124 in the marker coordinate system Σm are acquired by coordinate conversion. The offset coordinates of the detection point 124 are the coordinates of the detection point 124 with respect to the origin 126 of the marker coordinate system Σm. The acquired offset coordinates of the detection point 124 are stored in the memory 144 as detection point data 162 for each identification number of the object marker 122. The positional relationship between the object marker 122 and the detection point 124 is fixed, and the positional relationship does not change even when the object 120 moves.

FIG. 2 shows a configuration example of the detection point data 162 stored in the memory 144. The detection point data 162 includes a marker identification number 202, a detection point offset coordinate 204, and a detection point type 206. The marker identification number 202 is an identification number assigned to each object marker 122 obtained by decoding the image of the object marker 122. The detection point type 206 indicates the type of object to which the object marker 122 is attached.

The object marker 122 can be attached to various objects 120 in various ways. FIGS. 3A to 3D illustrate the relationship among various types of objects 120, object markers 122 attached to the objects 120, detection points 124, and object marker coordinates 126. FIG. FIG. 3A shows a case where the object 120 is a glove or a hand. FIG. 3B shows a case where the object 120 is a part. FIG. 3C shows a case where the object 120 is a label.

FIG. 3D illustrates the object 120 with a plurality of object markers 122 attached thereto. As an example, a plurality of object markers 122 a 1 and 122 a 2 (hereinafter collectively referred to as object markers 122 a) are arranged on the same side of the object 120. Even if the camera 110 cannot capture one object marker 122a1 due to a part of the worker's body or a shield, if the other object marker 122a2 can be captured by the camera 110, the position of the detection point 124 is determined. Can be set. As another example, a plurality of

object markers

122a, 122b, 122c are arranged on different sides of the object 120. The plurality of cameras 110 are arranged so that images can be taken from different angles of the object 120, respectively. The camera 110a images the side of the object 120 where the object marker 122a is attached, and the side where the

object markers

122b and 122c are attached becomes the blind spot of the camera 110a and cannot be imaged. The camera 110b takes an image of the side where the object marker 122a is attached, which is the blind spot of the

cameras

110a and 110c, and the camera 110c is attached of the object marker 122c which is the blind spot of the

cameras

110a and 110b. Each side is imaged. By arranging a plurality of cameras 110 at different angles of the object 120, it is possible to reduce blind spots where the object marker 122 cannot be imaged.

In the above embodiment, the offset coordinates of the detection point 124 with the object marker 122 as a reference are set. Next, the configuration of the control system 400 configured to support the worker's work using the set offset coordinates of the detection point 124 will be described.

<First Embodiment>
FIG. 4A is a schematic configuration diagram of a control system 400 according to the first embodiment of the present disclosure. In the present embodiment, the control system 400 compares the three-dimensional position of the detection point 124 in a predetermined coordinate system with the preset three-dimensional position of the detection point 124 to determine whether the operator is working correctly or not. To do. Further, if the worker's work is correct or incorrect, the control system 400 sends a control signal to the control object 460 based on a preset sequence, and a response signal from the control object 460. Receive. FIG. 4A shows a case where the object 120 and the control object 460 (tool 464) are the same (tool 470).

FIG. 4B is a schematic configuration diagram including functional blocks of the control system 400 according to the first embodiment of the present disclosure shown in FIG. 4A. The control system 400 includes a camera 110, one or more tools 470 each having an object marker 122 attached thereto, and a computer 140.

The computer 140 includes a processor 142 and a memory 144 as main components. The processor 142 includes a position / orientation detection unit 410, a position / orientation comparison unit 420, a sequence control unit 430, and a control signal generation unit 440. The components included in the processor 142 are just one example of expressing functions performed by the processor 142 as specific modules. The memory 144 includes detection point data 162, target coordinate posture data 452, and a sequence table 454.

The computer 140 is connected to one or more control objects 460. The computer 140 outputs a control signal to the control object 460 and receives a response signal from the control object 460. The computer 140 and the control object 460 may be directly connected, or a wired communication interface such as LAN (Local Area Network), or a wireless communication interface such as WiFi (Wireless Fidelity) or Bluetooth (registered trademark). It may be connected via.

Each control object 460 includes a tool controller, 462, and a tool 470. The tool controller 462 receives a control signal from the computer 140 or sends a response signal to the computer 140. When receiving a control signal from the computer 140, the tool controller 462, for example, validates the tool 470 (allows various operations of the tool 470) or sets various parameters of the tool 470. When an operation using the tool 470 is performed, the tool controller 462 receives various signals such as a tightening torque from the tool 470. Based on various signals from the tool 470, the tool controller 462 determines whether or not the work using the tool 470 has been successful, and the like, and sends a response signal to the computer 140, for example, a signal indicating that the work has been completed. Is output.

FIG. 5 is a flowchart of a method 500 performed by the control system 400 according to the first embodiment. In the present embodiment, a case will be described in which the object 120 and the control object 460 are the same (the tool 470 in FIG. 4A). Hereinafter, a flowchart of the method 500 will be described with reference to FIG.

Processing starts at step 502. In step 504, the processor 142 (position / orientation detection unit 410) analyzes the image of the object marker 122 attached to the tool 470 (here, the tool 470 as an object) from the image captured by the camera 110, and performs predetermined processing. The three-dimensional position and orientation of the object marker 122 in the coordinate system are calculated. The predetermined coordinate system may be a camera coordinate system Σc set in the camera 110, or a coordinate system Σb (shown in FIG. 8) based on a separately provided reference marker to be described later. Then, the processor 142 calculates the posture of the tool 470 as an object from the posture of the object marker 122 (step 504). The processor 142 may set the posture of the object marker 122 itself as the posture of the tool 470, or may set the posture of the inspection point 124 obtained by multiplying the posture of the object marker 122 by the rotation matrix as the posture of the tool 470. Further, the processor 142 calculates the position of the detection point 124 in a predetermined coordinate system based on the offset coordinates 204 of the detection point 124 of the detection point data 162 stored in the memory 144 and the three-dimensional position of the object marker 122. (Step 504).

Processing proceeds to step 506. The processor 142 (position / orientation comparison unit 420) reads target coordinate / orientation data 452 from the memory 144. The target coordinate posture data 452 stores target coordinates 612 of the detection point 124 in a predetermined coordinate system and a target posture 614 of the tool 470 in the predetermined coordinate system. The target coordinate 612 is the correct coordinate of the detection point 124 in a predetermined coordinate system, and the target posture 614 is the correct posture of the tool 470 in the predetermined coordinate system. The processor 142 compares the coordinates of the detection point 124 detected in step 504 with the target coordinates 612, and compares the posture of the tool 470 detected in step 504 with the target posture 614. A method for setting the target coordinates 612 and the target posture 614 will be described later.

Processing proceeds to step 508. The processor 142 (position / orientation comparison unit 420) matches the coordinates of the current detection point 124 in the predetermined coordinate system with the target coordinates 612 and the current attitude of the tool 470 in the predetermined coordinate system, and the target attitude. Whether or not 614 matches is determined. Alternatively, as another example, the processor 142 (position / orientation comparison unit 420) determines that the difference between the coordinates of the current detection point 124 in the predetermined coordinate system and the target coordinates 612 is within the predetermined position range 604 and Whether the difference between the current posture of the tool 470 and the target posture 614 in the coordinate system is within a predetermined posture range 606 is determined. The position range 604 is a range in which the detected 124 points are determined to be in the correct position, and the posture range 606 is a range in which the detected point 124 is determined to be in the correct posture. Note that the state of the tool 470 may be compared with either one of the target coordinates 612 and the target posture 614, and the match may be determined.

According to the present disclosure, in addition to comparing the coordinates of the detection point 124 with the target coordinates 612 (position range 604), the posture of the tool 470 is compared with the target posture 614 (posture range 606). Therefore, for example, in the sequence of tightening the bolt using the tool 470, the bolt can be tightened at the correct position and the correct posture, and a problem caused by the bolt entering obliquely can be avoided.

6A illustrates the relationship among the object marker 122, the detection point 124, the target position 602, the position range 604, the marker coordinate system Σm, and a predetermined coordinate system (here, the camera coordinate system Σc). The position range 604 is indicated by a sphere centered on the target position 602 in FIG. 6A. The shape of the position range 604 is not limited to a sphere, and may be various shapes such as a hemisphere or a cube. Instead of the camera coordinate system Σc, a reference marker coordinate system Σb (shown in FIG. 8) based on the reference marker may be used as a predetermined coordinate system.

FIG. 6B shows a configuration example of the target coordinate posture data 452 stored in the memory 144. The target coordinate posture data 452 includes at least a sequence number 610, a target coordinate 612 of the target position 602, a target posture 614, and a marker identification number 202. These values are preset. The sequence number 610 is a number assigned to each process defined in the sequence. The target coordinate 612 is indicated by coordinates (xt, yt, zt) in a predetermined coordinate system, and the target posture 614 is indicated by a vector of three orthogonal axes (P1, Q1, R1) in the predetermined coordinate system. A set including a target coordinate 612, a target posture 614, and a marker identification number 202 corresponds to one sequence number. A plurality of sets may be associated with one sequence number. In this case, a plurality of different objects 120 (here, tool 470) can be compared simultaneously with the corresponding target position and target posture. The target coordinate posture data 452 may further include a preset position range 604 and a posture range 606 corresponding to each sequence number. In FIG. 6B, as an example, the position range 604 is indicated by the radius of a sphere centered on the target coordinate 612, and the posture range 606 is an angle centered on each axis of the three orthogonal vectors indicating the target posture 614. Indicated.

A method for setting the target coordinates 612 and the target posture 614 will be described. The operator operates the tool 470 to acquire a camera image when the detection point 124 of the tool 470 is set to the target position 602 with the correct position and posture. From the acquired camera image, a three-dimensional position and orientation of the object marker 122 in a predetermined coordinate system are calculated. A target posture 614 is obtained from the calculated posture of the object marker 122. The target posture 614 may be the posture of the object marker 122 itself, or may be the posture of the detection point 124 obtained by multiplying the posture of the object marker 122 by a rotation matrix. Based on the offset coordinates 204 of the detection point from the object marker 122 and the three-dimensional position and orientation of the object marker 122, the coordinates of the detection point 124 in a predetermined coordinate system are calculated. The calculated coordinates of the detection point 124 are target coordinates 612.

Return to Fig. 5 again. If “Yes” in step 508, the processor 142 generates a coincidence signal and the process proceeds to step 510. On the other hand, if “NO” in the step 508, the process returns to the step 504.

When the coincidence signal is generated, in step 510, the processor 142 (sequence control unit 430) refers to the sequence table 454. The processor 142 determines whether or not the control object 460 (here, the tool 470 as the tool 464) needs to be controlled according to the contents set in the sequence table 454, and determines that the control is necessary ( In step 512, “Yes”), the process proceeds to step 514. On the other hand, if it is determined that the tool 470 does not require control, the process proceeds to step 518.

In step 514, the processor 142 (control signal generation unit 440) generates a control signal for the control object 460 and outputs the control signal to the control object 460. The control signal is a digital binary signal as an example, and indicates that the sequence has started or that the sequence has ended.

Control object 460 executes a predetermined process according to the control signal. As an example, the control object 460 (tool controller 462) enables the operation of the tool 470 as the tool 464. The worker performs work using the enabled tool 464. The tool controller 462 determines whether the work using the tool 470 is completed or successful based on a signal output from the tool 470 as a result of the work. When it is determined that the work using the tool 470 has been completed or succeeded, the tool controller 462 (control object 460) outputs a response signal to the control signal generation unit 440.

In step 516, when the processor 142 (control signal generation unit 440) receives the response signal from the control object 460, the processor 142 (control signal generation unit 440) outputs the response signal to the sequence control unit 430. The response signal is a digital binary signal as an example, and indicates that the input terminal of the control object 460 is active or the input terminal is inactive. The computer 140 can set the state of the input terminal of the control object 460 as a process start condition or an end condition in the sequence table 454.

Processing proceeds to step 518. When the processor 142 (sequence control unit 430) refers to the sequence table 454 and determines that there is no next process, the process ends (step 520). On the other hand, if it is determined in step 518 that there is a next process, the process returns to step 504.

As another example, in step 514, the control signal generation unit 440 generates data including a program number n corresponding to the process to be executed by the tool 470 as a control signal and outputs the data to the control object 460. The tool controller 462 reads a program corresponding to the acquired program number n, and parameters specified in the program, for example, when the tool 470 is a tightening tool, a tightening torque, a tightening angle, a rotational speed, whether reverse rotation is possible, Setting parameters such as the number of retries at the time of a tightening error are set in the tool 470, and the tool 470 is validated. When an operation is performed using the enabled tool 470, the tool 470 outputs various signals, for example, a tightening torque when the tool is actually operated, to the tool controller 462. The tool controller 462 determines whether the signal acquired from the tool 470 matches the set parameter. If the tool controller 462 determines that the signal matches, the tool controller 462 generates a processing completion message and outputs it to the computer 140. In step 516, when the computer 140 receives the processing completion message from the tool 470, the computer 140 refers to the sequence table 454 and ends the processing when determining that there is no next processing (step 520). On the other hand, if it is determined in step 518 that there is a next process, the process returns to step 504.

The computer 140 of the present disclosure does not hold parameters unique to the tool 464 such as the tool 470, for example, tightening torque, and the tool controller 462 holds control parameters unique to the tool 464. The computer 140 inputs and outputs general-purpose signals (control signals and response signals) that do not depend on the type of the tool 464 with the control object 460. Therefore, the computer 140 can be connected to various tools 464 (tools, signal lights, switches, etc.) or tools 464 manufactured by various manufacturers without changing the setting according to the type of the tool 464. The computer 140 can control various types of tools 464 by using the tool controller 462, and can flexibly cope with the process design of production sites using various tools. Replacement of 464 can be facilitated.

As shown in FIG. 4B, the control object 460 (tool 464) may be the same as or different from the object 120. Specific examples of the object 120 when the control object 460 and the object 120 are different are a glove (FIG. 3A), a part (FIG. 3B), a label (FIG. 3C), and a helmet (FIG. 11). Specific examples of the tool 464 when 120 is different are an indicator light, a switch, and the like. For example, when the object 120 is a glove and the control object 460 is an indicator light, the tool 470 as an object is replaced with a glove in steps S502 to S510 shown in FIG. 5, and the tool as a tool is replaced in steps S512 to S519. Tool 470 is replaced with an indicator light.

FIG. 7 shows an example of the sequence table 454. The sequence table 454 includes a sequence number 610, target coordinate posture data 452, a start condition 702, an end condition 704, and a control target output 706. The sequence table 454 is preset by the operator via the display operation unit 146 (FIG. 1A). The start condition 702 is a condition for starting the process for the control object 460, the end condition 704 is a condition for ending the process for the control object 460, and the control object output 706 is a signal generated by the computer 140. Indicates the port to output. Each of the start condition 702 and the end condition 704 includes one of “automatic”, “position and orientation”, “control input X”, and “time T”. The condition “automatic” indicates that the sequence process is unconditionally started or ended when the sequence process is selected. The condition “position / orientation” indicates that, when the sequence process is selected, the sequence process is started or ended by a coincidence signal from the position / orientation comparison unit 420. The condition “control input X” indicates that, when the sequence process is selected, the sequence process is started or ended according to the response signal received from the control object 460. The condition “time T” indicates that, when the sequence process is selected, the sequence process is started or ended after the set time T has elapsed.

As an example of a combination of the start condition 702, the end condition 704, and the control target output 706, the start condition 702 “automatic”, the end condition 704 “position and orientation”, and the control target output 706 “indicator” will be described (sequence number 001). In the process of sequence number 001, the object 120 is a hand, the tool 464 is an indicator light, and the object 120 and the tool 464 are different. First, the process of sequence number 001 is entered unconditionally (start condition). When the hand (object 120) is placed within the predetermined position range, the process proceeds to the next sequence process (end condition). The computer 140 sends a control signal to the control object 460. A specific indicator lamp (tool 464) is turned on based on the control signal. Then, the process proceeds to the next sequence process.

As another example of the combination of the start condition 702, the end condition 704, and the control target output 706, the start condition 702 “position and orientation”, the end condition 704 “external input 1”, and the control target object 460 “none” will be described ( Sequence number 002). In the process of sequence number 002, both the object 120 and the tool 464 are tools. First, a screw at a specific location is tightened at a correct position and orientation using a specific tool (object 120), and the process of sequence number 002 is entered (start condition). When an external input 1 (response signal) indicating completion of tightening from a specific tool is obtained, the process ends (end condition) and proceeds to the next sequence process.

The configuration and the like of the control system 400 according to the first embodiment of the present disclosure have been described above. Next, the configuration of the control system according to the second to fifth embodiments of the present disclosure will be described.

Second Embodiment
When the camera 110 moves in the global coordinate system Σg or the target position 602 changes, the target coordinates 612 and the target posture 614 with respect to the camera coordinate system Σc change. In the second embodiment, as shown in FIG. 8, a reference marker coordinate system Σb is used as a predetermined coordinate system. In the reference marker coordinate system Σb, a target coordinate 612 and a target posture 614 with respect to the reference marker coordinate system Σb are fixed.

8 includes an object marker 122, a detection point 124, a target position 602, a reference marker 802, a reference marker coordinate system Σb (Xb, Yb.Zb), and a marker coordinate system Σm (Xm, Ym, Zm). The positional relationship of is illustrated. The reference marker coordinate system Σb is a coordinate system having the reference marker coordinate 804 as the origin. The reference marker 802 is fixed in position and orientation relative to the target position 602 even when the camera 110 moves or the target position 602 changes in the global coordinate system Σg. Further, the reference marker 802 is disposed at an arbitrary position that can be imaged by the camera 110.

Hereinafter, various modifications of the second embodiment of the present disclosure will be described with reference to FIGS. 9A to 9C. FIG. 9A shows a schematic configuration of the control system 900A when the parts 912 arranged on the conveyor 910 move, and FIGS. 9B and 9C show schematic configurations of the

control systems

900B and 900C when the camera 110 moves, respectively. In the second embodiment, both the object 120 and the control object 460 (tool 464) are tools 470.

FIG. 9A shows a first reference marker 802a attached and fixed to a part 912 on the conveyor 910, and a second reference marker 802b attached and fixed on the conveyor 910. When the conveyor 910 moves, both the first reference marker 802a and the second reference marker 802b move. When the positional relationship between the conveyor 910 and the component 912 is fixed, at least one of the first reference marker 802a or the second reference marker 802b is used. When the positional relationship between the conveyor 910 and the component 912 changes, the first reference marker 802a attached to the component 912 is used. The computer 140 compares the coordinates of the detection point 124 in the reference marker coordinate system Σb (referenced to the first reference marker 802a in FIG. 9A) with the target coordinates 612 of the target position 602, and in the reference marker coordinate system Σb. The posture of the tool 470 as an object is compared with the target posture 614. When the computer 140 determines that the tool 470 is in the correct position and posture, the tool controller 462 outputs a control signal to the tool 470.

9B, unlike the example in which the camera shown in FIG. 9A is fixed, the camera 110 is attached to the object 120, and the camera 110 moves together with the tool 470 as an object. The camera 110 images the first reference marker 802a attached to and fixed to the component 912. The computer 140 compares the coordinates of the detection point 124 in the reference marker coordinate system Σb with the target coordinates 612, and compares the posture of the tool 470 in the reference marker coordinate system Σb with the target posture 614. According to the present embodiment, even if the target position 602 is set at a site that cannot be seen by the operator, the camera 110 can capture the vicinity of the target position 602, so whether the detection point 124 has approached the target position 602. Can be determined.

9C, unlike the example in which the camera shown in FIG. 9A is fixed, the camera 110 is attached to the helmet 932 worn by the worker 930, and the camera 110 moves as the head of the worker 930 moves. The camera 110 captures an image of at least one of the first reference marker 802a and the third reference marker 802c attached to and fixed to the component 912. The first reference marker 802a is attached and fixed to the component 912 on the work table 940, and the third reference marker 802c is attached to the work table 940 fixed in the global coordinate system Σg. The computer 140 uses at least one of the first reference marker 802a and the third reference marker 802c to determine the coordinates of the detection point 124 in the reference marker coordinate system Σb (in FIG. 9C, the third reference marker 802c is the reference). The target coordinates 612 of the target position 602 are compared, and the posture of the tool 470 as an object in the reference marker coordinate system Σb is compared with the target posture 614.

<Third Embodiment>
In the third embodiment, similarly to the second embodiment, the target coordinates 612 and the target posture 614 change with respect to the camera coordinate system Σc (Xc, Yc, Zc). However, in the third embodiment, unlike the second embodiment, the target coordinate 612 is corrected to the target coordinate in the camera coordinate system Σc using the position detection device 920 without using the reference marker 802. In the second embodiment, both the object 120 and the control object 460 (tool 464) are tools 470.

FIG. 9D schematically illustrates a configuration of a control system 800E according to the third embodiment of the present disclosure. In FIG. 9D, the component 912 is arranged on the conveyor 910 to which the position sensor 914 is attached. A control system 900E according to the third embodiment includes a camera 110, an object 120, a computer 140, a position sensor 914, and a position sensor PLC (Programmable Logic Controller) 820. The position sensor 914 detects the movement of the conveyor 910 and detects the current position Cp of the conveyor. The position sensor PLC 820 converts the current conveyor position Cp from the starting point of the conveyor into a physical length (m) and outputs it to the computer 140. Then, the computer 140 calculates the current position of the conveyor in one process when the length on the conveyor in one process is Lp (m). The current position of the conveyor within one process is a remainder obtained by dividing the current position Cp (m) from the starting point of the conveyor by Lp (m). The computer 140 holds a three-dimensional unit vector V (Ux, Uy, Uz) indicating the traveling direction of the conveyor in the camera coordinate system Σc. The computer 140 uses the current conveyor position in one process and the three-dimensional unit vector V indicating the conveyor traveling direction in the camera coordinate system Σc to calculate the target coordinates 612 stored in the memory 144. The target coordinate in the camera coordinate system Σc is corrected. The corrected target coordinate is obtained by Q = ((Cp mod Lp) × V + P. Here, P is the target coordinate 612 of the target position 602 stored in the memory 144 in the camera coordinate system Σc. The target coordinates thus used are used for comparison with the coordinates of the detection point 124 in the camera coordinate system Σc, where mod is a remainder, and the unit vector V is temporarily provided with a reference marker 802 on the conveyor 910, for example, It is acquired by photographing the reference marker 802 at regular time intervals with an installed camera.

<Fourth embodiment>
FIG. 10 is a diagram schematically illustrating a configuration of a control system 1000 according to the fourth embodiment of the present disclosure. In the fourth embodiment, it is determined whether or not a plurality of different types of objects 120 (workers' hands and tools) are in the correct positions or postures using images captured by the fixed camera 110. In the case of the position and posture, the control object 460 (tool 464) is used to assist the operator's work.

The camera 110 includes an operator's hand 120a with an object marker 122a, a tool 464c (object 120b) with an object marker 122b, a bolt bucket 1002 with target positions 602ta and 602tb set, and a target position. The component 912 in which 602tc1 to tc4 are set is imaged. The object marker 122a and the object marker 122b have different identification numbers, and the detection point 124a corresponds to the object marker 122a, and the detection point 124b corresponds to the object marker 122b. The operator is identified by the identification number of the object marker 122a, and the tool 464 is identified by the identification number of the object marker 122b.

As an example, the control system 1000 takes a bolt (not shown) from a specific bolt bucket 1002 (sequence process 1) by a specific worker, and the specific worker sets the bolt to the part 912. In addition, the operator uses a specific tool to assist in tightening the arranged bolt (sequence process 3). Hereinafter, the contents of the sequence processes 1 to 3 will be described in detail.

1. Sequence Process 1: Bolt Extraction In sequence process 1, the object 120 is an operator's hand, and the control object 460 is the

indicator lamps

464a and 464b. In the sequence process 1, the computer 140 activates (lights) the tool (indicator light) 464a, and the specific operator identified by the object marker 122a attached to the glove has a specific position in which the target position 602ta is set. Prompt to collect bolts from the bolt bucket 1002. When the detection point 124a set on the glove approaches the target position 602ta of the bolt bucket 1002, the computer 140 deactivates the tool 464a (turns off the indicator light) and proceeds to the next process. In FIG. 10, the

tools

464a and 464b (indicator lights) are connected to the computer 140, but the

tools

464a and 464b may be connected to the tool controller 462. In this case, the

tools

464a and 464b are turned on or off in response to a control signal from the tool controller 462.

2. Sequence process 2: Bolt arrangement In the sequence process 2, the object 120 is an operator's hand, and the control object 460 is an operator's hand. In the sequence process 2, when the detection point 124a set for the glove approaches the target position 602tc1 of the part 912, the computer 140 determines that the collected bolt is arranged at the correct target position 602tc1, and proceeds to the next process.

3. Sequence process 3: bolt tightening In sequence process 3, both the object 120 and the controlled object 460 are tools 470. In the sequence process 3, when it is detected that the detection point 124b of the specific tool 470 gripped by the operator approaches the target position 602tc1 in a target posture, the computer 140 outputs a control signal to the tool controller 462. The tool controller 462 sets the tightening program 1 based on the control signal. When the bolt is tightened with a torque equal to the tightening torque set in the tightening program 1, the tool 470 sends a tightening completion signal to the tool controller 462. When the tool controller 462 detects the tightening completion signal, the tool controller 462 outputs a response signal to the computer 140 and proceeds to the next step.

4). Sequence processing 4: Bolt sampling (second)
The computer 140 performs the same process as the sequence process 1 on the tool 464b instead of the tool 464a and on the target position 602tb instead of the target position 602ta.

5). Sequence process 5: Bolt arrangement (second)
The computer 140 performs the same process as the sequence process 2 on the target position 602tc2 instead of the target position 602tc1.

6). Sequence processing 6: bolt tightening (second)
The computer 140 performs the same process as the sequence process 3 on the target position 602tc2 instead of the target position 602tc1.

As described above, the control system 1000 can support the worker's work so that the worker performs various processes defined in the sequence process using the specific tool 464 or the specific object 120.

Note that the contents of the sequence processes 1 to 6 are examples and are not limited thereto, and various changes can be made according to the type of the object, the type of the control target, and the like. As an example, when the object 120 is a component (FIG. 3B) and the control object 460 is an indicator lamp, the control system 1000 causes the operator to set a specific indicator 120 identified by the object marker 122 as a preset indicator lamp. It is possible to assist the user in arranging at a specific part position where is lit. As another example, when the object 120 is a label (FIG. 3C) and the control object 460 is an indicator lamp, the control system 1000 turns on an indicator lamp in which a specific label identified by the object marker 122 is set in advance. The worker can be assisted to paste it at the specified position.

<Fifth Embodiment>
FIG. 11 is a diagram schematically illustrating a configuration of a control system 1100 according to the fifth embodiment of the present disclosure. The control system 1100 includes a first computer 140a connected to the first camera 110a, a second computer 140b connected to the second camera 110b, a tool controller connected to the first and second computers 140b, One object 120a (helmet 932) and a second object 120b (tool 470) are provided. The first computer 140a and the second computer 140b are connected to the tool controller 462 via an arithmetic unit 950 that outputs a logical product of outputs from these computers. In the fifth embodiment, the control system 1100 uses a combination of the first camera 110a whose position is fixed in the global coordinate system Σg and the second camera 110b whose position is not fixed.

The control system 1100 according to the fifth embodiment is different from the embodiment shown in FIG. 9C in that no reference marker is attached to the component 912 or the workbench 940. In the fifth embodiment, instead of providing a reference marker on the part 912 or the work table 940, two-step coordinate conversion is performed using the first camera 110a that is in a fixed positional relationship with the part 912.

The positional relationship between the first camera 110a and the component 912 is fixed, and the first object marker 122a attached to the helmet 932 and the target position 602ta in the three-dimensional space are imaged. The first computer 140a calculates the coordinates of the first detection point 124a set on the helmet 932 as an object from the position and orientation of the first object marker 122a. Then, the first detection point 124a is compared with a predetermined position range of the target position 602ta, and the posture of the helmet 932 is compared with a predetermined posture range to determine coincidence.

The second camera 110b is fixed to the helmet 932 worn by the worker 930, and images the object marker 122b and the part 912 in which the target position 602 is set. The second computer 140b calculates the coordinates of the second detection point 124b set on the tool 470 from the position and orientation of the second object marker 122b attached to the tool 470 as an object. Then, the second computer 140b compares the second detection point 124b with the position range of the target position 602tb, compares the posture of the tool 470 with the position range of the target posture, and determines coincidence. The arithmetic device 850 outputs the logical product of the coincidence signals from the first computer 140 a and the second computer 140 b to the tool controller 462. Through a series of these processes, the coincidence determination result by the first camera 110a and the first computer 140a and the coincidence determination result by the second camera 110b and the second computer 140b are combined, and directly by the fixed first camera 110a. In addition, the position and orientation of the target position 602tb in the coordinate system fixed to the part 912 can be detected and the coincidence can be determined without photographing the part 912. In the fifth embodiment, the processing by the first computer 140a and the processing by the second computer 140b are performed, and further the logical product of the processing results by the respective computers is taken, resulting in two-step coordinate transformation. Become. According to the present embodiment, although the first camera 110a can photograph the first object marker 122a, the first camera 110a cannot photograph the component 912 directly, and the second camera 110b can photograph the component 912. In the present embodiment, the target position can be detected even in such a complicated work environment.

According to the fifth embodiment, the fixed first camera 110a can image the object marker 122a that moves together with the non-fixed second camera 110b, and calculate the position and orientation of the first object marker 122a. As an example, when assembling an automobile or the like, the worker 930 may perform work by inserting a hand and a tool into the vehicle while being outside the vehicle. In such a case, the head of the worker 930 outside the vehicle can be photographed by the fixed camera 110a, but the hand and tool 120 of the worker inserted in the vehicle cannot be photographed by the fixed first camera 110a. Sometimes. The hand and tool of the worker 930 can be photographed by an unfixed second camera 110b attached to the helmet 932 of the worker. The fifth embodiment is useful when a range that cannot be captured by a fixed camera can be captured by a non-fixed camera.

Although the embodiments of the present disclosure have been described above, the above-described embodiments of the present invention are for facilitating the understanding of the present disclosure, and do not limit the present disclosure. The present disclosure can be changed and improved without departing from the gist thereof, and the present disclosure naturally includes equivalents thereof. In addition, any combination of the embodiment and the modified example is possible within a range where at least a part of the above-described problems can be solved or a range where at least a part of the effect can be achieved, and is described in the claims and the specification Any combination or omission of each component is possible.

DESCRIPTION OF SYMBOLS 100 ... Offset setting system 110 ... Camera 112 ... Object marker 120 ... Object 122 ... Object marker 124 ... Detection point 126 ... Object marker coordinate 130 ... Setting tool 132 ... Setting marker 134 ... Reference position 140 ... Computer 142 ... Processor 144 ... Memory DESCRIPTION OF SYMBOLS 150 ... Marker coordinate setting part 152 ... Position detection part 154 ... Coordinate conversion part 162 ... Detection point data 202 ... Marker identification number 204 ... Offset coordinate 400, 900A, 900B, 900C, 1000, 1100 ... Control system 410 ... Position and orientation detection part 420 ... Position and orientation comparison unit 430 ... Sequence control unit 440 ... Control signal generation unit 452 ... Target coordinate orientation data 454 ... Sequence table 460 ... Control object 462 ... Tool Controller 464 ... Tool 470 ... Tool 602 ... Target position 604 ... Position range 612 ... Target coordinate 614 ... Target posture 616 ... Marker identification number 802 ... Reference marker 804 ... Reference marker coordinate 910 ... Conveyor 912 ... Parts 914 ... Position sensor 920 ... Position Detection device

Claims

Obtaining a position and orientation of the object marker in a predetermined coordinate system from an image obtained by imaging the object marker attached to the object;
Based on the coordinates of the detection point of the object from the object marker in the marker coordinate system and the position and posture of the object marker, the position of the detection point and the posture of the object in the predetermined coordinate system are calculated. Steps,
Determining whether the position of the detection point is within a predetermined position range in the predetermined coordinate system and the posture of the object is within a predetermined posture range in the predetermined coordinate system;
A method comprising:
A step of outputting a control signal for a control object when the position of the detection point is within the predetermined position range and the posture of the object is within the predetermined posture range; The method according to claim 1, wherein the signal is a signal that does not depend on a type of the control object.
The method according to claim 2, further comprising a step of obtaining a response signal from the control object in response to the control signal, wherein the response signal is a signal independent of a type of the control object.
The predetermined coordinate system is a camera coordinate system when the predetermined position range and the predetermined posture range do not change with respect to a camera coordinate system of a camera that images the object marker. Method.
The predetermined coordinate system is a reference marker coordinate system when the predetermined position range and the predetermined posture range change with respect to a camera coordinate system of a camera that images the object marker. the method of.
When the predetermined position range and the predetermined posture range change with respect to the camera coordinate system of the camera that images the object marker, the target coordinates of the predetermined position range and the target posture of the predetermined posture range are: The method according to claim 1, wherein each of the target coordinates and the target posture in the camera coordinate system is corrected.
The position and orientation of the object marker in a predetermined coordinate system is detected from an image obtained by imaging the object marker attached to the object, the coordinates of the detection point of the object from the object marker in the marker coordinate system, and the object marker A position and orientation detection unit for detecting the position of the detection point and the orientation of the object in the predetermined coordinate system based on the position and orientation of
A position and orientation comparison unit that determines whether the position of the detection point is within a predetermined position range in the predetermined coordinate system and the posture of the object is in a predetermined target posture in the predetermined coordinate system;
A device comprising:
And a control signal generator that outputs a control signal for a control target when the position of the detection point is within the predetermined position range and the posture of the object is within the predetermined posture range. The apparatus according to claim 7, wherein the control signal is a signal that does not depend on a type of the control object.
The control signal generation unit further obtains a response signal from the control object according to the control signal, and the response signal is a signal independent of the type of the control object. The device described.
Calculating the position and orientation of the object marker in a predetermined coordinate system when the detection point of the object to which the object marker is attached overlaps the reference position of the setting tool;
Setting the coordinates of the detection point of the object based on the object marker in a marker coordinate system based on the reference position and the position and orientation of the object marker;
A method comprising:
The method according to claim 10, wherein the reference position is arranged at a position away from a setting marker attached to the setting tool.