WO1995017995A1 - Position and attitude detecting method, apparatus for practicing the same, and flexible production system using the same apparatus - Google Patents

Abstract

This invention aims at providing a production system of a high autonomy, which is constructed as follows. A simulated work environment scene generating means (20) is adapted to simulatively synthesize, when a visual robot (15) and a working robot (14) are in designed positions, a scene, which should be photographed by the visual robot (15), on the basis of the data stored in a work environment data storage means (29), and the visual robot (15) is adapted to actually photograph the scene of a work environment (1). A work environment scene comprehension means (22) is adapted to compare the real image with the simulated scene and detect deviations between the positions and attitudes of the working robot (14) and visual robot (15). Accordingly, even when an absolute reference point is not imaged on the visual robot (15), the position and attitude of an object of the work can be detected and corrected, and a production system of a high autonomy can be implemented.

Description

 Position 方法 ι? (· Flexible production system using posture detection method and device

 The present invention relates to a field of a production system that utilizes a computer to automate a product assembling operation, and a method and an apparatus for detecting a position and a posture of an object to be operated by a robot system without tees. A book on flexible production systems using this. In particular, a pseudo-work environment model created by computer graphics based on product design data, machining / assembly equipment data, and peripheral device data, and an actual work environment obtained by a visual device using a TV camera were compared. In contrast, the present invention relates to a position / posture detection method and apparatus suitable for realizing a function capable of autonomous work, and a flexible production system using the same. Background art

Until now, the production system was a low-mix, high-volume production system for supplying large quantities of uniform products. However, in order to meet diverse customer needs, the emphasis on value has shifted to product differentiation and individualization, and with the diversification of products, multifunctionality, and shortened product life, many products There is a demand for the development of a production system for small-lot production. Therefore, in order to supply products that meet the needs of customers in a timely manner, orders, design, connecting the whole of the manufacturing system of diversified soft city corresponding to the production) ^t'l:. High off Rekishiburu production Systems are being demanded. In addition, the total assembly process for home-m products is mainly manual work, However, it is difficult to reduce the number of people, so the demand for automating this is iff ', レ,.

 In the traditional deer method, robots work in a teaching playback style. Although it is becoming increasingly possible to determine the robot's operation sequence offline, it is common practice to perform online positioning for ^ H f reports. In addition to the need to stop for a long time, it also requires skilled workers to operate the robot, and if many robots are used, it takes a long time to start up the entire production line.

 On the other hand, in order to realize a flexible production system, it is necessary to change the equipment configuration and the position of the robot based on instructions from the work planning gate. For this purpose, it is necessary not to bring the robot to the correct position, but to make the robot autonomously act as if it is at the incorrect position. As a conventional example of such a configuration, Japanese Patent Laid-Open Publication No.

15 The first prior art). The mouth control method according to the first prior art is intended to give information about a moving object existing in the work environment に to the robot and to cause the mouth pot to perform an appropriate action. For this purpose, a knowledge database was used to store knowledge such as action procedures and actions according to various situations.

 ? 0 In addition, a simulated cocoon image of the working environment is displayed in real time, and the real environment and the simulation environment are displayed as overlays, and the position and posture of the simulation environment model are displayed in real environment. A conventional example in which a person can modify the position and orientation data of a simulation environment model to ί? Π so that it matches that of a real environment is disclosed in Japanese Unexamined Patent Publication No. - Five

^{2Γ, ■ Γ) I 9 '} 1 JP (hereinafter, described in the second referred to as the prior art). This first conventional technique aims at remote control of a mouth bot, and image analysis itself is not performed. People were going.

 In addition, even if the image of the object 'ii' is hidden by the image of the first operation ffl, the object can be recognized, or the relative position between the operation object and the operation hand can be determined. A conventional example of displaying a {π} image that enables human recognition is described in Japanese Patent Application Laid-Open No. H08-108 (hereinafter referred to as a third prior art). Again, MJ's work was 1 = 1, and image analysis itself was done by humans.

 By the way, you may want information widely at a certain point. In that case, it can be made possible by twisting the direction of the TV camera. This prior art is described in Japanese Patent Application Laid-Open No. 2-83194 (hereinafter, referred to as a fourth prior art). However, this conventional technique is intended to display a robot that can be easily viewed by a person who remotely controls a robot, and has not been considered in image analysis.

 As described above, in order to realize a flexible production system that can supply products that meet customer needs in a timely manner and that is compatible with low-volume, high-mix, low-volume production that links the entire system of orders, design, and manufacturing. To do so, the configuration of production equipment must be changed, and the position of the robot must be changed. In order to manufacture under these conditions, it is necessary to increase the reliability of the entire equipment. To do this, confirm that the work target is in the expected position according to the work plan, and if there is a deviation from the expected position, know the deviation and take action to respond to the deviation It is necessary to give autonomy to production systems.

However, none of the above-mentioned fourth prior arts has been neglected to realize the I: characteristic of this production system. In particular, in the conventional technique of η ·, the knowledge database stores the knowledge such as the robot's action procedure and the action based on various situations, and exists in the work environment. The thing that makes the mouthbot do the correct action based on the information that mimics the object)), and where to find the scene input device ^ j to pray the work cycle obtained as iff jf but what good sea urchin glance Eru must be known in advance. "in the prior art, and how visible the work object, the peripheral device • ^r etc. or obtain Μ, if the mouth box Bok visible force ,, part What does it look like in ' ^: w, I understand it-there are no steps, and I did not consider analyzing the actual working environment.

 In order to analyze the actual work environment, it is necessary to use the widest possible range of information as densely as possible. In the fourth prior art, a TV camera

The ability to use a wide range of information by turning 10 directions', because at that time, the work environment captured as information is simply displayed on the screen However, it cannot be used as work environment analysis. In view of the above-mentioned problems of the prior art, the tree invention analyzes the working environment and corrects the position / posture deviation of the robot or the like based on the analyzed results.

1) An object of the present invention is to provide a method of detecting a position and a posture of a photographed object, which can accurately grasp the position and the posture of the object and automatically correct the deviation. Another object is to provide a position / posture detecting device capable of accurately implementing the above method, and another object is to provide a flexible production system having high flexibility capable of coping with high-mix low-volume production. To provide.

 20

 Disclosure of the invention

In the tree invention method, shooting of the object in the working environment by the scene detecting means is performed by changing the position of the principal point of the camera lens in the scene detecting means as a center, and a plurality of し た ιΠί images obtained from-' The image is projected onto a plane and synthesized into one wide-angle image, and: fi, _f JI j is output to 'Factory': J¾ and upper fi ' Upper position ^ · Assuming that you are in the posture, the above scene detection means is shooting 7) The Ίill image is formed into a pseudo Ι'1'J, and the object to be photographed on the pseudo image is

The position of each characteristic m κ of I '. ^ And — li'f ¾ detection ^ stage

|: In the image, the (, ',: " ^ν ί. Ί: no i i .'Ϊ” of each feature point corresponding to the above feature points on the ttU object in the image is extracted, and the based on the above-mentioned scene detection means and the upper ^{r), ϋ ', w \} h \') ί, ν 1;. '.':.! · ¾, v and that Osamu to 1 by the pair ^ things to ·, ii i iii) The deviation is increased by the amount corresponding to! 'ί;ti'i: Either the detection means or the design data for moving the object is corrected.

 Further, in the method of the present invention, at least two scene inspection means capable of capturing an image of an object to be processed and a processing device for processing the object (and their peripheral devices) are installed in advance. The positions and postures of each of the working environments consisting of these two scene inspection means, the work object, the processing equipment, and the peripheral equipment at the predicted time are simulated as images based on pre-designed data. Then, when the working ring reaches the time axis at which the pseudo image was generated, the actual working environment is imaged by each scene inspection means, and the data of the actually shot working environment and If there is a discrepancy between the two and the data that generated the pseudo image, correct the position and orientation in the work environment based on the discrepancy and IE. Modify each corresponding design data And having to perform one of the possible.

0 The position of the tree invention ft · Attitude detection device i: scene detection means for photographing the object, the information: actual data indicating the actual position of the detection means, and the design of the object Assuming that the object to be photographed is in the design position and posture described above, and that the object to be photographed is in the design position and posture described above, the li'f output stage will be photographing. An image is pseudo-created using the actual data and the r, set-il data, and the position of each feature point on the object in both the pseudo-images, and The scene detection means detects the object to be photographed. ■ H / ί / ιί! The deviation between the above-mentioned feature points on the object to be photographed and the corresponding に お け る 'ι: point ί,' /:, ': in the' removed inn 'image is calculated, and the deviation is calculated based on the deviation. The position of the object to be photographed; andii) a means for performing either of ii) correcting the corresponding design data in the working environment by the amount of the deviation. Features.

 In addition, the position detection device for detecting the position of the tree can process the work object, the processing device for processing the work object, these peripheral equipment, the work object, the processing device, and the peripheral device. A work environment having at least two scene inspection means, a work object, a processing device, a peripheral equipment, a design data storage device storing information on the shape of each scene inspection means and characteristic points, and a work object , Machining equipment, peripheral equipment, work procedure data for each scene inspection means, work process data, and operation plan data, which are stored in the design data storage and work plan data storage, respectively. Based on the data obtained, the position and orientation scenes at each predicted time in the working environment are simulated, and when the simulated time axis is reached, the actual working environment Each scene In addition to imaging by the inspection means, each position and orientation of the work environment is detected by comparing the position and posture scene of the work environment in which the image data is generated in a pseudo manner, and the detected deviation is absorbed. And an assembling station control means for correcting the position and posture of each of the working wheels in the direction.

In the flexible production system of the invention of wood, the work object and the work object are processed |! A working device having at least two scene inspection means capable of imaging the work object, the peripheral device, the work object, the processing device, and the peripheral forgery; and a work object, the processing device, the peripheral device, A design data storage device that stores the information that corresponds to the shape and characteristic points of each scene inspection means, and the work procedure data and work process data for the work object, machining, peripheral equipment, and each scene inspection stage. To 1 '! 1¾ Π ·: Work plan data storage device that stores road data in a flash, and design-data t £ ¾ ^ and Π; total data record It is based on the data stored in the device. , (Vm ¾)>-) at the position of the predicted time ^. At the time when the posture scene was pseudo-generated and the pseudo-generated position reached the axis, the actual work was performed. The environment is imaged by each of the inspection means, and the image data is compared with the position / posture scenes of the simulated work environment to compare the position / posture of the work environment. A position / posture detection device is provided which has assembly station control means for detecting a deviation and correcting each position / posture in the working environment in a direction to absorb the detected deviation. Autonomous flexible production system

For each system, it is necessary to confirm whether the work has been performed according to the work plan and whether the next work can be performed as planned for each work. For that purpose, it is necessary to first detect the three-dimensional position and posture of the work object from the two-dimensional image obtained by the scene input sensor. In addition, it is necessary to always use the scene input sensor to see the work point or the like that you want to check.

15 It is necessary to change the position of the scene input sensor to an appropriate position as necessary, and to perform a so-called “looking-in” operation. For that purpose, it is necessary to accurately detect and correct the position of the scene input sensor. In this case, a specific reference point is provided, and the position cannot always be detected using the reference point. This is because the scene input sensor moves as needed and is 20.

 Therefore, in the tree invention, the scene input sensor and the actual position and orientation of the object are detected and corrected by the method described below, and their outlines will be described.

First, when the ji input sensor is in the specified position 'on' at the position specified on the work plan, in the case of the work plan, the design data and the work Computer graphics based on total data ス ίΐ ΐ ίΐ ΐ ΐ ίΐ 生成 生成 生成 生成 生成Here, the scene input sensor preferably has a zoom function.

Then, nii i: in the performed pseudo work, select a feature iili of the expected feature point such as a color or an edge, and in a real work environment, π′I ′, ( ¹ ! In this ¾, the work ^ 垸 is already I.

 Many production facilities such as production lines are also known. Therefore, it is possible to always prepare a feature point that can be easily processed. By capturing the scene by widening the scene input sensor with the zoom function, it is possible to identify some of the feature points. From the difference between the simulated work environment screen position of the characteristic point whose spatial coordinates are known and the actual work environment screen position captured by the scene input sensor, it is possible to determine the coarse position of the space の of the scene input sensor. it can. If necessary, if a scene is captured by a scene input sensor with a further enlarged zoom function, a precise three-dimensional spatial position can be determined.

5 Finally, superimpose the actual work environment screen and the pseudo work screen to check whether they match. If all scenes match and there is no difference from the work plan, the work has been performed according to the work plan. On the other hand, if there is a difference, the pattern matching process is performed for each part or each assembly part based on the design data and work plan data to determine which part is different. If the posture is different, change the corresponding data in the database. If only the posture of the work robot is different. If only the posture is different, change the position and posture of the work robot. In addition, when the background of the entire scene or the work robot is out of alignment, the position and orientation of the scene input device are changed, and the scene of the actual work environment is modified to match the scene of the computer graphics. Work plan The work is performed as per the 'work plan'.

Next, based on the two-dimensional image obtained by the human sensor, 'Li'i jil: The human-power sensor (television camera), which is displaying the image, and the method of detecting the dimensional position and orientation of the work object being photographed will be explained in detail. .

In KiKiwamu pseudo created from pre-given position coffer data such as ^{", (1:: Ϋ; Γ'ίϋ¾} ρΐίϋΓιί t, from the real work樂環¾ lotus ¾ of both而位^ obtained by Ίίίϋ input sensor Figure 4 shows a model of the working environment that is the basis of the following explanations.The robot and the work are assumed to be rectangular in the explanation. In this work environment, one unit as a scene input sensor is referred to as “object j”.

There is a self-propelled robot equipped with 10 TV cameras and 4 objects. The TV camera should be set to the position and posture where the object can be seen.

 As shown in Fig. 4, a static coordinate system independent of each individual object is provided in the work environment. This conflict coordinate system was introduced simply to facilitate processing such as coordinate transformation. On the other hand, TV cameras are

It has a visual coordinate system of 1. The four objects also have independent object coordinate systems, and the coordinates of the feature points (vertexes) of each object are defined on the object coordinate system. On the other hand, each coordinate of an object in the visual coordinate system and the object coordinate system (hereinafter, the visual coordinate system and the object coordinate system are sometimes collectively referred to as “local coordinate system”).

In general, the origin of the 20-kind is defined on the coordinate system of conflict.

As described above, while the origin of the object coordinate system is defined on the stationary coordinate system, the coordinates of the vertices of the object are defined on the object coordinate system, so that the movement of the "object" is defined. However, it can be simply grasped as g of the “object coordinate system” on the stationary coordinate system. Therefore, the object coordinate system? The coordinates of the feature points of each object do not change at all, as in the case of the vertices defined on the object. Each of these coordinate systems Figure 5 shows the relationship between the two. The width of the figure, S is the origin of the stationary coordinate system, C is the origin of the visual coordinate system, and is the ^ point of the object coordinate system.

 In addition, in the next child space I :. (in this case, on the stop coordinate system), the position of the roll coordinate system: · Six parameters are required to specify the posture

In other words, as the parameter of the position ^, the origin of each local coordinate system must be?> The position on the stationary coordinate system ^ (x, y, z). Therefore, the rotation angles 0x, 0y, and 0z of each oral coordinate system around the X, y, and z axes of the stationary coordinate system are required. The origin C of the visual coordinate system in (C x,

10 C y, C z), and the orientations of the visual coordinate system around the X, y, and z axes of the stationary coordinate system are denoted by G 1, 0 2, 63. Similarly, let the position of the origin P of the object coordinate system on the stationary coordinate system be (P x, P y, P z), and the object around the x, y, and z axes of the stationary coordinate system. The attitude of the coordinate system is shown as 0α, θβ, and 0γ.

 15 Next, creation of the pseudo work environment screen will be described. To create the simulated work environment screen, all of the previously given three-dimensional data on the shape and feature points of the object, the position and orientation where the object should be placed, etc. are temporarily stored in the static coordinate system. (In this case, the conversion is not limited to the stationary coordinate system, but may be converted to any one of the coordinate systems.)

This is done by transforming them into a visual coordinate system and then into a two-dimensional image captured by a television camera.

 To convert the coordinates defined on the above visual coordinate system to the conflict coordinate system, first, translate by “x, Cy, Cz”, and then move the y-axis, X The axis and z axis are rotated by (? 2, Θ1, 03, respectively)

^ You can do it. To convert the coordinates defined on the object coordinate system to the stationary coordinate system, first, translate by (P x, P y, P z) )} ¾, jVf. For y, x¾, and zli in the IΚ system, only 0/3, 0n, ί) γ should be applied.

 This E E E

 Z X Y n:

 Equations 1 and 2 show C c c ¾ Λ. Note that here

To convert 化 to Di ' ¹ , 5; 11 () 1 to 8. 0 5 0 1. This is denoted by i

 h. The coordinate transformation matrix τ c s for converting, ':, (on the SC

'(: -T """(C _X , C) .. C _Z ) R ₍ , T (y, G ₂ ) () T (XO,) ΐ Πζ.θ ·))

C ₂ C ₃ + S, S ₂ S3 S, S ₂ C ₃ -C ₂ S3 c, s ₂ c _x \

C) S ₃ C, C., -Si Cy S2 O ϋ

_{S, C 2 S.vS 2 C3 S} 2 S iS, C 2 CC, C 2 c 7. ( Number 1) oo

 Can be expressed as Trans in the equation is a translation transformation matrix. R ot is a rotation-transformation transformation matrix, and R ot (y, θ 2) rotates 0 2 times around the y-axis c s oo

15 means that o ool

 Similarly, a coordinate transformation matrix T p S that transforms a point on the object coordinate system for the controversial coordinate system into a stationary coordinate OOOL coordinate system is

if '= Trnm

(Equation 2) can be expressed.

 20 Therefore, if any point E (E x, E y, E z, 1) on the controversial coordinate system is transformed into the m ft coordinate system,

E (number 3)

'2 · ^Γ , In addition, the position vector O (O Xp, O Yp, 任意 I) is transformed into the ¾ party ^ coordinate system, the coordinates o _c are

(A: o (number)

And

 10. “Coordinate data collected on the visual target system by using the conversion formulas of several to four are converted to two-dimensional images (corresponding to screens captured by a TV camera) as described above. You.

 Here, a description will be given of a method of obtaining screen position coordinates when converting the coordinates Ec of an arbitrary point E converted on the visual coordinate system into the two-dimensional image.

The coordinates Ec of an arbitrary point on the visual coordinate system form an image on the point G of the image pickup し て through the lens as shown in FIG. The visual coordinate system is set so that its origin is located at the center of the lens. Points 1 and 2 in the figure are on a plane perpendicular to _Z c 座標 in the visual coordinate system. Similarly, points 3 and 4 are also in a plane perpendicular to the zc axis of the visual coordinate system. Therefore, the triangle formed by points 0, 1 and 2 in the figure is similar to the triangle formed by points 0, 3 and 4. Therefore, point G on the imaging surface is

…… (Equation ⁵ )

Here, SX is a coefficient representing the apparent focal length in the X direction, and S y is a coefficient representing the apparent point in the y direction!) "Where S x and S y are apparent, '. 'Α · · ί / | i I and I'-This is not the distance from the principal point of the lens to the imaging position of the TV camera, but the image of the object imaged on the imaging surface of the TV camera. Faith The, 'I is converted into pixel columns can be punished Present in grayed / digital Hen换at [pi I-calculation machines'Atsu-> -, until ί'ϋι' ϊίϋ of If I \ ^k) was} et i, This is because it means the apparent distance from the principal point of the lens.

 Similarly, a point G p on the screen of an arbitrary point o on the object coordinate system is

-= (Equation 6) |, 丫 /, EYC, Syj 0 / .c.

 By using the above-mentioned equations consisting of Equations (1) to (6), based on the design data,

Figure 10 shows how the object looks on the TV camera. When there are multiple objects at the same screen position, the distance to the television camera can be determined based on the distance information represented by OZc, and only the object that is positioned on the near side can be displayed. In this way, a pseudo work environment screen can be generated. Second, the TV camera obtained the two images obtained by shooting the actual working environment.

This section describes how to obtain three-dimensional space information by using one-dimensional screen position information.

 The three-dimensional space position information is obtained by comparing the simulated work environment screen obtained by the above processing with the work screen actually obtained by shooting with the TV camera.

ず れ can be obtained by using iili. In this case,

The difference between the two is extracted as the deviation of the position and orientation of the TV camera itself, and the difference is extracted as the deviation of the position and orientation of the captured object, etc. Processing ^ etc. Therefore, in the following, both processes will be described separately.

 First, the case where the TV camera is treated as a shift

In this case, it is assumed that the actual position of the object is known and that the posture is known. If the liL television force camera is in the position and attitude as designed, the force that the object can see on iii テ レ ビ ι メ of the GG television force camera Create ί. Then, the position and orientation of the television camera are determined by processing the created pseudo image and the execution surface by J-Dagger. The comparison process is performed by performing the following mathematical operation.

 わ か る As can be seen from Equations 3 and 5, the G components GX and G y in Equation 5 are the position (C x, C y, C z) of the origin C of the visual coordinate system on the stationary coordinate system. , Postures 0 1, 0 2, 0 3, and apparent focal lengths S x, S y. When the visual coordinate system (television camera) is moved on the stationary coordinate system, the actual position of the TV camera (CX, Cy ', Cz', θ1 ', 02', 63 ', Sx ', S y') and the design position (C x, C y, C z, 01, 02, 03, S x, S y). z Θ Θ 1, ひ 厶 2, 0 0 3, S S x, S S y). At this time, the actual values of the parameters C x ′, C y ′, C 7, ′, θ 1 ′, 0 2 ′, 0 3 ′, S χ ′, S y ′

 Q = C-, + m Cj i = x, y, z

θ _j θ '' + m 9j j = shi 2, 3 (Equation 7)

_{S k = Sk + ASk k =} x, and the relationship of y holds. At this time, the position of the point on the screen is

G _X + AG _X

 G (number 8)

G _Y + AG _Y / Here, AGx and ΔGy are positions on the screen generated by the difference between the actual position and the design position. When the parameter errors △ C χ, ΔC y, Δ C ζ, mm θ 1, △ 0 2, mm fl 3, mm S x, Δ S y are sufficiently small, the position difference G is _Z

(Number 9)

For example, the first term of the mumm GX can be expressed as ^-S Exc Exc 3E _ZC , ΔC _X (Equation 10) dCx ¾c \ dc _x ¾c ac _{x from} Equation 5. Further, 5 E xc / SC x, δ E zc / δ CX included in the equation 10 are, according to equations 3 and 1,

3C _X

 cEyc

3E _C OCx ^ c)-'

 E

^ Cx cE _Z c 3Cx

 acx

 1

-Κ _ο (ζ, -0 ₃ ') RoT (x, -Oi) Ro ( _y , -0 ₂ ^Tra " ^{s (} ' ^Cx '" ^Cy, ' ^{C /)} . The same applies to the term.

From equation (9) above, two relational expressions hold for one feature point. n boku The metaphysical i-difference Δ G n

AG ,, = R _n · X (Equation 12). Here, Δ Ο η, X, and R n are the screen position difference の] column of the feature point, the parameter error matrix, and the partial differential coefficient matrix, which are expressed by the following Equations 1 Λ ′ to 15, respectively. .

 = (AGxi .AGy.AGxn.AGY ,,) '(Equation 13)

X = (AC _X , mu CY'ACz'At 'AG ^' AC .ASx, mu S _Y ) (Equation 14) dG i 9Gx

3s _x as dS y (number 15)

9G _Yn 3G _Yn

as _x as _y ;

 By the way, A G n in Equation 13 is obtained from the screen positions of the feature points of the pseudo image and the real image. By substituting the design data into Equation 15, the partial differential coefficient matrix R n can be calculated. Therefore, the parameter error X can be obtained by the following equation from Equation 13.

X = (Ri R ,,) ' ¹ ' Rj 'AG _n (Equation 16) This means that simultaneous equations are solved by the least-squares method using the pseudo-inverse matrix as / π. Accuracy increases. Conversely, if there are at least 4 feature points,

 -1

 (R η Τ · R η) can be calculated, and the parameter error X can be obtained. In addition,

R η Τ is the transpose of R η. Here, Equation 12 is an approximate equation when the parameter error is assumed to be sufficiently small, and Equation 16 is a solution of the approximate equation. The actual solution is to substitute the approximate solution into Equation 7 to update the design value, substitute this into Equation 15 and perform a repeat operation to find Equation 16 to converge within a certain range Ask for. By this calculation, the position and orientation of the visual coordinate system (television camera) on the stationary coordinate system and the apparent focal length can be determined. Hereinafter, this is simply referred to as “TV camera positioning method”.

 Next, a process will be described in which a difference between the work screen and the simulated work screen is defined as a shift in the position and orientation of the target object. In this case, it is assumed that the actual position and orientation of the television camera are known in advance. In addition, as the simulated work screen, when the above-mentioned object is in the position and orientation according to the predetermined design, an image showing how this object looks on the screen of the TV camera is created. I do. Then, the position and orientation of the object are determined by comparing the image with the actual screen. The comparison process is actually performed by performing the following mathematical operation.

 The components G p X and G py of G p in Equation 6 are the position (P x, P y, P z) of the origin P of the object coordinate system on the stationary coordinate system, and the posture Θ, θ β , 0 y and the apparent focal length of the TV camera SX, S y, which can be represented by eight parameters.

When the object coordinate system (object) is moved on the stationary coordinate system, the actual position of the object (P x ′, P y ′, P z, θ α ′, θ β, θ γ ′, S χ ', S y') and the design position (P x, P y, P z, Θ a, θ β, Θ y, S x, S y) , Pz, Δ0α, AΘft, Δ0γ, ΔSχ, ΔSy), the difference between the pseudo work screen and the screen actually shot is Is generated. Therefore, in the same way as the TV camera position determination method described above, if the parameter error is obtained from the screen position difference and the partial differential coefficient matrix, the actual position, posture, and apparent focal length of the object on the stationary coordinate system can be obtained. Can be determined. Hereinafter, this is simply referred to as “object position determination method”.

 As described above, the difference between the above-described TV camera position determination method and this object position determination method is that the former is based on the feature point information of a plurality of objects on the screen, and the position of the self (TV camera) is determined. Whereas, the latter is a point in which the position and orientation of the target object itself are determined based on the feature point information of the target object. -The principle of the object position determination method described in the above description is based on the assumption that the actual position and orientation of the TV camera are accurately known. If the exact position is not known, a pseudo image viewed from the lens principal point different from the actual TV camera will be created, and as a result, the accuracy of the object position determination will be degraded. For the same reason, the above-mentioned TV camera position determination method is based on the premise that the actual position and orientation of the object position are accurately known. In other words, the two depend on each other, and accurate position detection etc. cannot be performed unless the actual position of either the TV camera or all the objects is accurately known in advance. .

However, when a TV camera is installed on the arm of the moving robot, it is expected that the TV camera and some of the objects will be off the design position at the same time. Therefore, there remains an inconvenient problem that the application of the object position determination method or the like is limited to a specific situation where the television camera is fixed, for example. Therefore, the inventor of the present application has also been able to solve the problem (1) to the extent that there is no practical problem by the first and second two methods described below. Here, the accuracy of determining the position of the TV camera depends on the correctness of the position of the object, so if the position of the object is determined based on the position of the TV camera determined based on position data that is not at the design position, Of course, the accuracy will be worse. Then, in order to determine the exact position of the object, first find the object that is misaligned, and then use the feature point information of the object as the rest of the design position, and once again use the TV camera Determine the position.

Further, the position / posture of the object that is displaced at the determined position of the TV camera 'is determined.

 First, the outline of the first technique of the present invention will be described.

 First, the position of the TV camera is temporarily determined using multiple objects. Then, those objects that are out of position are searched for. Then, the position of the TV camera is determined again using the feature point information of the object located at the design position. After that, the position and orientation of the object whose position has been shifted are determined this time at the re-determined position of the TV camera.

 Hereinafter, the method will be described in detail with reference to FIG. In this case, in the initial state, among the four objects P 0, P 1, P 2, and P 3, P 0, P 1, and P 2 are in the correct positions as designed, but the objects P 3 and P 3 It is assumed that the TV camera is out of the design position.

 [Step a 1]

Creates a screen that can be seen when all four objects PO, P 1, P 2, P 3, and the TV camera are at the design position, using the design data that we have in advance. . Then, using the created pseudo image and the image actually taken by the TV camera, the TV camera position determination method described above (in this case, the positions and postures of the objects P0 to P3 are all Perform assuming that the camera is in the correct position.) That is, the position of the origin of the visual coordinate system on the stationary coordinate system and the posture of the visual coordinate system are obtained. Hereinafter, the position and orientation of the television camera obtained here will be simply referred to as “estimated camera position”.

 However, since the feature point information of the object P 3 whose position is displaced is also used, the position テ レ ビ attitude of the TV camera obtained here, that is, the estimated camera position is different from the actual position on the stationary coordinate system. Are not exactly the same.

 [Step a 2]

 Subsequently, the estimated power obtained in step a. Using a design data which has a television camera at the camera position, and which can be seen when each object is at the design position, using the design data which has in advance. Create a pseudo. Then, using the created pseudo image and the image actually taken by the TV camera, the above-described object position determination method is used (in this case, it is assumed that a TV camera is actually present at the estimated camera position). The position and orientation of each object P0 to P3 on the stationary coordinate system, that is, the position of the origin of each object coordinate system and the orientation of each object coordinate system on the stationary coordinate system are provisional. Is decided. Hereinafter, the position and orientation of the object obtained here are simply referred to as “estimated object position J”.

 However, the position of the estimated object obtained in this way does not always match the actual position and orientation of the objects P0 to P3 on the stationary coordinate system. This is because the coincidence between the estimated camera position assumed when performing the object position determination method and the actual position of the television camera is not always true.

 [Step a 3]

In the three-dimensional space on the stationary coordinate system, each object is set to the above-mentioned estimated object position, and the television camera is set to the estimated camera position. Was However, the setting in this case is performed in a simulated manner, and is not a setting in which a television camera or each object is particularly set. After that, the above objects are moved to the similar position '1-J up to the design position. In this case, along with the movement of each of the objects, the TV camera also moves while maintaining the relative position r, i system of m! Π between the mΐ \ ΐ \ object ^ and the camera position ^. The camera moves in a simulated manner, and the position of the TV camera when each object is at the above-mentioned setting in- (, '/:' Attitude. The position of the TV camera obtained here is referred to as “; The next estimated camera position is called j.

 The details of the specific arithmetic processing when the above processing is actually performed will be described below. Assuming that the estimated force position obtained in step a 1 above is C 2 x, C 2 y, C 20 z, 0 2 1, fl 2 2, 0 2 3, it is defined on the controversial coordinate system The matrix for converting the coordinates into the stationary coordinate system can be expressed as shown in the following Expression 17 from Expression 1.

_{'1 2 = T ra "s} (. C 2x, C 2y, C 27) · R 0 T (y, G 22) · 0 T (x, e 2 i) - RoT (z, 6 23) ...... ( (Equation Π) Also, assuming that the estimated object position for the object P 0 obtained in the step a2 is PO x, PO y, Ρ Ο 0, 0 0 α, 600, 0 γ γ, the object A matrix for transforming the coordinates defined on the ^ coordinate system into the stationary coordinate system can be expressed by the following equation (18) from equation (2).

· R ₀ T (x, e _0a ) · R ₀ T (z, e ₀ J …… (Equation 18) The feature point sequence X n on the object P 0 is converted to a visual coordinate system by 0. Assuming that Y n, Y n is expressed as in Equation 19.

Υη

· Χ _η …… (Equation 19) On the other hand, ί) the position when the target object Ρ 0 is at the design position. Attitude, that is, the design position of the target object P 0 is represented by P t 0 x, P t O y, P t 0 z, 0 t O a, ί ί 0 β ί ί) t Ο γ, the coordinate transformation matrix to the controversial coordinate system is expressed by the following equation (20) from the above number. Can be.

'if

-o (y, e _{l0 [} ] () · RoT (x, 0 _lO a) _-R.T . θ _ι0γ )… ”-( _Equation 20) The actual position of the telecamera [1 · Assume that the posture is C3X, C3y, C:? 7, 0Λ-J32, 033. In addition, a matrix for transforming the target (defined in the visual coordinate system I.) into the stationary coordinate system can be expressed as in the following Equation 21.

'ic3 = _rais (c ₃ x, C3Y, c ₃ z)-Ro'r (ye ₃₂ '), R ₀ T (x, e _3l ). R ₀ T (z, e ₃₃ ) …… (Equation 21) Also, assuming that the feature point sequence X η on Πίίtarget object Ρ 0 is transformed into an actual visual coordinate system as Υ η ′, Υ η ′ is expressed as in Equation 22.

Υη = () ¹ · if · X _n …… (Equation 22) Here, if it is assumed that the exact same scene is shown on the screen, that is, if Y n = Y n ', then the following equation 2 3 The relationship should hold.

· X _n = ¹ · T · X _n …… (Equation 23) Therefore, the following equation (2 4) holds. ) ^-1 · = = (. 'If …… (Equation ²⁴ ) Now, C 2 x, C 2 y, C 2 z, 0 2, 0 2 2, 0 2 3, PO x,

P 0 y, P O z, 0 0 a, 0 0 β, 0 γ γ, P t O x, P t O y, P t

Since 0 z, 0 t 0 α 0 t 0/3, and 0 t 07, are already known, Equations 19, 20 and 22 are uniquely obtained. Then, by using this result and Equation 24 above, Equation 21 can be obtained as Equations 25 and 26 below. I C3-\ ^l C2l · Γθ · I ¹ ! J a21 ^a 22 «23, ϋ 0 0 1 / ...... (number 26)

 [Step a 4]

 Next, the distance between the secondary estimated powers of each object is calculated.

 [Step a 5]

 Here, it is possible to judge which object is out of position by checking the coincidence of the secondary estimated force position obtained for each object. In other words, the second estimated force mera positions calculated using the objects P 0, p i, and P 2 where the actual position and the design position match should all coincide. On the other hand, the second estimated camera position calculated using the object P3 located at a position deviated from the design position should not match the other. Therefore, if an object having a distance outside the permissible range of the quantization error is determined, an object that is not at the design position can be determined.

 [Step a 6]

 Using the characteristic point information of the objects P 0, P 1, and P 2 that have been found to be actually at the design position, the position of the TV camera is determined accurately again.

 [Step a 7]

 Using the exact position of the television camera determined in step a6 above, the position and orientation of the object P3 outside the allowable range are determined.

 By the above procedure, the position of the TV camera and the object not at the design position can be determined with high accuracy.

 Next, the second method will be described.

In this method, a TV camera position determination method is applied to each object, Determine the position of the TV camera. Hereinafter, the position of the TV camera obtained here is referred to as “provisional position”.

 After that, the same processing as [Step a5] to [Step a7] of the first method is performed. That is, it is possible to determine an object that is not at the design position by checking the consistency of each provisional position for each object obtained here. In this case, the method of determining the consistency is not limited to the method described in steps a4 and a5.

 In addition, in both the first and second methods, it is necessary in principle to use three or more objects. This is because there is no partner to compare with one object, and it is not possible to determine which is more accurate with two objects. Also, in the above description, it has been described that a pseudo image is created in each place, but this is a conceptual description, and does not necessarily need to be actually an “image”.

 Next, another method for determining the position of the television camera or the object using the television camera and the object will be described.

First, the relationship when one TV camera and one object are used will be described. In the stationary coordinate system, when the visual coordinate system is moved, the actual position of the TV camera (Cx ', Cy', Cz ', 01', 02 ', θ3, Sχ', Sy ' ) And the actual position of the object (PX ', Py', P, 0h, θβ, θγ ') and the design position of the TV camera (CX, Cy, Cz, 01, 02) , 03, Sx, Sy) and the design position of the object (Px, Py, Ρ ζ, Θa, θ0, θγ) C z, m 0 1, m 0 2, m 3, m S x, m S y) and (Δ Ρ, Δ P y, Δ Ρ ζ, Δ 0, Α θ β, 0 0 y) It is assumed that a problem has occurred. At this time, between the actual position of the parameter and the design position and deviation, GG

 Γ P X y

.. (Number 27)

| The relationship holds. At this time, the position of the point G ρ 'on the screen is

G X + Δϋρχ (Equation 28)

Therefore, when the parameter deviation is sufficiently small, the position difference G is

AS, as _y

(Number 29) ds _x no

 It can be expressed as. The screen position difference ΔG pn of this feature point on the object is expressed as AG „= RpnXp ....

 …… (Equation 30) where Δ G pn is the screen position difference vector of the feature points,

Λ (; ι '„= (AGPXI, mu G |, Y |, ·' · ■, AG x„, Δΰ γη) (Equation 31) It is shown by. X p is a parameter error vector, and

Xi '= (cX _P , οΧρ, sX _P ) ^T :'

 (Equation 32). Where sXp, cXp, oXp are the parameter error vector of the apparent focal length of the parameter ¾, the position of the camera (P: the parameter error vector of the posture, the parameter of the object In the error vector,

sXi> = (Δδχ, ASy) ¹ (Equation 33)

CXP-(AC _X , AC _y , AC _Z , Δθ,, Δ0 ₂ , Δθ ₃ , S _X , AS _y ) (Equation 34) () ^{X1, =} ( ^Δ? Χ '△, ΔΡ,, Δθ _α , Derutashitaro, is represented by the [Delta] [theta] _gamma) ^T (number 35).

Further, R ρ η is a partial derivative matrix,

θϋ χι 9Gpxi

dC _x dC 3Θ, 3S _X ,

 (Number 36)

3GpYn 9GpYn) GpYn 3G |, Y n 9GpY r

DC, DCv 3Θ, dS _x as. By the way, AG pn in Equation 31 is obtained from the screen position difference between the feature points of the pseudo image and the real image. Also, by substituting the design data into Equation 36 above, the partial differential coefficient matrix R pn can be calculated. Therefore, the parameter error X p can be obtained from the following equation from the equation (30).

(Equation 37) Next, a method for determining the positions of a plurality of television cameras and a plurality of objects will be described with reference to two television cameras and an object of 2 俩 as an example. Before this explanation, the relationship between one TV camera and one object described above is summarized as follows. To

 ^ Derivative matrix R pn

1.1 1,1

 Rr ,, = cR oR Ριι sR 1, ”(Equation 38).

 Here, c R pn, o R pn, and s R pn are the partial differential coefficient matrix related to the position and attitude parameters of the TV camera, the partial differential coefficient matrix related to the position and attitude parameters of the object, and the TV camera, respectively. 7 is a partial derivative matrix related to an apparent focal length parameter of FIG. Also, the parameter error vector Xp is

XP = (cX, ¹ ,, oX _v ^l , sX, ^1. ] (Equation 39).

 Here, c X p, o X p, and s X p are the position and attitude parameter vector of the TV camera, the position of the object and the attitude parameter vector, and the magnification parameter vector of the television camera, respectively.

 Next, when two TV cameras and two objects are used, the screen position difference vector ΔG a 1] of the feature points on the object and the partial differential coefficient matrix R a 1 relating to the parameters are used. The relationship between 1 and the parameter error vector X a 1 1 is

AG _a || = a R _a "· Xaii ...... (number 40) the following equation. Here, AG all, R ail, X a 1 1 , respectively,

_{AG all = (AGpn, AGp n} 2, AGp '1, Δΰί ...... ( number 41)

, 1 (number 42)

2 \ T

 M = (cXi,, cX?>, OX {>, oXf,, sX {,, sX, (Equation 43)

Do it again. The exponents i, j on the right shoulder of expressions 1, 1, 1, 2, etc. negate that they relate to the object of 桥 桥 |: | for the i 11 television force camera.

 Γίί Since the parameter number 40 has redundant parameters, it cannot be solved with parameter errors as it is. Therefore, an example in which one of the following objects is fixed in a stationary coordinate system and solved is described. When the first object is fixed to the controversial coordinate system, the partial differential coefficient matrix Ra1 1 of Equation 42 and the parameter error vector Xa1 1 of Equation 43 are

(Number 44)

(Equation 45). Therefore, the above equation 40 is

A, ' _a ll = R _a ||. A _a i | (Equation 46) From this, Equation 4 5 shows that the parameter error X a 1 1 is

, Ding

 X all = URallj · Rail) · Rpn · mum Gall (Equation 47). Here, Equation 46 is an approximate equation when the parameter error is sufficiently small, and Equation 47 is a solution of the approximate equation. By performing the convergence calculation in the same manner as the TV camera position determination method, the positions of the plurality of TV cameras and the position of the object can be determined.

Next, when two or more TV cameras have a predetermined constraint relationship, how to determine the position of the TV camera and the object is described below. La and 2 fl.'il will be explained using an example.

 Since two TV cameras are in custody, the で は, '/: ί? ¾ attitude of the first TV camera is used as a parameter here, and the first object is stationary. An example of solving for l¾i system is described. Second TV camera position

· The posture parameter is the position of one TV camera. · The posture is limited by the parameter.Therefore, the partial differential coefficient matrix Ra1 1 at this time is

Rail (number 48)

Can be expressed as And the parameter error X a 1 1 at this time is.

C _Λ Η == (cX, θΧ, sX, sX / (Equation 49)), the screen position difference vector AG a 11 1 of the feature point on the object, and the partial differential coefficient matrix Ra The relationship between 1 1 and the parameter error vector X a 1 1 is

A j _a "= K _a 1 | · A -all (Equation 50) By the way, the screen position difference vector Δ G a 11 of the feature point on the object is the feature point between the pseudo-two images and the real image. By substituting the design data into the above equation (4), the partial differential coefficient matrix R a 1] for the parameters can be calculated. Therefore, the parameter error vector X a 11 1 is expressed as ΙΙ-(. L | ^T -Rail) '· Rail-AG _a "(Equation 51). Here, Equation 50 is the approximate equation when the parameter error is sufficiently small, and Equation 5 5 is the solution of the approximate equation. As in the case of the above-mentioned “TV power camera positioning-common method”, multiple convergence The position of the television camera and the object can be determined when there is a predetermined relationship between ηπ of the camera and the camera.

Brief description of drawings

No.! FIG. 1 is a basic configuration diagram of an embodiment of the present invention. FIG. 2, _c 3 Figure illustrates a hard Douea configuration example of the assembling stations the control system shown in FIG. 1 is an illustration of one example of a working environment shown in .1 FIG. FIG. 4 is an explanatory diagram of a work environment model according to the present invention. FIG. 5 is an explanatory diagram of the relationship between the coordinate systems in FIG.

 FIG. 6 is an explanatory diagram showing a perspective transformation of an arbitrary point Ec on the visual coordinate system to an imaging surface.

 FIG. 7 is a flowchart showing a process for determining the position and orientation of the television camera and the object. FIG. 8 is an explanatory view showing a posture changing device having two rotation axes.

 FIG. 9 is an explanatory diagram showing a posture changing device having three rotation axes. FIG. 10 is an explanatory diagram showing a posture changing device having two rotation axes and one linear moving means.

 FIG. 11 is an explanatory view showing a posture changing device having three rotation axes and one linear moving means.

 FIG. 12 is an explanatory view showing a posture changing device when the position of the rotation center of the camera lens does not coincide with the position of the principal point.

 FIG. 13 is an explanatory diagram for projecting the imaged feature points on the same plane when the position of the rotation center of the camera lens and the position of the principal point in the posture changing device do not match.

 FIG. 14 is an explanatory diagram in which a plurality of television cameras are arranged at the same radial position with respect to the principal point position.

Fig. 15 shows the arrangement of multiple TV cameras arranged around the principal point. It is explanatory drawing when it sees from a point position.

 FIG. 16 is an explanatory diagram when measuring the position of each feature point when capturing each feature point of the object.

 7th] is an explanatory diagram of measuring the position of one feature point of an object using two attitude change equipment each having a scene detecting means.

 FIG. 18 is an explanatory diagram in the case where one point on a known curved surface is obtained by one scene detecting means.

 FIG. 19 is an explanatory diagram of a coordinate system in the case where one point on a known curved surface is similarly obtained by scene detection means.

BEST MODE FOR CARRYING OUT THE INVENTION

 Hereinafter, an embodiment of a flexible production system to which a position / posture detection device of an object to be imaged is applied to implement a position / posture detection method according to the present invention will be described with reference to the accompanying drawings. First, the basic configuration of a flexible production system will be described.

 As shown in FIG. 1, the flexible production system according to the present embodiment includes a work environment 1, an assembly station control device 2, a design data storage device 3, and a work plan data storage device 4, as shown in FIG. It is configured.

The work environment 1 is a work object 12 as a work, peripheral devices 13, a work robot 14 as a processing device, the power of the work robot 14, a posture change having a visual robot 15, and a television camera 34. Apparatus 33 consists of 3 and other structures. The visual robot 15 captures images of the work object 12, the peripheral device 13, the work robot 14, and the television camera 34, and the television camera 34 captures the work object 12. , The peripheral device 13, the working robot 14, and the visual robot 5], and all of the captured images are hereinafter referred to as “objects”. The peripheral device 13 supports and transports the work object 12 in the work environment 1, and the details will be described with reference to FIG. The work robot 14 has a hand effector that works on the work object 12, so-called mouth bot hand and other processing tools, and assembles the work object 12 using these. The work robot 14 itself is configured to be movable. The visual robot 15 has a means of detecting a scene similar to a television camera 34 to capture an object to be photographed in the work environment 1 as image data. It is configured to be movable as in 4.

 The posture changing device 33 is a device that changes the posture of the television camera 34. The TV camera 34 is a scene detection means for taking in the work environment 1 as image data, like the video camera of the visual robot 15. The assembly station control device 2 determines the position of the object based on the work environment data captured by the visual robot 15 and the television camera 34 and the design data storage device 3 and the work plan data storage device 4. The posture shift is detected, and the object is moved and corrected in a direction to absorb the detected shift. The details will be described later. Further, when detecting a shift in the position and orientation of the object to be photographed, the assembly station control device 2 changes and corrects the design data by the amount of the shift.

 The design data storage device 3 stores information on the shapes of products and robots, their sizes, feature points, and the like, that is, stores product design data, robot design data, and peripheral device design data.

 The work plan data storage device 4 stores work procedure data, work process data, and operation route data for the work object 12, the work robot 14, the peripheral device 13, the visual robot 15, and the posture change device 33. Is stored.

Next, the structure of the assembling station control device 2 will be described in more detail. As shown in FIG. 1, the assembling station control device 2 includes, as shown in FIG. Processing means 2 1, working environment scene Solution 22, Peripheral device control 23, Work robot control 24, Visual robot control 25, Operation command generation 26, Work environment database construction ^ Stage 28, Work It comprises environmental data storage means 29, image data storage means 30, both-image combining means 31, and attitude changing device control means 32.

 The work environment database construction means 28 constructs a work environment database from the data stored in the design data storage device 3 and the work plan data storage device 4 and the information obtained by the work environment scene understanding means 22. Things. The data constructed here is configured to be stored in the work environment data storage means 29.

 The simulated work environment scene generation means 20 has a function of simulating a work environment scene at a specific time during the work by using the data stored in the work environment data storage means 29. . In the following description, the work environment scene generated simulated based on the data in this manner is referred to as a “pseudo scene image”.

 The image processing means 21 processes the image data of the actual working environment obtained by the television camera and the television camera 34 of the attitude changing device 33 as the scene detecting means of the visual robot 15 and the working environment. It has a function of outputting to the scene understanding means 22. The image processing means 21 also has a function of storing a plurality of image data in the image data storage means 30. In the following description, an image of the actual working environment obtained through the scene detecting means is referred to as a “real scene image” corresponding to the pseudo scene image.

The two-image combining means 31 has a function of combining a single high-definition image or a wide-angle high-definition image based on a plurality of image data stored in the image data storage means 30. These single high-definition images and wide-angle high-definition images are selectively executed with the support of the work environment scene understanding means 22 described later. You. In the following description, a single high-definition image or an image synthesized on a single high-definition image based on multiple images of the actual working environment obtained through the visual robot 15 and the TV camera 34 will be referred to as “high-definition synthesis”. These are referred to as “images” or “wide-angle composite images”, but these are also considered real scene images.

 The work environment scene analysis means 22 compares the pseudo scene image generated by the pseudo work environment scene generation means 20 with the real scene image obtained by the visual robot 15 or the like, thereby obtaining the work object 1 2 It is designed to detect misalignment of the position and posture of the etc.

 Further, the operation command generation means 26 has a function of generating an operation command based on the information obtained by the work environment scene understanding means 22, and, based on the generated operation command, a peripheral device control means. 23 is a private edge device 13, a work robot control means 24 is a work robot 14, a visual robot control means 25 is a visual robot 15, and a posture changing device control means 32 is a posture. The change devices 33 are controlled individually.

 Next, a specific hardware configuration for implementing the assembly station control device 2 will be described with reference to FIG.

The assembly station control device 2 includes a system bus 101, a bus control device 102, a central processing unit 103a, a main storage device 103b, a magnetic disk device 104, and a keyboard 105. , Display 106, image generation device 107, image processing device 108, TV camera 109, zoom motor 'focus motor / iris motor control device 110, zoom motor. Lens system with lens motor 1 1 1, motor 1 1 6, pulse generator 1 1 3, counter 1 1 2, motor drive 1 1 5, DA converter (digital / analog converter) 1 1 4 It consists of an AD converter (analog-to-digital converter) 117, a sensor 118, and a network 119. These are generally used in the market. Since they are the same, the description of each detail is omitted.

 Here, among the components of the assembly station control device 2, the work environment database construction means 28, the work environment data storage means 29, the image data storage means 30, the image synthesis means 31, the work environment The scene understanding means 22, peripheral device control means 23, work robot control means 24, visual robot control means 25, motion command generation means 26, and attitude change device control means 32 are mainly composed of This is realized by the central processing unit 103a, its main storage unit 103b, and the magnetic disk unit 104. The pseudo working environment scene generating means 20 is realized mainly by the image generating device 107. Further, the image processing means 21 is mainly realized by the image processing device 108. However, since each component does not have these functions independently, but achieves these functions in close cooperation with others, the correspondences raised here are not necessarily strict. .

 It should be noted that the design data storage device 3 and the work plan data storage device 4 in FIG. 1 exchange data with each other via a network 119. The television camera 109 and the lens system 111 shown in FIG. 2 are mounted on the visual robot 15 and the posture changing device 33 instead of the assembly station control device 2 shown in FIG. It is. Similarly, the motor 116 is mounted on the peripheral device 13 or the working robot 14 shown in FIG. 1 and drives them.

 Next, details of the working environment 1 will be described with reference to FIG.

 In the figure, table-shaped robots 301a and 301b are for mounting the work object 12 and constitute peripheral devices 13 shown in FIG. . Each of the robots 301a and 301b is configured to move by itself as needed.

The arm-shaped robots 302a, 302b, and 302c are each It has an arm that can control its movement. At the end of the arm of each robot, there are hand effectors 30 21 a, 30 21 b, and a telecamera 3 for performing some work such as machining and assembling on the work 12. 0 2 2 is provided. Here, the robots 32 0 2 a and 30 2 b having the hand effectors 30 21 a and 30 21 b are the working robots 14 shown in FIG. The robot 302c having 22 is the visual robot 15 shown in FIG.

 The television camera 302 corresponds to the television camera 109 and the lens system 111 in FIG. Each of the robots 302a to 302c also has a configuration in which it travels by itself as necessary, similarly to the robots 301a and 301b in the form of the Tabnoke type.

 On the other hand, a television camera 34 is arranged at a position near each of the robots so that they can be imaged. The television camera 34 is installed in a posture changing device 35. This television camera 34 also corresponds to the television camera 109 and the lens system 111 in FIG. The work transfer robots 303a and 303b are for mounting the work object 12 and moving between the processes. Similarly, peripheral devices 13 are formed.

 Next, the operation of the flexible production system having the above configuration will be described.

In the flexible production system, the equipment configuration is changed dynamically according to the product design information and the type and quantity of the product, and processing and assembly are performed according to the online instructions from the work planning department as per the work plan. . In each process, the peripheral device 13, the work robot 14, the visual robot〗 5, and the posture changing device 33 shown in FIG. Based on the peripheral device control means 23, the work robot control means 24, It is controlled by the sensation bot control means 25 and the attitude change device control means 32. This will be described in detail with reference to FIGS. 2 and 3. The central processing unit 103a generates an operation command (digital signal). The operation command is sent to the D / A converter 114 via the system bus 101, where it is converted into an analog signal, and then sent to the motor driving device 115. The motor drive unit 115 drives the motor 116 in accordance with the operation command (analog signal), and the robot 310 as a peripheral device 13 0 a-3 0 1 b and the work robot 14 The robot 302a-302b as the robot, the robot 302c as the visual robot 15 and the posture changing device 35 are operated. In response to these operations, the counters 112 and the noiseless generator 113 also operate, and the robots 310a, 310b, 302a, 302b, and 302b, respectively. Used to control the c and attitude changer 35.

 Thus, for example, the robots 302a and 302b use the hand effectors 302la and 302lb to assemble the work object 12 placed on the robot 301b. Will be done. At this time, the visual robot 300c having the television camera 302 and the television camera 34 installed in the posture changing device 35 monitor the assembling work.

 The above-mentioned television cameras 302, 34 have a zoom motor, a focus motor, and a lens system 1 1 1 with an iris motor controlled by a zoom motor control device 110. If you want to see the work being monitored precisely, change the orientation of the TV cameras 302 and 34 to the desired positions using the robot 302c and the posture change device 35, A high-definition composite image is obtained by making the lens system 1 1 1 telephoto.

Note that data stored in the magnetic disk 104 is used as data indicating the shape and size of the parts to be assembled necessary for the work. Ma Similarly, as an operation command to each of the robots 302a · 302b, 310a · 301b and the hand effector 3002a / 302b. For example, the moving path, arrangement, assembling work procedure data, work process data and the like stored on the magnetic disk 104 are used.

 In the operation as described above, even for the same part, there is a difference between each individual due to a variation in a machining process in the middle. Therefore, the larger the number of parts to be assembled, the larger the degree of positional deviation tends to be, and an error occurs accordingly. This error is unavoidable even if each robot operates according to the control data.

 In this embodiment, the position and posture of each work robot 14 and the like are monitored by the visual port robot 15 and the television camera 34 to correct the position and posture. The procedure for monitoring the position / posture and correcting it will be described below.

 First, the work environment database construction means 28 shown in FIG. 1 is used to analyze the work environment from the data stored in the design data storage device 3 and the work plan data storage device 4. The data is read as work environment data, and the read data is registered in work environment data storage means 29 (in FIG. 2, it becomes magnetic disk 104).

Then, based on the work environment data stored in the work environment data storage means 29, the work robot 14 and the like are set to the position and orientation according to the work environment data by the pseudo work environment scene generation means 20. A pseudo-scene is generated, that is, a pseudo-scene image at a specific time during the work, which would be captured by the visual robot 15 in the event that there was. In FIG. 2, the processing environment stored in the magnetic disk 104 by the central processing unit 103a, the main storage device 103b, and the image generating device 107 in FIG. This is to generate a pseudo scene image using the data. Then, the pseudo working environment scene generating means 20 outputs the generated pseudo scene to the working environment scene understanding means 22 shown in FIG. Since the pseudo scene generation processing has already been described in the section of the operation, the description thereof is omitted here.

 On the other hand, when the visual robot 5 and the television camera 34 take an image on the same time axis as the pseudo scene image generated above, the two image processing means 21 perform predetermined processing on the taken image. Then, while outputting to the working environment scene understanding means 22 as a real scene image, a plurality of image data as a real scene image are stored in the image data storage means 30. In FIG. 2, the image processing device 108 processes an image obtained by the television camera 109 and outputs the processed image to the central processing device 103a.

 The image combining means 31 combines the plurality of image data stored in the image data storage means 30 into one composite image, and outputs the combined image to the working environment scene understanding means 22. . This is to be processed by the central processing unit 103a in FIG.

 The work environment scene understanding means 22 includes a pseudo scene image generated by the pseudo work environment scene generation means 20 and a real scene image of the work environment 1 obtained by the visual robot 15 and the television camera 34 (wide-angle image). (Including high-definition composite images or high-definition composite images) and extract the differences. In other words, information about the feature points of each of the work object 12, peripheral device 13, work robot 14, visual robot 15, and television camera 34 is extracted from the actual scene image, and the information about the feature points and the pseudo information are extracted. By comparing with the scene image, the positions and postures of the respective parts 12 to 15 and 34 are specified, and the position / posture deviation is detected. The processing for specifying the position / posture has already been described in the section of the operation, and a description thereof will be omitted. ·

As a result of the above processing, if there is a difference between the pseudo scene image and the real scene image, The work environment scene understanding means 22 changes the work environment data itself of the work environment data storage means 29 內 based on the pseudo scene image so as to match. As a result, in the subsequent operations of the parts 12 to 5 and 34, the pseudo scene image and the real scene image match.

 Or, the operation target] 2, peripheral device 13, work robot 14, visual port robot 15, and television camera 34, an operation command is issued to correct the displacement of the position / posture displacement. Send a command to generation means 26. As a result, the operation command generating means 26 issues an operation command corresponding to the command, and operates each of the units 12 to 15 and 34 so as to match the pseudo scene image.

 In the above processing, the work target 12, peripheral device 13, work robot 14 (3 O la, 30 1 b), visual robot 15, and TV camera 34 move during the work of each process. It is performed every time a specific operation is performed, that is, each time one specific operation is performed. As a result, the work target 12, the peripheral device 13, the work robot 14, the visual robot 15, and the television camera 34 are performed while checking the work position each time the work position is moved, so that the work is performed according to the work plan. Work can be carried out. In addition, if the above-described object position determination method is executed for each object (the work robot 14 etc.), it is possible to know the relative positional relationship and distance between them.

As described above, in the above embodiment, each of the work object 12 as the work environment 1, the peripheral device 13, the work robot 14, the visual robot 15, and the television camera 34 is provided. The position and orientation are predicted based on the work environment data, and the pseudo scene images of the predicted parts 12 to 15 and 34 are obtained, and the obtained pseudo scene image and the actual captured parts 1 are obtained. The actual scene images 2 to 15 and 34 are compared, and if there is a difference between the pseudo scene image and the actual scene image, the corresponding parts 12 to 15 and 34 are compared. Correct the direction to absorb the deviation, or correct the work environment data by the deviation Therefore, the actual work environment can be clearly analyzed. In addition, these repairs;; · Since the analysis of the working environment when making changes is also performed automatically, each unit 12 to 15 and 3 can be operated accurately using only numerical data. Eliminates the need for online teaching.

 Therefore, the production system can be operated autonomously. In addition, since it is not necessary to set the work objects and the robots with high precision, the equipment can be manufactured at low cost. Furthermore, it is not only economical to stop the production line for a long time, but also the production time can be shortened.

 In addition, the scene detection means consisting of the visual robot 15 and the TV camera 34 is independent of the work robot 14 and the peripheral devices 13, and the work environment is controlled while operating the work robot and the peripheral devices. By taking in the scene, the scene can be accurately judged, so that the manufacturing time can be reduced as much as possible.

 In addition, since the position of the scene detection means can be accurately determined based on the feature points in the work environment obtained by the scene detection means, the relative relationship between any two persons in the work environment can be accurately determined. You can ask.

 Next, FIGS. 8 to 11 show an embodiment of the posture changing device 33. FIG. First, a specific configuration for realizing the posture changing device 33 will be described with reference to FIG.

The attitude changing device 33 includes a base 401, a first rotation axis 402, a second rotation axis 400, and a television camera 404. That is, the first rotating shaft 402 is mounted on the base 401, and the output section 402a is rotated about the vertical axis. A second rotating shaft 400 is attached to the tip of this output unit 402 via a base 400a, and the rotating shaft 40: 3 has an L-shaped output unit _4003a. Rotate around the horizontal axis. Then, a television camera 404 is fixed to the tip of the output section 403a. In this case, the extension Y of the output section 402 a of the rotation axis 402 of 笫 1 and the extension line X of the output section 400 a of the second rotation axis 400 are orthogonal to each other. The center axis Z of the force camera lens 405 in the TV camera 404 is located at a position D perpendicular to each other on the extension line, and the rotation center of the camera lens 405 coincides, and the coincident position D is arranged so that the principal point of camera lens 405 also coincides with D. Therefore, the center axis Z of the first rotation axis 402 and the second rotation axis 400 and the center axis Z of the camera lens 405 are at one point at the principal point (D) of the force camera lens 405. They are arranged to intersect.

 The first and second rotating shafts 402, 403 are both composed of motors with reduction gears, and are directly connected to high-resolution encoders. Since the rotation of the motor is transmitted via the reduction gear, one rotation on the output shaft side of the reduction gear is equivalent to 100 rotations on the motor side, and if an encoder that divides one rotation of the motor into 400 rotations is used, the reduction gear High accuracy can be obtained such that one rotation of the output shaft is divided into 400,000.

 The rotating shafts 402, 403 are connected to the attitude changing device control means 32 of the assembling station control device 2 as shown in FIG. 1, and the motor driving device 115 as shown in FIG. Driven by The pulse encoders attached to the rotating shafts 402 and 403 are counted by the counter 112 and used to determine the position and orientation of the television camera 404.

In this embodiment, as described above, the posture changing device 33 has the first rotating shaft 402 and the second rotating shaft 400, and furthermore, the output section 400 of the first rotating shaft 402. The center of rotation of the camera lens of the television camera is located at a position D perpendicular to the extension line Y of the 2a and the output section of the second rotation axis 400. When the first and second rotating axes 402, 403 are selectively driven, the TV camera 404 is centered on the rotation center of the camera lens 405. It has a swing function that rotates and moves. Thus, the detection accuracy when detecting the object to be photographed can be improved.

 In addition, since the position D coincident with each other forms the principal point of the power camera lens 405, it is not affected by lens characteristics such as distortion of the camera lens 405, and more accurate detection is possible. Becomes possible.

 The embodiment shown in FIG. 9 is obtained by adding one rotating shaft 406 to the embodiment shown in FIG. That is, in this case, the output portion 400 a of the second rotating shaft 403 is connected to the third mounting portion 407 via the mounting plate 407 and the L-shaped support portion 408 provided on the mounting plate. A rotation axis 406 is installed, and a television camera 404 is attached to an output section (not shown) of the rotation axis 406. The television camera 406 is driven by driving a third rotation axis 406. 4 rotates around the horizontal axis orthogonal to the second rotation axis 400 on the mounting plate 407 and the support section 408, that is, around the rotation center of the camera lens 405. I have. In this case, the force lens is located at the intersection D between the extension Y of the output section 402 of the first rotation axis 402 and the extension X of the output section 400 of the second rotation axis 400. The center of rotation of the camera lens 405 is matched with the center of rotation of the camera lens 405, and the principal point of the camera lens 405 is located at the intersection D of the center of rotation and the extension lines Y and X.

 According to this embodiment, the television camera 400 has the first to third rotating shafts 402, 403, and 406, and by these driving, the television camera 405 can be subtly rotated. Therefore, as compared with the embodiment shown in FIG. 8, it is possible to improve the detection accuracy when detecting an object to be photographed.

FIG. 10 is a view in which a translation axis is added to the posture changing device 33 of the embodiment of FIG. That is, a mounting plate 410 is attached to the output section 400 a of the second rotating shaft 400, and a parallel translation shaft 409 made of a cylinder is installed at one end of the mounting plate 410. A television camera 404 is connected to the parallel moving section 409a of the moving axis 409. Accordingly, the television camera 404 is driven in the same plane as the second rotation axis 403 by driving the translation axis 409. It can be moved linearly in a direction orthogonal to the rotation axis 400.

 Further, the camera lens 400 of the television camera 400 has a center axis Z having an extension Y of the output section 402 a of the first rotation axis 402 and a second rotation axis 4. It is arranged at the position D where it crosses with the extension X of the output part 40 3 a of 0 3, and the principal point is located on the sub-axis Z.

 According to this embodiment, the television camera 404 can be rotated by driving the first and second rotating shafts 402, 403, and the force of the television camera 404, the lens 4 05 moves linearly in the same plane as the second rotation axis 4003 in the direction orthogonal to the rotation axis 4003, so that the focal position of the camera lens 400 can be easily and quickly adjusted. Can be.

 FIG. 11 is obtained by adding a rotating shaft 406 similar to the embodiment of FIG. 9 to the embodiment of FIG. That is, on the mounting plate 410 attached to the second rotating shaft 403, the third rotating shaft 406 is installed via the supporting portion 408, and by the driving of the rotating shaft 406 The television camera 404 rotates about the rotation center of the camera lens 405 with respect to the support portion 408. Therefore, according to this embodiment, as compared with the embodiment shown in FIG. 10, the television camera 404 is orthogonal to the second rotation axis 400 on the same plane as the second rotation axis 400. In addition to the linear movement in the direction, it also rotates around the center axis Z of the camera lens 405, so that the detection can be performed more precisely at any time.

By the way, usually, the rotation center of the camera lens 405 and the principal point often do not coincide. Therefore, when the principal point of the camera lens 405 and the center of rotation cannot be matched, the feature point is aligned with a specific position in the screen, for example, the center of the screen using the posture changing device. The rotation axis angle data is used. A posture changing device using two axes as shown in Fig. 8 will be specifically described with reference to Fig. 12. First, as a coordinate system of the posture changing device, a vertically downward axis is defined as Y, horizontal. The axis of direction is X, the intersection of these two axes Let 舢 be the 舢 '(and 舢 that intersect with them. Let 、 i | £ of the first and second rounds> b \ ΐΙΐ A 0 2, Λ 0 Λ' be φ 1, In this case, the force of the lens core wire Ί ο ο 1 (Υ, 1) R ot (X, φ 2) Ζ is given by the following equation.

ROT (y .t,) ROT (x. ^ ₂ ) z

 Becomes When this is used as image information of a plane, for example, as shown in FIG. 13, it is sufficient to re-project the image onto a plane Z = 1 parallel to the X and Y planes. Assuming that the projected coordinates on the plane are G X and G Y, G X and GY are obtained as follows. s 1 η ψ i c o s φ

G _x = · = t Ά n ψ (number 53)

Gv =-(number 54)

 C O S i C O S 2

In the same way, the number of feature points to be used is calculated in the same way, and all of them are cast on the same flat ifli I eight J]; :: i-ru. Use this デ ー タ data as one image data < Next, the advantage of using a posture changing device to set a feature point to an arbitrary point on the screen will be described. If a telephoto camera is used with a zoom lens mounted on the camera lens 405, focus cannot be achieved. If the feature points are blurred. Also, simply shifting the focus will cause the principal point of the lens to shift. However, if the feature point is set to an arbitrary point on the metaphor, it is not necessary to know the position of the principal point of the lens. Therefore, there is an advantage that the parameter term to be measured can be reduced. In addition, it is generally difficult to manufacture a lens without distortion, and it is said that using the entire surface of an image will lead to a reduction in measurement accuracy. The change device has advantages.

 Next, a plurality of TV cameras are arranged in three dimensions, a wide area is simultaneously captured as images using a high-power camera lens, processed into one high-definition image, and moved by a posture changing device. It is time-consuming. An example of this will be described with reference to FIGS. 14 and 15. Fig. 14 shows TV cameras arranged in an arc, with the principal points of each camera lens aligned to one point. Fig. 13 shows a television camera arranged on a spherical surface, viewed from the center line of the camera at the center.

 Next, the procedure for measuring the characteristic points of the captured data will be described with reference to FIG. Assuming that the posture measurement object in the figure is composed of three boxes, i, mouth, and c, these three boxes have feature points with clear transformers. This is indicated by a “+” mark in the figure. These feature points are measured in advance in the coordinate system of each of the three boxes. If the workpiece is machined as designed, it is natural that the design values may be used instead of the measured values.

The mouth, the box of c, and the box of c are made random. In this case, the measurement is to know the position and orientation of the box entrance and the box c with respect to-桥 Is Rukoto. The posture changing device 33 in the figure is driven so that the characteristic points of each box, mouth, and c come to a specific position on the screen, for example, the center of the screen, and detect the rotation angle at that time. Next, using the rotation angle, the projection is re-projected to a point on the same plane according to Equations 53 and 54 described above. That is, the position and orientation of the attitude changing device 33 are set with the box as the origin. Then, the parameters of the box mouth and the crocodile posture are obtained by the procedure described above.

 By the way, when using the posture changing device '33, the minimum number of required feature points will be described. It is necessary to find the posture changing device 33 for a certain object, for example, the box in Fig. 16. There are six parameters: the position of the origin (CX, Cy, Cz) of the posture change equipment 33 and the posture (01, Θ2, 03) related to the direction of the coordinate axes. On the other hand, the image information of feature points is two coordinate values of each feature point on the same plane. Therefore, if there are three feature points, six parameters of the position / posture of the posture changing device 33 with respect to the object can be obtained.

 This will be described in detail using mathematical expressions. The attitude changing device 33 shown in FIG. 16 is a two-axis type similar to that shown in FIG. 12, and its coordinate system is also the same as that shown in FIG. The rotation angles of the first and second rotating shafts 402 and 403 are denoted by Φ 1 and </> 2, respectively. At this time, the direction of the central axis of the camera lens 405 can be expressed by the above equation (52). In order to use this as image information of a plane, reprojection is performed on a plane Zc = 1 parallel to the XcYc plane. Assuming that the coordinates on the projected plane are G X and G Y, G X and G Y are obtained as Equations 53 and 54.

 The position of the origin C of the coordinate system of the posture changing device on the stationary coordinate system is (CX, CY, CZ), and the posture is the rotation angle around the X, Y, and Z axes of the stationary coordinate system. Represented by 3.

Also, the position of the origin P of the coordinate system of the position / posture object on the stationary coordinate system is (Px, Py, Ρ ζ), and the posture is around the X, Υ, and Ζ axes of the stationary coordinate system. 0 a, 0 β,. On the other hand, the apparent focal point of the attitude changing device / 1 · isx, s γ is given by Ζ c = 1, taking φ-plane of 行 c, γ c ψ ίπί, so SX =], SY = 1 is a fixed value.

 The parameters decided here are the one TV camera described in Sakugawa and

• (Cv. 4 ^{The object is a river V, when it is | and the lord. Therefore, the meta position i 'of the feature point can be expressed by Equation 29. However, the cause of the screen position difference is the position If only the attitude measurement object, Equation 29 is changed to Equation 55 below.

 (Number 5 5)

+

These six parameters are the difference factors, and measurement can be performed if there are at least three characteristic points on the position and orientation measurement object.

The above has dealt with the case of measuring each of the feature points on the same plane, in turn, the position of the measuring method _s one feature point will be described in the position of one feature point of the object (X, Upsilon , Z). On the other hand, the parameters of one scene detection means are (G x, G y) because they are plane images. Therefore, the scene detecting means cannot measure the position of one feature point unless there are two or more. When two or more scene detection sensors are used, the method for determining the positional relationship between the two is described in Section 17I.

In Fig. 17, the two attitude changers 3 3 1 and 3 3 2 are attached respectively. Ilta 'Ιι'ί Π1 ·

Output stage (television camera) There are two units, Λ and キ, and the Caribbean River leak 33 and the object to be measured 334. Two attitude change devices:! 1,]] 2 have two axes as shown in Fig. 8. On the other hand, the workpiece 3 for the calibration 3 3 3 has a plurality of calibration marks 3 Λ Λ a at the position of 怠: negligible; each mark 3 3 3 a is a dimension measuring machine in advance. By measuring in the work coordinate system, their relative positions are captured as data.

 Each of the scene detecting means Λ and B detects the mark 33 33 a on the calibration work 33 33 and detects the mark 33 33 a on the work coordinate 33, for example, the scene detecting means B and the calibration means in the coordinate system of the scene detecting means A. The three-dimensional position and orientation of the workpiece 3 3 3 are measured. By this operation, the relative position of the two scene detecting means A and B is determined.

 Next, how to find the length L between the two points P and Q of the measurement object 3 3 4 of the block will be described. The P point of the measurement object 3334 is set to the center of both sides by the scene detection means A and B, respectively, and the angle of the rotation axis in the posture changing devices 331 and 332 at that time is obtained. From this angle, one point on each plane is obtained, and three-dimensional coordinates are obtained by the method described above. In the same way, find the Q point. From the three-dimensional coordinates of the points P and Q, the distance L between the points P and Q is determined. Here, a method of obtaining the three-dimensional position will be described in more detail.

 For example, when obtaining the coordinates of the point P, if the angle of the rotation axis of the posture changing device 331 when the scene detection means A matches the point P with a specific position on the screen is φ 1 A, 2Λ, From the following Equations 5 6 and 5 7,

GX _Λ = ta η φ 1 _Λ ( _Equation 5 6)

^> Υ ^ = — ta 11 ψ i _Λ / COS ψ Ι 数 (Equation 5 7) and the meta-targets GX Λ and G Υ ま る are obtained Similarly, assuming that the angle of the rotation axis of the scene detection means B (A degree of rotation ¾li of the posture changing device 332) is 1 13 and φ 2 B,

G X „= t a η ψ,„… (Equation 5 8)

G y _B =-tan </) _{1 B} 'cos… (Equation 5 <7), and the coordinates GXB and GYB on Ψ-iTii are obtained.

 Let P X, P y, and P z be the design positions of point P in the stationary coordinate system, and let P x ′, P y ′, and P z ′ be the actual positions. If the displacement at that time is ΔP X, mm P y, ΔP z, the following relational expression is obtained.

Δ Ρ χ = Ρ ' _χ - _{Ρ χ} … ( _Equation 6 0)

AP y = _P 'y- _{P y} … (Equation ₆

Δ P _z = P 'z - in p _z ... (6 2) Here, the difference of the AG XA between the actual screen position and screen position in the design,厶GXB, When AGYB, holds the following equation .

厶^a V ^厶³ ... ( "

^C

Um _X one… (a few um " ^d '

 Then, Equations 63 to 66 are solved as simultaneous equations of ΔP x, ΔP y, and Δ Ρ ζ. These Δ Ρ χ, ΔP y, and Δ Ρ 近似 are approximate values, and from the relationship of the aforementioned equations 60 to 62, the following equation is obtained.

 Ρ 'χ = Ρ χ + Δ Ρ χ… (Equation 6 7)

P 'y = P y + A P y… (Equation 6 8)

Ρ 'ζ = Ρ ζ + Ρ Ρ……… （閲… （………… 数 （数 （………………… （（（（… （………………… 数 数 数 P 数 数 P 数 数 数 数 数By returning this Τ · ΙιΙΠ, we get the exact positions Ρ X ', P y', P z ' be able to. Thus, if the three-dimensional coordinates of each feature point are known, it is self-evident that by combining the three-dimensional coordinates, it is possible to define where the object in the air is located. You can also.

 Next, a method of obtaining a known point on the illi meta by one scene detecting means will be described with reference to FIG. A known surface is represented by Ζ = f (X, y). The following description is based on the assumption that the position and orientation of the scene detection means with respect to the curve have already been determined.

 In FIG. 18, it is now known from the value of the posture changing device 331 that the solid going from the scene detecting means 180 to the point B is known. To obtain the coordinates of point B at this time, as shown in Fig. 19, on the surface represented by (xO, y0) in the coordinates (X0, y0, z0) of point A, Find point C of Next, a vector a that connects points on a curved surface obtained in the vicinity of the point C, for example, (χ Ο + Δχ Ο, y0), (χ0-Δχ0, y0) is obtained. In the same manner, the vector b connecting the points on the surface obtained by (X0, y0 + Δy0), (xO, yO—AyO) is obtained. The tangent plane of point C can be defined by the outer product of these vectors a and b. Let the intersection of this tangent plane and the vector from A to B be D (X 1, y 1, z 1). Find the plane tangent to the point E on the curve represented by (xl, yl, z1), and find the intersection F of this plane and the vector from A to B. By repeating this operation, the coordinates of point B on the curved surface can be obtained.

By the way, as shown in FIG. 18, if it is desired to define a certain area on the curved surface by using the scene detection means 180, a line along the hatched area in the figure is used as the scene detection means 180. The area on the three-dimensional free-form surface can be defined by defining on the screen obtained in step 1 and finding the corresponding point on the surface for one pixel on the screen by the method described above. Industrial applicability

 As described above, according to claims 1 to 4 of the present invention, the actual work environment scene and the pseudo work scene in design are analyzed, and each work environment in the work environment is analyzed based on the analysis result. The position and posture in the work environment are configured so that the actual control data can be changed and each position and posture in the work environment can be corrected to match the design values. · It has the effect of automatically correcting the position deviation with high accuracy.In addition, it is necessary to conduct online teaching of the processing equipment by automatically analyzing the work environment when making corrections and changes. There is also an effect such as disappearance.

 Further, according to claims 5 to 11 of the present invention, there is an effect that the method according to claims 1 to 4 can be appropriately performed. Particularly, according to claims 8 to 11, the scene detecting means and The camera lens of a typical TV camera can change its attitude with its principal point at the center, enabling more accurate detection without being affected by lens characteristics such as camera lens distortion. It is effective, and it is economical because a commercially available and inexpensive television camera can easily obtain high-definition images at a wide angle.

 Furthermore, according to claims 12 to 17 of the present invention, since the position / posture detection device according to claims 5 to 11 is provided, the production system can be operated independently, and the work environment can be improved. It is not necessary to manufacture each part with high precision, so the equipment can be made cheaper, and there is no need to stop the production line for a long time. It can provide an economical system.

Claims

The scope of the claims

 1. When the scene detecting means and the object to be photographed deviate from a predetermined design position / posture, the position / posture detecting method corrects the deviation. The subject is photographed by changing the principal point position of the camera lens in the scene detection π M i stage to a sub-center, and projecting the obtained images onto a single plane to obtain one image. When the scene detection means and the object to be photographed are assumed to be in a predetermined design position / posture, the image which would be photographed by the scene detection means is synthesized with the angle of view image. A position of each feature point on the object on the pseudo image, the position of each feature point on the object on the pseudo image, and the position on the object on the composite image actually captured by the scene detection means. Calculate the amount of deviation between each feature point and the position of each corresponding feature point Correcting the position and orientation of the scene detecting means and the object based on the shift amount based on the object; and correcting the position and orientation of the scene detecting means and the object by an amount corresponding to the shift amount. A position / posture detection method characterized by performing one of correction of design data for moving an object.

2. In advance, at least two scene inspection means capable of imaging the work object, the processing apparatus for processing the work object, and these peripheral devices are installed, and these two scene inspection means are installed. The position at the predicted time of each of the working environments consisting of the work object, the processing device, and the peripheral device at each predicted time. The posture is artificially generated as an image based on the data designed in advance. When the environment reaches the time axis at which the pseudo image was generated, the actual work environment was imaged by each scene inspection means, and the data of the actually captured work environment and the data at which the pseudo image was generated were compared. If there is a difference between the two, 5 correct each position / posture in the work environment based on the difference, and adjust the corresponding design data in the work environment by the amount corresponding to the difference. Fix A position / posture detection method characterized by performing any one of the following.

3. At least two scene inspection means capable of imaging the work object, the processing device for processing the work object, and these peripheral devices are installed in advance, and these two scene inspection devices are installed. The position / posture scene at each predicted time of the working environment consisting of the means, the work object, the processing device, and the peripheral device is pseudo-generated as an image based on the previously designed data, and then, When the work environment reaches the time axis at which the pseudo image was generated, the actual work environment is imaged by each scene inspection means, and the data of the actually imaged work environment and the pseudo image are generated. When there is a difference between the two, the respective positions and postures in the work environment are corrected based on the difference, and each corresponding design data in the work environment is corrected by the difference described above. To do In the following, it is assumed that the above-described pseudo-generation process and the process of correcting each position and orientation in the work environment are performed each time each of the work environments 內 performs one specific operation. Characteristic position / posture detection method.

4. The position / posture characterized in that the actually captured work environment data is a single wide-angle and high-precision image. Detection method.

 5. In the position / posture detection device that determines the position / posture of the object in the work environment,

Scene detection means for photographing the object, actual data indicating the actual position of the scene detection means, and storage means for storing design data indicating the design position and orientation of the object. Assuming that the object to be photographed is in the design position / posture, the image that would be taken by the scene detection means is simulated using the actual data and the design data. A position of each feature point on the object on the pseudo-created image, and a position on the image of the scene detection means actually photographing the object. The shift between the above-mentioned feature points on the object and the positions of the corresponding feature points is extracted, and the position and orientation of the object are corrected based on the shift. Means for performing either one of correcting design data of an object to be photographed.

6. A work environment having at least two scene inspection means capable of imaging the work object, a processing device for processing the work object, peripheral devices thereof, and the work object, the processing device, and the peripheral device. A design data storage device that stores information on the shape and characteristic points of the work object, processing equipment, peripheral equipment, and each scene inspection means, and work on the work object, processing equipment, peripheral equipment, and each scene inspection means. Based on the work plan data storage device storing the procedure data, the work process data, and the operation route data, and the data stored in the design data storage device and the work plan data storage device, each of the work environment data is predicted in advance. The position / posture scene at the time that was generated is artificially generated, and when the artificially generated time axis is reached, the actual work environment is imaged by each scene inspection means, and The position and posture of the work environment are compared with the position and posture scenes of the work environment in which the image data was generated in a simulated manner, and the position and posture deviations of the work environment are detected. A position / posture detection device comprising: an assembling station control means for correcting each position / posture.

7. A work environment having at least two scene inspection means capable of imaging the work object, a processing device for processing the work object, peripheral devices thereof, and the work object, the processing device, and the peripheral device. A design data storage device that stores information on the shape and characteristic points of the work object, processing equipment, peripheral equipment, and each scene inspection means, and work on the work object, processing equipment, peripheral equipment, and each scene inspection means. A work plan data storage device that stores procedure data, work process data, and operation route data, respectively, and a data stored in the design data storage device and the work plan data storage device, each of which is predicted in advance in the work environment. The position / posture scene at the time is simulated, and when the simulated time is reached, the actual work environment is imaged by each scene inspection means, and the image data is simulated. Each position and posture of the work environment is compared with the generated position and posture of the work environment, and the position and posture of the work environment are detected, and each position and posture in the work environment is detected in a direction to absorb the detected deviation. Assembly station control means for correcting, and the assembly station control means artificially generates a work environment scene at a specific time during the work based on the design data storage device and the work plan data storage device. Simulated work environment scene generation unit and image synthesis for synthesizing one high-precision image based on actual image data captured by each scene inspection unit when the same time axis as a specific time during work is reached A work environment scene understanding unit that compares the pseudo work environment scene generated by the pseudo work environment scene generation unit and the high-precision image obtained by the image synthesis unit, and detects a shift in each of the positions and postures in the work environment; It has an operation command generation unit that selectively drives the work object, processing equipment, peripheral devices, and each scene inspection means in order to absorb positional shifts in the work environment in response to a command from the work environment scene understanding unit. Position and attitude detection device.

 8. At least one of the scene detecting means in the working environment is constituted by a television camera installed in a posture changing device, and the posture changing device has a first rotation axis rotatable around the axis. And an output section is disposed orthogonal to the output section of the first rotation axis, and has a second rotation axis to which the television camera is attached, and a center line of the first rotation axis output section and a second rotation axis. The center axis of the camera lens of the TV camera is arranged at a position that intersects with the center line of the rotation axis output unit of 2, and the rotation center of the camera lens is matched with the camera lens. 8. The position / posture detecting device according to claim 6, wherein the principal points are arranged so as to coincide with each other.

9. At least one of the scene detection means in the work environment changes posture. The attitude changing device comprises a first rotation axis rotatable around an axis, and an output section orthogonally arranged to the output section of the first rotation axis. A second rotating shaft, and an output unit mounted orthogonal to the second rotating shaft and the first rotating shaft with respect to an output unit of the second rotating shaft, and a second rotating shaft to which the television camera is attached. And a television set at a position where the center line of the first rotation axis output unit, the center line of the second rotation axis output unit, and the center line of the third rotation axis output unit intersect each other. The center axis of the camera lens in the camera is arranged, the center of rotation of the camera lens is matched, and the principal point of the camera lens is located at the coincident position. Claim 6 characterized in that the posture is changed to the center. Or the position / posture changing device according to 7.

 10. At least one of the scene detecting means in the working environment is constituted by a television camera installed in a posture changing device, and the posture changing device is a first rotation axis rotatable around an axis. A second rotating shaft whose output section is arranged orthogonally to the output section of the first rotating shaft; and an output section corresponding to the second rotating shaft and the first rotating shaft with respect to the output section of the second rotating shaft. A linear movement axis attached to the television camera along the movement direction, the center line of the first rotation axis output unit, and the second rotation axis. The center axis of the camera lens of the TV camera is arranged at a position that intersects with the center line of the output unit, the center of rotation of the camera lens is matched, and the principal point of the force lens matches the matched position. The main lens is mainly Position position and orientation changing device according to claim 6 or 7, wherein the configured such that the posture change around the.

At least one of the scene detecting means in the working environment is constituted by a television camera installed in a posture changing device, and the posture changing device is a first rotation rotatable around an axis. Shaft and the output section to the output section of the first rotary axis A second rotating shaft arranged orthogonally, and the output unit is attached to the output unit of the second rotating shaft so as to be linearly movable in a direction orthogonal to the second rotating shaft and the first rotating shaft. A first linear movement axis; and a third rotation axis, the output section being disposed along the linear movement direction of the output section with respect to the output section of the linear movement axis, and having the television camera mounted thereon, The center axis of the camera lens of the television camera is arranged at a position where the center line of the rotation axis output section of the second rotation axis output section and the center line of the third rotation output section cross each other. The center of rotation of the camera lens is matched, and the principal point of the camera lens is arranged so that it coincides with the coincident position, and the camera lens is configured to change its attitude around the principal point position. Change of position and posture according to claim 6 or 7 apparatus.

12. The work object, a processing device for processing the work object, peripheral devices thereof, and at least two scene inspection means capable of imaging the work object, the processing device, and peripheral devices. The work environment and the design data storage device that stores information on the shape and characteristic points of the work object, processing equipment, peripheral devices, and scene inspection means, and the work object, processing device, peripheral devices, and scene inspection means Based on the work plan data storage device that stores the work procedure data, work process data, and operation route data, and the data stored in the design data storage device and the work plan data storage device, each of the work environment data is stored in advance. A position / posture scene at the predicted time is artificially generated, and when the artificially generated time axis is reached, the actual work environment is imaged by each scene inspection means, and Each position and posture of the work environment in which the imaging data was generated in a simulated manner.The position and posture of each work environment were compared with the scene, and the displacement of each position and posture of the work environment was detected. A flexible production system comprising: a position / posture detection device having assembly station control means for correcting the position / posture.

1. A work object, a processing device for processing the work object, a peripheral device thereof: if, and at least two scene inspection means capable of imaging the work object, the processing device, and peripheral devices. Work environment, work object, processing equipment, peripheral equipment: Design data storage device that stores information on the shape and characteristic points of each scene inspection means, work object, processing equipment, peripheral equipment, each scene inspection Based on data stored in the work plan data storage device, which stores the work procedure data, work process data, and operation route data for the means, and the design data storage device and the work plan data storage device, each of the work environment The position / posture scene at the time predicted in advance is artificially generated, and when the artificially generated time axis is reached, the actual work environment is imaged by each scene inspection means. The position and orientation of each work environment in which the image data was generated in a simulated manner.The position and orientation of each work environment were compared with the scene, and the displacement of each position and posture of the work environment was detected. Assembling station control means for correcting the position and orientation, and the assembling station control means stores a work environment scene at a specific time during the work based on the design data storage device and the work plan data storage device. A simulated work environment scene generation unit that generates a simulated work environment, and, when the same time axis as a specific time during work is reached, a single high-precision image is generated based on actual image data captured by each scene inspection unit. By comparing the image synthesis unit to be synthesized with the simulated work environment scene generated by the simulated work environment scene generation unit and the high-precision image obtained by the image synthesis unit, the deviation of the work environment and the position and orientation in the environment is detected. A work environment scene understanding unit and an operation command for selectively driving a work object, a processing device, a peripheral device, and each scene inspection means in order to absorb a positional shift in the work environment according to a command from the work environment scene understanding unit. A flexible production system comprising a position / posture detection device having a generation unit.

1. The position / posture detection device uses a means for detecting each scene in the work environment. At least one of them is composed of a television camera installed in a posture changing device, and the posture changing device has a first rotation shaft rotatable around an axis, and an output portion having a first rotation shaft. A second rotation axis to which the television camera is attached, and a center line of the first rotation axis output section and a center line of the second rotation axis output section. The center of the camera lens is located at the position where the camera lens intersects with the camera, and the center of rotation of the camera lens is matched with the camera lens so that the principal point of the camera lens coincides with the matching position. 14. The flexible production system according to claim 12, wherein the flexible production system is arranged.

15. At least one of the scene detecting means in the working environment in the position / posture detecting device is constituted by a television camera installed in the posture changing device, and the posture changing device is rotatable around an axis. A first rotating shaft, a second rotating shaft having an output section orthogonal to the output section of the first rotating shaft, and an output section having a second rotating shaft with respect to the output section of the second rotating shaft. A second rotation axis and a first rotation axis, and a third rotation axis to which the television camera is mounted, and a center line of the first rotation axis output unit and a second rotation axis. The center axis of the camera lens in the television camera is arranged at a position where the center line of the shaft output section and the center line of the third rotation axis output section intersect with each other, and the rotation centers of the camera lenses are matched with each other. Make sure that the principal point of the camera lens matches the corresponding position Disposed to claim 1 2 or 1 3 flex reluctant production system wherein the camera lens is characterized in that the center due urchin configured to change the posture to the main point position.

16. At least one of the scene detecting means in the working environment of the position / posture detecting device is constituted by a television camera mounted on the posture changing device, and the posture changing device rotates around the axis. A possible first rotary axis; a second rotary axis, the output of which is arranged orthogonal to the output of the first rotary axis; The force unit is attached to the output unit of the second rotating shaft so as to be linearly movable in a direction intersecting the second rotating shaft and the first rotating shaft, and the front television camera is attached along the moving direction. A center axis of the camera lens of the television camera at a position where the center line of the first rotating shaft output section and the center line of the second rotating shaft output section intersect each other. In addition, the rotation center of the camera lens is matched, and the principal point of the camera lens is arranged so as to coincide with the coincident position, and the camera lens is configured to change its attitude around the principal point position. 14. The flexible production system according to claim 12, wherein:

 17. At least one of the scene detection means in the work environment in the position / posture detection device is constituted by a television force lens installed in the posture change device, and the posture change device rotates around an axis. A possible first rotating shaft, a second rotating shaft whose output section is arranged orthogonal to the output section of the first rotating shaft, and an output section corresponding to the output section of the second rotating shaft. A linear movement axis mounted so as to be linearly movable in a direction orthogonal to the second rotation axis and the first rotation axis; and an output section extending along the linear movement direction of the output section with respect to the output section of the linear movement axis. And a third rotation axis to which the television camera is attached, and a center line of a first rotation axis output section, a center line of a second rotation axis output section, and a third rotation output. The center axis of the power lens in the TV camera at a position that intersects with the center line of the part In both cases, the center of rotation of the camera lens is matched, and the camera lens is arranged so that the principal point of the camera lens coincides with the coincident position, and the camera lens is configured to change its attitude around the principal point position. The flexible production system according to claim 12 or 13, characterized in that: