JPH07237158A - Position-attitude detecting method and device thereof and flexible production system - Google Patents

Abstract

PURPOSE:To provide a highly autonomous production system. CONSTITUTION:When a visually sensing robot 15 or a work robot 14 on the like is situated in a position according to a design, a pseudo-work environment scene creating means 20 pseudo-synthesized a scene supposed to be photographed by the visually sensing robot 15 according to data stored in a work environment data storage means 29, on the one hand, the visually sensing robot 15 actually photographs a scene of a work environment 1. A work environment scene understanding means 22 compares an actual image and a pseudo-scene with each other, and detects dislocation of a position and an attitude of the visually sensing robot 15 or the like. Thereby, even in a condition where an absolute reference point is not reflected on the visually sensing robot 15, since a position and an attitude of a work object or the like can be detected and corrected, a highly autonomous production system can be realized.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a field of a production system utilizing a computer for automating a product assembling work, and a method for detecting a position / orientation of an object to be imaged which causes a robot system to work without teaching. The apparatus and a flexible production system using the apparatus.
In particular, compare the simulated work environment model created by computer graphics based on product design data, processing / assembly device data, and peripheral device data with the actual work environment obtained with a visual device using a TV camera. The present invention relates to a position / orientation detection method and apparatus suitable for realizing a function capable of autonomous work, and a flexible production system using the same.

[0002]

2. Description of the Related Art Conventional production systems have been small-item mass-production systems for supplying a large amount of uniform products. However, in order to meet diverse customer needs, the importance of value is shifting to product differentiation and individualization, and with the diversification of products, the increase in functionality, and the shortening of product life, a wide variety of products have been developed. Development of a production system corresponding to small-volume production has been required. Therefore, in order to supply products that meet the needs of customers in a timely manner, a flexible production system with high flexibility has been demanded that is compatible with small-lot production of a wide variety of products by connecting an overall system of orders, design, and manufacturing. . Further, although the total assembling process of home electric appliances mainly involves manual labor, it is difficult to increase or decrease the number of workers according to the seasonal demand fluctuation, and therefore, there is a strong demand for automation.

In the conventional small-volume mass-production method, the robot operates by the teaching playback method.
It has become possible to determine the robot operation sequence offline, but online teaching is generally used for position information, and it is necessary to stop the production line for a long time during teaching, and robot operation is not possible. Skilled workers are also required. Further, when many robots are used, it takes a long time to start up the entire production line.

On the other hand, in order to realize a flexible production system, it is necessary to change the equipment structure or the position of the robot based on an instruction from the work planning department. For this purpose, it is necessary not to bring the robot to an accurate position but to make the robot autonomously act when it is in an incorrect position. As a conventional example of such a configuration,
There is a method described in Japanese Patent Laid-Open No. 3-73284 (hereinafter, referred to as a first conventional technique). The robot control method according to the first prior art is intended to give the robot information about a moving object existing in the work environment to cause the robot to make an appropriate action. Therefore, as a knowledge database, knowledge of action procedures and actions according to various situations was stored.

Further, the simulated image of the work environment is displayed in real time, the real environment and the simulation environment are overlaid, and the simulation is performed when the position / orientation of the simulation environment model is different from that of the real environment. A conventional example in which a person can easily correct the position / orientation data of an environment model so as to match that of an actual environment is disclosed in Japanese Patent Laid-Open No. 3-55194 (hereinafter, referred to as second prior art). Have been described. The second prior art is intended for remote control of a robot, and the image analysis itself is performed by a person.

Further, even when the image of the operation object is hidden by the image of the operation hand, the operation object can be recognized, or a person can recognize the relative positional relationship between the operation object and the operation hand. A conventional example of displaying an image is described in Japanese Patent Laid-Open No. 3-60989 (hereinafter, referred to as a third conventional technique). This conventional example is also intended for remote operation, and the image analysis itself was performed by a person.

By the way, there are cases where information is desired widely at a certain point. In that case, it can be made possible by swinging the direction of the TV camera. This conventional example is described in Japanese Patent Application Laid-Open No. 2-83194 (hereinafter referred to as the fourth conventional technique). However, this conventional technique is for displaying the robot so that it can be easily seen by a human who remotely controls the robot, and is not considered for image analysis.

[0008]

As described above, it is possible to supply products that meet the needs of customers in a timely manner and to receive orders.
In order to realize a highly flexible production system that is compatible with small-lot production of a wide variety of products that connects the entire design and manufacturing system, it is necessary to change the configuration of the production equipment and the position of the robot. In order to manufacture under such conditions, it is essential to increase the reliability of the entire equipment. To do this, make sure that the work target exists at the desired position according to the work plan, and if it deviates from the desired position, know the deviation and perform the operation corresponding to the deviation. It is necessary to give autonomy to the production system.

However, none of the above-mentioned first to fourth prior arts considers the realization of the autonomy of this production system. In particular, in the first conventional technology, the knowledge database stores the knowledge such as the action procedure of the robot and the motion according to various situations,
Based on information about moving objects existing in the work environment,
It is necessary to know in advance where the scene input device is and what the scene looks like in order to analyze the work environment captured as the scene, although the robot is caused to make an appropriate action. In the first conventional technology, there is no specific way to understand what the work target looks like, what peripheral devices look like, what the robot looks like, what part it looks like, etc. I didn't consider it.

Further, in order to analyze the actual work environment, it is necessary to use information in the widest possible range with high density. In the fourth conventional technology, a wide range of information is used by changing the direction of the TV camera. At that time, the work environment captured as information is simply displayed on the screen. Therefore, it cannot be used as an analysis of the work environment.

In view of the above problems of the prior art, the present invention analyzes the work environment and accurately grasps the position / orientation deviation of the robot or the like based on the analysis result.
Accordingly, it is an object of the present invention to provide a position / orientation detection method for an object to be photographed, which can automatically correct the position / orientation deviation with high accuracy. Another object is to provide a position / orientation detecting device that can accurately implement the above method, and yet another object is to provide a flexible manufacturing system with high flexibility capable of supporting high-mix low-volume production. To do.

[0012]

According to the method of the present invention, an object to be photographed in the work environment is photographed by the scene detecting means by changing the position of the principal point of the camera lens in the scene detecting means. It is assumed that the plurality of images are projected onto one plane and combined into one wide-angle image, and that the scene detection means and the object are in a predetermined design position / orientation. In this case, the images that would have been captured by the scene detection means are pseudo-synthesized, and the position of each feature point on the object to be photographed on the pseudo image and the scene detection means actually capture the image. The amount of deviation between each of the characteristic points on the object and the position of the corresponding characteristic point on the obtained composite image is extracted, and the scene detection means and the position of the object are extracted based on the amount of deviation.・ Correct the posture with reference to the above object Then, either one of the scene detection means and the design data for moving the object to be photographed is corrected by an amount corresponding to the shift amount.

Further, in the method of the present invention, at least two scene inspection means capable of picking up an image of the work object, the processing device for processing the work object, and these peripheral devices are installed in advance. The position / orientation of the work environment including the scene inspection means of the table, the work object, the processing device, and the peripheral device at each predicted time is pseudo-generated as an image based on predesigned data, and then, , When the work environment reaches the time axis at which the pseudo image is generated, the actual work environment is imaged by each scene inspection means, and the data of the work environment actually imaged and the data that generated the pseudo image are When there is a deviation between the two, the respective positions / postures in the working environment are corrected based on the deviation, and the corresponding design data in the working environment is corrected by the amount of the deviation. And having to perform any one of the.

In the position / orientation detecting apparatus of the present invention, the scene detecting means for photographing the object, the actual data indicating the actual position of the scene detecting means, and the designed position / orientation of the object. Assuming that the storage means storing the design data indicating the above and the object to be photographed are in the designed position / orientation, the image that the scene detection means is likely to capture is The position of each characteristic point on the object to be photographed is pseudo-created using the design data, and the image of the object actually photographed by the scene detection means is displayed on the image. Finding a deviation between each feature point on the object and the position of each corresponding feature point, correcting the position / orientation of the object on the basis of the deviation, and within the work environment by the amount of the deviation. Modify each corresponding design data of And having a means for executing one of the be.

In the position / orientation detecting device of the present invention,
Work object, processing device for processing the work object, peripheral devices thereof, work environment having at least two scene inspection means capable of imaging the work object, the processing device and the peripheral device, and the work object Design data storage device that stores information about the shapes and characteristic points of objects, processing devices, peripheral devices, and each scene inspection means, and work procedure data, work process data for work objects, processing devices, peripheral devices, and each scene inspection means. ，
Based on the work plan data storage device that stores the movement path data, and the data stored in the design data storage device and the work plan data storage device, the position / orientation scene at each pre-predicted time in the work environment is calculated. Pseudo-generated, when reaching the pseudo-generated time axis, the actual work environment is imaged by each scene inspection means, and each position / orientation of the work environment in which the imaged data is pseudo-generated Assembling station control means for detecting a deviation of each position / orientation of the work environment in comparison with the scene and correcting each position / orientation in the work environment in a direction to absorb the detected deviation. And

In the flexible production system of the present invention,
Work object, processing device for processing the work object, peripheral devices thereof, work environment having at least two scene inspection means capable of imaging the work object, the processing device and the peripheral device, and the work object Design data storage device that stores information about the shapes and characteristic points of objects, processing devices, peripheral devices, and each scene inspection means, and work procedure data, work process data for work objects, processing devices, peripheral devices, and each scene inspection means. ，
Based on the work plan data storage device that stores the movement path data, and the data stored in the design data storage device and the work plan data storage device, the position / orientation scene at each pre-predicted time in the work environment is calculated. Pseudo-generated, when reaching the pseudo-generated time axis, the actual work environment is imaged by each scene inspection means, and each position / orientation of the work environment in which the imaged data is pseudo-generated A position / orientation detecting device is provided which detects a deviation of each position / orientation of the work environment as compared with the scene and corrects each position / orientation in the work environment in a direction of absorbing the detected deviation.

In the flexible production system, at least one of the scene detecting means in the working environment of the position / orientation detecting device is composed of a television camera installed in the attitude changing device, and the attitude changing device is used. The device has a first rotating shaft rotatable about an axis, an output unit arranged orthogonal to the output unit of the first rotating shaft, and
A second rotating shaft to which the television camera is attached, and a camera lens of the television camera at a position where a center line of the first rotating shaft output portion and a center line of the second rotating shaft output portion intersect each other. The center axis is arranged, the rotation centers of the camera lenses are aligned, and the principal points of the camera lenses are aligned at the matching positions.

[0018]

In the autonomous flexible production system, it is necessary to confirm for each work whether or not the work is performed according to the work plan and whether the next work can be performed as scheduled. For that purpose, first, it is necessary to detect the three-dimensional position and orientation of the work target from the two-dimensional image obtained by the scene input sensor. Further, in order to always see the work point or the like to be confirmed by the scene input sensor, it is necessary to change the position of the scene input sensor to an appropriate position and perform a so-called "peek" operation, if necessary. For that purpose, it is necessary to accurately detect and correct the position of the scene input sensor. in this case,
It is not possible to set a specific reference point and always use the reference point to detect the position. Because the scene input seisa is
This is because they are moved as necessary.

Therefore, in the present invention, the scene input sensor and the actual position / orientation of the object are detected and corrected by the method described below, and the outline thereof will be described.

First, when the scene input sensor is directed to the position designated in the work plan in the direction specified, the design environment and work plan data are used to determine what the work environment looks like. Originally, a pseudo work environment screen is created using computer graphics. Here, the scene input sensor preferably has a zoom function.

Subsequently, expected feature point candidates such as colors and edges are selected in the generated pseudo work environment screen, and feature points corresponding to the candidates are searched for in the actual work environment. In this case, the work environment is known at the production site. Also, many manufacturing facilities such as production lines are known. Therefore, it is possible to always prepare a characteristic point that serves as a reference for easy image processing.

By making the scene input sensor with a zoom function wide-angle and capturing the scene, some of the feature points can be identified. The rough position in the space of the scene input sensor can be determined from the difference between the pseudo work environment screen position of the feature point whose spatial coordinates are known and the actual work environment screen position captured by the scene input sensor. Further, if necessary, if a scene is captured by a scene input sensor having a zoom function further expanded, a precise three-dimensional spatial position can be determined.

Finally, the actual work environment screen and the pseudo work screen are overlapped and it is confirmed whether or not they coincide with each other. If all the scenes match and there is no difference from the work plan, it means that the work is being performed according to the work plan. On the other hand, if there is a difference, which part is different is checked by performing pattern matching processing for each part or assembled part based on the design data and work plan data. If it is different, change the corresponding data in the database, or if only the position / orientation of the work robot is different,
Change your posture. If the entire scene or the background of the work robot is out of alignment, the position and direction of the scene input device are changed, and the scene in the actual work environment is corrected to match the scene in computer graphics. Work is done. Next, based on the two-dimensional image obtained by the scene input sensor, the three-dimensional position / orientation of the scene input sensor (television camera) itself capturing the image or the work target being captured. The method of detecting the is described in detail.

In the present invention, the position and orientation in the three-dimensional space are determined from the difference between the pseudo work environment screen created from the position data given in advance and the actual work environment screen position obtained by the scene input sensor. is doing.

First, FIG. 4 shows a model of a work environment on which the following description will be based. Here, for the sake of explanation, the robot or the work is shown as a rectangular parallelepiped and simply referred to as an “object”. In this work environment, there is a self-propelled robot equipped with one TV camera as a scene input sensor and four objects. The TV camera should be set in such a position and posture that the object can be seen.

As shown in FIG. 4, a static coordinate system independent of each individual object is provided in the work environment. This static coordinate system is introduced simply to facilitate processing such as coordinate conversion. On the other hand, TV cameras have their own visual coordinate system. Also, the four objects
Each has an independent object coordinate system, and the coordinates of the feature points (for example, vertices) of each object are defined on the object coordinate system. On the other hand, the origin of each coordinate system itself for an object in the visual coordinate system and the object coordinate system (hereinafter, the visual coordinate system and the object coordinate system may be collectively referred to as "local coordinate system") is , Is generally defined on a stationary coordinate system. In this way, the origin of the object coordinate system is defined on the stationary coordinate system, while the coordinates such as the vertices of the object are defined on the object coordinate system. , Can be regarded as the movement of the “object coordinate system” on the stationary coordinate system. Therefore, like the vertices on the object defined in the object coordinate system, the coordinates of the feature points of each object do not change at all, and the arithmetic processing can be simplified. The relationship between these coordinate systems is shown in FIG. In the figure, S
Is the origin of the stationary coordinate system, C is the origin of the visual coordinate system, and P is the origin of the object coordinate system.

Further, in the three-dimensional space (in this case, on the stationary coordinate system), six parameters are required to specify the position / orientation of the local coordinate system. In other words, the position parameter requires the position (x, y, z) of the origin of each local coordinate system on the stationary coordinate system, and the attitude parameter x, the stationary coordinate system.
Rotation angle θx of each local coordinate system around y and z axes,
θy and θz are required. Therefore, in the following description, the position of the origin C of the visual coordinate system on the stationary coordinate system is (Cx, Cy, Cz), and x of the stationary coordinate system is used.
The orientation of the visual coordinate system around the axes, y-axis, and z-axis is θ
Shown as 1, θ2, θ3. Similarly, the position of the origin P of the object coordinate system on the stationary coordinate system is (Px, Py, Pz)
Further, the postures of the object coordinate system around the x-axis, the y-axis, and the z-axis of the stationary coordinate system are shown as θα, θβ, and θγ.

Next, the creation of the pseudo work environment screen will be described. The creation of the pseudo work environment screen is performed by once temporarily collecting all of the three-dimensional data given in advance regarding the shape and characteristic points of the target object, the position and the posture where the target object should be placed, etc.
Convert to a static coordinate system (in this case, the conversion is not limited to the static coordinate system, but it is sufficient to convert to any one coordinate system.) After that, convert this again to the visual coordinate system, Furthermore, it is performed by converting into a two-dimensional screen imaged by a television camera.

To convert the coordinates defined on the visual coordinate system to the stationary coordinate system, first, (Cx, Cy, C
z) and then the y-axis of the stationary coordinate system,
It is only necessary to rotate the x-axis and the z-axis by θ2, θ1, and θ3, respectively. Similarly, when converting the coordinates defined on the object coordinate system to the stationary coordinate system, first, (P
(x, Py, Pz), and then θβ, θ for the y-axis, x-axis, and z-axis of the stationary coordinate system, respectively.
It is sufficient to rotate only α and θγ.

Equations 1 and 2 show this processing as a mathematical expression. In order to simplify the formula, si
Let nθi be represented by si and cos θi be represented by ci.

The coordinate conversion matrix TcS for converting the point on the visual coordinate system to the stationary coordinate system is

[0032]

[Equation 1]

Can be expressed as Trans in the equation is a translation transformation matrix. Also, RoT is a rotation movement conversion matrix,
Rot (y, θ2) means rotating by θ2 around the y-axis.

Similarly, a coordinate transformation matrix TpS for transforming a point on the object coordinate system with respect to the stationary coordinate system to the stationary coordinate system.
Is

[0035]

[Equation 2]

Can be expressed as

Therefore, an arbitrary point E on the stationary coordinate system
Coordinates Ec when (Ex, Ey, Ez, 1) are transformed on the visual coordinate system are

[0038]

[Equation 3]

It becomes

Similarly, the coordinate Oc when the position vector O (OXp, OYp, OZp, 1) of an arbitrary point on the object coordinate system is converted on the visual coordinate system is

[0041]

[Equation 4]

It can be expressed as

The coordinate data collected on the visual coordinate system using the conversion formulas of the above equations 1 to 4 is the two-dimensional screen (corresponding to the screen photographed by the television camera) as described above.
Is converted to. Here, a method of obtaining screen position coordinates when the coordinates Ec of the arbitrary point E converted on the visual coordinate system is converted on the two-dimensional screen will be described.

Coordinates Ec of an arbitrary point on the visual coordinate system
Forms an image at a point G on the imaging surface via a lens, as shown in FIG. The visual coordinate system is set so that its origin is located at the center of the lens. Point 1 and point 2 in the figure
Is in the plane perpendicular to the zc axis of the visual coordinate system. Also,
The points 3 and 4 are also in the plane perpendicular to the zc axis of the visual coordinate system. Therefore, the triangle formed by the points 0, 1, and 2 in the figure is similar to the triangle formed by the points 0, 3, and 4. Therefore, the point G on the imaging surface is

[0045]

[Equation 5]

Is obtained as Here, Sx is a coefficient representing an apparent focal length in the x direction, and Sy is a coefficient representing an apparent focal length in the y direction. Here, what is called the apparent focal lengths of Sx and Sy is that this is not the distance from the principal point of the lens to the image pickup surface of the TV camera, but the video signal of the object formed on the image pickup surface of the TV camera. This is because it means the apparent distance from the principal point of the lens to the screen in the computer obtained by converting into a pixel array that can be processed by the computer by analog / digital conversion.

Similarly, an arbitrary point O on the object coordinate system
The point Gp on the screen is

[0048]

[Equation 6]

Is obtained as

By using each of the above-mentioned relational expressions 1 to 6, it is possible to calculate how the object looks on the television camera based on the design data. When there are a plurality of objects at the same screen position, the distance to the television camera can be determined based on the distance information represented by OZc, and only the object on the front side can be displayed. In this way, the pseudo work environment screen can be generated.

Next, using the two-dimensional screen position information obtained by the television camera photographing the actual work environment,
A method for obtaining the three-dimensional spatial position information will be described.

The three-dimensional spatial position information is obtained by comparing the pseudo work environment screen obtained by the above processing with the work screen actually obtained by the television camera and extracting the shift between the two screens. be able to. In this case, the detected deviation between the two screens is
The processing content and the like are different between the case of extracting as the posture deviation and the case of extracting as the position / orientation deviation of the photographed object or the like. Therefore, in the following, both processes will be described separately.

First, the position of the TV camera itself,
The processing when it is handled as a posture shift will be described. In this case, it is assumed that the actual position / orientation of the object is known in advance. Also, as a pseudo work screen,
When the television camera is in a predetermined position and posture as designed, an image showing how an object looks on the screen of the television camera is created. Then, the position / orientation of the television camera is determined by comparing the created pseudo image with the actual screen. Then, the comparison process is actually performed by performing the following mathematical operation.

As can be seen from the equations (3) and (5),
The G components Gx and Gy of the equation 5 are on the stationary coordinate system,
The position (Cx, Cy, Cz) of the origin C of the visual coordinate system, the postures θ1, θ2, θ3, and the apparent focal lengths Sx, Sy of 8
It can be represented by one parameter. When the visual coordinate system (TV camera) is moved on the stationary coordinate system,
Actual position of TV camera (Cx ', Cy', Cz ',
θ1 ′, θ2 ′, θ3 ′, Sx ′, Sy ′) and design positions (Cx, Cy, Cz, θ1, θ2, θ3, Sx, S)
y), the deviation (ΔCx, ΔCy, ΔCz, Δθ
1, Δθ2, Δθ3, ΔSx, ΔSy) have occurred. At this time, the actual values of the parameters Cx ', Cy', C
Between z ', θ1', θ2 ', θ3', Sx 'and Sy',

[0055]

[Equation 7]

The following relationship holds. At this time, the position of the point G'on the screen is

[0057]

[Equation 8]

It becomes Here, ΔGx and ΔGy are position differences on the screen caused by the difference between the actual position and the design position. Parameter error ΔCx, ΔCy, ΔCz, Δθ1, Δθ2,
When Δθ3, ΔSx, and ΔSy are sufficiently small, the position difference ΔG is

[0059]

[Equation 9]

It can be expressed as Here, for example, ΔG
From the equation 5, the first term of x is

[0061]

[Equation 10]

It can be expressed as Also included in the number 10,
δExc / δCx and δEzc / δCx are equations 3 and 1
Than,

[0063]

[Equation 11]

Is obtained. The same applies to the other terms.

From the above equation (9), two relational expressions hold at one feature point. The screen position difference ΔGn of n feature points

[0066]

[Equation 12]

Will be written. Where ΔGn, X,
Rn is a screen position difference matrix of feature points, a parameter error matrix, and a partial differential coefficient matrix, each of which is expressed by the following equations 13 to 15:
It is shown by.

[0068]

[Equation 13]

[0069]

[Equation 14]

[0070]

[Equation 15]

By the way, ΔGn in the equation 13 is obtained from the screen position difference between the characteristic points of the pseudo image and the real image. Also, the partial differential coefficient matrix Rn can be calculated by substituting the design data into the equation 15. Therefore, the parameter error X is obtained from the following equation 13 by the following equation.

[0072]

[Equation 16]

This means that the simultaneous equations are solved by the least square method using the pseudo inverse matrix, and the accuracy increases as the number of feature points increases. On the contrary, if there are at least 4 characteristic points, the number 1
(RnT · Rn) ^{−1 in} 6 can be calculated, and the parameter error X can be obtained. Note that RnT is a transposed matrix of Rn.

By the way, the equation 12 is an approximate expression when the parameter error is sufficiently small, and the equation 16 is a solution of the approximate expression. The actual solution is obtained by substituting this approximate solution into Equation 7 to update the design value, substituting this into Equation 15 and re-obtaining Equation 16 by iterative calculation to converge within a certain range. By this calculation, the position and orientation of the visual coordinate system (television camera) on the stationary coordinate system and the apparent focal length can be determined. Hereinafter, this will be simply referred to as the "television camera position determination method".

Next, description will be made regarding the processing in the case where the deviation between the work screen and the pseudo work screen is treated as the deviation of the position / orientation of the object. In this case, it is assumed that the actual position / orientation of the TV camera is known in advance. Further, as the pseudo work screen, an image showing how the target object looks on the screen of the television camera is created when the target object is in the predetermined position and posture as designed. Then, the position / orientation of the object is determined by comparing the image with the actual screen. The comparison process is actually performed by performing the following mathematical operation.

The Gp components Gpx and Gpy of the equation 6 are
It can be represented by eight parameters of the position (Px, Py, Pz) of the origin P of the object coordinate system on the stationary coordinate system, the postures θα, θβ, θγ and the apparent focal lengths Sx, Sy of the television camera. . On the stationary coordinate system,
When the object coordinate system (object) is moved, the actual position of the object (Px ′, Py ′, Pz ′, θα ′, θβ ′,
θγ ′, Sx ′, Sy ′) and design positions (Px, Py,
Pz, θα, θβ, θγ, Sx, Sy), deviations (ΔPx, ΔPy, ΔPz, Δθα, Δθβ, Δθγ,
If ΔSx, ΔSy) has occurred, a difference corresponding to the deviation occurs between the pseudo work screen and the screen actually photographed. Therefore, similar to the above-described television camera position determination method, if the parameter error is obtained from the screen position difference and the partial differential coefficient matrix, the actual position, orientation, and apparent focal length of the object on the stationary coordinate system are determined. be able to. Hereinafter, this will be simply referred to as "object position determining method".

As described above, the difference between the TV camera position determining method and the object position determining method is that the former position of the self (TV camera) is determined based on the feature point information of a plurality of objects in the screen. On the other hand, the latter is a point that determines the position / orientation of the object itself based on the feature point information of the object.

The object position determining method described in the above description is based on the principle that the actual position / orientation of the television camera is accurately known. This is because if the exact position is not known, a pseudo image viewed from the lens principal point different from the actual TV camera will be created, and as a result, the accuracy of the object position determination will deteriorate. For the same reason, the TV camera position determination method described above also requires that the actual position / orientation of the object position is accurately known. That is, the two are dependent on each other, and accurate position detection and the like cannot be performed unless the actual position of either the television camera or all the objects is known in advance.

However, in the case where a TV camera is installed on the arm of the moving robot, it is expected that some of the TV camera and the object are displaced from the designed positions at the same time. Therefore, there remains an inconvenient problem that the application of the object position determining method and the like is limited to a specific situation where the television camera is fixed, for example. Therefore, the inventor of the present application was able to solve this problem to the extent that there is no practical problem by the first and second types of methods described below.

Since the accuracy of determining the position of the TV camera depends on the correctness of the position of the object, if the position of the object is determined by the position of the TV camera determined based on the position data not in the designed position. , Of course, the accuracy becomes worse.
Therefore, in order to determine the accurate position of the target object, first, the target object that is out of position is searched for, and then the position of the TV camera is checked again using the feature point information of the target object according to the remaining design position. To decide. Further, the position / orientation of the target object that is displaced at the determined position of this television camera is determined.

First, the outline of the first method of the present invention will be described.

First, the position of the television camera is provisionally determined using a plurality of objects. Then, one of these objects that is out of position is searched for. After that, the position of the television camera is re-determined again using the feature point information of the target object that is located at the designed position. Further, after this, the position and orientation of the displaced object is determined at the re-determined position of the television camera.

The method will be described in detail below with reference to FIG. In this case, in the initial state, the four objects P0,
It is assumed that P0, P1, and P2 of P1, P2, and P3 are at the correct positions according to the designed positions, but the object P3 and the television camera are displaced from the designed positions.

[Step a1] Four objects P0, P
1, P2, P3, and the television camera are all created in a pseudo manner using the design data that they have in advance, which screens would appear when they were in the design position. Then, using the created pseudo image and the image actually shot by the television camera, the above-described television camera position determination method is used (in this case, the positions and orientations of the objects P0 to P3 are all in correct positions). The position and orientation of the TV camera, that is, the position of the origin of the visual coordinate system on the stationary coordinate system and the orientation of the visual coordinate system are obtained. Hereinafter, the position and orientation of the TV camera obtained here will be simply referred to as “estimated camera position”.

However, since the feature point information of the displaced object P3 is also used, the position / orientation of the television camera obtained here, that is, the estimated camera position is the actual position on the stationary coordinate system. Is not a perfect match.

[Step a2] Subsequently, the above step a
A screen that will be seen when the TV camera is located at the estimated camera position obtained in 1 and each object is located at the design position is artificially created using the design data that is stored in advance. Then, using the created pseudo image and the image actually shot by the TV camera, the object position determination method described above is used (in this case, it is assumed that the TV camera is actually present at the estimated camera position). The position and orientation of each object P0 to P3 on the stationary coordinate system, that is, the position of the origin of each object coordinate system on the stationary coordinate system and the attitude of each object coordinate system are provisionally determined. Hereinafter, the position and orientation of the object obtained here will be simply referred to as “estimated object position”.

However, the estimated object position thus obtained does not always match the actual position / orientation of the objects P0 to P3 on the stationary coordinate system. This is the estimated camera position assumed when performing the object position determination method,
This is because the match with the actual position of the TV camera is not always established.

[Step a3] In the three-dimensional space on the stationary coordinate system, each object is set to the estimated object position and the television camera is set to the estimated camera position. However, the setting in this case is performed in a pseudo manner, and the TV camera and each object are not actually installed. After that, each of the above-mentioned objects is pseudo moved to the design position. In this case, the television camera also moves in a pseudo manner while maintaining the relative positional relationship established between the estimated object position and the estimated camera position as the respective objects move. The position / orientation of the TV camera when it is in the design position is obtained. The position of the TV camera obtained here is hereinafter referred to as "secondary estimated camera position".

Specific contents of arithmetic processing when the above processing is actually performed will be described below.

The estimated camera position obtained in step a1 is set to C2x, C2y, C2z, θ21, θ22, θ23.
Then, the matrix for coordinate conversion of the coordinates defined on the stationary coordinate system to the stationary coordinate system can be expressed by the following Expression 17 from Expression 1.

[0091]

[Equation 17]

Further, the object P0 obtained in the step a2
The estimated object position for P0x, P0y, P0
If z, θ0α, θ0β, and θ0γ, the matrix for converting the coordinates defined on the object coordinate system to the stationary coordinate system can be expressed by the following Expression 18 from Expression 2 above. .

[0093]

[Equation 18]

When Yn is a transformation of the feature point sequence Xn on the object P0 on the visual coordinate system, Yn is given by
It is represented as 9.

[0095]

[Formula 19]

On the other hand, the position / orientation when the object P0 is at the design position, that is, the design position of the object P0,
Pt0x, Pt0y, Pt0z, θt0α, θt0β,
When θt0γ, the coordinate conversion matrix to the stationary coordinate system can be expressed by the following Expression 20 from Expression 2 above.

[0097]

[Equation 20]

Here, if the actual position / orientation of the television camera is C3x, C3y, C3z, θ31, θ32,
It is assumed that it is θ33. Further, a matrix for converting the coordinates defined on the visual coordinate system into the stationary coordinate system can be expressed by the following Expression 21.

[0099]

[Equation 21]

Also, the characteristic point sequence Xn on the object P0.
Let Yn ′ be the result of conversion on the actual visual coordinate system, and Yn ′ is expressed as in Eq.

[0101]

[Equation 22]

At this time, if the same scene is displayed on the screen, that is, if Yn = Yn 'is considered,
The relationship of the following Expression 23 should be established.

[0103]

[Equation 23]

Therefore, the relationship of the following equation (24) is established.

[0105]

[Equation 24]

Now, C2x, C2y, C2z, θ21,
θ22, θ23, P0x, P0y, P0z, θ0α, θ
0β, θ0γ, Pt0x, Pt0y, Pt0z, θt0
Since α, θt0β, and θt0γ are known,
The above equations 19, 20, and 22 are uniquely obtained. Then, using this result and the equation 24, the equation 21
Can be obtained as in the following Expressions 25 and 26.

[0107]

[Equation 25]

[0108]

[Equation 26]

[Step a4] Next, the distance between the secondary estimated camera positions obtained for each of the objects is calculated.

[Step a5] Here, by observing the coincidence of the secondary estimated camera positions obtained for each object, it is possible to determine which object is displaced. That is, all the second-order estimated camera positions calculated using the objects P0, P1, and P2 whose actual position and design position match should match. On the other hand, the second calculated using the object P3 located at a position deviated from the design position
The next estimated camera position should not match the others. Therefore, if an object having a distance outside the allowable range of the quantization error is discriminated, an object that is not at the design position can be determined.

[Step a6] Using the characteristic point information of the objects P0, P1 and P2 which are actually found on the design position, the position of the television camera is accurately determined again.

[Step a7] Using the accurate position of the television camera determined in step a6, the position and orientation of the object P3 outside the allowable range are determined.

With the above procedure, the positions of the object not located at the designed position and the television camera can be determined with high accuracy.

Next, the second method will be described. In this method, the position of the TV camera is determined by applying the TV camera position determination method for each object. Hereinafter, the position of the TV camera obtained here will be referred to as a “provisional position”.

After that, the same processing as [Step a5] to [Step a7] of the first method is performed.
That is, it is possible to discriminate an object that is not at the design position by checking the coincidence of each provisional position for each object obtained here. Note that the method of determining the matching in this case is not limited to the method shown in steps a4 and a5.

In both the first method and the second method, it is necessary in principle to use three or more objects. This is because if there is one target object, there is no other party to compare with, and if there are two target objects, it is not possible to determine which is more accurate. Further, in the above description, it has been described that pseudo images are created at various places, but this is a conceptual meaning, and does not necessarily have to be an "image".

Next, another method for determining the position of the television camera or the object using the television camera and the object will be described.

First, the relationship when one TV camera and one object are used will be described. In the stationary coordinate system, when the visual coordinate system is moved, the actual position of the television camera (Cx ′, Cy ′, Cz ′, θ1 ′, θ2 ′, θ
3 ', Sx', Sy ') and the actual position of the object (P
x ′, Py ′, Pz ′, θα ′, θβ ′, θγ ′),
TV camera design position (Cx, Cy, Cz, θ1, θ
2, θ3, Sx, Sy) and design position (P
x, Py, Pz, θα, θβ, θγ), deviations (ΔCx, ΔCy, ΔCz, Δθ1, Δθ2, Δ).
θ3, ΔSx, ΔSy) and (ΔPx, ΔPy, ΔP
z, Δθα, Δθβ, Δθγ) has occurred. At this time, between the actual position of the parameter and the design position and deviation,

[0119]

[Equation 27]

The following relationship holds. At this time, the position of the point Gp 'on the screen is

[0121]

[Equation 28]

It becomes: Here, when the deviation of the parameters is sufficiently small, the position difference ΔGp is

[0123]

[Equation 29]

It can be expressed as The screen position difference ΔGpn of this feature point on the object is

[0125]

[Equation 30]

We will write Here, ΔGpn is the screen position difference vector of the feature point,

[0127]

[Equation 31]

It is shown by. Xp is a parameter error vector,

[0129]

[Equation 32]

It is shown by. SXp in this,
Among the parameter errors, cXp and oXp are the parameter error vector of the apparent focal length, the parameter error vector of the position and orientation of the camera, and the parameter error vector of the object.

[0131]

[Expression 33]

[0132]

[Equation 34]

[0133]

[Equation 35]

It is shown by.

Further, Rpn is a partial differential coefficient matrix,

[0136]

[Equation 36]

It is shown by. By the way, the ΔGpn of the equation 31 is obtained from the screen position difference between the feature points of the pseudo image and the real image. Further, the partial differential coefficient matrix Rpn can also be calculated by substituting the design data into the above equation 36. Therefore, the parameter error Xp is obtained by the following equation from the equation (30).

[0138]

[Equation 37]

Next, a method for determining the positions of a plurality of television cameras and a plurality of objects will be described by taking two television cameras and two objects as an example. Prior to this description, the relationship between one TV camera and one object described above will be summarized as follows.

The partial differential coefficient matrix Rpn is

[0141]

[Equation 38]

Let us say that. Where cRpn, oRpn, s
Rpn is a partial differential coefficient matrix related to the position / orientation parameter of the TV camera, a partial differential coefficient matrix related to the position / orientation parameter of the object, or a partial differential coefficient matrix related to the apparent focal length parameter of the TV camera. In addition, the parameter error vector Xp is

[0143]

[Formula 39]

Let us say that. Where cXp, oXp, sXp
Are the position / orientation parameter vector of the TV camera, the position / orientation parameter vector of the object, and the magnification parameter vector of the TV camera, respectively.

Next, in the case of using two TV cameras and two objects, the screen position difference vector ΔGall of the feature points on the object, the partial differential coefficient matrix Rall regarding the parameters, and the parameter error vector Xall Relationship is

[0146]

[Formula 40]

The following equation is obtained. Where ΔGall, Ral
l and Xall,

[0148]

[Formula 41]

[0149]

[Equation 42]

[0150]

[Equation 43]

It is re-expressed as follows. Where 1,1,1,2
The right shoulder indices i, j of etc. are meant for the jth object for the ith television camera.

In the equation 40, since the parameters are redundant,
As it is, it cannot be solved due to parameter error. Therefore, an example will be described in which any one of the following objects is fixed in the stationary coordinate system and solved. When the first object is fixed to the stationary coordinate system, the partial differential coefficient matrix Rall of the equation 42 and
The parameter error vector Xall of the equation 43 is

[0153]

[Equation 44]

[0154]

[Equation 45]

It becomes: Therefore, the number 40 is

[0156]

[Equation 46]

From this, the equation (45) gives the parameter error Xall as

[0158]

[Equation 47]

It can be obtained with. Here, Expression 46 is an approximate expression when the parameter error is sufficiently small, and Expression 47 is a solution of the approximate expression. Similar to the TV camera position determining method, the positions of the plurality of TV cameras and the position of the object can be determined by performing the convergence calculation.

Next, regarding the method of determining the positions of the TV camera and the object when there is a predetermined constraint relationship between the plurality of TV cameras, two TV cameras and two objects are used. An example will be explained.

Since the two TV cameras are constrained, an example in which the position / orientation of the first TV camera is used as a parameter and the first object is fixed in the stationary coordinate system and solved is described here. To do. Since the position / orientation of the second TV camera is constrained by the position / orientation of the first TV camera, the partial differential coefficient matrix Rall at this time is

[0162]

[Equation 48]

It can be expressed as Further, the parameter error Xall at this time is

[0164]

[Equation 49]

The relationship between the screen position difference vector ΔGall of the feature points on the object and the partial differential coefficient matrix Rall regarding the parameters and the parameter error vector Xall is as follows:

[0166]

[Equation 50]

It becomes: By the way, the screen position difference vector ΔGall of the feature points on the object is obtained from the screen position difference of the feature points between the pseudo image and the real image. Further, by substituting the design data into the equation (48), the partial differential coefficient matrix Rall regarding the parameter can be calculated. Therefore, the parameter error vector Xall is

[0168]

[Equation 51]

It is obtained by Here, Equation 50 is an approximate expression when the parameter error is sufficiently small, and Equation 5
1 is a solution of the approximate expression. Similar to the “TV camera position determination method”, the positions of the TV camera and the object can be determined by performing the convergence calculation when there is a predetermined constraint relationship between the plurality of TV cameras.

[0170]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a flexible production system to which a position / orientation detecting device for an object to be imaged is applied in order to carry out the position / orientation detecting method according to the present invention will be described below with reference to the accompanying drawings. First, the basic configuration of the flexible manufacturing system will be described.

The flexible production system of this embodiment is roughly divided into a work environment 1, an assembly station control device 2, a design data storage device 3, and a work plan data storage device 4, as shown in FIG. Is configured.

The work environment 1 is a work target 1 as a work.
2, peripheral device 13, work robot 14 as a processing device
In addition to the above, the visual robot 15, a posture changing device 33 having a television camera 34, and other structures are included. Then, the visual robot 15 images the work target 12, the peripheral device 13, the work robot 14, and the television camera 34, and the TV camera 34 detects the work target 12, the peripheral device 13, the work robot 14, and the visual robot 15. It is designed to capture images, and hereinafter, all of these captured images will be referred to as “objects”. In addition, peripheral device 1
The reference numeral 3 supports and conveys the work target 12 in the work environment 1, the details of which will be described with reference to FIG.

The work robot 14 has a hand effector that acts on the work target 12, a so-called robot hand, other processing tools, and the like, and uses this to assemble the work target 12. The work robot 14 is configured to be movable itself. The visual robot 15 has a scene detection unit that forms a TV camera similar to the TV camera 34 in order to capture an object to be imaged in the work environment 1 as image data, and is configured to be movable like the work robot 14. .

The posture changing device 33 is a device for changing the posture of the television camera 34. Then, the TV camera 34
Is a scene detection means for capturing the work environment 1 as image data, like the video camera of the visual robot 15.

The assembling station control device 2 determines the position / orientation of the object to be photographed based on the work environment data picked up by the visual robot 15 and the television camera 34, and the design data storage device 3 and the work plan data storage device 4. Deviation is detected, and the object to be photographed is moved and corrected in a direction to absorb the detected deviation. The details will be described later. Further, when the assembly station control device 2 detects a deviation of the position / orientation of the object to be photographed, it changes the design data by the amount of the deviation and corrects it.

The design data storage device 3 stores information on the shapes of products, robots, etc., their sizes and characteristic points, that is, product design data, robot design data,
Stores peripheral device design data.

The work plan data storage device 4 is the work target 1
2, work robot 14, peripheral device 13, visual robot 1
5, work procedure data, work process data, and motion path data for the posture changing device 33 are stored.

Next, the assembly station controller 2
More specifically, the configuration of the assembly station control device 2, as shown in FIG. 1, the assembly station control device 2, the pseudo work environment scene generation means 20, the image processing means 21, the work environment scene understanding means 22, the peripheral device control means 23, The work robot control means 24, the visual robot control means 25, the motion command generation means 26, the work environment database construction means 28, the work environment data storage means 29, the image data storage means 30, the image synthesis means 31, and the posture changing device control means 32. It is configured to have.

The work environment database construction means 28 constructs a database relating to the work environment from the data stored in the design data storage device 3 and the work plan data storage device 4 and the information obtained by the work environment scene understanding means 22. It is a thing. The data constructed here is stored in the work environment data storage means 29.

Then, the pseudo work environment scene generation means 20.
Has a function of artificially generating a work environment scene at a specific time during work by using the data stored in the work environment data storage means 29. In the following description, the work environment scene pseudo-generated based on the data in this way is referred to as a “pseudo scene image”.

The image processing means 21 is a television camera as the scene detection means of the visual robot 15 and the posture changing device 3.
3 has a function of processing the image data of the actual work environment obtained by the TV camera 34 of No. 3 and outputting it to the work environment scene understanding unit 22. The image processing means 21 also has a function of storing a plurality of image data in the image data storage means 30.
In the following description, the image of the actual work environment obtained by the scene detecting means will be referred to as "actual scene image" in correspondence with the pseudo scene image.

The image synthesizing means 31 has a function of synthesizing one high-definition image or a wide-angle high-definition image based on a plurality of image data stored in the image data storage means 30. The single high-definition image and the wide-angle high-definition image are selectively executed by the support from the work environment / scene understanding means 22 described later. In the following description, one high-definition image or an image combined into one high-definition image based on a plurality of images of the actual work environment obtained through the visual robot 15 and the TV camera 34 is referred to as a “high-definition combined image” or “high-definition combined image”. Although they are called "wide-angle composite images", these are also considered as real scene images.

The work environment scene understanding means 22 compares the pseudo scene image generated by the pseudo work environment scene generation means 20 with the real scene image obtained by the visual robot 15 or the like to determine the position / position of the work target 12 or the like. The deviation of the posture is detected.

Further, the operation command generation means 26 has a function of generating an operation command based on the information obtained by the work environment scene understanding means 22, and the peripheral device control means 23 based on the generated operation command. , The work robot control means 24 controls the work robot 14, the visual robot control means 25 controls the visual robot 15, and the posture changing device control means 32 controls the posture changing device 33.

Next, with reference to FIG. 2, a specific hardware configuration for realizing the assembly station control device 2 will be described.

The assembly station controller 2 includes a system bus 101, a bus controller 102, and a central processing unit 103.
a, main storage device 103b, magnetic disk device 104, keyboard 105, display 106, image generation device 1
07, image processing device 108, television camera 109, zoom motor / focus motor / iris motor control device 110, lens system 111 with zoom motor / focus motor / iris motor, motor 116, pulse generator 113, counter 112, motor drive device 115, DA converter (digital / analog converter) 1
14, AD converter (analog / digital converter) 11
7, a sensor 118, and a network 119. Since these are the same as those generally used in the market, the detailed description of each is omitted.

Among the constituent elements of the assembly station control device 2, the work environment database construction means 2 is used.
8, work environment data storage means 29, image data storage means 30, image composition means 31, work environment scene understanding means 22,
The peripheral device control means 23, the work robot control means 24, the visual robot control means 25, the motion command generation means 26, and the attitude changing device control means 32 are mainly the central processing unit 10.
3a and its main storage device 103b and magnetic disk device 1
It is realized by 04. The pseudo work environment scene generation means 20 is mainly realized by the image generation device 107. Further, the image processing means 21
Are mainly realized by the image processing device 108. However, each component does not have these functions by itself, but achieves these functions in close cooperation with others, so the correspondence relationship described here is not necessarily strict.

Data is exchanged with the design data storage device 3 and the work plan data storage device 4 in FIG. 1 via the network 119. Also, FIG.
The TV camera 109 and the lens system 111 shown in FIG.
Is mounted on the visual robot 15 or the posture changing device 33 instead of the assembly station control device 2 shown in FIG. Similarly, the motor 116 is mounted on the peripheral device 13 and the work robot 14 shown in FIG. 1 and drives them.

Details of the work environment 1 will be described below with reference to FIG. In the figure, a table type robot 301a,
Reference numeral 301b is for mounting the work target 12, and constitutes the peripheral device 13 shown in FIG. Each of these robots 301a and 301b is configured to move and move as needed.

In addition, the arm-shaped robots 302a, 30
Each of 2b and 302c has an arm whose movement can be controlled freely. Then, at the tip of the arm of each robot, the hand effectors 3021a and 3021 for performing work such as processing and assembling on the work target 12 are performed.
b, a television camera 3022 is provided. here,
Robot 3 having hand effectors 3021a and 3021b
02a and 302b are the work robots 14 shown in FIG. 1, and the robot 302c having the television camera 3022 is the visual robot 15 shown in FIG.

Note that the television camera 3022 is the same as that shown in FIG.
TV camera 109 and lens system 111 in
It corresponds to. And each of these robots 30
2a to 302c are also table-shaped robots 301a and 30
Similar to 1b, it is configured to move by itself when necessary.

On the other hand, a television camera 34 is arranged at a position near each of the robots so as to be able to image them. The television camera 34 is installed in the posture changing device 35. The television camera 34 also corresponds to the television camera 109 and the lens system 111 in FIG. In addition, the work transfer robots 303a, 30
3b is for carrying the work target 12 and moving between each process, and forms the peripheral device 13 like the robots 301a and 301b.

Next, the operation of the flexible production system having the above configuration will be described. In the flexible manufacturing system, the equipment configuration is dynamically changed according to the product design information and the type and quantity of the product, and processing and assembling are performed according to an online command from the work planning department according to the work plan. In each process, the peripheral device 1 shown in FIG.
3, the work robot 14, the visual robot 15, and the attitude changing device 33, based on the operation command generated by the operation command generating means 26, the peripheral device control means 23, the work robot control means 24, the visual robot control means 25, the attitude changing device. It is controlled by the control means 32.

This will be described concretely with reference to FIGS. 2 and 3. The central processing unit 103a generates an operation command (digital signal). The operation command is sent to the DA converter 114 via the system bus 101, converted into an analog signal here, and then sent to the motor drive device 115. The motor drive device 115 moves the motor 116 according to the operation command (analog signal), and the robots 301a and 301b as the peripheral device 13, the robots 302a and 302b as the work robot 14, and the visual robot 15
The robot 302c and the posture changing device 35 are operated. Corresponding to these operations, the counter 112 and the pulse generator 113 also operate and the robots 301a, 3a
01b, 302a, 302b, 302c and the posture changing device 35 are used for control.

Thus, for example, the robots 302a, 302
02b uses the hand effectors 3021a and 3021b to assemble the work target 12 placed on the robot 301b. At that time, the visual robot 302c having the TV camera 3022 and the TV camera 34 installed in the posture changing device 35 monitor the assembling work.

The television cameras 3022 and 34 are provided with zoom motors, focus motors, focus motors, iris motors controlled by the controller 110.
If the user has a lens system 111 with an iris motor and wants to see the monitored work precisely, the robot 302c and the posture changing device 35 change the direction of the television cameras 3022 and 34 to a desired position, and the lens system By setting 111 to the telephoto position, a high-definition composite image is obtained.

The data indicating the shape and size of the parts to be assembled which are necessary for the work are the magnetic disk 1
The one stored in 04 is used. Also, similarly,
The operation command to each of the robots 302a, 302b, 301a, 301b and the hand effectors 3021a, 3021b includes, for example, movement route, placement, assembly work procedure data,
The work process data and the like stored in the magnetic disk 104 are also used.

In the operation as described above, even if the same parts are used, there are differences among the individual parts due to variations in the machining process in the middle. Therefore, the greater the number of parts to be assembled, the greater the degree of positional deviation, and the more errors there are. This error cannot be avoided even if each robot operates according to the control data.

In this embodiment, each work robot 14
The position and orientation of the visual robot 15 and the TV camera 3
The position / orientation is corrected by monitoring with 4. The procedure for monitoring the position / orientation and correcting it will be described below.

First, the data necessary for analyzing the work environment from the data stored in the design data storage device 3 and the work plan data storage device 4 by the work environment database construction means 28 shown in FIG. Is read as work environment data, and the read data is registered in the work environment data storage means 29 (in FIG. 2, it becomes the magnetic disk 104).

Then, when the work robot 14 or the like is in the position / posture according to the work environment data based on the work environment data stored in the work environment data storage means 29 by the pseudo work environment scene generation means 20. Artificially generate a scene that the visual robot 15 will capture.
That is, a pseudo-scene image at a specific time during work is generated. This processing content is shown in FIG.
3a, the main storage device 103b, and the image generation device 107 generate a pseudo-scene image using the work environment data stored in the magnetic disk 104.

Then, the pseudo work environment scene generation means 20
Outputs the generated pseudo scene to the work environment scene understanding unit 22 shown in FIG. The process of generating the pseudo-scene has already been described in the section of the above-mentioned action, and the description thereof will be omitted here.

On the other hand, when the visual robot 15 or the television camera 34 captures an image on the same time axis as the generated pseudo-scene image, the image processing means 21 performs a predetermined process on the captured image and then executes the actual processing. While outputting to the work environment scene understanding means 22 as a scene image, a plurality of image data as an actual scene image are stored in the image data storage means 30. In FIG. 2, the image processing device 108 is the television camera 1
The image obtained at 09 is processed and output to the central processing unit 103a.

The image synthesizing means 31 synthesizes a plurality of image data stored in the image data storage means 30 into one synthetic image, and then outputs the synthetic image to the work environment / scene understanding means 22. This will be processed by the central processing unit 103a in FIG.

In the work environment scene understanding means 22, the pseudo scene image generated by the pseudo work environment scene generation means 20 and the real scene image of the work environment 1 obtained by the visual robot 15 and the television camera 34 (a wide-angle composite image). , Or a high-definition composite image is included), and the difference is extracted. That is, information on the feature points of each of the work target 12, the peripheral device 13, the work robot 14, the visual robot 15, and the television camera 34 is extracted from the real scene image, and the information on the feature points and the pseudo scene image are compared. Thus, the position / orientation of each of these parts 12 to 15 and 34 is specified, and the position / orientation deviation is detected. Since the processing such as the identification of the position / orientation has already been described in the above-mentioned action column,
The description here is omitted.

As a result of the above processing, when there is a difference between the pseudo scene image and the real scene image, the work environment scene understanding means 22 causes the work environment data storage means 29 which is the source of the pseudo scene image so that they coincide with each other. Change the work environment data itself. As a result, in the subsequent operations of the respective units 12 to 15 and 34, the pseudo scene image and the real scene image match each other.

Alternatively, the work target 12, the peripheral device 13,
Work robot 14, visual robot 15, TV camera 3
Among the four items, a command is sent to the motion command generating means 26 in order to correct the position / orientation deviation that has occurred. As a result, the motion command generation means 26 issues a motion command in accordance with the command and causes the respective units 12 to 15 and 34 to operate so as to match the pseudo scene image.

The above processing is performed by the work target 12 and the peripheral device 1.
3. The work robot 14 (301a, 301b), the visual robot 15, and the television camera 34 are executed each time the work is performed, that is, each time one specific work is performed. As a result, the work target 12, the peripheral device 13, the work robot 14, the visual robot 15, and the television camera 3
Since 4 and the work are performed while checking the work position each time the work is performed, the work can be performed according to the work plan. If the above-described object position determining method is executed for each object (work robot 14 or the like), it is possible to know the relative positional relationship and distance between them.

As described above, in the above embodiment, the work target 12 as the work environment 1, the peripheral device 13, the work robot 14, the visual robot 15, and the television camera 3 are used.
4 and the position / orientation of each of the parts 4 and 4 are predicted based on the work environment data, and the predicted pseudo-scene images of the respective parts 12 to 15 and 34 are calculated, and the calculated pseudo-scene image and the parts 12 that are actually imaged are acquired. ~ 15, 34 of the real scene images are compared with each other, and as a result of the comparison, if there is a deviation between the pseudo scene image and the real scene image, the corresponding units 12 to 15, 34 are
The actual work environment can be clearly analyzed because the work environment data is corrected by correcting the deviation so as to absorb the deviation or by the deviation. Further, since the work environment is automatically analyzed when these corrections / changes are made, the respective parts 12 to 15 and 34 can be accurately operated only by the numerical data, so that the online teaching is not necessary. .

Therefore, if the position / orientation detecting device as described above is used in a production system, the production system can be operated autonomously. Further, since it is not necessary to install the work target and the robot with high accuracy, the equipment can be manufactured at low cost. Furthermore, not only is it unnecessary to stop the production line for a long time, it is economical, and the manufacturing time can be shortened.

In addition to this, the scene detecting means including the visual robot 15 and the television camera 34 is made independent of the work robot 14 and the peripheral device 13, and the work environment is taken in while operating the work robot and the peripheral device so that the scene is accurately obtained. Since it can be determined, the manufacturing time can be shortened as much as possible.

Also, since the position of the scene detecting means can be accurately obtained based on the feature points in the work environment obtained by the scene detecting means, the relative relationship between any two persons in the work environment can be obtained. Can be accurately determined.

Next, the posture changing device 3 is shown in FIGS.
Example 3 will be described. First, a specific configuration for realizing the attitude changing device 33 will be described with reference to FIG. The posture changing device 33 includes a base 401 and a first rotary shaft 4
02, a second rotary shaft 403, and a television camera 404. That is, the first rotation shaft 402 is mounted on the base 401, and the output portion 402a thereof is configured to rotate around the vertical axis. A second rotary shaft 403 is attached to the tip of the output unit 402a via a pedestal 406a, and the L-shaped output unit 403a of the rotary shaft 403 is rotated about a horizontal axis. A television camera 404 is fixed to the tip of the output unit 403a.

In this case, the output unit 4 of the first rotary shaft 402
02a extension line Y and the output unit 40 of the second rotation shaft 403.
The central axis Z of the camera lens 405 in the television camera 404 is located at a position D orthogonal to the extension line X of 3a, and is orthogonal to each other on the extension line, and the rotation center of the camera lens 405 is coincident with that. The principal point of the camera lens 405 is arranged so as to coincide with the coincident position D. Therefore, on the first rotary shaft 402 and the second
The rotation axis 403 of the camera lens 405 and the central axis Z of the camera lens 405 are arranged so as to intersect at a principal point (D) of the camera lens 405.

The first and second rotary shafts 402 and 403
Both are composed of a motor with a reduction gear, and are directly connected to a high-resolution encoder. Since the rotation of the motor is transmitted through the reduction gear, one rotation on the reduction gear output shaft side is equivalent to 100 rotations on the motor side, and if an encoder that divides one rotation of the motor into 4000 is used, one rotation on the reduction gear output shaft is performed. It is possible to obtain a precision as high as dividing the rotation into 400,000.

The rotary shafts 402 and 403 are as shown in FIG.
Attitude changing device control means 3 of the assembly station control device 2
2 and also as shown in FIG.
Driven at 15. The pulse encoders attached to the rotary shafts 402 and 403 are counted by the counter 112 and used to determine the position / orientation of the television camera 404.

In this embodiment, the posture changing device 33 has the first rotary shaft 402 and the second rotary shaft 403 as described above, and the extension Y of the output portion 402a of the first rotary shaft 402 is used. Extension line X of the output portion 403a of the second rotation shaft 403
The rotation center of the camera lens 405 of the television camera 404 is arranged at a position D orthogonal to the rotation center of the television camera 404, and when the first and second rotation axes 402 and 403 are selectively driven, the rotation center of the camera lens 405. Since the television camera 404 has a swinging function of rotating around, the detection accuracy when detecting an object to be photographed can be increased by the swinging function.

Moreover, since the coincident positions D are the main points of the camera lens 405, the lens characteristics such as the distortion of the camera lens 405 are not affected, and more accurate detection can be performed.

The embodiment shown in FIG. 9 is obtained by adding one rotary shaft 406 to the embodiment shown in FIG. That is, in this case, the mounting plate 4 is attached to the output portion 403a of the second rotating shaft 403.
07, the third rotation shaft 406 is installed through the L-shaped support portion 408 provided on the mounting plate, and the television camera 404 is installed at the output portion (not shown) of the rotation shaft 406. The television camera 404 is rotated on the mounting plate 407 and the supporting portion 408 by driving the rotation shaft 406 of the third rotation axis 406 around the horizontal axis orthogonal to the second rotation axis 403, that is, around the rotation center of the camera lens 405. ing. In this case, the rotation center of the camera lens 405 is aligned with the intersection position D between the extension line Y of the output portion 402a of the first rotation shaft 402 and the extension line X of the output portion 403a of the second rotation shaft 403, and The principal point of the camera lens 405 is located at the intersection position D between the rotation center and the extension lines Y and X.

According to this embodiment, the first to third rotary shafts 402, 403 and 406 are provided, and by driving these,
Since the television camera 405 can be delicately rotated, the detection accuracy when detecting the object to be photographed can be further improved as compared with the embodiment shown in FIG.

FIG. 10 shows a posture changing device 33 of the embodiment shown in FIG.
The parallel movement axis is added to. That is, the mounting plate 410 is attached to the output portion 403a of the second rotating shaft 403, the parallel movement shaft 409 including a cylinder is installed at one end of the attachment plate 410, and the parallel movement portion 4 of the movement shaft 409 is installed.
The television camera 404 is connected to 09a. Therefore, the television camera 404 can be linearly moved in a direction orthogonal to the rotation axis 403 in the same plane as the second rotation axis 403 by driving the translation shaft 409.

The central axis Z of the camera lens 405 of the television camera 404 is the extension line Y of the output section 402a of the first rotary shaft 402 and the output section 4 of the second rotary shaft 403.
It is arranged at a position D intersecting with the extension line X of 03a, and the principal point is located on the central axis Z thereof.

According to this embodiment, the television camera 404 can be rotated by driving the first and second rotating shafts 402 and 403, and the camera lens 4 of the television camera 404 can be moved.
05 is in the same plane as the second rotary shaft 403,
Since it moves linearly in a direction orthogonal to 3, the camera lens 4
The focus position of 05 can be adjusted easily and in a short time.

FIG. 11 shows the embodiment of FIG. 10 with a rotary shaft 406 similar to that of the embodiment of FIG. That is, the second
A third rotating shaft 406 is installed on a mounting plate 410 attached to the rotating shaft 403 of the supporting unit 408 via a supporting unit 408.
08, the camera lens 405 is rotated about the center of rotation.

Therefore, according to this embodiment, as compared with the embodiment shown in FIG. 10, the television camera 404 moves linearly in a direction orthogonal to the second rotation axis 403 on the same plane as the second rotation axis 403. Since the camera lens 405 is pivotally moved around the central axis Z, more precise detection can be performed.

By the way, normally, the camera lens 405
Often the center of rotation and the principal point do not match. Therefore,
When the principal point of the camera lens 405 and the rotation center cannot match, the characteristic point is aligned with a specific position in the screen, for example, the center of the screen by using the posture changing device, and the angle of the rotation axis of the posture changing device at this time. Use the data. A posture changing device using two axes as shown in FIG. 8 will be specifically described with reference to FIG. 12. First, as a coordinate system of the posture changing device, a vertically downward axis is Y and a horizontal axis is X. , Z is the axis that intersects these two axes and is orthogonal to them. Further, the rotation angles of the first and second rotation shafts 402 and 403 are ψ1 and ψ2, respectively. At this time, the direction R of the center line of the lens
ot (y, ψ1) Rot (x, ψ2) Z is given by the following formula.

[0227]

[Equation 52]

[0228] When this is used as image information of a plane, for example, as shown in FIG. 13, it may be re-projected onto a plane Z = 1 parallel to the X and Y planes. When the projected coordinates on the plane are G _X and G _Y , G _X and G _Y are obtained as follows.

[0229]

[Equation 53]

[0230]

[Equation 54]

In this case, the number of feature points to be used is determined in the same manner, and all the feature points are projected on the same plane. The data of this plane is used as one image data.

Next, the advantage of aligning a feature point to an arbitrary point on the screen using the posture changing device will be described. When a telephoto lens is used for the camera lens 405, the focus must be adjusted. If so, the feature points will be blurred. In addition, the principal point position of the lens will deviate just by shifting the focus. However, if the characteristic points are aligned with arbitrary points on the screen, it is not necessary to know the position of the principal point of the lens. Therefore, there is an advantage that the number of parameters to be measured can be reduced. In addition, it is generally considered difficult to produce a lens without distortion, and using the entire surface of an image is said to lead to a decrease in measurement accuracy. The device has advantages.

Next, by arranging a plurality of television cameras three-dimensionally, capturing a wide range simultaneously as an image using a high-magnification camera lens, processing it into one high-definition image, and changing the movement time in the posture changing device. I lost it. An example of this will be described with reference to FIGS. In FIG. 14, a plurality of television cameras 501 to 505 are arranged in an arc, and the position of the principal point D of each camera lens is adjusted to one point. FIG. 15 shows a television camera arranged on a spherical surface, as seen from the center line direction of the central camera in FIG.

Next, the procedure for measuring the characteristic points of the imaged data will be described with reference to FIG. Assuming that the posture measurement object in the figure is composed of three boxes a, b, and c, these three boxes have distinctive feature points with a contrast. This is indicated by a "+" mark in the figure. These characteristic points are measured in advance in the coordinate system of each of the three boxes. When the workpiece is machined as designed, it goes without saying that the design value may be used instead of the measured value.

The boxes of A, B, and C are random. In this case, the measurement is to know what the position and orientation of the box B and the box C are with respect to the largest box a. The posture changing device 33 in the drawing is driven so that the characteristic points of the boxes A, B, and C are located at a specific position on the screen, for example, the center of the screen, and the rotation angle at that time is detected. Then, using the rotation angle, the image is re-projected to a point on the same plane by the above-mentioned equations 53 and 54, that is, the position and orientation of the posture changing device 33 and the positions of the boxes B and C with box A as the origin. The attitude parameter is obtained by the procedure described above.

By the way, when the posture changing device 33 is used, the minimum required number of feature points will be described. The posture changing device 3 for a certain object, for example, box A in FIG.
The parameters that need to be calculated by 3 are the posture changing device 33
There are six positions of the origin (Cx, Cy, Cz) and postures (θ1, θ2, θ3) related to the directions of the coordinate axes. On the other hand, the image information of the feature points is two coordinate values of each feature point on the same plane. Therefore, if there are three feature points, six position / posture parameters of the posture changing device 33 with respect to the object can be obtained.

This will be described in detail using mathematical expressions. Figure 1
The posture changing device 33 of 6 is the same as that shown in FIG.
The axis type is used, and the coordinate system is the same as in FIG.
The rotation angles of the first and second rotation shafts 402 and 403 are respectively ψ
1 and ψ2. At this time, the direction of the central axis of the camera lens 405 can be expressed by the above equation 52. In order to use this as plane image information, the image is re-projected onto the plane Zc = 1 parallel to the XcYc plane. If the projected coordinates on the plane are G _X and G _Y , then G _X and G _Y can be obtained as equations 53 and 54.

The position of the origin C of the coordinate system of the posture changing device on the stationary coordinate system is (C _X , C _Y , C _Z ), and the posture is a rotation angle θ1 around the X, Y, and Z axes of the stationary coordinate system. , Θ2, θ3
It is represented by.

Further, the position of the origin P of the position / orientation object coordinate system on the stationary coordinate system is (Px, Py, Pz), and the orientation is a rotation angle θα around the X, Y, and Z axes of the stationary coordinate system. ，
It is represented by θβ and θγ. On the other hand, the apparent focal lengths S _X and S _Y of the posture changing device are Zc on the plane parallel to the Xc and Yc planes.
Since = 1 is set, S _X = 1 and S _Y = 1 are fixed values.

The parameters determined here are the same as when one television camera and one object described in the operation are used. Therefore, the screen position difference of the feature points is
It can be represented by 9. However, if the cause of the screen position difference is only the position / orientation measurement object, the expression 29 is changed to the following expression 55.

[0241]

[Equation 55]

These 6 parameters are the difference factors,
If there are at least three characteristic points on the position / orientation measurement target, measurement can be performed.

Although the case where each feature point is measured on the same plane has been described above, a method of measuring the position of one feature point of the object will now be described. The position of one feature point has three parameters (X, Y, Z). On the other hand, one scene detection unit has two parameters (Gx, Gy) because it is a plane image. Therefore, the position of one feature point cannot be measured unless there are two or more scene detecting means. A method of obtaining the positional relationship between two or more scene detection sensors will be described with reference to FIG.

In FIG. 17, two posture changing devices 33 are provided.
Two scene detection means (television cameras) A and B respectively attached to 1 and 332, and a calibration work 3
33 and a measurement object 334. The two posture changing devices 331 and 332 have two rotation axes as shown in FIG. On the other hand, a plurality of calibration marks 333a are pasted at arbitrary positions on the calibration work 333, and each mark 333a is measured in advance in the work coordinate system by a coordinate measuring machine, and the relative marks The position is captured as data.

Each of the scene detection means A and B detects the mark 333a on the calibration work 333 and detects the tertiary coordinate of the scene detection means B and the calibration work 333 in the stationary coordinate system, for example, the coordinate system of the scene detection means A. Measure the original position and orientation. By this operation, the relative positions of the two scene detection means A and B are determined.

Next, obtaining the length L between the two points P and Q of the measurement object 334 in the figure will be described. Each of the scene detection means A and B aligns the point P of the measurement object 334 with the center of the screen, and obtains the angle of the rotation axis in the posture changing devices 331 and 332 at that time. From this angle, one point on each plane is obtained, and three-dimensional coordinates are obtained by the method described above. Similarly, the Q point is also obtained. The distance L between the points P and Q is obtained from the three-dimensional coordinates of the points P and Q. Here, how to obtain the three-dimensional position will be described in more detail.

For example, when the coordinates of the point P are obtained, the angles of the rotation axes of the posture changing device 331 when the scene detecting means A aligns the point P with a specific position on the screen are ψ ₁ ^A and ψ ₂ ^A. For example, according to the following equations 56 and 57,

[0248]

[Equation 56]

[0249]

[Equation 57]

As a result, the coordinates G _X ^A and G _Y ^A on the plane are obtained.

Similarly, if the angles of the rotation axes of the scene detecting means B (angles of the rotation axis of the posture changing device 332) are ψ ₁ ^B and ψ ₂ ^B ,

[0252]

[Equation 58]

[0253]

[Equation 59]

As a result, the coordinates G _X ^B and G _Y ^B on the plane are obtained.

The design position of the point P in the stationary coordinate system is represented by Px, P
Let y, Pz, and the actual positions be Px ', Py', Pz '. If the positional deviations at that time are ΔPx, ΔPy, and ΔPz, the following relational expressions are obtained.

[0256]

[Equation 60]

[0257]

[Equation 61]

[0258]

[Equation 62]

Here, the screen position on the design and the actual screen
The difference from the position is ΔG_X ^A, ΔG_X ^B, ΔG _Y ^BThen,
The relational expression holds.

[0260]

[Equation 63]

[0261]

[Equation 64]

[0262]

[Equation 65]

[0263]

[Equation 66]

Then, these equations 63 to 66 are expressed by ΔPx,
Solve as a simultaneous equation of ΔPy and ΔPz. This ΔPx,
ΔPy and ΔPz are approximate values, and the above-mentioned formulas 60 to 6
From the relationship of 2

[0265]

[Equation 67]

[0266]

[Equation 68]

[0267]

[Equation 69]

By using the relation of Px ', Px
Update y'and Pz '. By repeating this procedure, the actual positions Px ', Py', Pz 'can be obtained.

As described above, if the three-dimensional coordinates of each feature point are known, it is obvious that by combining these, it is possible to define where an object in space is. Therefore, it is necessary to create a three-dimensional map. You can also

Next, a method of obtaining one point on a known curved surface in advance by one scene detecting means will be described with reference to FIG.
A known curved surface is represented by Z = f (x, y) in advance.
Further, the following description will be given on the assumption that the position / orientation of the scene detection means with respect to the curve has already been obtained.

Referring now to FIG. 18, the scene detection means 180 is now available.
A vector from the point to the point B is known from the value of the posture changing device 331. To obtain the coordinates of point B at this time,
As shown in FIG. 19, coordinates of point A (x0, y0, z0)
The point C on the curved surface represented by (x0, y0) is obtained.
Next, in the vicinity of this point C, for example (x0 + Δx0, y
0), (x0-Δx0, y0), a vector a connecting points on the curved surface is obtained. Similarly, (x0,
A vector b connecting points on the curved surface obtained by (y0 + Δy0), (x0, y0−Δy0) is obtained. The tangent plane at the point C can be defined by the outer product of the vectors a and b. The intersection of this tangent plane and the vector from A to B is D (x1, y
1, z1). A plane in contact with a point E on the curve represented by (x1, y1, z1) is obtained, and an intersection F of a vector from this plane and A to B is obtained. By repeating this operation, the coordinates of point B on the curved surface can be obtained.

By the way, as shown in FIG. 18, if a certain area on a curved surface is to be defined by using the scene detecting means 180, a line along the diagonally shaded area in the figure is drawn as the scene detecting means 18.
It is possible to define a region on a three-dimensional free-form surface by defining on the screen obtained with 0 and finding the corresponding point on the curved surface for one pixel on the screen by the method described above.

[0273]

As described above, according to the first aspect of the present invention.
According to 4, the actual work environment scene and the pseudo work scene in the design are analyzed, and each position / orientation in the work environment is grasped based on the analyzed result to obtain the actual control data. Since it is configured so that it can be changed or each position / orientation in the work environment can be corrected to match the design value, the position / orientation deviation in the work environment can be automatically corrected with high accuracy. There is an effect, and there is also an effect that it is not necessary to perform online teaching of the processing apparatus by automatically analyzing the work environment when making corrections and changes.

According to claims 5 to 11 of the present invention,
There is an effect that the method according to claims 1 to 4 can be carried out accurately, and particularly according to claims 8 to 11, the camera lens in the television camera as the scene detection means can change its attitude around the principal point position. Therefore, there is an effect that more accurate detection can be performed without being affected by lens characteristics such as distortion of the camera lens, and it is possible to easily obtain a high-definition image in a wide angle with an inexpensive TV camera on the market. It is affordable and economical.

Further, according to the twelfth to seventeenth aspects of the present invention, since the position / orientation detecting device according to the fifth to eleventh aspects is provided, the production system can be operated independently, and each of the working environment can be operated. Since it is not necessary to manufacture with high precision, the equipment can be made less expensive, and there is no need to stop the production line for a long time, resulting in a flexible and economical system that can handle high-mix low-volume production. Can be provided.

[Brief description of drawings]

FIG. 1 is a basic configuration diagram of an embodiment of the present invention.

FIG. 2 is a diagram showing a hardware configuration example of an assembly station control device shown in FIG.

FIG. 3 is an explanatory diagram of an example of the work environment shown in FIG.

FIG. 4 is an explanatory diagram of a work environment model of the present invention.

5 is an explanatory diagram of a relationship between each coordinate system in FIG.

FIG. 6 is an explanatory diagram showing perspective transformation of an arbitrary point Ec on the visual coordinate system onto an imaging surface.

FIG. 7 is a flowchart showing a process of determining the position / orientation of a television camera and an object.

FIG. 8 is an explanatory diagram showing a posture changing device having two rotation axes.

FIG. 9 is an explanatory diagram showing a posture changing device having three rotation axes.

FIG. 10 is an explanatory view showing a posture changing device having two rotating shafts and one linear moving means.

FIG. 11 is an explanatory diagram showing a posture changing device having three rotation axes and one linear movement unit.

FIG. 12 is an explanatory diagram showing a posture changing device when the positions of the rotation center of the camera lens and the principal point do not match.

FIG. 13 is an explanatory diagram for projecting the imaged feature points on the same plane when the positions of the rotation center of the camera lens and the principal point in the posture changing device do not match.

FIG. 14 is an explanatory diagram in which a plurality of television cameras are arranged at the same radial position with respect to the principal point position.

FIG. 15 is an explanatory diagram when a plurality of television cameras arranged around the principal point position are viewed from the principal point position.

FIG. 16 is an explanatory diagram for measuring the position of each feature point when the feature point of an object is imaged.

FIG. 17 is an explanatory diagram for measuring the position of one feature point of an object by using two posture changing devices each having a scene detection unit.

FIG. 18 is an explanatory diagram in the case of obtaining one point on a known curved surface by one scene detection means.

FIG. 19 is an explanatory diagram of a coordinate system in the case of similarly obtaining one point on a known curved surface by the scene detection means.

[Explanation of symbols]

1 ... Work environment, 2 ... Assembly station control device, 3 ...
Design data storage device, 4 ... Work plan data storage device, 1
2 ... Work target, 13 ... Peripheral device, 14 ... Work robot,
Reference numeral 15 ... Visual robot, 20 ... Pseudo work environment scene generation means, 21 ... Image processing means, 22 ... Work environment scene understanding means, 23 ... Peripheral device control means, 24 ... Work robot control means, 25 ... Visual robot control means, 26 ... operation command generation means, 28 ... work environment database construction means, 29 ...
Work environment data storage means, 30 ... image data storage means,
31 ... Image synthesizing means, 32 ... Posture changing device control means, 3
3, 331, 332 ... Attitude changing device, 34, 180, 4
04 ... TV camera, 405 ... Camera lens, 101 ...
System bus, 102 ... Bus control device, 103a ... Central processing unit, 103b ... Main storage device, 104 ... Magnetic disk device, 105 ... Keyboard, 106 ... Display,
107 ... Image generating device, 108 ... Image processing device, 109
... TV camera, 110 ... Zoom motor / focus motor / iris motor control device, 111 ... Lens system, 112 ... Counter, 113 ... Pulse generator,
114 ... DA converter, 115 ... Motor drive device, 116
... Motor, 117 ... AD converter, 118 ... Sensor, 11
9 ... Network, 301a, 301b ... Table type robot, 302a, 302b, 302c ... Arm type robot, 3021a, 3021b ... Hand effector, 3022
... TV cameras, 303a, 303b ... Work transfer robots, 401, 411, 421, 431 ... Base, 40
2,403,412,413,414,422,42
3, 432, 433, 436 ... Rotating shafts, 402a, 40
3a, 409a ... Output unit, 425, 435 ... Parallel movement shaft, 424, 434 ... Parallel movement shaft drive motor.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical display location B25J 13/08 A G01B 11/00 H 11/26 H G05B 19/418 G06T 7/00 1/00 G06F 15/64 M

Claims (17)

[Claims] 1. A position / orientation detecting method for correcting the deviation when the scene detecting means and the object to be photographed are deviated from a predetermined designed position / orientation, in the work environment by the scene detecting means. The subject is photographed by changing the position of the principal point of the camera lens in the scene detection means as a center, and the obtained plurality of images are projected onto one plane to form one wide-angle image. Assuming that the scene detection means and the object to be photographed are in a predetermined designed position / orientation, a pseudo image is created by the scene detection means. The positions of the respective feature points on the object to be photographed on the pseudo image, and the characteristic points on the object to be photographed on the composite image actually photographed by the scene detection means. The amount of deviation from the position of the feature point is calculated and Correcting the position / orientation of the scene detection means and the object to be photographed based on the amount of displacement based on the object to be photographed, and moving the scene detection means and the object to be photographed by an amount corresponding to the amount of deviation. A position / orientation detection method characterized by performing either one of modifying design data. 2. At least two scene inspection means capable of picking up an image of a work object, a processing device for processing the work object, and peripheral devices thereof are installed in advance.
The position / orientation of the work environment including the scene inspection means of the table, the work object, the processing device, and the peripheral device at each predicted time is pseudo-generated as an image based on predesigned data, and then, , When the work environment reaches the time axis at which the pseudo image is generated, the actual work environment is imaged by each scene inspection means, and the data of the work environment actually imaged and the data that generated the pseudo image are When there is a deviation between the two, the respective positions and orientations in the work environment are corrected based on the deviation, and the corresponding design data in the work environment corresponding to the deviation are corrected. A position / orientation detection method characterized by performing either one of correction and correction. 3. At least two scene inspection means capable of imaging a work target, a processing device for processing the work target, and peripheral devices thereof are installed in advance, and these 2
The position / orientation scene at each predicted time of the work environment including the scene inspection means of the table, the work object, the processing device, and the peripheral device is pseudo-generated as an image based on predesigned data. Then, when the work environment reaches the time axis at which the pseudo image was generated, the actual work environment is imaged by each scene inspection means, and the data of the work environment actually imaged and the data that generated the pseudo image are generated. If there is a deviation between the two, the respective positions / postures in the work environment are corrected based on the deviation, and the corresponding design data in the work environment is corrected by the amount of the deviation. Either of the above is performed, and thereafter, the pseudo generation process and the process of correcting each position / orientation in the work environment are executed each time each of the work environments performs one specific operation. Characterized by Position and orientation detection method. 4. The position according to claim 1, wherein the data of the work environment actually captured is one wide-angle and high-accuracy image. -Attitude detection method. 5. A position / orientation detecting device for determining the position / orientation of an object to be photographed in a work environment, wherein a scene detecting means for photographing the object to be photographed and an actual position indicating the actual position of the scene detecting means. If it is assumed that the data and design data indicating the design position / orientation of the object to be photographed are stored, and the object to be photographed is in the design position / orientation, the scene detection means An image that is likely to be captured by using the actual data and the design data is pseudo-created, and the position of each feature point on the object to be captured on the pseudo-created image and the A shift between the position of each feature point corresponding to each feature point on the subject and the position of each feature point on the image actually captured by the scene detection means on the subject is extracted, and the subject is photographed based on the shift. Correcting the position / posture and the above Amount corresponding position and posture detecting device, characterized in that it comprises a means for executing one of the modifying the design data of the object to be photographed of. 6. A work having a work target, a processing device for processing the work target, peripheral devices thereof, and at least two scene inspection means capable of imaging the work target, the processing device, and the peripheral device. A design data storage device that stores information about the environment, the work object, the processing device, the peripheral device, the shape and characteristic points of each scene inspection means, the work object, the processing device,
A work environment based on a work plan data storage device that stores work procedure data, work process data, and motion path data for peripheral devices and each scene inspection means, and data stored in the design data storage device and work plan data storage device. The position / orientation scene at each of the pre-predicted times of the inside is pseudo-generated, and when the pseudo-generated time axis is reached, the actual work environment is imaged by each scene inspection means, and Each position / posture of the work environment is compared with each position / orientation scene of the work environment in which the imaging data is artificially generated.
A position / orientation detecting device comprising: an assembly station control means for detecting a deviation of the attitude and correcting each position / orientation in the work environment in a direction to absorb the detected deviation. 7. A work having a work object, a processing device for processing the work object, peripheral devices thereof, and at least two scene inspection means capable of imaging the work object, the processing device and the peripheral device. A design data storage device that stores information about the environment, the work object, the processing device, the peripheral device, the shape and characteristic points of each scene inspection means, the work object, the processing device,
A work environment based on a work plan data storage device that stores work procedure data, work process data, and motion path data for peripheral devices and each scene inspection means, and data stored in the design data storage device and work plan data storage device. The position / orientation scene at each of the pre-predicted times of the inside is pseudo-generated, and when the pseudo-generated time axis is reached, the actual work environment is imaged by each scene inspection means, and Each position / posture of the work environment is compared with each position / orientation scene of the work environment in which the imaging data is artificially generated.
An assembly station control means for detecting a deviation of the posture and correcting each position / posture in the work environment in a direction to absorb the detected deviation, and the assembly station control means comprises a design data storage device and A pseudo work environment scene generation unit for pseudo generating a work environment scene at a specific time during work based on the work plan data storage device, and when each scene inspecting means reaches the same time axis as the specific time during work The image combining unit that combines one high-precision image based on the actual captured image data is compared with the high-precision image by the pseudo-work environment scene and image combining unit by the pseudo-work environment scene generating unit, and the work is performed. A work environment scene understanding unit that detects each position / orientation deviation in the environment, and a work object that absorbs the position deviation in the work environment according to a command from the work environment scene understanding unit, Engineering system, peripheral device, the position and posture detecting device characterized by having an operation command generating unit for selectively driving each scene inspection means. 8. At least one of the scene detection means in the work environment is composed of a television camera installed in an attitude changing device, and the attitude changing device is a first rotation rotatable about an axis. An axis and an output part disposed orthogonally to the output part of the first rotation shaft and having a second rotation shaft to which the television camera is attached, and a center line of the first rotation shaft output part. The center axis of the camera lens in the television camera is arranged at a position intersecting with the center line of the second rotation axis output section, the rotation center of the camera lens is made to coincide, and the main position of the camera lens is made to coincide with the center. The position / orientation detection device according to claim 6 or 7, wherein the points are arranged so as to coincide with each other. 9. At least one of the scene detection means in the work environment is composed of a television camera installed in the attitude changing device, and the attitude changing device is a first rotation rotatable about an axis. An axis, a second rotating shaft whose output part is arranged orthogonal to the output part of the first rotating shaft, and an output part which is the second rotating shaft.
And a third rotary shaft to which the television camera is mounted, the first rotary shaft having a second rotary shaft and a third rotary shaft which are mounted orthogonal to the second rotary shaft and the first rotary shaft. The center axis of the camera lens of the television camera is arranged at a position where the center line of the camera section, the center line of the second rotation axis output section, and the center line of the third rotation axis output section intersect with each other. 7. The constitution is such that the rotation centers are made to coincide with each other, and the principal points of the camera lens are arranged so as to coincide with each other so that the camera lens changes its posture around the principal point position. Or the position / orientation changing device described in 7. 10. At least one of the scene detection means in the work environment is composed of a television camera installed in an attitude changing device, and the attitude changing device is a first rotation rotatable about an axis. An axis, a second rotating shaft whose output part is arranged orthogonal to the output part of the first rotating shaft, and an output part which is the second rotating shaft.
And a linear movement shaft on which the television camera is attached along the movement direction, the linear movement shaft being attached so as to be linearly movable in a direction orthogonal to the second rotation axis and the first rotation axis with respect to the output portion of the rotation axis. , The center axis of the camera lens in the television camera is arranged at a position where the center line of the first rotary shaft output portion and the center line of the second rotary shaft output portion intersect with each other,
It is characterized in that the center of rotation of the camera lens is made to coincide with each other, and the principal points of the camera lens are arranged so as to coincide with each other, and the camera lens is changed in posture about the principal point position. The position / orientation changing device according to claim 6. 11. At least one of the scene detection means in the work environment is composed of a television camera installed in an attitude changing device, and the attitude changing device is a first rotation rotatable about an axis. An axis, a second rotating shaft whose output part is arranged orthogonal to the output part of the first rotating shaft, and an output part which is the second rotating shaft.
A linear movement shaft mounted so as to be linearly movable in a direction orthogonal to the second rotation axis and the first rotation axis with respect to the output section of the rotation axis, and the output section outputs the output to the output section of the linear movement axis. And a third rotation shaft to which the television camera is attached, the center line of the first rotation shaft output part and the center line of the second rotation shaft output part being arranged along the linear movement direction of the part. And the center axis of the camera lens in the TV camera is arranged at a position where the center line of the third rotation output section and the center line of the third rotation output section intersect with each other.
It is characterized in that the center of rotation of the camera lens is made to coincide with each other, and the principal points of the camera lens are arranged so as to coincide with each other, and the camera lens is changed in posture about the principal point position. The position / orientation changing device according to claim 6. 12. A work having a work target, a processing device for processing the work target, peripheral devices thereof, and at least two scene inspection means capable of imaging the work target, the processing device and the peripheral device. Design data storage device that stores information about the environment and the work object, processing device, peripheral device, shape and feature points of each scene inspection means, and work procedure for the work object, processing device, peripheral device, and each scene inspection means Based on the work plan data storage device that stores data, work process data, and motion path data, and the data stored in the design data storage device and the work plan data storage device, at each predicted time in the work environment. The position / orientation scene of the above is pseudo-generated, and when the pseudo-generated time axis is reached, the actual work environment is imaged by each scene inspection means, and Of each position / orientation of the work environment by comparing each position / orientation scene of the work environment in which the simulated data is generated, and each position in the work environment in the direction of absorbing the detected misalignment. A flexible production system comprising a position / attitude detection device having an assembly station control means for correcting the attitude. 13. A work having a work target, a processing device for processing the work target, peripheral devices thereof, and at least two scene inspection means capable of imaging the work target, the processing device and the peripheral device. Design data storage device that stores information about the environment and the work object, processing device, peripheral device, shape and feature points of each scene inspection means, and work procedure for the work object, processing device, peripheral device, and each scene inspection means Based on the work plan data storage device that stores data, work process data, and motion path data, and the data stored in the design data storage device and the work plan data storage device, at each predicted time in the work environment. The position / orientation scene of the above is pseudo-generated, and when the pseudo-generated time axis is reached, the actual work environment is imaged by each scene inspection means, and Of each position / orientation of the work environment by comparing each position / orientation scene of the work environment in which the simulated data is generated, and each position in the work environment in the direction of absorbing the detected misalignment. And an assembly station control means for correcting the posture, and the assembly station control means artificially generates a work environment scene at a specific time during work based on the design data storage device and the work plan data storage device. A pseudo work environment scene generation unit, and an image composition unit that composes one high-precision image based on the actual imaging data imaged by each scene inspection means when the same time axis as the specific time during work is reached. , A work environment scene understanding unit that detects the deviation of each position / posture in the work environment by comparing the pseudo work environment scene generated by the pseudo work environment scene generation unit and the high-precision image by the image synthesis unit. Position / orientation including a work object, a processing device, a peripheral device, and an operation command generation unit that selectively drives each scene inspection means in order to absorb a positional deviation in the work environment according to a command from the work environment scene understanding unit. A flexible production system including a detection device. 14. At least one of the scene detecting means in the work environment in the position / orientation detecting device is composed of a television camera installed in the attitude changing device, and the attitude changing device rotates about an axis. A possible first axis of rotation,
The output unit is arranged orthogonal to the output unit of the first rotation shaft, and has a second rotation shaft on which the television camera is attached, and the center line of the first rotation shaft output unit and the second rotation shaft are provided. The center axis of the camera lens of the TV camera is placed at a position that intersects with the center line of the rotation axis output section, the rotation center of the camera lens is made to coincide, and the principal point of the camera lens is made to coincide with that position. The flexible production system according to claim 12 or 13, wherein the flexible production system is arranged as follows. 15. At least one of the scene detecting means in the work environment in the position / orientation detecting device is composed of a television camera installed in the attitude changing device, and the attitude changing device rotates about an axis. A possible first axis of rotation,
A second rotary shaft having an output portion arranged orthogonal to the output portion of the first rotary shaft, and the output portion having the second rotary shaft and the first rotary shaft relative to the output portion of the second rotary shaft. And a third rotation shaft on which the television camera is mounted, the center line of the first rotation shaft output unit, the center line of the second rotation shaft output unit, and the third rotation shaft. The center axis of the camera lens in the TV camera is placed at a position that intersects with the center line of the axis output section, the center of rotation of the camera lens is aligned, and the principal point of the camera lens is aligned at the aligned position. 14. The flexible production system according to claim 12, wherein the camera lens is arranged so as to change the posture around the principal point position. 16. At least one of the scene detecting means in the work environment in the position / orientation detecting device is composed of a television camera installed in the attitude changing device, and the attitude changing device rotates about an axis. A possible first axis of rotation,
A second rotary shaft having an output portion arranged orthogonal to the output portion of the first rotary shaft, and the output portion having the second rotary shaft and the first rotary shaft relative to the output portion of the second rotary shaft. And a linear movement shaft on which the television camera is attached along the movement direction, and the center line of the first rotation shaft output unit and the second rotation shaft output unit are provided. The center axis of the camera lens in the TV camera is placed at the position where it intersects with the center line, the rotation center of the camera lens is aligned, and the principal point of the camera lens is aligned at the aligned position. 14. The flexible production system according to claim 12, wherein the camera lens is configured to change its posture around the principal point position. 17. At least one of the scene detecting means in the work environment in the position / orientation detecting device is composed of a television camera installed in the attitude changing device, and the attitude changing device rotates about an axis. A possible first axis of rotation,
A second rotary shaft having an output portion arranged orthogonal to the output portion of the first rotary shaft, and the output portion having the second rotary shaft and the first rotary shaft relative to the output portion of the second rotary shaft. A linear movement shaft mounted so as to be linearly movable in a direction orthogonal to the output unit, and an output unit arranged along the linear movement direction of the output unit with respect to the output unit of the linear movement shaft, The television camera has three rotation shafts, and the center line of the first rotation shaft output unit, the center line of the second rotation shaft output unit, and the center line of the third rotation output unit intersect with each other. In addition to arranging the central axis of the camera lens in, the center of rotation of the camera lens is made to coincide, and the principal point of the camera lens is made to coincide with the coincident position. 13. The structure according to claim 12, wherein the structure is changed. 3 flexible manufacturing system according.